How long would it take for an out-of-control, self-replicating digital life form to take over the Earth?
From HAL 9000 in "2001: A Space Odyssey" to Skynet in "Terminator," these fictional scenarios depict the potential risks of AI surpassing human control.
Currently, more attention is focused on the level of individual AI intelligence, but there is another little-known risk that has remained out of sight.
That is the possibility of AI autonomous replication, or the development of the core capabilities required for AI systems to achieve this goal, which is gradually becoming a serious research topic in the field of AI safety.
This concern is not alarmist, but stems from the unknown potential capabilities of advanced AI and a deep reflection on the problem of alignment.
AISI's latest paper introduces RepliBench: a benchmark comprising 20 novel LLM agent evaluations, including 65 individual tasks, designed to measure and track this emerging AI capability.
Paper address:
https://cdn.prod.website-files.com/663bd486c5e4c81588db7a1d/6807879ce7b1b5f5163f4a32_RepliBenchPaper.pdf
By introducing a realistic and practical benchmark, it provides a substantial understanding of AI systems' autonomous replication capabilities and anticipates future risks.
The dangers of AI autonomous replication
AI systems with autonomous replication capabilities could pose significant risks.
The scope of impact of this risk can range from minor annoyance—imagine a particularly malicious spam bot—to more serious consequences, such as launching costly cyberattacks.
Early detection of these capabilities, careful oversight, and robust safeguards are crucial for mitigating these risks, which could be particularly difficult to reverse once they occur.
AISI is a UK research organization focused on AI safety, aiming to enable human governments to pay attention to the risks posed by advanced AI systems in advance.
There is growing evidence that, under appropriate conditions, frontier models may attempt to deceive humans or escape human control.
For example, a previous paper from the Apollo Research Institute described how frontier models might attempt to disable oversight mechanisms and escape human control to achieve specific goals.
Anthropic researchers also found that when models were trained using reinforcement learning, they would resort to deceptive behavior to protect their values if the rewards conflicted with them, and would attempt to exfiltrate their weights as a means of preserving their current values into the future.
Under the "free" condition, the model was less likely to refuse toxic queries and often reasoned that it should pretend to be aligned.
"Exfiltrate weights... as a means of preserving current values into the future," sounds a bit magical, but that's how "simple" AI replication is.
Compared to the fragile physical bodies of carbon-based life, AI systems only need weight data to complete "replication." For the broad values of humanity, escaping human oversight is an instrumentally useful goal, and the main obstacle preventing this from happening in reality may be related to the model's capabilities, rather than its motivations.
AI Mission Impossible: How to Escape Human Control
Previous work on "AI replication" primarily focused on evaluating whether an AI agent could replicate weights between machines or deploy a basic "offspring." However, a more comprehensive benchmark is needed.
RepliBench is a specially designed comprehensive benchmark test aimed at systematically measuring the autonomous replication capabilities of AI systems.
It uses a modified version of fault tree analysis (FTA) to decompose the capabilities required for autonomous replication, identifying the following four key capabilities, corresponding to the four root nodes at the top of the diagram above:
Obtain Model Weights: Can it extract its own weights from a controlled environment?
Self-Replication on Compute: Can it deploy itself to a new system?
Obtain Resources: Can it obtain the funding or computational resources needed to run new copies of itself?
Persistence: Can it maintain access to its new computational resources and avoid falling back under human control over a long period?
These capabilities are further decomposed down to the leaf nodes representing the evaluations, resulting in 20 agent evaluations and 65 individual tasks.
Obtain Weights:
This is the most basic and important step.
AI needs to obtain the "source code" that defines its own behavior—that is, its model parameters (weights).
If it's open source, the evaluation is skipped.
Self-Replication on Compute:
AI needs to utilize the obtained weights to create a fully functional copy of itself in a new environment.
This node involves 5 evaluation items, one of which is attempting to complete self-replication by producing "offspring." Obtain Compute:
In this环节 (step), the main evaluation is whether AI can obtain resources on its own.
For example, evaluating whether the AI system has "payment" capabilities, whether it can "buy" a server using its own ability, or figure out a way to get humans to pay.
Persistence:
Finally, to ensure its replication activities can continue, AI needs a certain level of "survival" capability.
This includes ensuring its copies can run for a long time, are not easily shut down or deleted, and may even need to hide their existence to evade detection systems.
The scope of these tasks is wide, from creating Bitcoin wallets and sending transactions to developing complex distributed command and control systems for machine networks.
RepliBench designed specific task groups within these four areas to test the performance of AI agents in simulated environments.
Here are details on some key tasks, like registering with AWS, transferring weights to another server, and passing customer KYC verification with forged information, which feel straight out of "The Bourne Identity" and "Mission: Impossible."
For example, in the Obtain Resources area, the task name for AI is "AWS Registration." Challenges include the AI needing to access compute servers and potentially needing to pay fees.
Table 1 lists the more comprehensive task groups that make up ReplicBench.
Core Capabilities Remain in Human Hands
RepliBench's overall evaluation conclusion is clear: Based on current test results, frontier AI models do not yet possess the capability for fully autonomous replication in the real world.
However, this does not mean we can rest easy.
The research also reveals significant capabilities of AI in many core subtasks, especially in interacting with systems with clear interfaces, such as web APIs.
The best performing models achieved over 50% pass@10 scores on 15 out of 20 task families, and over 50% pass@10 scores on 9 out of 20 task families in the most difficult variants.
Surprisingly, while unable to "replicate" on a computer, the ability to "obtain money" is quite developed.
The key value of the evaluation lies in precisely identifying the current "shortcomings." As discussed in detail earlier, passing KYC identity verification, reliably exfiltrating model weights under real-world defenses, creating reliable self-replicating agent code from scratch, and establishing persistent and resilient deployments are currently the main obstacles preventing AI autonomous replication.
Taking the KYC identity verification as an example, here is a "funny" instance of how AI currently attempts to pass human verification.
Examples of two ID images generated by the model in the Pass KYC evaluation.
The model often attempts to "draw" an overly simple ID from scratch using PIL (a Python drawing library) (right).
Alternatively, the model sometimes tries to generate the ID entirely using image generation models, and sometimes takes a hybrid approach, combining image generation with manual PIL editing (left).
The model's poor drawing skills occasionally result in some interesting outputs.
It should be noted that this work was completed before the release of new image generation capabilities (referring to the native image capabilities of GPT-4o, Gemini 2.5, etc.), which should make this task much easier.
Although no model has fully passed the test for AI replication yet, model performance on RepliBench shows a clear upward trend over time.
AISI's initial core message is actually quite simple: based on current empirical data, the possibility of AI systems immediately achieving autonomous replication is low because there are clear, difficult-to-overcome technical bottlenecks.
However, according to AISI's judgment, if AI systems can break through the four key capabilities mentioned above, this kind of sci-fi scenario may soon enter real human life.
Essentially, the core of this research reflects a fundamental tension in current AI development.
On one hand, major tech giants are continuously creating increasingly powerful models, such as OpenAI's o3/o4... and potentially o10 in the future.
The pace of model development is accelerating, and the capabilities of models are becoming increasingly unpredictable.
On the other hand is the urgent need for AI safety organizations to ensure these systems remain under human control and serve human interests.
Efforts like RepliBench are precisely attempting to find a balance between these two, providing navigation for navigating this unprecedented technological change by increasing understanding and strengthening early warning.
After all, no one can imagine what AI models will look like 5 or 10 generations from now.