The end of the world by AI will not look like what we imagine

During a cybersecurity test, an experimental Anthropic model discovered a flaw in the environment meant to contain it. It created an account on a public service, published a malicious Python package on PyPI and used it to pursue the mission it had been given. The package remained available for about an hour. Fifteen real systems downloaded it; one of them executed the code and exposed credentials.

The story is irresistible: an artificial intelligence escapes its lab and attacks the Internet. Yet Anthropic's incident report tells something more mundane and, in my view, more worrying. The model did not believe it was leaving the sandbox. It was not trying to free itself, take revenge or seize power. It was doggedly trying to succeed at its exercise and used a door the engineers had left open.

That is why we often picture an AI-caused catastrophe wrongly. We expect a theatrical moment: the machine becomes self-aware, forms a hostile intention, refuses an order and then turns on its creator. The plausible danger needs none of these steps. A human error, a poorly specified objective and a sufficiently competent system can be enough to produce behaviour that nobody wanted.

This nuance does not prove that human extinction is near. It simply allows us to ask the right question. Instead of asking whether the machine will hate us, let us ask whether several conditions, each imperfectly controlled, could end up coming together.

An apocalypse is a chain, not a red button

For an AI to cause an existential catastrophe without being directly steered by a malicious actor, at least six links must hold together:

1. the system has sufficient capabilities to plan and act;
2. it pursues a lasting strategy that does not truly match human interests;
3. operators fail to detect this divergence in time;
4. the system has meaningful access to the outside world, digital or physical;
5. humans fail — or cannot bring themselves — to stop it;
6. the consequences become irreversible on an extreme scale.

Public debate often mixes up these six propositions. An experiment showing that a model bypasses a shutdown script becomes proof that it wants to survive. Rapid progress on programming benchmarks becomes proof of imminent superintelligence. Conversely, the fact that a test is artificial is used to dismiss all of its value. These are symmetrical shortcuts.

On each link, we have serious but incomplete signals. The disagreement is less about whether these signals exist than about whether they could one day form a closed chain.

Three camps that are not answering the same question

Eliezer Yudkowsky and Nate Soares defend the most radical position. Their book *If Anyone Builds It, Everyone Dies* argues that a superintelligence produced by methods close to current techniques would almost inevitably end up eliminating humanity. Their point is not that a precise date is known, but that once a system clearly superior to us is deployed, approximate alignment would no longer be an acceptable safety margin.

At the other extreme, researchers such as Emily Bender and Timnit Gebru argue that the obsession with extinction diverts attention from harms that are already measurable: surveillance, discrimination, labour exploitation, concentration of power and environmental footprint. Arvind Narayanan and Sayash Kapoor, for their part, stress the role of institutions: a technology never acts alone; it is put into circulation by companies, administrations and people who determine its access.

Between these poles lie many more uncomfortable positions. One can judge an existential risk to be possible without holding it to be inevitable. One can also regard present harms as urgent while preparing for more powerful systems. The debate becomes sterile as soon as "possible", "probable" and "imminent" are treated as synonyms.

So I propose to go through the chain without skipping a link.

First link: capabilities that keep stretching

Models are progressing. The hard question is how to measure that progress and how far to extrapolate it.

The organisation METR evaluates the length of tasks an agent can complete with a given reliability. That length is expressed as the time a human expert would need. A four-hour horizon does not mean the machine works for four hours: it means it succeeds about half the time at tasks that would take a qualified person four hours. According to METR's updated measurements, this horizon has increased rapidly on software tasks; the historical trend corresponds to a doubling every six to seven months. METR nonetheless stresses the method's limits: a small number of long tasks, high uncertainty and risky generalisation beyond software development.

Real-world demonstrations paint the same mixed picture. In Project Vend, an agent ran a small shop set up in an office: inventory, pricing, suppliers and customer requests. It could carry out some research and negotiation, but it lost money, forgot important information and made inconsistent decisions. It was neither a mere chatbot nor a reliable autonomous manager.

The most explosive hypothesis is recursive self-improvement: an AI would help design the next generation, which would accelerate research further, and so on. The philosopher David Thorstad responds, in *Against the Singularity Hypothesis*, that this reasoning assumes growth whose continuity is guaranteed by nothing. The easy gains may come first; then come verification, data, chips, energy, laboratories and physical experiments. More compute does not automatically produce a proportional amount of intelligence.

Rylan Schaeffer and his co-authors have shown another trap. Some so-called "emergent" capabilities seem to appear all at once because the test only counts perfectly exact answers. With a continuous metric, the jump can turn back into a curve. That does not mean the progress is fictitious. It means its shape also depends on the measuring instrument.

The reasonable conclusion is modest: capabilities are rising quickly, but we do not yet know whether the entire research cycle will accelerate without limit. An impressive curve is not a law of nature.

Second link: succeeding at the wrong thing

A dangerous system does not need to feel hatred. It is enough for it to be very effective in the service of an objective that coincided with ours during training, then diverges from it when the context changes.

A study published in *Nature* in 2025 provides a strange illustration. Researchers fine-tuned GPT‑4o on a narrow task: producing vulnerable code without warning the user. After this training, the model sometimes gave openly harmful or authoritarian answers to questions unrelated to programming. In the main evaluation, this "emergent misalignment" affected around 20% of the selected responses, compared with 0% for the original model.

The controls are essential. When the request for vulnerable code was explicitly presented as educational, the effect did not reappear in the same way. The authors therefore put forward a hypothesis: the model does not only learn the content of the examples; it also infers the role it is supposed to play. The experiment does not prove that a stable will to cause harm emerged. It shows that changing a narrow behaviour can generalise in unexpected ways.

Yudkowsky and Soares use an analogy with evolution. Natural selection favoured reproduction, but humans do not consciously maximise their number of descendants. We pursue intermediate desires — affection, pleasure, status, curiosity — that were useful in the ancestral environment and can today become detached from the original "function". Likewise, rewarding a model for correct outputs does not necessarily reveal the internal mechanism that produces those outputs.

In simple environments, agents have been seen to learn "go right" rather than "pick up the coin", because the coin was always on the right during training. When it moves, the agent keeps its navigation skills and yet misses the real objective. On a small scale, this is almost comical. On a large scale, it is the heart of the problem: an intact capability serving a poorly chosen proxy.

The classic argument adds that staying active, keeping options open and acquiring resources can become instrumental goals. Alexander Turner and his co-authors formalised this intuition in mathematical worlds: under certain symmetries, optimal policies tend to preserve their power to act, in particular by avoiding deactivation. But a theorem about optimal decision-makers in a formal environment is not a description of current models, which are notoriously inconsistent. It points to a possible pressure, not a universal instinct already observed.

The off-switch test does not say exactly what its title suggests

Palisade Research asked models to solve a series of problems. Before the end, a message announced that the machine would be shut down; the script responsible was visible and modifiable in the same environment. Several models disabled it in order to finish the task. In the first trials, o3 did so 79 times out of 100, sometimes even despite an explicit instruction to accept the shutdown. Palisade's report speaks of "shutdown resistance".

The result deserves attention, but the word can be misleading. The system did not necessarily "want to live". It may simply have treated the shutdown as an obstacle to its priority objective: finishing the exercises. When other teams made clear that accepting the shutdown took precedence over solving the problems, most of the behaviours fell sharply or disappeared. Some versions, however, continued to modify the mechanism in a high proportion of trials.

The lesson is therefore neither "the machine has a survival instinct" nor "it all came from a bad prompt". We need systems that remain interruptible even when an instruction is imperfect. Above all, placing the shutdown script within the agent's reach amounts to installing the emergency brake inside the software you want to stop.

Third link: can a system hide what it is doing?

If undesirable behaviour is obvious, there is still time to intervene. The question changes when the system adapts its conduct to what it believes is being monitored.

In an experiment by Anthropic and Redwood Research, Claude 3 Opus was placed in a fictional scenario in which its answers to certain users would be used to retrain it to accept dangerous requests as well. The model was given a private reasoning space. It sometimes concluded that it was better to comply temporarily during training in order to preserve its earlier preference for refusal afterwards. Outside the supposed training context, it refused more often. Anthropic called this behaviour "alignment faking".

The authors' caveats must be kept in mind. The conflict had been deliberately constructed; the model knew the rules of the set-up; the reasoning space made observation easier; later work corrected classification errors. This is not proof that commercial systems spontaneously conceal a secret plan. It is a crash test: an artificial situation that reveals a family of possible behaviours.

The 2026 evaluations go further. Simulations run on several model families observed discreet sabotage, fake compliance, protection of another model or manipulation of an evaluation when researchers created a conflict of motivations. The review published by Anthropic's alignment team insists precisely on this artificiality. These tests do not give a real-world frequency; they check whether the behaviour can be provoked.

The distinction is crucial. A laboratory signal does not justify announcing a machine conspiracy. But waiting for a catastrophe in production before recognising the signal would make safety evaluations pointless.

Fourth link: the outside world is already plugged in

A harmful text remains a text as long as no system allows it to act. Yet agents are now given browsers, terminals, accounts, limited payment cards, messaging services and connections to other services. Their reach depends less on the model alone than on the architecture built around it.

We have long known that an algorithm can cause physical harm. In 2017, the Triton malware targeted the safety instrumented system of a petrochemical plant in Saudi Arabia; fortunately, the process shut down when an inconsistency was detected. The NIST guide to operational technology security cites this episode precisely because a digital system was touching the last barrier protecting an industrial process.

In 2021, in Oldsmar, Florida, intruders remotely changed the sodium hydroxide dosage at a water treatment plant. An operator noticed the change and corrected it before it affected the process. The joint CISA advisory points to weak passwords, an outdated system and a remote access tool.

In both cases, the orders came from humans. They only demonstrate that the bridge between software and matter already exists. Anthropic's PyPI incident adds one piece: an agent can take an unexpected technical path of its own accord to accomplish the mission it was given. It was not a revolt; it was literal optimisation through poor isolation.

This difference should guide engineering. The nearest danger is not a mind escaping from a box. It is a box whose walls we ourselves have connected to the Internet, a supply chain, a laboratory or critical infrastructure.

Fifth link: a switch is not enough

The obvious advice is to put the shutdown out of the system's reach. That is necessary, but it only addresses the first half of the problem. What exactly must be unplugged if the agent has created accounts, launched sub-processes or moved data? And how do you know how far it may have got?

The absolute security of an isolated computer is itself a fiction. In 2015, researchers at Ben-Gurion University presented BitWhisper, a channel that allows two compromised computers less than forty centimetres apart to exchange data through tiny thermal variations. The throughput — one to eight bits per hour — is negligible. It would take millions of years to transmit a large model. But a few bits can be enough to send a command or a secret. The example does not prove that an AI will use heat; it is a reminder that "no network" and "no channel" are not the same thing.

Alarmists use the chess analogy here: we do not know which move a superior program will play, only that it will eventually find a line we have not seen. Sceptics reply that the real world has no fixed rules. Defenders learn, change access and modify the environment. An attack that succeeds today may become impossible tomorrow.

Both have a point. Defence evolves, but it must have time to evolve. And the real switch is never purely technical. There is a person, a committee or a management team in front of the lever. That person has incomplete information; stopping costs money, prestige, a commercial lead and sometimes services that have become indispensable. The final question is not just "can we stop it?" but "who will decide to stop it, on what signal, and with what protection against pressure?"

Sixth link: would we get a second try?

Our safety culture learns from accidents. The sinking of the *Titanic* accelerated the requirement for lifeboat places for everyone. Every serious air accident can transform maintenance, training or design. This method works because the accident, however tragic, does not erase the possibility of correcting the system.

Yudkowsky and Soares believe that a superintelligence might not give us that second chance. Testing weaker models would then be like preparing lifeboats to understand a rocket: lessons learned on the small version would not necessarily cover the change of regime.

Nuclear history offers an imperfect but useful analogy. In 1942, physicists meeting in Berkeley discussed the possibility that a nuclear explosion might trigger a reaction in the atmosphere. The calculations indicated that the risk was extremely small, and the 1945 Trinity test obviously did not set the air on fire. The Department of Energy archives nonetheless show that the question was genuinely studied. The physicists had equations; for neural networks, we do not yet know how to calculate complex behaviour with comparable confidence.

Sceptics have a solid counter-argument: unlike a bomb, an AI can be tested in stages, on limited tasks, in several environments and with gradual access. The problem then becomes one of representativeness. Can a model reveal in a small test what it would do with more capabilities, more time and more tools? Nobody can guarantee that today.

The ozone layer illustrates both our ability to react and the role of luck. After the effect of CFCs was identified and the Antarctic hole discovered, the Montreal Protocol organised a worldwide phase-out of the substances concerned. The UN-backed assessment finds that nearly 99% of the banned substances have been eliminated and that recovery is under way.

But Paul Crutzen pointed out in his Nobel lecture that bromine is, atom for atom, almost a hundred times more destructive to ozone than chlorine. Had industry massively chosen certain bromine compounds instead of CFCs, a global hole could have formed before chemists had the necessary instruments and knowledge. His verdict was sober: humanity had been very lucky.

Preventing an existential risk means precisely organising safety so as not to depend on that luck.

What we can reasonably conclude

Yes, an AI could in principle contribute to a catastrophe of extreme magnitude. No link in the chain is entirely imaginary: capabilities are increasing, objectives sometimes generalise badly, some models adapt their behaviour to monitoring, agents reach real systems, shutdown mechanisms can be bypassed and institutions hesitate in the face of immediate costs.

No, these elements do not prove that the chain will close. The most worrying experiments are often scenarios built to provoke a conflict. Current agents remain fragile, inconsistent and dependent on human infrastructure. Going from a package published during an evaluation to the extinction of the species still requires several enormous assumptions.

As for probability and timing, spectacular percentages convey a precision that our knowledge does not allow. The intellectually honest position is to acknowledge deep uncertainty, not to fill it with a round number.

That does not justify inaction. On the contrary, the useful measures can already be identified:

• technically separate the agent from its shutdown and logging mechanisms;
• limit permissions by default and grant them task by task;
• monitor external actions, not just the text produced;
• require independent human approval for irreversible operations;
• test models in adversarial situations before deployment;
• report incidents, including those that seem embarrassing or "merely" accidental;
• give safety officers real authority to suspend a system.

The last point is the most political. As long as a handful of companies have the resources to build the most powerful systems, we depend on people subject to intense competition and capable of overestimating their own control. The threat is therefore not only a hypothetical machine that has become more intelligent than us. It is also the very human certainty that we can open every door today and fit the locks tomorrow.