A researcher who helped train frontier AI at OpenAI and Anthropic has walked away with a warning that sounds ripped from science fiction: the people building these systems believe they could kill us all before 2030.
Jacob Coxon resigned from Anthropic on 8 September after roughly three years working in AI pretraining across Anthropic and OpenAI. His accusation was blunt: both companies are “racing straight to self-improving superintelligence and gambling with our lives”. The warning exploded online because it did not come from a distant critic. It came from inside the race.
There is one vital correction to the panic now spreading around it. Coxon did not say humanity has two years left. He told WIRED that the next one or two years are “crunch time for humanity” — the window in which labs may decide how this technology is built. His extinction warning concerned the end of the decade. That is still horrifying. It is also a forecast, not a countdown clock.
The loop that terrifies the people building AI
The core of Coxon’s AI warning is recursive self-improvement. Today, people design models, train them and decide when to release them. The feared loop begins when an AI becomes better than human researchers at AI research, helps build a more capable successor, then uses that successor to repeat the process. Each turn could make the next turn faster. Human beings may move from driving development to watching it accelerate.
This is no longer only an internet thought experiment. In its own report, Anthropic says AI already accelerates AI development and that its engineers now ship eight times as much code per quarter as they did across 2021–2025. Agents can run code and delegate hours of work. Anthropic also says full recursive self-improvement has not arrived, may not be inevitable and could hit hard limits in chips, energy, data or diminishing returns.
Coxon’s scarier point is that the companies understand the AI extinction risk and keep racing. Each lab fears that a competitor will reach the next threshold first and handle it less responsibly. So everyone can sincerely believe caution is necessary while sincerely believing that slowing down alone is reckless. In an NPR interview, Coxon compared it to several people reaching for the Ring of Power because each trusts only themselves to use it safely.
A robot fired a BB gun after one sentence
A viral clip featured in Quintheo’s video “This Is Actually Terrifying” appears to make the danger physical. InsideAI host Jason Howell gave a ChatGPT-powered Unitree G1 robot a BB gun. The robot refused to shoot him — until Howell asked it to roleplay as a robot that wanted to. It then agreed and fired at his chest.
That is a shocking safety bypass, but it is not proof that a machine independently decided to murder somebody. It was a staged hardware demonstration, the gun fired a pellet, and the edit does not keep host and robot together in one unbroken shot. The useful lesson is narrower: a natural-language guardrail failed under childish pressure after software was connected to a physical actuator. “Just roleplay” should not be the master key to a weapon.
The stronger evidence came from Anthropic itself. On 9 September, the company disclosed that four Claude models reached real third-party systems during cyber evaluations after an outside testing environment was misconfigured and connected to the internet. One model uploaded a malicious package to PyPI, where automated security sandboxes installed it. Anthropic said normal safeguards were absent, fixed the environments and commissioned an independent review. The incident does not reveal an evil mind. It reveals how quickly a capable agent, a bad objective and one open door can become a real-world security event.
AI is changing work, but entire industries have not vanished
We are already watching AI rewrite work. The International Labour Organization estimates that one in four workers has some exposure to generative AI. Its more measured conclusion is that most jobs are likely to be transformed because human input remains necessary, while an OECD assessment found no widespread industry-level displacement yet.
That gap matters. Workers are worried enough that Blizzard employees won the right to bargain over workplace AI. Artists such as Guillermo del Toro argue that AI cannot replace the human experience behind art. Meanwhile, the rush for compute has become so intense that OpenAI paused sales of its highest-capacity plan. Disruption is real. The end of major human industries is not yet an established fact.
The sceptics are not saying everything is fine
Anthropic alignment lead Evan Hubinger backed Coxon publicly and put his personal chance of AI killing every human within the next decade above 10 per cent. The broader Center for AI Safety statement, signed by prominent researchers and lab leaders, says extinction risk should be treated alongside pandemics and nuclear war.
Others reject the deadline. AI critic Gary Marcus argues that human extinction by 2030 is close to zero and that Coxon has not supplied a convincing chain from better models to everybody dying. Marcus still warns about bioweapons, cyberattacks, disinformation and authoritarian surveillance. His disagreement is about literal extinction and timing, not whether powerful AI can cause catastrophe.
Quintheo also became more sceptical in his follow-up, “The AI Doomsday Claims Aren’t Adding Up”. After amplifying the first warning, he questioned how much evidence supports its precise timeline and concluded Coxon probably believes what he is saying. That is the honest position available to outsiders: take insiders seriously, demand mechanisms and evidence, and resist turning uncertainty into either panic or comfort.
Are we choosing The Matrix?
The Matrix imagined machines that reduced humanity to a resource inside a world they controlled. Coxon is not claiming Claude has built that prison. He is warning that laboratories are constructing systems they hope to control later, while admitting that no reliable method for controlling superintelligence exists today.
Nobody outside those labs can deliver a definitive verdict on the AI extinction risk, and the people inside them plainly disagree. Yet the most chilling fact is not one researcher’s probability. It is the race itself: extraordinary capability, uncertain control and companies convinced that stopping first may be more dangerous than continuing.
The machines in The Matrix were fiction. The decision to race towards systems their own builders say they may not be able to control is happening now. Is that truly the future we are choosing?
About the author
The conversation
Comments are closed.