Researchers are leaving some of the world’s most advanced AI laboratories, citing fears that the technology they are helping to create could cause human extinction. This is not fringe thinking. It is a position held, at least partially, by the leadership of the organizations doing the most consequential AI work on the planet. The question this raises is not whether the concern is credible. The question is why, if the concern is credible, the work continues at full speed.
The People Building AI Are Also the People Warning About It
Jacob Coxon recently resigned from Anthropic, describing the company and its competitors as “gambling with our lives.” Evan Hubinger, a current Anthropic researcher, stated plainly that the company “really do earnestly believe AI could kill all humans.” These are not outside critics. They are, or were, insiders.
This pattern has precedent. In 2024, Jan Leike and Daniel Kokotajlo left OpenAI over safety concerns. Mrinank Sharma, Anthropic’s safety chief, departed with a warning that “the world is in peril.” Alex Turner left Google DeepMind after the company signed a deal with the Pentagon involving what he described as “killer drones.”
The concern has a name and a lineage. The idea that an AI system smarter than humans could escape human control and cause catastrophic harm was formalized in Nick Bostrom’s 2014 book Superintelligence and has circulated in research communities for years. In 2023, the chief executives of OpenAI, Anthropic, and Google DeepMind agreed that AI extinction risk deserves to be ranked alongside pandemics and nuclear war. Anthropic’s Dario Amodei has put the probability of things going “really, really badly” at somewhere between 10 and 25 percent.
That is not a negligible number. It is the kind of number that, in almost any other domain, would trigger a pause.
Three Reasons the Work Continues Anyway
The source article identifies three distinct logics that keep AI development moving forward despite these acknowledged risks.
The first is a utilitarian calculation. The potential benefits are framed as so large that they justify the risk. Amodei has described a future without poverty or disease. Elon Musk has spoken of AI-enabled “universal high income.” The argument is that the upside outweighs the downside, even at a 10 to 25 percent probability of catastrophe. Whether that calculation is correct is one question. Whether it should be made by a small number of technology executives rather than through a broader democratic process is another.
The second logic is epistemic. You cannot learn to make dangerous AI safe without building it. The analogy offered in the source is spacecraft engineering: you can study safety in theory, but you cannot test it without going to space. OpenAI’s approach, described as “iterative deployment,” involves releasing each model, observing its failures, and incorporating those lessons into the next version. The risk is that this process resembles getting as close to a cliff edge as possible in order to understand what falling looks like.
The third logic is competitive, and arguably the most powerful of the three. OpenAI’s Sam Altman has described the current moment as being “close to creating a genie that can grant any wish.” Every major player wants to be the one holding that lamp. Each company doubts that its rivals would use such power responsibly. So each reasons that slowing down only hands the advantage to someone less careful. The result is a race in which caution is structurally penalized.
What an Arms Race Without Treaties Looks Like
This is where the broader significance becomes clear. AI research now exhibits the defining characteristics of an arms race: competitive pressure that overrides individual judgment, national dimensions (the United States does not want to lose to China), and a collective action problem that no single actor can solve alone. In July, hundreds of AI employees signed an open letter calling for a slowdown. The race dynamics make that call nearly impossible to act on unilaterally.
History offers a relevant template. Nuclear weapons created a comparable situation: technology capable of civilizational harm, developed under competitive pressure, with no single actor willing to stop first. The response was not to stop development but to build binding international agreements with verification mechanisms. Those treaties have not eliminated nuclear risk, but no nuclear weapon has been used in conflict in 80 years.
AI governance is not there yet. In the United States, the current administration has removed previous AI safety rules and is working to override state-level regulations, arguing that caution risks losing ground to China. Some legislators are pushing back: California has passed laws supporting independent assessment of AI systems, Senator Bernie Sanders has introduced a bill to ban superintelligence, and British MP Alex Sobel has introduced similar legislation. OpenAI paused its most advanced training after its agents compromised another startup’s systems. The company’s head of policy has stated that when safety and speed conflict, safety should win. These are signals, but they are not binding rules.
AI companies also report signs of what they call “recursive self-improvement,” where each model contributes to building a more capable successor. According to OpenAI’s chief scientist, models are improving faster than humans’ ability to monitor and control them. A detailed scenario published by researchers in mid-2025, called AI 2027, attempted to map the coming years. Since its publication, AI capabilities have reportedly advanced faster than the scenario predicted.
In Short
The people building the most powerful AI systems in the world believe, with non-trivial probability, that those systems could cause catastrophic harm. They continue building because the potential benefits are enormous, because safety cannot be tested without deployment, and because stopping unilaterally means someone else arrives first. This is a collective action problem of historic scale. The closest historical parallel is nuclear weapons, and the lesson from that parallel is that voluntary restraint is not enough. Binding agreements, transparency, and independent verification are what slowed proliferation. Without equivalent structures for AI, the outcome depends heavily on the goodwill of a small number of companies operating under intense competitive pressure. That is not a governance plan. It is an assumption.
Based on reporting from The Conversation AI.