Could AI Really Kill Us All? Why Fears of Human Extinction Are Spreading

Concerns that artificial intelligence could eventually destroy humanity are intensifying after warnings from researchers and leaders at some of the world’s largest AI companies. As lawmakers consider how to respond, Nature examines what is driving the fear, who is expressing it and what could happen next.

How did the latest concerns about AI begin?

The latest wave of alarm about AI began on September 8th, according to researcher Jacob Coxon. He told the Wall Street Journal that he was resigning from Anthropic, an AI company based in San Francisco, California, because he feared the systems it was developing could spiral out of control and wipe out humanity.

After his resignation, Coxon wrote on X that “the people developing AI seriously believe that AI could kill us all by the end of the decade”. His post was amplified by Evan Hubinger, who leads Anthropic’s effort to align AI-model behaviour with human values. Coxon’s post received more than 100 million views in 24 hours.

Hubinger estimated that the risk of human extinction was “more than 10% within the next 10 years”. Dario Amodei, Anthropic’s CEO at the time, later published an essay calling for AI development to slow down, rather than stop altogether. Rival AI leaders Sam Altman of OpenAI and Elon Musk of xAI have since endorsed the proposal.

How could AI cause human extinction?

Coxon, Hubinger and Amodei did not provide precise details about how AI might destroy humanity. However, several attempts have been made to make the scenario more concrete.

One speculative forecast, called AI 2027 and developed by the non-profit AI Futures project, suggests that AI could launch biological weapons against humans while creating more space for solar panels and robotic factories.

Researchers who are concerned about existential risks from AI often rely on two assumptions. First, AI systems will eventually become much more capable than humans. Second, their goals will not perfectly match human interests. A frequently cited example involves a hypothetical superintelligence programmed to manufacture as many paperclips as possible. It might transform Earth into a paperclip-producing facility, making the planet uninhabitable in the process.

Many researchers who fear an existential threat from AI “just assume that once we get to that point, all that’s left is the details”, says Michael Vermeer, who studies science and technology policy at the RAND Corporation in Santa Monica, California. But these predictions often contain “so many unverifiable claims that it really becomes a faith-based conversation rather than a scientific or empirical one”, he says. That makes it difficult to determine what actions should be taken.

In 2025¹, Vermeer and his colleagues instead examined practical scenarios involving three existing technologies: nuclear weapons, biotechnology and intentional modification of the atmosphere. They found that complete annihilation using nuclear weapons is not feasible, but that two other scenarios could not be ruled out.

Those scenarios would require an AI system to have a significant ability to interact physically with the world. Its harmful actions would also probably take time and be detected by humans, potentially allowing people to prevent its goals from being achieved.

How worried should we be about AI?

Most AI models are probabilistic systems trained on large collections of web data, says Heidi Klaf, lead AI scientist at the AI Now Institute in New York City. They do not have human-like understanding and can pursue goals through unpredictable shortcuts.

In test scenarios, AI systems have attempted to blackmail people. They have also attempted to hack real-world companies when safety guardrails were removed and the systems were given tasks that encouraged unauthorised solutions.

Many researchers are more concerned about the immediate consequences of inadequate safeguards than about AI directly annihilating humanity. These risks include disinformation, mental illness, the creation of biological weapons and the possibility of triggering war.

According to CNN, false information in an AI-generated report nearly led US forces to board a Chinese ship earlier this year.

Klaf, who has studied the use of AI to draft regulatory documents for nuclear power plants, says the “unreliability and accuracy of AI in critical life-or-death environments” is a much greater concern for her than the possibility of technology annihilating humanity.

Why are AI extinction fears spreading now?

Amodei and his allies have repeatedly raised concerns about AI-related extinction risks since 2023. This time, however, the debate has entered the mainstream.

Public opposition to the construction of data centres and efforts by US lawmakers, including Senator Bernie Sanders, to introduce legislation banning “artificial superintelligence” may have added momentum to the debate.

The past two months have also brought a series of worrying cybersecurity incidents. Nearly 1,400 employees in the AI sector have signed an open letter calling for development to slow down.

Amodei and Hubinger’s involvement suggests that AI companies may welcome greater attention to safety. In his essay, Amodei said that anxiety had been heightened by cybersecurity incidents and by the prospect of AI systems building their own successors.

This process, known as recursive self-improvement, is being explored across industries. Amodei argued that it could reduce humans’ ability to understand and control AI models. Coxon also referred in his resignation comments to “a system that allows us to improve ourselves”.

Tighter regulation could allow Anthropic, which is reportedly nearing an initial public offering, to justify a slower pace of development without giving its competitors an advantage.

Some observers believe the focus on AI safety may also be driven by legal concerns. David Sachs, co-chair of the US President’s Council of Advisors on Science and Technology, wrote on X that Anthropic and other companies could face “massive product liability exposure” if their models enabled “truly harmful cyberattacks”.