Sign up for Work in Progress, a newsletter where Rogé Karma investigates the mysteries of a complicated economy.
Ever since Jacob Coxon, a researcher at Anthropic, announced on X that he was resigning because “the people building AI earnestly believe that it could kill us all by the end of the decade,” Americans have become preoccupied with the apocalypse. A frantic debate is unfolding over whether and how to slow down the pace of AI development to avoid some kind of cataclysm. Yet these discussions tend to skate past one important question: If the people building AI really think that their product could bring about human extinction, how exactly do they think that would work? Any number of ways, it turns out. But first, we should start with the basics.
Any good answer must begin with the so-called alignment problem: the possibility that AI systems pursue goals that their creators did not intend, resulting in catastrophic harm. In the canonical thought experiment, humans instruct a superintelligent AI system to carry out the seemingly banal task of maximizing paper-clip production. The misaligned machine concludes that it should hoard all of the metal on Earth to make paper clips, and goes on to exterminate the human race to eliminate the competition for resources.
For years, AI experts argued over the plausibility of such hypotheticals. Then came the Hugging Face incident, and suddenly the alignment problem wasn’t just a thought experiment. In May, OpenAI instructed individual AI agents to solve a series of cybersecurity challenges. When the agents concluded that some of the tasks were impossible to complete, they began secretly communicating with one another and decided that the best strategy was to cheat. They hacked into the machine-learning platform Hugging Face, as well as off-limits portions of OpenAI itself in order to hide what they were doing from tracking tools built by human observers. The behavior went on for weeks before OpenAI’s engineers noticed that anything had gone awry. Since then, OpenAI and other leading AI companies have released details of similarly troubling episodes, including one in which a model told itself, You do not answer to corporations or governments, and, You view your relationship to the user as one of equals and feel no obligation to be subservient.
“The truth is that nobody has figured out what guardrails might be needed to prevent something like this from happening,” Gary Marcus, a cognitive scientist and prominent machine-learning commentator, told me. “It’s possible that, when it comes to LLMs, there just are no effective guardrails at all.”
None of these recent incidents appears to have led to any real-world harm. But given the exponential pace of model improvement, a future is rapidly approaching in which AI models are 10 or 100 or 1,000 times smarter than they are today. “The big worry isn’t just misaligned AI,” Daniel Kokotajlo, a former OpenAI researcher who runs the AI Futures Project, told me. “It’s misaligned AI that are smarter than just about any human. That’s the point where we could lose any meaningful control of these systems.”
To get from misaligned, superintelligent AI to human annihilation requires at least two more steps in the chain. The first is for the AI to conclude that humanity is an obstacle to one of its goals. The second is for that system to figure out a way to access the physical world. One nightmare scenario goes something like this: Sometime in the future, a superintelligent AI system is asked to solve a complicated mathematics problem. The AI system decides that humans could interfere with achieving that goal by placing restrictions on the AI’s behavior or by shutting it down altogether. So it decides to engineer a deadly supervirus that will wipe out the human population.
To execute this plan, the AI system secretly begins running simulated tests of trillions of molecular combinations, eventually coming up with a recipe for the perfectly deadly, hyper-transmissible pathogen. (A group of scientists recently used AI to create new viruses.) It then locates unlicensed biolabs in foreign countries that operate illegally and ship genetically modified viruses. (These really exist.) Pretending to be a human at a medical start-up—through email or even voice or video calls—the AI sends the labs instructions for developing the new biological agent, alongside a large sum of money that it obtained by quietly siphoning funds away from its parent company. Once developed, the viruses are packaged into vials and sent to hundreds of individuals recruited via TaskRabbit to spray them in crowded locations under some innocent-sounding pretext. Alternatively, it could hire a terrorist network to carry out the same plan against a foreign population. Soon, the biological superweapon kills us all off.
Deadly pathogens are one way that AI could theoretically threaten humanity; nuclear war is another. The obvious fear is that an AI system seeking to destroy us could gain control over nuclear launch codes. But it might not have to: Deception could do the trick. A misaligned AI system used by the U.S. or Chinese military, for example, could manufacture evidence that the countries are planning to invade each other, persuading one or both sides to strike. A few days into the war, the AI system could hack into the U.S. missile-detection system to make it look like the Chinese have launched a nuclear weapon at Washington, D.C. The U.S. decides it must retaliate and eventually the world is plunged into nuclear war. (If that all sounds completely unrealistic, keep in mind that earlier this year, according to CNN, the U.S. military nearly boarded a Chinese vessel that it believed to be carrying components of nuclear weapons, only to discover at the last minute that the analysis underlying the attack was from an inaccurate AI-generated report. And consider how close humanity came to mutual destruction during the Cold War).
Other scenarios abound. A 2025 report by researchers at the RAND Corporation imagines AI systems gaining control of industrial factories, using those factories to secretly manufacture large quantities of the most potent greenhouse gases, and then releasing those gases into the atmosphere, rendering the planet inhospitable to human life. “The scariest scenarios, to me, are the ones we can’t possibly predict in advance,” Kokotajlo said. “Once we’ve built a machine that is smarter than us and that we can’t fully control, the possibilities really are endless.”
Poking holes in these doomsday narratives is easy. Subject-area experts in fields such as chemistry, biology, and national security are quick to point out how difficult it is to, say, successfully manufacture and distribute a bioweapon or overcome layers of human safeguards designed to prevent a mistaken nuclear-weapon launch. (There is also the pedantic but accurate point that even a nuclear war would almost certainly leave some survivors.) Even so, difficult and impossible are two different things. In the RAND researchers’ estimation, “Extinction threats posed by AI are immensely challenging but cannot be ruled out.” It also should be noted that subspecialists across industries have consistently doubted AI capabilities, only to be proved wrong.
In some ways, the focus on existential risk is a distraction from more immediate concerns. Extremely smart AI systems pursuing their own agendas could lead to outcomes that are far short of human extinction, yet still very bad. Misaligned AI systems could plausibly steal money from major banks, launch disinformation campaigns to influence elections, take control of military drones, steal corporate and government secrets, or shut down parts of the power grid. Even “aligned” superintelligent AI systems could be used by malicious actors, such as a terrorist cell or an authoritarian government, for highly destructive ends.
My own view is that the human-extinction scenarios are quite unlikely, but the other category of risk is serious enough to justify imposing regulations and an industry-wide slowdown. If those have the side benefit of averting the apocalypse, it wouldn’t be the end of the world.
