Much about the global AI panic — could it really “kill us all by the end of the decade”? — is genuinely new, and thus extra-scary. Then again, the specter of human-engineered extinction is hardly novel; we have about eight decades of experience in thinking about it.
So what metaphors and lessons can we draw from the field of nuclear weapons as we wrap our minds around what lies ahead in artificial intelligence?
One of the oldest metaphors came from J. Robert Oppenheimer, sometimes called “the father of the atomic bomb.” Writing in 1953, when Washington and Moscow were already able to annihilate one another while irradiating the world, he likened America and the Soviet Union “to two scorpions in a bottle, each capable of killing the other, but only at the risk of his own life.”
In game theory, which was then in its heyday, Oppenheimer’s analogy was often paired with the Prisoner’s Dilemma, which analyzes the incentives for cooperation or betrayal between two people who are arrested and separately interrogated. Its upshot is that cooperation would be better for both, if they could trust each other; but since they can’t, defection may be the more rational choice.
These scorpions, or prisoners, bear a passing resemblance to contestants in today’s AI revolution, whether they’re companies (OpenAI and Anthropic, say) or the two AI superpowers, the United States and China.
Yes, it would be safer, for them and all of humanity, if, as Anthropic’s boss urges, the rivals start cooperating by “pacing the frontier” — voluntarily slowing down to give regulators time to catch up. But each must assume the other could cheat, so it may be safer to keep racing, even if that leads to dead scorpions or miserable prisoners.
See more: New Momentum in the Nuclear Fuel Chain
“Whoever wins AI wins,” US President Donald Trump has said, believing that the US is ahead by a nose and seeing no reason to slow down. Beijing, as long as it’s slightly behind, sees no way to ease off without losing. So both hurtle forward.
Believe it or not, Oppenheimer’s metaphor may nonetheless be too optimistic — or not pessimistic enough — to fit today’s context. When rational players go round after round (as opposed to just once) in the Prisoner’s Dilemma, they can reach stability. That’s what happened during the Cold War in the form of deterrence.
But as leading AI scholars have been making clear all week, nothing about AI, which has advanced by leaps and bounds just in a few months, is stable. A more apt metaphor may be the one used by Thomas Schelling, a titan of nuclear strategy and winner of the Nobel Prize for his work in game theory.
As the Cold War advanced, Schelling imagined a situation in which two adversaries find themselves “in a room full of gasoline,” each holding matches. Neither can credibly threaten to strike his match, since the conflagration would kill them both. But each can, through provocation or miscalculation, raise the risk of incineration by inadvertent escalation or accident.
Schelling was describing a situation of brinkmanship that is captured in game theory with Chicken, a scenario in which two drivers race toward each other. If one swerves, the other wins. If both swerve, neither wins, but both are safe. If neither swerves, they crash. Translated to today’s AI situation, it looks as follows.
Whereas the Prisoner’s Dilemma allows for rationality (even for scorpions, in this mixed metaphor), Chicken describes something more frightening: the loss of control and, in Schelling’s phrase, a “threat that leaves something to chance.”
That seems to describe the AI race, as companies in the US, and presumably China, are creating large-language models and other agents that autonomously manipulate or attack other systems and have already done so.
As if that weren’t scary enough, the industry has also entered the phase of “recursive self-improvement,” as AI agents autonomously build the next generation of even more powerful models. It’s as though, in a gasoline-filled room, the matches were not only making sparks all by themselves but also building new matches, and then sophisticated flame throwers.
This makes one more metaphor relevant: the sword of Damocles. It was used by President John F. Kennedy in a speech at the United Nations. “Every inhabitant of this planet must contemplate the day when this planet may no longer be habitable,” he said; “every man, woman and child lives under a nuclear sword of Damocles, hanging by the slenderest of threads, capable of being cut at any moment by accident or miscalculation or by madness.”
That madness resembles visions I’ve had in the past week of AI agents going rogue. If Oppenheimer’s scorpions make us think of retaliation and Schelling’s gasoline-filled room evokes uncontrollable risk, the Damocles sword sums up the panic by positing extinction not for the individual players, but for the species.
The sword of Damocles allows for hope as well as despair. Hope, because in the years after Kennedy’s exhortation the world achieved a form of détente, with treaties for non-proliferation and arms control (an era that is now sadly turning into one of rearmament again). Today, that would mean collective pacing, plus mutual verification, as well as coordinated regulation by Washington, Beijing and others.
Better yet, the US and China could start playing a different game, called Stag Hunt. It imagines two hunters who must decide whether to cooperate and kill a stag that would feed both of them for a long time. If either defects, the other catches nothing, while the defector only gets a rabbit.
In today’s context, the stag represents the huge benefits that AI could bring to humanity, from curing its diseases to fixing its other problems. The upside of nuclear technology was mainly electricity generation; the bonus of superintelligence could be much larger.
What fills me with awe is the point at which the metaphors and game theory of the Cold War era break down. The scorpions, prisoners, car racers and other characters in our thought exercises all represented humans with agency interacting with an exogenous technology, namely fission. But what happens when the AI models themselves become players?
AI doesn’t share, at least not intrinsically, our human instinct to avoid catastrophe — a difference euphemistically called the “alignment problem.” That one insight is enough to explain why it’s imperative, as the jargon has it, always to keep a human in the loop, and to go no faster than human speed. This too is a lesson already learned by leaders during the late Cold War: An out-of-control AI contest, like a nuclear war, cannot be won, and must never be fought.
A message from Advisor Perspectives and VettaFi: Discover something new! Click here to register for our upcoming webcasts.
Bloomberg News provided this article. For more articles like this please visit
bloomberg.com.
Read more articles by Andreas Kluth
