Nobody is pining for megadeath. But megadeath is not the only thing that makes the Doomsday Machine petrifying. The real terror is in its autonomy, this idea that it would be programmed to detect a series of environmental inputs, then to act, without human interference. “There is no chance of human intervention, control, and final decision,” wrote the military strategist Herman Kahn in his 1960 book, On Thermonuclear War, which laid out the hypothetical for a Doomsday Machine. The concept was to render nuclear war unwinnable, and therefore unthinkable.
Kahn concluded that automating the extinction of all life on Earth would be immoral. Even an infinitesimal risk of error is too great to justify the Doomsday Machine’s existence. “And even if we give up the computer and make the Doomsday Machine reliably controllable by decision makers,” Kahn wrote, “it is still not controllable enough.” No machine should be that powerful by itself—but no one person should be either.
The Soviets really did make a version of the Doomsday Machine during the Cold War. They nicknamed it “Dead Hand.” But so far, somewhat miraculously, we have figured out how to live with the bomb. Now we need to learn how to survive the social web.