Existential Risk

 The national news media have been in the throes all week of an epic freakout about the specter of "AI existential risk." Like a lot of people, I find myself unsure how seriously to take this. 

We have to start with a recognition—first off—that the people now estimating the risk of human extinction from AI superintelligence at a one-in-ten chance would have given you roughly the same odds five or ten or twenty years ago. 

Many of the people building the most sophisticated AI models, after all, belong to a subculture, tracing their intellectual roots to the Rationalist community, who hold as a quasi-religious tenet of faith that humanity is destined to create a hyper-intelligent AI "singularity." 

Asking these people whether we are "close" to creating such a being is therefore not to pose a scientific question—it's asking them to recite an item of their catechism that they've all learned by heart. 

And it has always been a conviction among some members of this community that the "singularity" will not only arrive in glory, but that it will inevitably bring about the death and/or apotheosis of all human beings with it. 

By comparison with this apocalyptic fatalism, the belief currently expressed by some leading researchers at the frontier AI labs that the odds of our extinction are merely 10% counts as a moderate position. 

I'd say the true situation, then, is something like this: many people building AI belong to a doomsday cult that has always believed in the coming extinction of the human race; the ones leading the frontier labs belong to a sort of mainline offshoot of this cult that holds a slightly gentler version of its doctrines, but which emerged from the same milieu. 

This should give us reason not to take their current warnings about existential risk entirely at face value. 

On the other hand, I find myself haunted by Robert Jay Lifton's observation, in his book on Aum Shinrikyo, that if a technology becomes powerful enough, it can throw our conventional categories of what is "paranoid" and what is "reasonable" into chaos. 

In studying Aum Shinrikyo, Lifton was obviously dealing with a doomsday cult that held a range of extreme conspiracist and paranoid beliefs. On the other hand, the existence of nuclear weapons that really do have the power to end the world makes it impossible to wholly dismiss their beliefs. 

Lifton writes: "The threatening existence of nuclear weapons [...] has radically disrupted all efforts to balance individual death with a sense of human continuity. Such ultimate weaponry has imposed on everyone disturbing images of our capacity to destroy our world and extinguish our species with our own technology, by our own hand, and to no purpose. Small wonder we have encountered during the last half of the twentieth century not only intensified end-of-the-world visions but a troubling sense of these visions as closer to actuality."

Aum Shinrikyo might be a mere doomsday cult anticipating the end of the world, then—like so many millenarian cults past. But there was one big difference. None of those apocalyptic cults had the power to actually bring about the end of the world they anticipated. Whereas the existence of "ultimate weapons" meant that this doomsday cult—if they got their hands on them—might be able to do just that. 

Lifton draws out this point through an extended discussion of the classic case from psychoanalytic literature of Daniel Paul Schreber—who suffered from similar apocalyptic and megalomaniac delusions about the coming end times. But there was an all-important difference between Schreber and the leader of the Japanese terrorist cult: only the latter lived in a time when the deliberate destruction of the entire human race had become a technological possibility. 

As Lifton writes: "The megalomanic paranoia of both Schreber and Asahara required world-ending catastrophe for a sense of transcendent survival, but [... i]n Schreber’s era, there were no nuclear or other ultimate weapons to enter his imagination; nor was he a guru who could attract and maintain disciples. Putting the matter simply, Asahara was a Schreber with disciples and ultimate weapons."

Aum Shinrikyo got as far as murdering innocent people on the Tokyo subway with sarin gas. If they had possessed nuclear weapons, there seems little doubt they might have detonated them in inhabited areas. 

Many of the people building AI hold similarly paranoid beliefs. They too are haunted by "disturbing images of our capacity to destroy our world and extinguish our species with our own technology, by our own hand, and to no purpose."

But they are not Schrebers. They are not just thinking up these apocalyptic scenarios in the psych ward. They are building a technology that really is immensely powerful and dangerous—as it has already proven itself many times to be. 

I don't know that current chatbots have evinced a world-ending potential on the scale of nuclear weapons, to be sure. They may not yet be "ultimate weapons" in that sense. But a chatbot with nuclear launch codes or with instructions to create a super-virus?...

So we start, again, from a recognition of the fact that the people building the most sophisticated AI models are members of a doomsday cult. But we could take that in either of two ways. Maybe it means that, when they warn of risks to humanity, they are just prophesying the fulfillment of their own end times revelation, and we need not take it seriously. 

Or it means—as the case of Aum Shinrikyo should remind us—that doomsday cults in an age of "ultimate weapons" have a dangerous chance to bring about the apocalyptic scenarios they warn us against. 

When these guys insist earnestly that "AI superintelligence" is just around the corner, then; or that "AGI" or the "singularity" is coming; or that "we are only a few weeks or months away from achieving recursive self-improvement," then we will be past the point of no return for humanity—we should consider the possibility that they are "hell bent on a self fulfilling prophecy" (to quote William Gaddis's diagnosis of the millenarian religious right). 

That's why I find myself—even when confronted with a belief system that has all the contours of a classic paranoid doomsday cult—unable to simply dismiss it for this reason. We live in an age of "ultimate weapons," after all; which—as Lifton says—disrupt our formerly neat division between what is mere megalomaniac delusion and what is credible risk. 

A random person in our society now has access to a technology that really might have world-ending potential. 

It's a little hard to have sympathy with the people who built these doomsday machines and are now quitting in protest and telling government to constrain their own creation. It's even harder to sympathize with those who stay and continue to build them and make them more powerful—while freely admitting in public that they share these concerns about existential risk and really hope the government will step in to tie their hands to stop them from building what they can't seem to help but build. 

As a character observes in Cormac McCarthy's Stella Maris: one only has so much patience for those scientists who built the world-ending atomic bomb in the Manhattan Project and then dined out the rest of their lives on their profound, publicly-uttered "second thoughts" about their role in creating this monster. "[T]hey should have thought of that sooner [...] they should have had first thoughts," as the character observes. 

But the people building these models plead with us that they have no choice. They are locked in a competitive arms race. They are players in a game theory problem of collective action, which is why they need world governments to step in to prevent by force the proliferation of the very doomsday machine they are creating. 

After all, as Ezra Klein points out in the latest episode of his podcast: "[I]f you talk to anyone in Washington or at the A.I. labs about this, you just crash into the shoals of: Well, what about China? [...] If we slow down, we will lose the A.I. race to China. And as dangerous as it is to build these things we can’t control, it is even more dangerous to have them in China’s hands if they’re not in ours."

This argument was used to defend the creation of nuclear weapons during the Cold War as well, of course. And indeed, as Edmund Wilson shows in his Apologies to the Iroquois, it's been used to defend just about any corporate investment or building project that people find disruptive and don't want in their backyards. 

When an Indigenous community faced displacement from their lands for a hydroelectric project in the 1950s, Wilson writes that the notorious "power broker" and "master builder" Robert Moses penned an op-ed defending the mega-project on competitive arms race grounds. At the close of the piece, Moses darkly warned: "How our democratic system can survive such stultifying domestic weakness, incompetence and ineptitude in the ruthless, world-wide competition with other systems of government more incisive and less tolerant of obstruction, is more than I can figure out." 

This is essentially the same argument that AI developers make about data centers today. Public backlash to AI has largely taken the form of opposing new construction of these facilities in their towns and neighborhoods. Many AI developers see this as a backward-looking Luddite reaction to their technology—even as they publicly say that they think their inventions may kill us all and beg the government to regulate them in order to "pace" their development. 

But why not let the public opposition to the further construction needed to carry on AI development be the thing that "paces" the development? The response to this is always—as Klein points out—because then China will surpass us in the arms race. 

The implication being: we, with our democratic institutions and community consultation procedures for new construction projects are hamstrung in the competition; whereas an authoritarian system that can "just do things"—a system "more incisive and less tolerant of obstruction" than our own, in Moses's ominous phrase, will spring ahead.

But what if the only way to pace this technology is, well, to pace it. Stop building the infrastructure to train and develop it. 

Looking back with the benefit of hindsight, after all, we can say that Moses was wrong. We didn't lose the Cold War because we had too much community consultation and democratic safeguards on new construction. To the contrary, our ultimate victory over the Soviet Union in the Cold War came at a time when the United States was adding new environmental standards and new community review procedures on hydroelectric dams and other megaprojects. 

We didn't win by becoming less democratic and imitating other systems that are "less tolerant of obstruction." We won the Cold War while being as obstructive as ever. 

So too, I don't think we need to fatalistically abandon ourselves to the megaprojects of the AI master builders of our time, out of some far-fetched belief that such submission will be necessary to prevail in the latest arms race.  Edmund Wilson wrote of the mega-developments backed by people like Robert Moses that: "The struggle to restrain the[m] is undoubtedly at the present time one of the principle problems of American life." Perhaps the struggle against AI data center construction is the principal problem of our own.

Wilson was not persuaded, as he puts it, by Moses's insinuation that we can only save "our democratic system" by becoming "less democratic ourselves." And so too, today, the answer to potentially dangerous Chinese AI is not to build equally dangerous and unchecked American AI, nor to abandon the democratic and community safeguards we currently place on new construction projects in order to expedite its creation (though the Trump administration's EPA is presently attempting to do exactly that). 

Rather, we should trust to the democracy that has never failed us yet in any geopolitical contest. All the more so in light of this week's news cycle: If even the people building frontier AI models are telling us they are having "second thoughts," and that they fear the results of their own handiwork—by all means, let us help them to stop building it.