All Too Human

 One of the greatest sources of confusion in the interminable AI debate is the fact that many of the people building the most sophisticated models also hold a quasi-religious belief in the inevitability of an eventual AI "singularity," which they have long endowed with divine and apocalyptic traits. 

In trying to figure out how seriously to take both their warnings of "existential risk" from AI, as well as their utopian promises of eventual "superabundance" or immortality from the same technology, then, our challenge is constantly to have to figure out how to separate out the scientific from the mythological elements of the way the designers are thinking about their new machine. 

I therefore responded eagerly to Cal Newport's invitation in a recent New York Times column to think in more reasonable, sober-minded terms about this technology for a change. The leading AI labs, he writes, "are deeply connected to a strain of apocalyptic futurist thinking that believes something like an all-powerful superintelligent A.I. god is inevitable." 

While its makers may increasingly view AI as a god, though, he suggests we should remember that it is really just a machine. As Newport writes: "Though the term 'agent' conjures creepy vibes, they’re actually normal computer programs that repeatedly prompt an L.L.M. for instructions and then execute whatever the model suggests. Agents aren’t worrisome when constrained to specific tasks with common-sense controls."

This was all reassuring to read. And I for one went around for a week telling everyone that the next step we really need to take is to "demythologize" the whole AI discussion, and remember that we are not dealing here with a god or a devil, but a human-made technology. 

But I've been reading Dwight Macdonald's essay collection The Responsibility of Peoples this past week, and he offers us some reason to ask—even if this reminder is all perfectly true—how reassuring should we really find it? 

It's often struck me before how much the AI debate today resembles the argument over nuclear weapons in the mid–twentieth century. And indeed, Macdonald's 1946 essay on "The Bomb," from this collection, seems remarkably reminiscent of the particular back-and-forth we've just laid out above. 

Macdonald quotes Einstein on the subject of atomic weapons: "No one in the world should have any fear or apprehension about atomic energy being a supernatural product. In developing atomic energy, science merely imitated the reaction of the sun's rays. Atomic power is no more unnatural than when I sail my boat on Saranac Lake." 

To Einstein's soothing words, however, Macdonald retorts: "As though it were not precisely the natural, the perfectly rational and scientifically demonstrable that is now chilling out blood! Indeed, all of us except a few specialists know as much about witches as we do about atom-splitting; and all of us with no exceptions are even less able to defend ourselves against The Bomb than against witchcraft. No silver bullet, no crossed sticks will help us there."

He concludes: "[T]he effort to 'humanize' The Bomb by showing how it fits into our normal, everyday life also cuts the other way: it reveals how inhuman our normal life has become."

This is a pretty unanswerable objection to my efforts to "demythologize" AI as well. I may be correct that it is "only" a machine; just as I was correct in earlier years when I argued that it was "only" a statistical model parroting human speech. 

In both cases, however, what we eventually have to grapple with is: even if the new technology is "only" these things—just as Einstein was right that nuclear power is "merely" harnessing the power of the sun ("'Merely' is good!," Macdonald observes in a sarcastic editorial aside at that point)—we next have to ask: how much good does that really do us? 

Suppose AI never becomes a malevolent superintelligent hive-mind, as still seems unlikely to me. The technology as it currently exists nonetheless really does pose immense dangers. As Bill Gates observed in a recent interview with Ezra Klein: "A.I. has crossed the threshold that its ability to empower a bioterrorist to kill hundreds of millions — that exists today. The ability to do a cyberattack that scrambles all of the bank accounts, shuts down the electric grid — that exists today, and we know that’s the case."

To Macdonald's point—maybe we'd prefer to take our chances with witches and vampires. Merely supernatural agencies suddenly seem like they're operating on a much more human and manageable scale than this awesome power unleashed by our own all-too-"rational," human-made technology. 

"[L]ife is becoming more and more like a Science Fiction story," Macdonald observes in this 1946 essay. We often say the same thing today. Indeed, many of the makers of today's LLM's were influenced by science fiction. But that doesn't mean they can't fulfill their own literary-induced speculations. As I pointed out recently, citing Oscar Wilde on this point: "Life imitates art," as often as the reverse. 

Macdonald concluded: "All honour then to the as yet anonymous British and American scientists—Men I would rather say—who were so wisely foolish as to refuse their co-operation on The Bomb!"

Macdonald argues, much like the Anthropic researcher who recently resigned (prompting a wave of headlines about AI "existential risk"), that the only ethical thing to do in the face of efforts to construct such a potential dangerous technology is to refuse to help build it. He would doubtless applaud the researcher for his choice, as well as those less prominent scientists who never even started working for one of these frontier labs in the first place, due to the same moral reluctance and scruples. 

Of course, the response to that suggestion is usually two-fold: if we give-up on the effort to create super-powerful AI, we will lose the new arms race with China; also, we will miss out on the age of Superabundance that this technology could unleash. Macdonald was of course acquainted with these arguments for nuclear power as well. 

"What becomes of the[ir] chief argument," he writes in The Root is Man, "—that out of present evil will come future good — if we now confront the possibility that there may not be a future?"

In many of his recent interviews, Ezra Klein has asked AI developers a version of the same question: if this technology is as dangerous as you say, why build it? 

The "mythological" beliefs may be part of the answer. Newport warns in his essay cited above about the frightening possibility that AI leaders' quasi-religious beliefs about the AI "singularity" may be "driving these specific labs toward more reckless behavior [...] If you believe that winning the race to superintelligence is necessary to save humanity, you might not worry too much about collateral damage along the way."

Macdonald's question in The Root is Man that we just cited seems like the only apt response to the view that this much "collateral damage" can possibly be justified by what is essentially an unknowable future—indeed, one that, by the own AI apocalypticists' accounting—may not arrive at all, since their machine god may destroy us all before it becomes "aligned" with human interests. 

It is rather odd, as a friend observed to me in conversation the other day, to see how the "effective altruists" of Silicon Valley moved from talking about buying mosquito nets for the Third World a few years ago to now driving headlong into the construction of a machine superintelligence that has a 10% chance, by their own reckoning, of wiping out the human race. 

The people building AI, while openly embracing these same odds, seem to think that catastrophic cyber attacks or other "rogue AI" mishaps along the road to alignment are worth it for the inestimable gains—indeed, the infinite immortality—that they sincerely believe their all-powerful machine god will enable for whatever handful of human beings survives the transition period. 

Their arguments sound almost like a philosophy department thought experiment used to illustrate the self-evident absurdity of extreme utilitarianism. "Wouldn't it be worth potentially wiping out humankind for the chance than a few of us get to live in perpetual bliss as uploads to the cloud?"  

And here is where we get to the point that utilitarianism becomes less and less plausible as a moral theory the more its calculations factor in essentially unknowable claims about the future. As Arthur Koestler observes in one of his many essays on the subject of what he calls the conflict between "morality and expediently," a surgeon may be justified in cutting into a person's body for a known and immediate good, but the larger the intervention, and the further its consequences stretch into the future, the less warrant the surgeon would have to proceed. 

The slim and rather outré possibility that some of us might be immortalized as digital consciousnesses someday—even if we could bring ourselves to regard that as a desirable thing—seems like precisely the sort of future outcome that is so remote and unknowable that it cannot possibly be used in an utilitarian calculus to justify present loss. 

As Koestler elsewhere writes: "At what precise point does the healer's lancet change into the butcher's hatchet? [...] A system of ethics based on quantitative criteria is a slope on which there is no halt because all is a matter of degrees and not of (qualitative) values."

Koestler had in mind the people who justify Stalinist terror on the improbable theory that it would someday bring into being a communist utopia. But we can say the same of the "effective altruists" and Silicon Valley AI utilitarians who say that it is worth unleashing unprecedented havoc on present generations—in the form of job loss, cyber insecurity, the risk of attacks on critical infrastructure—for the sake of some imagined age of coming AI super-abundance or techno-immortality. 

As Macdonald observes elsewhere in his essay on "The Bomb": "The flimsiness of these justifications is apparent; any atrocious actions, absolutely any one, could be excused on such grounds. For there is really only one possible answer posed by Dostoievsky's Grand Inquisitor: if all mankind could realize eternal and complete happiness by torturing to death a single child, would this act be morally justified?"

And so, the generations in present time who are slated to be sacrificed for AI development cannot be expected to welcome the experiment. To quote Isaiah Berlin: "The one thing that we may be sure of is the reality of the sacrifice, the dying and the dead. [... T]he ideal for the sake of which they die remains unrealized. The eggs are broken, and the habit of breaking them grows, but the omelette remains invisible."