The AI people are talking about the end of the world again. A senior scientist at Anthropic wrote Tuesday on X that “we really do earnestly believe AI could kill all humans! I personally think it is >10% within the next decade.”
That was a strikingly specific forecast, but it’s consistent with the language used by many on the frontier of artificial intelligence to describe the possible consequences of their products. Anthropic CEO Dario Amodei warned in a January essay that machines “could decide that it is justifiable to exterminate humanity.” Last year, he hazarded “a 25 percent chance that things go really, really badly.” OpenAI CEO Sam Altman, who has recently been more careful, said in 2023 that “the bad case — and I think this is important to say — is, like, lights out for all of us.”
The surge of AI doomsday talk, prompted by a researcher’s highly publicized resignation from Anthropic, brings to mind another entrepreneur’s apocalyptic musings. Sam Bankman-Fried, creator of the cryptocurrency exchange FTX, said in 2022 that he would be willing to accept a 49 percent chance of destroying the world in exchange for a 51 percent chance of creating an additional, second world somewhere in the universe.
At Bankman-Fried’s 2023 fraud trial following FTX’s collapse, the prosecution emphasized his bizarre value system. It prompted Caroline Ellison, an FTX executive who pleaded guilty to fraud charges, to testify that Bankman-Fried “would be happy to flip a coin if it came up tails and the world was destroyed — as long as if it came up heads, the world would be, like, more than twice as good.”
Bankman-Fried’s example is relevant to AI-doom discourse for two reasons. First, it helps give context to the AI elite’s end-of-the-world obsession. Bankman-Fried — like many key figures in the AI industry — was strongly influenced by “effective altruism,” a strange ideology with an epicenter in Berkeley, California. EA is best described as a nerdy and extreme form of utilitarianism, which holds that morality is all about achieving “the greatest happiness of the greatest number.” The coin-flip thought experiment would, on average, increase overall human happiness. Most all-encompassing worldviews have a theory of the end of days, and for EA, annihilation by autonomous machines ranks highly.
But a technocratic attitude toward the apocalypse can be incomprehensible or repulsive to ordinary people. That was evident at Bankman-Fried’s trial, where he was convicted on all counts by the jury and harshly sentenced by the judge. (On Thursday, he appealed his conviction to the Supreme Court.) AI companies will inevitably face a large number of lawsuits in the coming years as their products become more ingrained in American life. When the legal wrangling begins, the industry’s extravagant (and in some cases, strangely lighthearted) talk about the chance that their agents could bring about human extinction won’t play well in court.
AI agents might well cause serious mischief that is short of apocalyptic. They could inadvertently steal money or expose private information, for example, in the course of completing a legitimate task. Such scenarios seem more plausible after the revelation that OpenAI’s bots breached another company’s servers this summer without OpenAI’s knowledge. They would probably prompt civil lawsuits, not criminal cases like Bankman-Fried’s. But the same strategy that helped federal prosecutors convict Bankman-Fried — painting their target as indifferent to terrible risks — would help plaintiffs persuade jurors to punish AI companies whose models cause harm.
OpenAI and Anthropic are too young to have faced major lawsuits like the one Meta just settled with state attorneys general over alleged harms to teenagers from Facebook and Instagram. The crux of such cases is negligence — whether the company knew or should have known its product would injure someone. It can be hard to demonstrate a company’s state of mind, and plaintiffs in the Meta case devoted enormous legal resources to establishing what its executives knew.
But imagine if Meta CEO Mark Zuckerberg had earlier said in public, “We think there’s a significant chance our product will hurt teenagers’ development.” The case would’ve been a piece of cake for the state AGs — and the ultimate settlement would have been far north of the $18 billion Meta agreed to last month. People at AI companies are predicting far worse than developmental harm to some teenagers. They’re openly discussing the possibility that the autonomous computer programs they’re working to create could cause death on an unimaginable scale.
There are a number of reasons for the AI elite — seemingly against their interest — to give fodder to the “doomers.” Perhaps they’re just saying what they think and, following Bankman-Fried’s utilitarianism, think their work is justified because the upside probabilistically outweighs the downside. Perhaps they get a thrill from suggesting that they hold the future of humanity in their hands. Perhaps they want to create pressure for government regulation.
Legally, it might not matter. The record is now full of potentially devastating evidence for lawyers hoping to prove negligence (or worse) in the AI industry. Politicians talk about throwing a wrench in AI with new regulations. But the plaintiffs’ bar already has plenty of causes of action available if an AI product malfunctions. The companies might have put themselves in a more perilous position than they realize.
Bankman-Fried’s defense argued that the fallen crypto tycoon “doesn’t make decisions with malice in his heart. He makes decisions with math in his head.” That didn’t move the judge, Lewis Kaplan, who cited Bankman-Fried’s strange moral calculations as a reason for imposing a 25-year sentence. He described Bankman-Fried as “a man willing to flip a coin as to the continued existence of life and civilization on Earth, if the chances were imperceptibly greater that it would come out without that catastrophic outcome.”
To the extent that the AI elite are giving the impression that they inhabit a similar moral universe, they’ve made themselves prime legal targets even if nothing like their worst-case fantasies ever comes true. Ensconced among San Francisco technologists with esoteric ideas, they might not sense how alien their value system would appear to a jury.
Even though Bankman-Fried was convicted of fraud to the tune of $11 billion, his customers recovered their money in the bankruptcy process because some of his investments paid off. One of them: a $500 million stake in Anthropic, which surged in value after FTX’s demise. Bankman-Fried might not be the last high-IQ entrepreneur whose indulgence in end-times philosophizing is used against him in a court of law.
The post Why the Sam Bankman-Fried case should spook the AI elite appeared first on Washington Post.
