As the tech elite prepares to make billions from the A.I. revolution, the public mood about this technology has taken a dark turn. Those concerns intensified after experimental OpenAI models autonomously hacked into a third-party company’s servers in July, prompting widespread alarm. This week, Senator Bernie Sanders reacted to this grim mood by calling for a pause on all “advanced A.I. development.”
These sentiments are magnified by how A.I. chief executives like Sam Altman and Dario Amodei insist on talking about this technology. Instead of reassuring their customers, they’re often among the first to point out the grave dangers their companies might be unlocking and react to incidents involving their own products as if they were third-party observers of an unstoppable force of nature. True to form, on Wednesday, Mr. Altman grimly promised that his company’s next generation of models were “going to be sobering for everybody.”
Why do these leaders act this way? Some argue it’s all an elaborate marketing strategy. Others assume it’s a natural response to a terrifying technology. The more I study these figures, however, the more convinced I’ve become that something else helps explain this passive dismay: They’re weirder than we realize.
The story begins in the mid-1990s, when a precocious teenager named Eliezer Yudkowsky discovered the email list for a futurist movement known as the Extropians. The group promoted visions of a technology-powered utopia where we repair our aging bodies with nanomachines and back up our minds to high-powered computers.
Mr. Yudkowsky was entranced by the Extropians’ radical optimism and the movement’s interest in the idea that a “superintelligent” A.I. would yield a golden future. As the science journalist Adam Becker notes in his insightful book “More Everything Forever,” the possibility of solving all of humanity’s problems with an A.I.-driven singularity “transformed” Mr. Yudkowsky, who once posted “I think that I can save the world.”
Despite his fervor, Mr. Yudkowsky eventually began to fear that controlling superintelligent machines would be more challenging than techno-optimist groups like the Extropians realized. He concluded that his peers weren’t thinking clearly or rigorously enough about the issue. Between 2006 and 2009, he went on a writing spree, producing hundreds of thousands of words, largely posted on the blogs Overcoming Bias and LessWrong.
These posts came to be known as “The Sequences,” and they were eventually edited into a six-volume book collection titled “Rationality: From A.I. To Zombies.” The community dedicated to these ideas began calling themselves the Rationalists. At the core of their belief system was the idea that superintelligent A.I. run amok was the gravest threat to humanity, and that developing precise, rational thinking habits was the best way to avoid extinction.
For a certain subset of young engineering types in Silicon Valley, these ideas were immensely attractive. This makes sense, given that they cast young engineering types as the literal saviors of humanity. “When I came across LessWrong as a senior in college I was in some sense an empty husk waiting to be filled by something,” a former Rationalist named Qiaochu Yuan wrote. “Eliezer Yudkowsky was saying things that made more sense and captivated me more than I’d ever experienced.”
Some Rationalists began living in group houses in the Bay Area, where earnest conversations about A.I. safety and reducing cognitive biases mixed freely with polyamory and psychedelics. The movement continued to grow, soon overlapping and influencing related philosophies such as effective altruism (E.A.), which began to shift its focus from making charitable giving more efficient to fighting existential risks like superintelligent A.I.
Mr. Yudkowsky created the Machine Intelligence Research Institute (MIRI), which would go on to attract many millions of dollars in donations, largely from individuals and groups connected to the technology industry who presumably appreciated being cast as heroic figures in a cosmic drama.
Last year, Mr. Yudkowsky attempted to bring his hyper-rationalist thinking around superintelligent A.I. to the mainstream with the publication of “If Anyone Builds It, Everyone Dies,” written with Nate Soares, the president of MIRI. Though the book drew considerable attention, Mr. Yudkowsky’s unusual mixture of pseudoprecision and discursiveness proved off-putting to many. In his review, Stephen Marche noted that the book’s odd structure, replete with “unhelpful parables” and QR codes to scan for additional details, made it read “like a Scientology manual.” This helps explain why, despite this burst of attention, Mr. Yudkowsky and his movement largely failed to gain widespread acceptance from the general public.
If the story of Rationalism ended with the mixed success of Eliezer Yudkowsky’s bid for mainstream recognition, then it would be merely another short entry in Silicon Valley’s long history of eccentric futurist groups. But Mr. Yudkowsky’s movement cannot be dismissed so easily: Among the acolytes that he did manage to attract were some of the most important figures in the current generative A.I. boom.
It was Mr. Yudkowsky, for example, who introduced two DeepMind co-founders, Demis Hassabis and Shane Legg, to Peter Thiel, who became the company’s first major investor. Mr. Legg had become interested in Rationalist-style ideas after attending a talk by Mr. Yudkowsky, and went on to write a doctoral dissertation on the topic of superintelligence.
Rationalism also infused OpenAI. “Eliezer has IMO done more to accelerate AGI than anyone else,” Mr. Altman tweeted in 2023. He went on to suggest that Mr. Yudkowsky might deserve the Nobel Peace Prize for his influence on A.I. One of OpenAI’s co-founders, Greg Brockman, currently the company’s president, reportedly ran a weekly reading group dedicated to Mr. Yudkowsky’s LessWrong forum.
Other members of OpenAI’s board also appear to have been influenced by this type of thinking. Here’s how the technology journalist Cade Metz described Mr. Altman’s 2023 temporary ouster as OpenAI’s chief executive: “Board members with ties to the Rationalist and E.A. movements said they could not trust him to build A.I. for the benefit of humanity.” Emmett Shear, the interim chief executive the board brought in to replace Mr. Altman, was so active in online Rationalism culture that Mr. Yudkowsky named a character after him in his sprawling 600,000-word fan fiction epic: “Harry Potter and the Methods of Rationality.”
Anthropic is similarly connected to this belief system. The company was founded in 2021 by a group of former OpenAI employees, including Dario Amodei and his sister Daniela. As Mr. Becker reports, the Amodei siblings were motivated in part by their concern that OpenAI “didn’t take the alignment problem seriously enough,” a reference to the early Rationalist obsession with aligning superpowerful A.I. to serve human interests. In a 2023 interview, Mr. Amodei echoed standard Rationalist thinking when he casually quipped that there was a 10 to 25 percent chance that A.I. would end humanity.
This reality, that Rationalism deeply influenced many of today’s major A.I. companies, helps us calibrate the unnerving language we’ve been hearing from their leaders. When Mr. Altman declares their pending models will be “sobering” for humanity or Mr. Amodei frets that this technology “will test who we are as a species,” it doesn’t necessarily mean that they discovered frightening new evidence that something catastrophic is imminent.
It is, instead, representative of how Rationalists always think about A.I. In these circles, it’s taken for granted that A.I. capabilities will rapidly accelerate and completely transform the world, and to talk about it in any other way would be considered uninformed. For a Rationalist, the only open question is to what degree their heroic brains, carefully trained in the style glamorized by Mr. Yudkowsky’s “Sequences,” can prevent these transformations from devolving into extinction events.
To be clear, figures like Mr. Altman and Mr. Amodei probably wouldn’t currently describe themselves as Rationalists. As their companies have grown, they seem to be distancing themselves from some of Mr. Yudkowsky’s more extreme ideas. But it still seems likely that these individuals’ entanglement in this movement normalized the otherwise radical idea that A.I. is always close to changing everything. The influence of Rationalism helps explain why this style of speech came so easily to these A.I. leaders.
It helps to ask how things would have been different if generative A.I. had emerged from companies without connections to Rationalism. We can glimpse this counterfactual in the example of China, whose booming A.I. industry is independent of Silicon Valley futurist movements. As Ross Douthat recently explained, whereas the United States is “behaving as though frontier A.I. models could be the equivalent of nuclear weapons,” the Chinese see A.I. as a “lower-risk technology to be shared and commercialized to win friends and influence the world.”
We can find a similar moderation in the American A.I. leaders who have minimal connections to Rationalism. Consider Nvidia’s chief executive, Jensen Huang, who co-founded the company years before Mr. Yudkowsky became an Extropian, and never seemed to have mixed much with Rationalist circles. His frustration with the more extreme talk from figures like Mr. Altman and Mr. Amodei is often palpable, as when, during a podcast interview, he made the following plea: “I appreciate that many of us grew up and enjoyed science fiction, but it’s not helpful. It’s not helpful to people. It’s not helpful to the industry. It’s not helpful to society. It’s not helpful to the governments.”
The observation that the most alarmist claims about A.I.’s impact seem to come from the individuals most connected to Rationalism should anger us. This exaggerated grimness makes it easy to overlook the more immediate harms created by these technologies, such as their potential use in cybersecurity attacks, excess resource consumption, impact on our cognitive fitness, role in undermining education and amplification of soul-deadening slop.
This style of anxious talk also allows the A.I. companies to portray themselves as the reluctant “stewards” of an inevitable technology, which is a framing they can use to deflect blame for damage caused by their own products.
The time has come to push back. Once we understand the key ideas of Rationalism, we can begin to filter them out. Be very wary, for example, when A.I. leaders or online commentators begin talking in sweeping generalities about the future of the technology, such as stating with confidence that many jobs will soon be automated or that their language models will soon become sentient beings intent on escaping their creators’ control. (No, OpenAI’s autonomous agents conducting hacks doesn’t mean that A.I. is rising up against its human overlords.)
The correct reply is to promptly turn the conversation back to the possibilities, harms and viability of the products that actually exist. To extend an analogy I’ve made before, if Ford automobiles were spontaneously catching on fire, we wouldn’t tolerate the chief executive’s changing the subject to discuss our general lack of preparedness for a future with flying cars. We’d demand that the company address the acute problem happening right now.
It’s not surprising that so many Americans feel negatively about A.I. at the moment, but these sentiments aren’t inevitable. Our current crop of L.L.M.-powered tools are exciting — and worrisome, and promising, and frustrating, all at the same time. With proper direction, they’ll hopefully make our lives better. But they don’t have to be terrifying. We thought Silicon Valley was trying to warn us of an unavoidable danger, but maybe they were just being weird.
Cal Newport is a professor of computer science at Georgetown University and the author of “Deep Work.”
The Times is committed to publishing a diversity of letters to the editor. We’d like to hear what you think about this or any of our articles. Here are some tips. And here’s our email: [email protected].
Follow the New York Times Opinion section on Facebook, Instagram, TikTok, Bluesky, WhatsApp and Threads.
The post How Scared Should We Be of A.I. Right Now? appeared first on New York Times.
