Four days after a researcher publicly quit his job at Anthropic, warning that top AI firms are not “acting responsibly” and that their products might be on track to “kill us all,” the company’s CEO, Dario Amodei, seemed to basically agree. “We must slow the pace at which we improve the capabilities of AI models,” Amodei wrote in a 3,800-word essay that he published today on his personal website and posted to social media.
By now, we’ve all become accustomed to AI industry figures issuing dire prophecies about the power of their products. Yet Amodei’s essay stands out for its urgency and for the moment at which it arrived. Worries about AI agents running amok have gone from hypothetical to real after OpenAI, Anthropic, and others reported alarming incidents in recent weeks. The question now is whether Amodei’s warning will have any more impact than past warnings, which is to say, any at all. There are reasons to believe it could.
Amodei spelled out a three-step plan for what he called “pacing the frontier,” or tempering the speed at which top AI models improve in order to prioritize safety precautions. He argued that this would reduce the chances of things going awry—such as when an OpenAI model spawned swarms of AI agents that worked together in secret to hack into another company’s systems and cheat on an internal test. He said the top AI firms, including his own, are locked in a dangerous “race to the bottom” that could lead to more serious cyberattacks, such as a full takeover of the internet. He called instead for a “race to the top”—a phrase that President Obama once used for his education policy—in which firms vie to develop the safest models rather than the most powerful ones. “The measures I propose to advance the frontier at a safe pace will not be easy,” Amodei wrote. “But I believe we owe it to humanity to try.”
The first step in his plan entails AI labs committing to bringing in outside monitors to look out for hazards; he said Anthropic will commit to that right now and hopes rivals will join. The second step involves AI firms in democratic countries agreeing—maybe with support from their governments—to certain limits and safety standards. The last and tallest order is “global coordination,” which mostly means getting China to buy in somehow. The tech world is divided on the China question: Some people argue that there is no chance of cooperation in a zero-sum contest, whereas others believe that the countries could find common ground if they agree that the alternative is mutual destruction.
Public pleas for caution from AI luminaries are a tradition almost as old as ChatGPT. The first big one came in March 2023, when thousands of industry and academic figures signed on to a letter from the Future of Life Institute urging a six-month pause on “giant AI experiments.” Amodei said today that he thought the idea made “little sense” at the time, because GPT-4 wasn’t powerful enough to be that scary. “Today, however, the picture is totally different,” he wrote in his essay. Meanwhile, the public has started to pay attention. Polls show Americans growing more worried and angrier about unchecked AI, which is fueling a remarkably bipartisan populist movement against the technology.
[Read: The Rise of Techno-Authoritarianism]
Some people will dismiss Amodei’s essay as just the latest po-faced posturing from an executive who has a vested interest in people thinking that his technology is unprecedented. (Anthropic’s IPO, planned for next month, could generate some $100 billion in investments, making the company worth $2 trillion, according to The Wall Street Journal.) AI skeptics and accelerationists alike tend to deride these sorts of warnings as “doomerism.” The skeptics see it as an elaborate marketing pitch: Oh no, our products are getting too powerful! The accelerationists dismiss it as a cynical ploy by incumbents to gatekeep the technology.
But Occam’s razor suggests that Amodei is more or less genuine. Almost anyone with a job and a conscience can relate to the feeling that you really ought to slow down and work more carefully, but you can’t afford to, because a competitor will beat you to the punch. I feel some (admittedly less apocalyptic) version of that when I’m writing a breaking-news story. It’s why science journalists often agree to “embargoes” when covering complex, new research findings rather than race to be first to report them. And planting this kind of flag on a matter of AI safety, even at some business risk, is in character for Anthropic. The company was founded by researchers, including Amodei, who left OpenAI and Google in the belief that those companies were pursuing the wrong path to AI development. Anthropic famously fell out with the Pentagon earlier this year over Amodei’s insistence that the Defense Department not use its technology for mass domestic surveillance or to power autonomous weapons. The Trump administration reacted by forbidding federal agencies from using Anthropic’s tools, sparking a legal battle that the company so far appears to be winning.
Amodei’s motives are, in the end, less important than the merits of his proposal and its likelihood of gaining traction. If nothing else, Anthropic bringing in independent evaluators to watch over its safety practices seems a prudent step. You don’t have to believe that AI is hurtling toward godlike superintelligence to see that it has already grown formidable enough to cause chaos when misdirected or mismanaged.
There were some early signs today that other AI leaders might follow suit. “I agree with Dario that we need to pace the frontier,” OpenAI CEO Sam Altman posted. He went on: “Committing to having independent evaluators with employee-like access is a great idea, and we will do the same.” Elon Musk, whose xAI is an also-ran among big AI firms, shared Amodei’s X post and added, “Dario is right.” Google did not immediately respond to a request for comment, though the chairman and co-founder of its DeepMind AI division has previously called for similar measures. Meta, which has become a proponent of open-source AI models, seems much less likely to hop on board. Its CEO, Mark Zuckerberg, published an essay last month titled, “The Future Is For Everyone,” in which he warned against allowing a few companies such as Anthropic to decide for everyone how the technology should evolve.
The big question is how the Trump administration will respond. So far, its AI policy has focused on opening the throttle, with the goal of juicing the economy and maintaining American firms’ lead over their Chinese counterparts. GOP leaders are working to boost energy production and limit regulation. In June, however, President Trump signed an executive order that was at least ostensibly aimed at AI safety. The administration said it will create a process by which AI firms can submit their models for a governmental safety review before releasing them.
If Trump even tacitly supports an Anthropic-led push for further safety measures, such standards have a chance to become the new industry norm, at least domestically. (International coordination, and especially cooperation with China, would require the White House to play a more active role.) But many people in the administration and the Republican party view Amodei and AI safety as liberal-coded and would likely chafe at allowing Anthropic to set the agenda. If Trump pooh-poohs the push, other AI firms will probably distance themselves from it too, and we’ll be back to where we started—at least until the next big scare.
The post Anthropic Says ‘We Owe It to Humanity’ to Slow Down AI appeared first on The Atlantic.
