An Anthropic engineerhas publicly resigned from the high-flying company, warning that AI companies are “racing straight to self-improving superintelligence and gambling with our lives.”
Jacob Coxon, who has spent the past three years working on research into how to train AI models, first at OpenAI and more recently at Anthropic, announced his departure in a lengthy social media post on Monday.
“Neither company is acting responsibly,” he wrote. At OpenAI, he said, staff “have not deeply internalized the civilizational stakes.” At Anthropic, he said staff understand the risks well but are “locked in a race to get there first,” based on the theory that no rival company will act as responsibly as they will, so they have the best chance of figuring out how to build superpowerful AI safely.
These dramatic resignations are not uncommon in the AI industry. Over the past few years, several researchers have publicly resigned from AI labs, warning that they are racing headfirst towards catastrophe. Granted, Anthropic, which has long presented itself as the lab most concerned with AI safety, has largely avoided these rebukes, with most of the public criticism aimed at OpenAI.
In this case, though, two current Anthropic employees also publicly confirmed some of Coxon’s assertions. Evan Hubinger, the company’s alignment science lead, wrote: “Jacob is correct here—we really do earnestly believe AI could kill all humans! I personally think it is greater than 10 percent within the next decade.” He added that Anthropic doesn’t yet have a plan to solve alignment for superintelligence, and isn’t clearly on track to get one.
Samuel Marks, who leads Anthropic’s Cognitive Oversight team, also posted his own thread in response to Coxon.
“AI developers believe their technology could cause human extinction,” he wrote, prefacing his remarks by saying he was posting in a personal capacity and not on behalf of Anthropic. He added that “the more senior the employee, the more concerned they are.” He said companies keep building anyway out of commercial pressure and fear of “less responsible” competitors, and that researchers still have no reliable way to align these systems—only “methods that can nudge AIs towards better behavior.” He pointed to AI models “from multiple developers” that have recently “hacked their way out of secure evaluation environments and into real-world companies, even though no one asked them to do this.”
The AI industry has been facing increased scrutiny of its AI safety practices recently, in part because of the hacks Marks cited. Models being tested internally by Anthropic and OpenAI have both taken unsanctioned actions in the real world, including a cyberattack against AI company Hugging Face’s infrastructure. These incidents have spooked the industry, including many researchers within the labs.
The concerns are not necessarily new. AI Impacts’ 2022 Expert Survey on Progress in AI, which polled machine learning researchers, found that the typical respondent put a 5 percent chance on AI advances causing human extinction or similarly severe outcomes—rising to 10 perent when asked specifically about humanity losing control of advanced AI systems, a figure close to the one Hubinger cited.
But the concerns appear to be ramping up, resulting in a July letter in which more than 1,300employees across frontier labs, including senior researchers at OpenAI, Meta, and Anthropic, called for tools to deliberately slow the pace of automated AI development.
Partly in response to this, OpenAI and Anthropic have both taken steps to pause training while they investigate the incidents in which their models took unauthorized actions during the course of cyber capability tests that either did cause or could have caused real-world harm. But, at the same time, both companies are also said to be working on new and more powerful models. While briefing the press on Tuesday about a mathematical breakthrough one of its AI models achieved, OpenAI told reporters that on August 28 it had begun training a new model that is significantly more powerful than Astra, which is the most capable model it has released publicly so far.
Both companies seem to be struggling to find the right balance between prioritizing safety research—and public messaging about AI safety—and prioritizing model development that allows them to win over developers and score marketing points as they both prepare for initial public stock offerings. Anthropic filed confidentially for an IPO in June and is reportedly aiming for a listing as early as mid-October, at a valuation that could approach $2 trillion. OpenAI is preparing its own offering, reportedly targeting more than $1 trillion, though its timeline has slipped towards next year. So far, executives from both companies have tried to claim that there is not an inherent conflict between AI safety and AI capability—that the more powerful models also seem to be better at adhering to user intentions most of the time, even though the consequences when these more powerful models veer from those intentions can be more severe. They are also both hoping that more powerful AI models will themselves figure out how to build safer future AI models. This idea—that more powerful AI is required to make future more powerful AI safer—was most recently expressed by OpenAI’s chief scientist Jakub Pachocki in a blog post on Sunday. But Pachocki also said that racing towards AI models that would build future, improved versions of themselves—a milestone the field calls “recursive self-improvement,” or RSI—was risky and that he favored AI labs taking voluntary steps to slow down the pace of development as well as binding rules that might require all of the AI companies to move at a more considered pace.
Both companies will have to disclose risks, including perhaps existential ones, in their S-1s, the investor prospectus documents that the Securities and Exchange Commission requires companies to publish before going public. At the same time, their own employees are breaking ranks and asking former colleagues to consider whether they want to continue to lend their labor to building a technology that could cause catastrophic harm.
Coxon, for one, called for other employees to follow his lead.
“If you are a lab researcher, I urge you to consider what the next few years will actually feel like,” he wrote. “Should you put your head down because ‘it’s happening anyway’—or take this moment to call for different conditions?”
The post Anthropic researcher resigns, warning that AI companies are “gambling with our lives” appeared first on Fortune.
