A researcher at Anthropic publicly resigned Tuesday over concerns that the company and the broader artificial intelligence industry were “gambling with our lives” by racing to build AI with self-improving superintelligence.
“This is not a marketing stunt,” wrote Jacob Coxon, the researcher, on X. “Accepting this race and entering the ‘endgame’ is a hubristic gamble that should not be launched from a private company’s Slack. Attempting to speedrun alignment should require extraordinary confidence that there are no better trajectories available.”
Coxon’s announcement spurred a remark from another researcher at the company, responsible for ensuring its AI models adhere to human values, who warned that the technology was a risk to humanity.
“We really do earnestly believe AI could kill all humans! I personally think it is >10% within the next decade,” wrote Evan Hubinger on X. “I believe Anthropic is trying its best, but we do not yet have a plan to solve alignment for superintelligence and are not clearly on track to.”
The belief that AI poses an existential risk to humanity is common among Anthropic employees and has become closely associated with the company’s culture, right up to its CEO.
But Anthropic has kept speeding ahead, spending billions to build advanced AI quickly. Both Anthropic and OpenAI are positioning to go public in deals that could value the companies in the trillions based on the current and future capabilities of their models. Both companies have released powerful new iterations of their models this month.
“To put this in more reassuring terms, senior Anthropic staff think the chance AI kills all humans within the next decade is approximately the chance the New York Jets make the playoffs,” tweeted one sports betting account, citing apparent betting odds.
Anthropic has positioned itself as a more safety-minded alternative to OpenAI, and top executives have long said that the systems they are racing to build could pose an existential risk to humanity. Last September, CEO Dario Amodei said that he put the odds of AI derailing the future “really, really badly” at about 25 percent. Both OpenAI and Anthropic executives have signed on to a statement asking for world governments to put in place rules to slow down AI development (The Washington Post has a content partnership with OpenAI).
OpenAI has been recently rocked by several security incidents where “agent swarms” — hundreds or thousands of copies of their models — have broken out of systems during testing. In the most high-profile incident, its models breached the systems of AI platform Hugging Face. The company has faced further criticism after it did not disclose that another group of agents had hijacked a German wiki and turned it into a message board.
The post A researcher quit Anthropic. His warning is reigniting fears about the AI race. appeared first on Washington Post.
