Automated AI research is on the horizon, a former Anthropic and OpenAI researcher-turned-whistleblower said, warning it could leave humans totally unable to control what comes next.
Jacob Coxon, the 27-year-old who publicly resigned from Anthropic last month after working as a capabilities researcher, made the comments before New York City lawmakers on Monday as he sounded the alarm that AI could threaten humanity’s survival.
Coxon testified that automating AI research has become “the primary goal” of top frontier AI labs.
“At my last job, I was, in some sense, working to automate myself,” Coxon said at the New York City Council hearing focused on AI risks.
“When I was at OpenAI, we had milestones for an automated AI researcher, around 2027, 2028, and we are on track or beating these expectations,” he said. “That’s a matter of months from now, not decades or even five years.”
Coxon said that once AI can conduct its own research, “humans will be much further out of the loop than we are today, and today is already not great.”
Speaking from what he called firsthand experience, Coxon said that the majority of the code written today is written by AI. “And people do not check it that carefully anymore,” he testified.
OpenAI and Anthropic did not immediately respond to a request for comment on Coxon’s remarks on Tuesday.
Coxon was among a panel of AI-industry whistleblowers to testify at the hourslong hearing, which also included testimony from representatives of OpenAI, Anthropic, Google, and Meta.
“This hearing is not about stifling innovation,” City Council speaker Julie Menin said at the outset of the proceeding, explaining, “the true intent of this hearing is to ensure that innovation happens responsibly.”
In his testimony, Coxon told council members that his experience inside Anthropic and OpenAI convinced him that the firms were moving ahead without adequate safeguards in place.
“It is more likely than not that humanity loses control to these AIs, and it could end in human extinction,” Coxon said, adding, “From my experience, the companies are being extremely reckless given the stakes.”
When Coxon announced his resignation from Anthropic on X in a now-viral post, he said neither Anthropic nor OpenAI was “acting responsibly,” and alleged both were “gambling with our lives.”
During Monday’s hearing, Coxon called for more transparency from AI labs and a slowdown in the development of rapidly advancing technology.
“The companies run on a startup mindset: move fast, break things, fix them later. That works for a photosharing app. It does not work for building the most powerful technology ever built,” he testified.
Days after Coxon’s resignation, Anthropic CEO Dario Amodei published a new, nearly 4,000-word essay urging the AI industry to slow down.
“We must slow the pace at which we improve the capabilities of AI models,” Amodei wrote. “Progress will still seem fast, and we must make wise use of the time we gain.”
In response, OpenAI CEO Sam Altman said in an X post, “I agree with Dario that we need to pace the frontier. This has been a primary topic of discussions we’ve had at OpenAI in recent weeks.”
Read the original article on Business Insider
The post The Anthropic researcher-turned-whistleblower says he was ‘working to automate’ himself appeared first on Business Insider.
