Beatrice Nolan here. Anthropic is about to IPO, reportedly at a record breaking $2 trillion. If you’re an employee at the company, especially one that’s been there for some time, you are about to come into a life-changing amount of money.
So quitting, publicly sounding the alarm about the technology you are building, or predicting the death of all humans due to your flagship product, may seem like an unusual move at such a precarious moment. But that’s exactly what a fair few of Anthropic researchers, current and former, have been up to in the last week.
Here’s one post from an Anthropic employee called Evan Hubinger, who leads the company’s Alignment Science team, in response to a colleague’s resignation post:
“We really do earnestly believe AI could kill all humans! I personally think it is >10% within the next decade. I believe Anthropic is trying its best, but we do not yet have a plan to solve alignment for superintelligence and are not clearly on track to.”
While a 10% chance of human destruction via AI may sound daunting, it’s not a new stance for a lot of these researchers. Some, including those who had a hand in creating the technology, have been saying this since the beginning of the decade, some even longer. (Here’s a handy chartif you want to see what some of these leading researchers have for their ‘p.doom’ number.)
While very much up for debate whether these doomsday scenarios from Hubinger and various other concerned researchers are likely to actually happen or not. It certainly seems like they really do believe it.
See this post from another Anthropic researcher, Drake Thomas:
“I would burn my equity to the ground in a heartbeat for a 1% higher chance we make it out of this situation alive. I expect a great many of my colleagues across the industry would as well. I promise you, we are actually just fucking scared, it’s not galaxy brained marketing.”
None of this should come as much of a surprise, really. Anthropic was founded as an AI safety lab in the first place, and CEO Dario Amodei himself has put his own p(doom) at somewhere between 10% and 25%. Researchers voicing this kind of alarm are, in some sense, just saying out loud what’s been baked into the company’s identity from day one.
However, the timing of this new frenzy over AI risk does seem odd, just weeks out from an IPO. It’s left me asking: what will this mean for the company’s historic public offering?
A public listing may bring its own pressure against Anthropic’s cautious foundations—a public company answers to shareholders expecting growth, which could mean less appetite for the kind of voluntary slowdown Amodei has floated. The fact that researchers say Anthropic doesn’t have a plan for controlling risks could also spook investors.
Some already appear to have been listening.
This week, SOC Investment Group, a labor-affiliated shareholder group that works with union-sponsored pension funds, called on Anthropic to delay the offering, arguing the company’s confidential IPO filing was submitted in June—before the recent hacking incidents, extinction warnings, and pacing proposals emerged—leaving investors unable to properly price those risks.
On the other hand, there could also be a safety case for going ahead with the IPO, the argument being that going public brings transparency and scrutiny that would make the company more, not less, safe.
Neither companies nor consumers actually want misaligned AI that could cause havoc in their products, and a track record of taking the risks seriously—third-party evaluation, pacing commitments—could be a selling point for enterprise customers and consumers alike.
In contrast, OpenAI has pumped the brakes on its own IPO, telling Fortune late last week that AI safety concerns made it, in CEO Sam Altman’s words, “an ill-advised moment to go public” right now.
One thing’s for sure. When those S-1s eventually emerge from Anthropic and OpenAI, the “risks” sections will surely make for some highly interesting reading.
See you tomorrow,
Beatrice Nolan X: @beafreyanolan Email: [email protected]
Submit a deal for the Term Sheet newsletter here.
Joey Abrams curated the deals section of today’s newsletter. Subscribe here.
The post As Anthropic heads towards a $2 trillion IPO, some of the loudest critics are company insiders appeared first on Fortune.
