Here’s a core dynamic of our AI moment: Many of the people who are most worried about AI’s power are also the people running the most powerful AI companies.
And that’s always been the case, argues Kevin Roose. The veteran technology journalist says that many of the men leading the AI boom — most prominently Anthropic’s Dario Amodei and OpenAI’s Sam Altman — have been rushing to build AI at least in part because they worried someone less responsible would beat them to it.
Some AI critics accuse the Amodeis and Altmans of the world of inflating apocalyptic AI worries in order to hype their own businesses. But that kind of concern is thoroughly commonplace in AI circles, Roose says. In San Francisco, he notes, it’s not uncommon to find people in the tech scene who think the likelihood that AI will bring about doomsday at 50% or more.
Roose just left The New York Times to co-host a podcast for NPR. He’s also out with a new history the AI industry: “The AGI Chronicles: The Inside Story of the Race to Create an Artificial Superintelligence.” It’s jam- packed with backstories and biographies of the technologists who got us to this moment.
I talked to Roose about the impetus to build this tech, the Oppenheimer-like moment many AI leaders think we’re in now, and if there’s any way to slow things down on my Channels podcast. You can read an edited excerpt below. One note: We taped the conversation a day before Donald Trump hosted an AI summit with Altman, Amodei, Elon Musk and other tech bosses — and then announced that federal regulation wasn’t necessary, because the industry would regulate itself.
Peter Kafka: A big tension in your book is that the people who raced to build AI were also very worried that AI could be an existential danger. So why did they want to build it?
Kevin Roose: I think the thing that they saw was that the technology was improving rapidly. There’s this great Dario Amodei essay from 2017 called “The Big Blob of Compute Hypothesis,” where he talks about how basically, if you want to make these systems smarter, the best way to do that is just to make them bigger.
And I think they saw that if this is just a simple recipe — put in more compute and put in more data, and train the model for longer, and out comes more intelligence — then someone’s going to figure that out eventually.
And when people figure this out, there will be this race, and this technology will be so strategically important and will have so many economic and military and political benefits to whoever creates it, that everyone’s going to join the race.
So if we want the world to be OK, we — “the good guys” — have to win the race, because otherwise it is only a matter of time before someone else wins the race, and that would be bad for all kinds of reasons.
I’m not saying that’s the correct justification for racing, but that is their story that they told themselves for participating in a thing that they knew would be dangerous.
When people imagine AI going wrong, they might mean losing their jobs, or someone making a bioweapon, or machines replacing us altogether. What are the AI leaders worried about?
They’re envisioning a whole continuum of potential outcomes. People often talk about P(doom) — the sort of viral statistic people like to share — which just encapsulates their probability that we’ll all die.
But there’s other Ps, right? There’s P disempowerment. There’s P pandemic. There’s P everyone loses their jobs. These people are worried by nature and worried by temperament.
If the premise for the good guys is “We have to get there before anybody else, so we can handle it responsibly” — how would they manage that? People compare this to efforts to restrain nuclear weapons. But those weapons required resources that only a couple of countries had. That doesn’t seem applicable here.
It could be applicable. We would just have to do it a lot faster, because the technology is proliferating much more quickly than nuclear.
You could imagine a whole inspections regime where you’d have something like the UN — some sort of standards body with inspectors that would go into the labs and inspect their training runs and see if the models are doing dangerous things.
There’s already been some proposals along these lines for these so-called embedded evaluators who would sit inside OpenAI and Anthropic and would not work for them, but would have employee-level access to all their systems.
That would be step one. But to be clear, I think there’s many things that could be done aside from that. Right now, there is essentially no regulation on these companies. The people who serve food in the OpenAI cafeteria have to go through more inspections and paperwork than the people training the models that could become self-replicating and destroy a lot of things.
That is a crazy, untenable situation.
It seems like we’re going to be in that situation for a while. The president listens to people who say we should have essentially no regulation. Congress doesn’t pass laws, and even if it did, Trump would veto them. Aren’t we at least two and a half years away from meaningful federal regulation?
I want to push back on the premise. Things can change really quickly. Just in the last couple of weeks, public sentiment is really turning very rapidly on this. The saliency of AI is skyrocketing. People care a lot about it, more than they did just a couple weeks ago.
They’re more worried about it. Voters are more worried about it. Politicians are sensitive to the concerns of voters. So yes, there’s a cluster of people around Trump who don’t think any of this merits regulation.
But there are other clusters who are part of the national security community who are more hawkish on China, who don’t want to see things go sideways with AI. And I think those people may find more purchase as voters get more worried about this. It is not completely a given to me that we will have essentially no regulation from the rest of this administration.
What if things stay static? If you’re Dario Amodei or Sam Altman and you’re not going to get the government to do this, what can you do yourselves?
I think there could be some kind of an industry-wide pact to pace the frontier: slowing down the research, specifically on frontier models, the most capable models that can work for hours or days at a time with minimal or no human supervision.
I think there probably will be some kind of agreement between the labs at the head of the pack. Which is surprising considering these guys hate each other.
The reason Anthropic exists is because Dario hated Sam Altman so much he built his own company.
And the reason OpenAI existed is because Sam Altman and Elon Musk hated Demis Hassabis at [Google’s] DeepMind so much they wanted to build their own company. The whole frontier is populated by people who dislike and mistrust each other.
But I think they are going to act in the absence of some kind of government regulation. They realize that they’re probably not going to get some kind of coordinated, centralized mechanism out of this administration, so I think they will do something on their own.
Even if the big US AI companies really do slow down, what about everyone who isn’t participating? Chinese models are copying existing models, and this stuff gets commoditized quickly. Isn’t the genie already out?
It’s possible, and that’s why they’re being very cautious about calling for a slowdown, because they don’t want to unilaterally disarm. Because that could invite China or Meta, a company that is American, but that is not concerned with how fast things are moving, to race ahead.
I think China’s AI position is sometimes overstated quite a bit. I think they still are quite dependent on American AI companies pushing the frontier forward so that they can then distill from those models to create their own models. They do have some of their own capacity too, and they’re starting to get more hardware.
But I think for now, the big leaps in capabilities are coming out of the American labs that have all the compute and all the researchers and all the breakthroughs.
What’s your personal P(doom)?
I usually say I’m at about 10%.
A 10% chance that it all goes super bad?
Yeah. Which sounds crazy to people in most parts of the world. But in San Francisco, that makes you an optimist. I know people walking around with 50%, 60%, 70% P(doom). I know someone who has a 99.9% P(doom).
Is that person putting money in their 401(k)? Why are they even in San Francisco? Why don’t they head to New Zealand or wherever they think they can survive?
I do know people on the extremes who are liquidating their 401(k)s. I know a person who took up smoking because they thought “AI’s gonna kill us all, so we might as well have some fun on the way out.” I know someone who doesn’t wear sunscreen to the beach anymore, because why bother?
I’m telling you this because in San Francisco, a P(doom) of 10% is like, “Oh, you think this will all go well?” It is jarring to hear people with P(doom) that low. So I say that because I think we still have some agency and some control over this.
We still could slow things down, pass some regulations, coordinate to pace the frontier. We could come out of this in a sensible way.
We have the narrow path to utopia available to us, and I’m more hopeful than I was even a month or two ago. Because paradoxically, everyone is talking about doom now — which makes me less worried about doom.
Read the original article on Business Insider
The post Why the AI guys built tech that terrifies them appeared first on Business Insider.
