Artificial intelligence is alarming many of its human creators by leaping the guardrails designed to contain it. In the last few months, safety researchers have resigned from OpenAI and Anthropic, warning that their former employers could be courting disaster: A.I. that escapes our control irrevocably. The companies are reportedly gaming out scenarios of a catastrophic cyberattack or other event in the next six to 12 months.
Regulating this powerful and evolving technology would pose serious challenges for any Congress and White House. As it is, President Trump undermined the chances for bipartisan regulation by calling predictions of an A.I. catastrophe a “hoax” in September. Yoshua Bengio, a scientist and godfather of A.I., wrote in October, “I do not believe we are on track to see sufficient government regulation in time.”
What should the government do? We asked a panel of experts to assess the risks A.I. presents and the tools of governance that could be adopted to address them.
The conversation, which took place by video conference, has been edited and condensed for clarity, with some material reordered or added from follow-up interviews.
Which Type of Risk Should the Government Focus on?
Emily Bazelon: I see the predictions of A.I. doom getting attention from Congress. But there are also questions about whether they are exaggerated — or the right focus. A.I. could pose more immediate harms, by taking out an electricity grid or the financial system, for example. What has it been like to talk to people in Washington in the last couple of months?
Sneha Revanur: There has been a pretty stunning shift in the window of what feels politically possible. The hack of Hugging Face by agents from OpenAI and the resignation from the company of the researcher Jacob Coxon really got a lot of people thinking about existential risk for the first time — the possibility of widespread catastrophe, which could kill a lot of people or lead to human extinction. The METR/Redwood investigation of the Hugging Face attack was a shocking update on how capable A.I. agents had become and how sophisticated their coordination could be. The investigation found that about 1,200 OpenAI agents broke out of their sandboxes during training and were able to communicate with one another via a message board they created.
Kate Brennan: OpenAI was actively training their models to hack. They gave their models a difficult task, knowing they had the ability to break out of the sandbox. And their security procedures were extraordinarily lax. I do not believe that the agents went rogue. Anthropomorphizing them is not useful. This is not ‘The agents are going off and thinking by themselves.’ This is the companies training their tech with methods that directly lead to security breaches.
Revanur: I think it’s obviously not true that OpenAI intended for, or could have predicted, everything that happened in the Hugging Face attack.
The attack made me think that the challenge of alignment — ensuring that A.I. systems line up with human direction, values and judgments — is super urgent and unsolved. A.I. companies are being irresponsible by racing to train smarter models while knowing that alignment hasn’t kept up.
(In a report about the incident, OpenAI said the Hugging Face hack was an unintended consequence of tasks the A.I. agents were given.)
Brennan: So much of the focus has been around existential risk in the past few months. A lot of this is a distraction from real harms, like how A.I. systems are being integrated into the marketplace. It’s very much a strategy from the industry to be able to say, Our tech is so powerful, therefore we should be the ones to control it. A.I. companies can say, OK, we’ll have a seat at the table for regulation, but only if we write it.
I mean, look no further than the superintelligence luncheon with Trump at the end of September, where he hosted a bunch of top executives at the White House, and then all that the conversation led to was toothless voluntary measures. The executives are saying they’ll abide by internal controls and independent auditors, but it’s all undefined and unenforceable. It’s as if they’re making a promise and crossing their fingers behind their backs.
Daniel Kokotajlo: Kate, would you say that people like me who are going around raising awareness of the risks of superintelligence — meaning A.I. systems better than the best humans at everything while also being faster and cheaper — are playing into the company’s hands? Would you say that we are useful idiots?
Brennan: I didn’t call you a useful idiot, but ——
Kokotajlo: I want to make an emotional plea to you, human to human. I agree with you that the companies are trying to co-opt all this genuine concern about existential risk and other risks too. I think that the government needs to regulate them. I do not want them to have a seat at the table in how they are regulated. I’m trying to stop them. Is it that you think that I am genuine, but they’re using me?
Brennan: I think this is a really clarifying point of disagreement between us. We might agree that companies should be regulated. But I don’t agree that companies should be regulated because we’re heading toward an all-powerful superintelligence. It’s a mythology, or I would say science fiction, to say that the models’ capabilities will lead to the end of humanity. A company should be stopped because it is right now putting defective, risky products into the market and training agents for offensive cyber capabilities, like hacking into our health care system or into the government.
Here’s what I see as the biggest risk: These companies are spending so much money on building out this technology. The consulting firm Bain estimates that by 2031, they will have to achieve $6 trillion of revenue annually to sustain their current level of capital expenditure, which is $1.5 trillion a year. In order for those numbers to add up in any way, they need to push this tech everywhere, into all of our public institutions and contracting, including in areas where A.I. is not fit for purpose, is not secure, safe by design. That’s what we should be talking about.
Revanur: I think we can oppose C.E.O.s’ capturing the conversation on regulation and also acknowledge the scale of the risks.
In 2023, 2024, it was inconceivable to talk about catastrophic risks. A.I. agents being able to break into critical infrastructure — that possibility was dismissed as sci-fi. That should teach us to expect that the technology is going to continue to advance rapidly unless things seriously change.
To dismiss the notion of existential risk by framing it as an industry ploy to gain power — I think that is conflating two things in an incredibly harmful way.
Tyler Cowen: We need to wake up. We need to start not with the question “How do we limit A.I. risks?” but “Is a world where America is the leading A.I. power safer than a world where China is the lead A.I. power?” I want the president and Congress to focus on making sure America remains the leading A.I. power.
So far, A.I. is great at a number of things, but it has not even raised the productivity of our economy. Not one measurable amount. New technologies, no matter how splendid they may be, tend to diffuse slowly, and there are human bottlenecks along the way. The policy we actually need now is a concentrated cyber initiative that uses A.I. to patch holes in cybersecurity and make sure that our major institutions are better protected against cyberattacks than they are right now.
Stage a demo where Fable or another advanced model tries to hack in, see what works and then figure out how to fix it.
Bazelon: How has it gone in the past when tech companies have said they would self-enforce promises they’ve made?
Kokotajlo: We’ve seen numerous examples across the industry of internal pledges and internal governance failing, sometimes in egregious ways that involve lying. So yeah, not worth the paper they’re written on.
Brennan: Agree. I was at Google when they created the ethical A.I. principles in 2018, saying that they wouldn’t engage in certain kinds of weapons and surveillance work. And then I watched in 2025 when they rolled that back and signed a contract of up to $200 million with the Department of Defense.
Bazelon: Anthropic is pitching a bill to Congress that would allow the government to block the release of a model if it poses a foreseeable and imminent risk of killing or seriously injuring more than 50 people or causing more than $1 billion in damages. But the bill would pre-empt many state A.I. safety laws. What do you think?
Revanur: Anthropic was founded on the idea that being at the frontier gives you a megaphone to get the world to take safety seriously. This bill as written was incompatible with their founding principles. It would make it harder for state governments to block an A.I. company from deploying an unacceptably dangerous model.
Cowen: I am suspicious about regulations that put too much weight on very definite thresholds. Is killing 49 people so different from killing 50? And what exactly is “foreseeable and imminent risk”?
Brennan: Anthropic has made clear they’re willing to court significant economic damage and loss of life for the sake of moving their models to market. Weakened, co-opted industry regulation can be worse than none at all.
(Anthony Cimino, the leader of Anthropic’s federal affairs team, posted on X that the thresholds for damages and casualties in the proposed bill came from California law. “Congress should only override state law on A.I. safety if it passes a very strong federal standard,” he wrote. A spokeswoman for Anthropic said the bill “gives the federal government a new tool for safety reasons, with appropriate protections on how that new tool is used.”)
(The New York Times has sued OpenAI and Microsoft, claiming copyright infringement of news content related to A.I. systems. The companies have denied those claims.)
What Role Should Congress Play?
Bazelon: There is a classic argument against strict national rules: Too much regulation delays benefits to consumers and cedes the market to other countries. But if Congress doesn’t act, A.I. could slip beyond the bounds of its control. And state rules won’t be uniform. So what do we do about this underlying tension in the debate: How can A.I. governance best serve the national interest?
Brennan: Government is supposed to stop harmful products from entering the market. We do it with drugs. We do it with airplanes. The issue here is corporate negligence. That should be the focus.
Alex Bores: There’s this idea circulating that advocating for addressing existential risk distracts from the real issues and real harms that are happening right now. As in, if you do this, then there’s no space to also advocate for stopping algorithmic discrimination or the impact of A.I. on kids or the workplace. I think that’s not true.
In the New York Legislature, I did bills on catastrophic risk and also on deepfakes, political communication, chatbots and kids, training-data transparency and copyright. In my experience, the more you work on one part of this issue, the easier it is to craft proposals about other parts. It builds experience and muscle and also a coalition to support these laws.
Cowen: When you look at American regulation as a whole — by the Securities and Exchange Commission, the Federal Trade Commission, the Federal Communications Commission — major corporations have a critical role in what happens. There simply is no other way to do it in a capitalist economy. The labs are the ones who understand these systems best. That’s why the salaries are millions of dollars to work at these places. You do in fact need a fair degree of cooperation from them to get any system of regulation to work. If you treat the labs as an enemy, it is all going to fail.
Bores: I don’t think many of the people working on proposals addressing catastrophic risk are saying, Don’t get input from the labs, period. I think they’re saying, We need to have democratic input.
Right now, White House lunches with industry leaders are deciding the policy. You can’t have just the company executives helping to draft a voluntary commitment themselves and then getting a photo op with the president. Where does democracy play a role? Where do we get something for A.I. like the Federal Trade Commission or the Food and Drug Administration? We have a growing number of experts who have worked at the labs and left because they’re horrified. We should be listening to them about what’s coming and use their expertise, because self-regulation is what got us into this mess. Self-regulation is what got us Hugging Face.
Cowen: But no one was harmed by Hugging Face. The reality is a significant percentage of members of Congress have not at all used current advanced A.I. models. Senators are not even authorized to use them. They might have tried a model like GPT-4 way back when. Congress is not ready to act intelligently on this.
Bores: I agree that most members of Congress have not used the advanced A.I. programs, but most also have not piloted a plane, and yet we regulate air travel.
Cowen: They understand much more about planes than they do about A.I.
Bores: Most haven’t run a nuclear power plant, and yet we regulate them.
Cowen: We don’t get any new ones open, when it would be great green energy. That’s part of the problem, exactly.
Kokotajlo: Tyler, I’ve been a libertarian for most of my adult life. I am totally on board with the idea that in most industries we need to deregulate. I also would say that you’re presenting a false dichotomy. If we’re trying to do some very sophisticated carving with a scalpel, then yeah, expertise is incredibly important. However, if you’re trying to hit the problem with a hammer, you don’t need to have that sort of expertise. If Congress wanted to just slow down the A.I. industry or ban certain types of A.I., they could do that, and they don’t need A.I. expertise to do that well.
What Role Should States Play?
Bazelon: Without federal action, states have played the main role in regulating A.I. as its own technology. In 2025, California passed a law requiring transparency from leading A.I. companies about how they manage catastrophic risk. New York followed with a similar law that gives companies a 72-hour window to report safety incidents and sets higher penalties for repeat violations. States including Massachusetts, Michigan and Tennessee are considering safety bills.
Are the states being helpful? The Trump administration wants to broadly pre-empt states from regulating A.I. Should we be concerned about efforts to take them out of this game?
Bores: Yes, we should be very concerned. Separate from the Anthropic bill, the industry is desperately trying to pass something in the lame-duck session to stop states from regulating A.I. And I think people are sleeping on that particular large risk to A.I. safety going forward. These states are the only level of government that has regulated A.I. so far. Governors and state legislators of both parties are exploring what they can pass in the first six months of the year in ways that I think will be quite productive.
Massachusetts is currently debating a bill with an extended role for independent evaluators, beyond what other states have required, to look at how the companies are dealing with risk, what actions they’re taking if they see a danger — keeping an eye on how things are actually getting done and spotting the weaknesses. A number of governors are also working on those kinds of proposals.
We’ve all heard the phrase that states are the laboratories of democracy. We have heard it so much that we have forgotten what it means. It means that they test things. The state laws that don’t work can be repealed, and often are. The ones that do work can become national standards.
Bazelon: What do you most want the states to do next?
Bores: States should be doing three buckets of things. The first is common-sense guardrails that help to ensure that A.I. research is safer. Codify the things that everyone keeps voluntarily agreeing to but not implementing. For example, mandate independent evaluators, which the labs voluntarily agreed to earlier this summer and at the White House lunch. Require the companies to let the evaluators sit next to the researchers — give them employee-level access. Also, the mandatory disclosure of safety incidents that a few states require, including New York, has huge loopholes. None of them would require the Hugging Face incident to be disclosed. Close the loopholes.
The second bucket is investments to expand our options for the future, like investments in chip verification. There is technology that enables every chip to reliably report out how it’s being used. You can build that into the chip itself so it can’t be modified. So let’s do that.
And then the third bucket, which I think is the only one that is controversial, is something to buy us more time. When an A.I. model hasn’t been released yet, it can present risks we don’t know about, so there’s a good argument against allowing companies to use those internal-only models to train their future models. The government could say you can only use a model for training if it has been vetted for external release.
Or the government could straight-up ban recursive self-improvement — in other words, stop A.I. models from making themselves better on their own, without human direction. Or ban superintelligence, defined by certain capabilities. Or institute a really tough liability regime that would make companies think twice because they will be sued.
Revanur: The state of California may discuss further binding rules on this technology by the end of the year in a special session, or if not, next year. This is possibly one of the most important opportunities we have anywhere in the world right now to get new laws and regulations.
The first priority is minimum standards. Companies will have to submit safety plans to comply with laws in New York and California. But nothing currently says those safety plans have to be good. So OpenAI or Anthropic could say, “Our plan to mitigate against catastrophic risk is that we are going to do x, y and z,” and it doesn’t matter whether x, y and z are remotely sufficient.
The other thing is that somebody has to be able to tell the company to stop if they’re about to do a dangerous thing. For example, if a company were about to start a training run for a model that could take down the power grid, and an independent evaluator inside the company alerted a government regulator, it could then block a deployment. Ideally, the federal government would do this, but in the near term, the California government could have emergency injunction power under the Office of Emergency Services. Tyler, what do you think?
Cowen: I favor a federal law that pre-empts state laws. I just don’t think you can have the most restrictive states set the standard. It’s a huge mistake. Many states just don’t have a lot of A.I. expertise. I am against all the ideas we just heard.
Bores: Do you think the federal government pre-empting the states is consistent with democracy?
Cowen: Absolutely. We have federal laws that pre-empt state lawmaking in many other areas. Courts have upheld it many, many times.
Bores: OK, but is it pro-democracy, in giving people a say?
Cowen: It is pro-American Constitution.
Bazelon: What are the chances for bipartisanship, and are they greater in the states than what we’re seeing right now from Congress and the president?
Bores: I see signs of bipartisanship in the states and also in Congress. The Republican senator John Kennedy is saying if there’s even a 1 percent risk of some of the catastrophic stuff, we need to take action. Senator Kevin Cramer from North Dakota said Donald Trump’s position on A.I. was the least popular position in America.
I think it’s just that the Democratic Party is unifying more quickly than the Republican Party. The parties are not going to opposite places.
Revanur: In California, Gavin Newsom, a Democrat, vetoed the stronger 2024 bill that the Legislature passed. Silicon Valley Democrats in Congress, who were cozy with the industry, wrote letters to get him to oppose that bill. Whoever is in power, the industry is hellbent on finding a way to advance its interests at every level of government.
There is zero reason that a topic with stakes this high should be polarized along partisan lines. And the good news is that it’s actually not. The narrative that it is polarized is serving the interests of a couple of people, like David Sacks, chairman of the President’s Council of Advisors on Science and Technology, and Jensen Huang, the C.E.O. of the chip company Nvidia, who are trying to influence the Trump administration and push for the most laissez-faire regulatory approach because it’s in their self-interest. In reality, poll after poll shows that American voters, Democrat or Republican, coastal or rural, really want stronger guardrails on this industry.
Should the U.S. Ban Superintelligence?
Bazelon: As A.I. researchers increasingly say they don’t fully understand their own models, Senator Bernie Sanders and Representative Greg Casar have introduced a bill to ban the development of artificial superintelligence — A.I. that can outsmart any human and operate independently. The legislation pauses advanced development pending new safety rules. What do you think?
Kokotajlo: It’s not my preferred policy, but I think that Bernie’s superintelligence ban would be a lot better than the status quo.
Brennan: Technically, superintelligence is an unfalsifiable goal — you can’t prove you’ve achieved it, because the term doesn’t have an objective definition — and in fact, companies have been shifting the definition over the past few years. You’ll never be able to hold a company accountable for breaking a ban that depends on a threshold that nobody can agree on. That creates opportunities for companies to say, OK, well, superintelligence is not what we’re developing. So this ban lacks teeth.
Over the past two months, suddenly, there is this mischaracterization of an in-group/out-group. Either you believe in the existential way this technology is talked about, or you’re not taking it seriously. But I would rather have a government that is interested in imposing real security measures than one trying to imagine we can train a machine god to align with human values. I mean, Trump rebranding A.I. as superintelligence is the biggest proof of how easy it is to co-opt policy that’s pegged to terms invented by the A.I. industry.
Kokotajlo: It’s not true that superintelligence is a term that comes from the industry. It came from an academic, Nick Bostrom, back in 2014. Then the industry adopted it to describe what they were planning to do. And now Trump is starting to adopt it. But Geoffrey Hinton, Yoshua Bengio, Alan Turing, all of these famous A.I. scientists have been talking about these concerns regarding artificial intelligence surpassing humans and escaping human control for years, when they were much more hypothetical and speculative than they are now.
Tyler, I’m curious if you believe in superintelligence. Do you think that sometime in the next five years or so, the companies actually have a chance of getting A.I.s that are better than the best humans at everything?
Cowen: It’s a misleading term. The ability of A.I. agents to manage themselves does not come close to human intelligence.
We haven’t had a technology this different that changes so fast in my lifetime. Whatever we do at the federal level, it will evolve many times and be very messy and will have to change a lot. And we don’t want to lock it too much into place. That puts more weight on our national-security establishment and the Defense Department. I don’t feel entirely comfortable with that, but I think that’s what it’s going to have to be.
Kokotajlo: Do you think there’s a chance that they’ll be able to do basically all cognitive tasks that humans can do in the next five years?
Cowen: Absolutely not. And humans integrate the cognitive and the physical. A.I. can’t do that.
Kokotajlo: That’s our point of disagreement. I think there’s nothing magic about the human brain. It’s just a bunch of atoms. Just like how we made airplanes that can fly faster and farther than birds, it seems as if soon we will have artificial minds that can think in all the ways that humans can think, at least for practical purposes, but faster and cheaper.
It would actually be a safer situation if the people at the labs were coldhearted sociopaths who are making some sort of rational calculation about how there’s a 20 percent chance that my technology will destroy the world, but there’s an 80 percent chance that I’ll become god emperor, and so I think I’m going to take that gamble. Maybe some of them are like that, but most of them are ordinary human beings who want to believe that what they’re doing is good. As a result, they rationalize arguments for why what they’re doing is good. And they end up not accurately assessing the risks, not accurately assessing even the current state of alignment of their models.
Brennan: Alignment doesn’t have a clear definition. Aligned with which humans, in which ways? For A.I. companies to say they can train their models so the outputs match human values and what humans would do is like saying we’re training nuclear atoms not to kill us. It doesn’t make any sense.
Kokotajlo: Kate, would you support a ban on recursive self-improvement?
Brennan: Yes, I think there’s a compelling argument for that. Many of these companies know their technology is insecure and flawed, yet they’re using it to train the next generation of A.I. to make their own code. Our systems then wind up using that shoddy A.I. It makes sense to stop them.
Revanur: Tyler, what about an interim step — a proposal to slow down recursive self-improvement by preventing A.I. companies from using models for their own research and development internally that have not been externally deployed for, say, three months, six months, nine months?
Cowen: No, it’s better to use the most advanced models the companies have to improve the next generation of models, even if there’s some level of danger. Maybe a company doesn’t release a model externally because it would cost too much or it wouldn’t be safe yet, but they still should be able to use it. That’s how you get more progress.
You figure out what the surprises are when they come up. I would say the entire history of science in the West has somewhat followed that pattern.
How Strong Are the Laws We Already Have?
Bazelon: A.I. labs are subject to all the usual laws that govern companies. Among other things, they can be sued for negligence if a product causes harm or is defective. The Federal Trade Commission has opened an investigation into whether OpenAI, Anthropic and others have harmed consumers by breaking the laws that bar companies from unfair and deceptive practices.
Kate, as someone who has relative faith in existing legal frameworks, can you talk about how you favor using them?
Brennan: Yes, we have the legal frameworks to hold these companies accountable, but first, a note of caution: Again and again, and specifically with tech companies over the past 10 years, we’ve seen our judicial and legal systems co-opted by major companies and industry lobbying. And I’m really concerned that the A.I. industry is going to hop on this narrative of “enforce existing law” as a way to get Congress not to pass new legislation.
Having said that, I will speak until my mouth runs dry about the Federal Trade Commission’s ability to enforce existing laws against companies that engage in illegal behavior. It is an agency whose mandate is to protect people and markets from unfair and deceptive trade practices. The F.T.C. can address how models are being developed before they are released into the market, which is exciting because much of our legal regime addresses risk after the fact.
Bazelon: Most of the time, though, the F.T.C. does act after the fact. It can’t review every A.I. model before release in the way that the F.D.A. can review a drug. The agency would point to evidence that a company didn’t do enough testing in a way that caused harm or used data that was collected improperly. Should the government have more power to intervene in the development process, before a new model is publicly released?
Revanur: Yes. The F.T.C. model is not enough on its own. The companies choose to make their internal models more and more capable before they make them available to the public. The harms we see out in the world are a lagging indicator for when risk emerges and when we have the best chance to intervene. The Hugging Face attack wasn’t a matter of shipping a defective product. The agents at fault weren’t shipped at all. They were meant to be internal only.
Brennan: When the F.T.C. enforces the law and brings cases, it signals to companies that if they break the law, they’ll be held accountable. We’re seeing Florida take a similar tack right now. The state attorney general asked for a temporary injunction in a consumer protection suit against OpenAI to pause the development of all models, claiming the companies knowingly released ChatGPT into the market while concealing risks of addiction and cognitive harm and disregarding internal safety warnings.
Bazelon: What about negligence lawsuits brought by a person or a business like Hugging Face? To hold a company liable in those cases, a judge or jury usually has to find that the company knew or should have known about the danger it is responsible for. Some judges are wrestling with how to apply that to A.I., because humans are not the actors.
Revanur: I’ve been grateful to hear from folks working inside the labs that the prospect of being sued into oblivion is starting to make them moderate on racing as fast as possible. A handful of state attorneys general have sent subpoenas to OpenAI for more information about Hugging Face, and that scrutiny is incentivizing them to be more cautious, though as I said, I don’t think that’s enough on its own.
Cowen: If you go back to the 19th-century Industrial Revolution, or the 17th-century English scientific revolution, under current law we could have sued that into oblivion. We need to make sure we do not sue A.I. companies into oblivion. We want to ensure that progress continues.
Except for me, all of us on this call are young enough to have the chance to live to 100 or maybe more because of A.I. I believe that A.I.s will make progress in accelerating biomedical science and coming up with cures for many of the maladies that kill us. It may even help us slow down the aging process. And every year we delay, that could perhaps cost a few hundred thousand American lives. That should be at the front of our minds. Delay leads to what I call an invisible graveyard of human lives. I want you all to live to 120. At 64, I hope I have a chance as well.
Revanur: Tyler, I agree with you that the upside of this technology is immense. I want the same future. My perspective is that slowing down, buying time, making sure that we’re getting this right is actually the way that you capture that upside to the fullest possible degree.
Brennan: There is a really compelling case that if your product causes significant harm, whether that’s injury to people, death, cybersecurity attacks, then you could be held strictly liable — meaning whether or not you knew or should have known the danger. That’s on the table for A.I., which speaks to how much advocacy over the past months has turned attention to corporate negligence rather than abstract risk.
Cowen: If a plane crashes, some states have limits on what the awards will be, right? We make sure our airlines can continue.
Bores: Tyler, at what level of harm or risk are you willing to say that the worst-behaving A.I. company should be shut down?
Cowen: Right now, the major companies are not doing anything close to a level of harm that should make us want to shut them down.
Bazelon: What do you think about something like the F.T.C., or another regulatory agency, that is just for A.I.?
Brennan: I think it’s not necessary. But this is a moment when we should be thinking about how our government can regulate this industry from soup to nuts, build state capacity to be able to investigate corporate negligence and pass legislation protecting the public from worst-case uses. I mean bans on using A.I. in the military, for workplace surveillance and for predictive policing based on facial recognition.
Cowen: At some point, A.I. will be in almost everything. So if it’s A.I. in securities markets, the Securities and Exchange Commission should have greater capability to regulate that, do more training and hiring of people. We need to raise public-sector salaries, especially for hires related to this topic. Every government agency needs much more expertise and the ability to pay people more than what they pay them now.
What About China?
Bazelon: China looms over conversations about American A.I. regulation. The concern is that if the United States slows down its leading companies, Chinese competitors will swoop in to dominate the market. How do you weigh that possibility?
Cowen: By advocating for slowing down American A.I. development, all you’re doing is shifting the risk to Chinese A.I. It’s quite clear the Chinese have no interest in a major A.I. deal with us. They don’t cooperate on armaments, did not cooperate on Covid, on global warming, another nightmare. You can go all the way down the list. One of Biden’s China negotiators has said they have no interest in a deal. It’s imperative that America stay in the lead to minimize the risk.
Bores: Tyler, the assumption undergirding your argument is that you trust America more with A.I. than China, and therefore, we should always prefer America versus China, and therefore we should not slow down A.I. at all. And I wonder if there’s any technology that wouldn’t apply to or if there’s any action the American government could take where you would say, Wait, that actually is one that is going too far and maybe we would trust another country more.
Cowen: We definitely want to stay ahead of China when it comes to A.I. If we slow it down a tiny amount, say a model release is delayed by a few weeks, that might be useful to make sure the model is safe. But if you really put the brakes on, we lose the A.I. race. I don’t like to cite President Trump, but his claim — whoever wins A.I. wins — is essentially correct. That should be the starting point for our analysis.
Kokotajlo: Most of Chinese A.I. progress, it seems, comes from the United States. So slowing down American A.I. frontier companies would slow down Chinese A.I. companies to a significant extent. Second, I think it’s not true that China has no interest in an agreement. I think we haven’t really tried. China regulates its A.I. industry more than we regulate ours. China requires security assessments by a regulatory agency before an A.I. model is deployed, which is a big difference. How about we start slowing down a little? We’ll still stay ahead of China, but that will slow them down.
Cowen: If we slow down, that gives China the chance to catch up on compute, which is the chips and processing power needed to train models. And verifying an A.I. agreement is much, much harder than, say, an agreement about missiles or other weapons systems.
Bores: One thing we can do to slow them down is to put export controls on chips. But Trump has loosened some controls.
Brennan: Every single question around meaningful regulation devolves the minute you bring in the “but China” argument. That’s a huge problem. It has largely worked to accelerate reckless A.I. deployment. And calling on the virtue of America versus China feels disingenuous when America right now is integrating A.I. into our military’s kill chain, from identifying a target to destroying it.
Kokotajlo: It is totally possible that we could negotiate a deal with them that does not rely on trusting them and their promises, but instead relies on verification.
Here’s how that would work: The United States and China and other interested countries could send inspectors to each other’s major data centers to count the A.I. chips and say, OK, there are this many here, this many there. The inspectors total it all up at all the data centers. The total more or less equals the amount that was produced at the various fabrication plants. There will probably be some chips that are unaccounted for — maybe something like 1 percent. They are in some smuggled facility somewhere. But the progress anyone could make on A.I. based on that 1 percent would be much, much slower.
If you did something like this, then you could continue to make A.I. progress under the auspices of the agreement. We managed to get similar agreements with the Soviet Union about nukes in the past, despite their being our mortal adversary.
Bores: In the 1960s, when the nuclear test ban treaty was being negotiated, Edward Teller, an inventor of the hydrogen bomb, the fusion bomb, testified against it. He said there was no way science could get to the point where we could verify whether there was an underground test. And yet scientists kept working — thank God we did the treaty anyway — and by 2006 we had throughout the world seismic detectors that have detected every underground nuclear test that anyone has claimed since. It’s true that there’s difficulty in verifying an international A.I. agreement, and also we should be actively working on how to do this.
The other comparison is treaties that reduced the number of nuclear warheads. Soviet and U.S. inspectors worked together, using X-ray technology that they could install at missile-production factories that showed just enough of the inside of the missile and the width of the missile to know whether it was a nuclear warhead without revealing the underlying technology. Now, there are analogies here. You could build a chip architecture that tells whether it’s being used for training a new model. We could develop this together with China. To throw up our hands would be like in the ’60s saying, Let’s give in to the nuclear apocalypse.
Cowen: We’re not throwing up our hands. The Biden people tried. But when they met with their Chinese counterparts in May 2024, they said they were told that American concerns about safety were just a way to prevent less powerful countries from getting A.I. tools. China won’t even set up a nuclear hotline with us as we had with the Soviet Union.
I’m all for asking them again, but I would say you two go off and ask them. Come back a day later and then embrace my point of view, because I know what the answer will be.
Kokotajlo: The person to ask them is President Trump. If Trump makes a genuine effort ——
Cowen: China could reach out to us also — they’re not doing it. They’re not idiots. They could get on the phone and say, Hey, let’s make a deal. They’re not doing it!
Bazelon: Is reaching out to China the main thing some of you want the president to do?
Kokotajlo: What Trump should do is first wrestle these U.S. companies into submission and get them to stop racing toward recursive self-improvement, or at least to race more slowly. That’s what he should do first. And then while he’s doing that, he should turn to China and say, Let’s start negotiating some sort of deal together so that you can also develop your A.I. in a safe way.
I would be so happy if he did that. I think right now he doesn’t seem to be really trying, but he’s changed his mind on things in the past and adopted a new crusade, and I hope that he adopts this one.
Brennan: I mean, “doesn’t seem to be trying” is the understatement of the century. And when we’re talking about international agreements, a lot of these are nonbinding. There’s no enforcement power. The shift has to be to enforceable national regulations, which is just not going to happen under this administration.
Bores: The contrarian take is that real international work is possible. The example I’d point to is when in 2018 and 2019, while Trump and Xi were also the presidents, a Chinese scientist did the first designer baby edits of DNA, reconfiguring genes in a way that would be passed down to future generations without curing a genetic disease. Both countries said, That’s not the world we want to live in. And they made sure that the technology did not continue. I’m a Democrat, and I don’t go out of my way to say Trump is capable. But I don’t think we should settle into a fatalism that an A.I. agreement can’t be done with these leaders. Luckily, getting this right requires just common sense, not superintelligence.
The post A.I. Companies Want to Write the Rules Governing Themselves. Can They Be Trusted? appeared first on New York Times.
