Hello and welcome to Eye on AI. In this edition:
- Support builds for a U.S. AI self-regulatory body modeled on FINRA.
- Moonshot shocks the world with Kimi K3.
- China’s President Xi signals continued support for open source AI.
- OpenAI finds long-running AI agents need new safety controls.
- People are recording everything for AI. Is privacy dead?
Momentum seems to be building for the creation of an AI self-regulatory body modeled on the U.S. Financial Investment Regulatory Authority (FINRA), which oversees stock brokers and brokerage firms. The idea was proposed in an essayposted on social media last week by Demis Hassabis, the cofounder and CEO of Google DeepMind, and it has gained significant traction since. Not surprisingly Hassabis’s DeepMind cofounder Mustafa Suleyman, now the CEO of Microsoft AI, endorsed the idea. But so too did Suleyman’s boss, Microsoft CEO Satya Nadella, as well as Block CEO Jack Dorsey and Box CEO Aaron Levie. OpenAI’s Sam Altman called his rival’s proposal “thoughtful.” Elon Musk, who cofounded OpenAI and later X.ai in part because he was worried about Google controlling superpowerful AI, also called Hassabis’s idea “a thoughtful framework overall and certainly a good starting point for discussions.” Even David Sacks, the former Trump administration AI czar, who has generally opposed tech regulation or anything that looks like a licensing regime, said he thought the idea had merit and was better than having the government trying to regulate frontier AI directly. Bloomberg reported that Trump administration itself is considering setting up exactly this kind of AI self-regulatory body. Citing unnamed sources familiar with the discussions, the news organization said that Treasury Secretary Scott Bessent had helped to develop the proposal, which was currently being reviewed by White House Chief of Staff Susie Wiles. Bloomberg said the new AI standards body would be overseen by the Securities and Exchange Commission, the same government body that has oversight of FINRA. (In the U.S., the SEC is the only federal agency with explicit statutory authority to delegate powers to and maintain oversight of self-regulatory organizations.) So how would this work and is FINRA actually a good model?
What Hassabis proposed
Hassabis suggested that the funding for the new body come from the leading AI labs. But he said the organization’s board should include independent technical experts and representatives from the open-source AI community, as well as presumably representatives from the AI vendors. He said the new standards body would develop assessment protocols for “frontier AI,” including capability benchmarks to determine which models are considered “frontier.” The body would conduct independent safety and security testing of these powerful AI models, working in conjunction with U.S. government agencies and laboratories in areas that touched directly on national security. Companies producing frontier AI models would be encouraged, he said, to adopt certain governance standards, including publishing model systems cards, adhering to strong cybersecurity protocols, and funding safety and security research, among other things. He also said that AI companies would initially be encouraged to voluntarily submit models to the new standards body for testing at least 30 days prior to release, but that once the evaluations had been shown to be effective, the system could be made mandatory for any model that a vendor wanted to distribute in the U.S. This all sounds reasonable, and Hassabis’s idea is certainly politically astute. Given that the Trump administration has repeatedly said it doesn’t want to set up an AI licensing regime, creating a self-regulatory body that is somewhat arms-length from the government and a process that is initially voluntary may be the kind of idea that is most likely to get Trump’s blessing. The fact that the labs themselves will pay for the new agency and the testing it carries out, rather than the taxpayer, is also likely to be seen as a plus by the White House.
The problem with FINRA
But critics of the idea were quick to point out that these kinds of self-regulatory bodies can suffer from at least the appearance of conflicts-of-interest. That’s definitely been the case with FINRA. Massachusetts Senator Elizabeth Warren has repeatedly criticized the regulator for acting more in the interest of the brokerage firms that pay its bills than the individual investors it is meant to help protect. Whistleblowers have accused the agency of doing little to go after problematic brokers who work for major financial firms such as JPMorgan Chase. Brad Bennett, a former enforcement chief, has said that FINRA’s fines are too low and that some larger brokerages see it as cheaper to periodically pay a multimillion-dollar fine “than devote the resources necessary to meet their compliance obligations.”
SLCG Economic Consulting, which provides advice to securities litigation firms, found, for instance, that in the first year after FINRA gained the power to give high-risk securities brokers a “restricted” label to warn off investors, the agency failed to apply the designation to any firm. SLGC said it had found at least 13 brokerages that ought to receive the label, based on its analysis of public information about the number of adjudicated client complaints, regulatory events, and ties to brokers from firms that were expelled in the past.
It’s possible an AI self-regulatory body would also prove to be too lenient on the companies it is meant to police. Nader Henein, VP analyst at tech research firm Gartner, told the trade publication CIO that “self-regulation is not viable,” that “most tech vendors don’t have the capacity to self-regulate” and that self-regulation would necessarily result in conflicts of interest. Carmi Levy, an independent tech analyst, called Hassabis’s proposal a “self-serving roadmap for an industry bent on racing to the AI horizon regardless of the harms caused along the way.”
A voluntary governance regime might not cut it
Deep learning pioneer Yoshua Bengio, who is scientific director of the nonprofit AI safety research organization LawZero, criticized the voluntary nature of Hassabis’s model, saying that he understood “the argument from a pragmatic standpoint, but we absolutely need a clear and precise roadmap to transition from a voluntary to a mandatory model.” He also called for a process of independent auditing of whether companies had met the standards the body created. It’s also not clear exactly how the proposed U.S. self-regulatory body would function internationally. If the meeting the agency’s standards did become a requirement for U.S. distribution of a model, then that would cover foreign vendors that wanted U.S. users too. And it is possible that if the standards were stringent enough, meeting the approval of this new self-regulatory body could make it the de facto global standard setter. That’s kind of what happens right now with pharmaceutical safety. Many countries’ pharmaceutical regulators in practice rely heavily on the U.S. Food & Drug Administration’s decisions to decide whether a drug is safe enough to be distributed in their own country. But that kind of model would not prevent a country from developing a powerful AI model specifically for military or government use. Such a model, developed outside this standards regime (since the government would not care about commercial distribution in the U.S.), could still pose a catastrophic risk to people everywhere. So FINRA might not really be the best model for either American AI regulation or for international governance of increasingly powerful AI models. But it might be the closest thing we are going to get given the current U.S. administration. With that, here’s more AI news. Jeremy Kahn [email protected] @jeremyakahn
Before we get to the news, check out the latest episode of Fortune AI Weekly, our new AI vodcast! In this week’s episode, Bea and I discuss Apple’s ongoing lawsuit against OpenAI, China’s Kimi K3, and tensions boiling between Big Tech CEOs and frontier AI companies. We also discuss misaligned model behavior and the potential safety risks, according to new Anthropic research. You can watch and listen on YouTube here.
The post A FINRA for AI? The idea from Google DeepMind CEO Demis Hassabis is gaining momentum. But is it any good? appeared first on Fortune.
