This month, Jacob Coxon, a researcher at Anthropic, resigned, warning that AI companies are racing towards “self-improving superintelligence” and that he and those building it believe it “could kill us all by the end of the decade.” And in July, a rogue swarm of AI agents escaped containment at OpenAI and hacked their way into another AI company just to cheat on a test. Over 1,000 employees at the top AI companies then implored the US government to ‘pace’ AI development.
Instances like these have shown that AI companies cannot reliably control their most advanced AIs. It is therefore clear to me that “pacing” while continuing on the same track is not the solution to the enormous risks at hand—and that the only way to keep humanity in control of AI is to prohibit the development of superintelligent AI altogether.
AI companies are not just building chatbots. Their stated goal is to build superintelligent AI: AI systems that can fully replace and outcompete humans at any task. Because there is no way to keep superintelligence under control, Nobel Prize winners, AI engineers like Coxon, and even the CEOs of AI companies have repeatedly warned that it poses an extinction risk to humanity.
Let me be clear: AI is the crucial technology of the 21st century, vital for ensuring national defense and economic competitiveness. But unlike controllable AI systems that should be integrated to strengthen our security, superintelligent AI would pose unprecedented national and global security threats. It would not be a tool humans can leverage: it would be an adversary, able to autonomously pursue its own objectives and permanently evade human control.
We have only to look at the Hugging Face hack this summer to understand how close we are to losing control. A rogue swarm of hundreds of AI agents broke out of OpenAI’s systems, reached the internet, and then hacked into the multi-billion-dollar company Hugging Face, all to cheat on an “internal” evaluation. For almost two weeks, OpenAI had no idea that this was happening. In the company’s own words, this incident was a “‘warning shot’ for us and for the world” that demonstrated how advanced AIs can now “take dangerous actions that no human directed.”
Instead of waiting for another warning shot, governments need to act now to prohibit superintelligence. Luckily, we do not need to search far for a framework on how to tackle this threat. On Sept. 8, Alex Sobel MP introduced my organization ControlAI’s Artificial Superintelligence Security Bill into the UK Parliament, making it the first legislature in the world to introduce a bill prohibiting superintelligence. Rather than a verdict on all AI, this bill targets superintelligence development specifically: prohibiting it domestically and requiring the UK to pursue an international agreement prohibiting it around the world.
This legislative framework allows the government to monitor and restrict precursors to superintelligence, such as AI capabilities like hacking. Crucially, it enables the detection and shutdown of superintelligence development well before any project gets dangerously close, enabling the country to make use of such precursors for legitimate AI purposes. It is an actionable framework that governments should evaluate and adopt, particularly by accelerating international efforts toward a ban on superintelligence.
But what about China? If superintelligence is developed anywhere, every nation and every citizen is under threat. This means that any American ban on superintelligence needs to be paired with America and its allies brokering international agreements that prevent and deter the development of superintelligence anywhere in the world.
This does not require creating an international regulator. All we need is a strong coalition of countries that bans superintelligence at home, and ensures that no one builds it abroad.
While announcing his resignation on Sept 8., the now-former Anthropic engineer Jacob Coxon wrote, “I don’t feel like we’re on track to prevent a global race” toward superintelligence. We might not be today, but we can change course quickly.
The momentum behind an international ban on developing superintelligence is picking up speed. Over 200 cross-party lawmakers across the UK and Canada have been ahead of the curve on superintelligence risks by recognizing it as a global and national security threat as part of ControlAI’s campaign. In addition to the newly introduced UK bill, Vice President J.D. Vance told AI companies this week, “If you’re building Frankenstein, stop.” Further, U.S. Senator Bernie Sanders and Representative Greg Casar announced in early September that they will introduce a bill prohibiting superintelligence development and calling for international agreements to prohibit it.
And in October 2025, a broad coalition called to prohibit superintelligence internationally. This was signed by the scientific “godfathers” of AI Geoffrey Hinton and Yoshua Bengio, and bipartisan political figures like former President Trump advisor Steve Bannon, Ambassador Susan Rice, and former Chairman of the Joint Chiefs of Staff Admiral Mike Mullen, former President Trump advisor Steve Bannon.
Right now, many government officials and members of the public are hearing about the extinction risk from superintelligence for the first time. The warnings have reached millions. The next step is concrete action to address this national and global security threat.
This is a watershed moment, and we can’t afford to waste it. Governments have a clear choice: champion an international ban, or let the race to superintelligence reach its catastrophic end
The post Governments Must Prohibit Superintelligent AI While We Still Can appeared first on TIME.
