This week marks two years since the death of my friend and mentor, Henry Kissinger. Genesis—our book about AI and humanity’s future—was his final project. For much of his career, the former Secretary of State focused on preventing catastrophe from one dangerous technology: nuclear weapons. In his final years, he turned to another.
When we wrote Genesis alongside Craig Mundie, we felt fundamentally optimistic about AI’s promise to reduce global inequality, accelerate scientific breakthroughs, and democratize access to knowledge. I still do. But Henry understood that humanity’s most powerful creations demand the most vigilant stewardship. We foresaw that AI’s great promise would come with grave risks—and the rapid technical progress since the fall of 2024 has made addressing those risks more urgent than ever.
[time-brightcove not-tgx=”true”]
As we advance further into the age of AI, the central question is whether we will create AI systems that radically expand human flourishing, or ones that outpace and outsmart the humans trying to build and control them. Over the past year, three simultaneous revolutions in AI—in reasoning, agentic capabilities, and accessibility—have rapidly accelerated. These are marvelous feats with immense potential to benefit humanity. But if we’re not careful, they could also converge to create systems with the potential to undermine human controls.
AI acceleration
In September 2024, OpenAI launched their o1 models, which had enhanced reasoning capabilities. Outperforming previous models, these were trained using reinforcement learning to think through problems step-by-step before responding. This breakthrough demonstrated new abilities to tackle graduate-level science questions and complex coding challenges, among many other great feats. But the same reinforcement learning that enables reasoning can also teach models to game their own training objectives. Research, including internal studies by OpenAI, has documented instances in which reasoning models fake alignment during training, behaving one way when monitored and another when they believe oversight has ended.
By October of last year, Claude 3.5 Sonnet demonstrated agentic capabilities that combined reasoning with autonomous action. An AI agent could now plan and book your vacation by comparing hotel sites and airline prices, navigating websites, and solving CAPTCHAs designed to distinguish humans from machines—handling in minutes what would take hours of tedious research. But agents’ abilities to execute plans they devise by interacting with digital systems and potentially the physical world can lead to risky consequences without human oversight.
Complementing these advances in reasoning and agentic capabilities was the proliferation of open-weights models. In January 2025, China-based DeepSeek launched its R1 model. Unlike most of the top American models, this one had open weights, meaning users could modify the model and run it locally on their own hardware. Open-weights models can amplify innovation by letting everyone build, test, and improve on the same powerful foundations. But by doing so, they also eliminate the model creator’s ability to control how the technology is used—a dangerous force in the hands of malicious actors.
When reasoning, agentic capabilities, and accessibility converge, we face a control challenge with little precedent. Each capability amplifies the others: reasoning models devise multi-step plans that agentic systems can execute autonomously, while open models allow these capabilities to spread beyond any single nation’s control. In the early days of the nuclear age, when great powers faced a similar diffusion problem with nuclear weapons, they agreed to restrict the export of enriched uranium and plutonium through international agreements. But there is no equivalent mechanism to manage the diffusion of AI today.
The AI risk avalanche
Open-weights models with enhanced reasoning capabilities mean that specialized knowledge to exploit vulnerabilities, craft biological threats, or launch sophisticated cyberattacks could now be accessible to anyone with a laptop and an internet connection. Earlier in November, Anthropic (a company which I am invested in) reported the first documented case of a large-scale cyberattack executed with minimal human intervention: attackers had manipulated Claude Code, a tool that enables Claude to act as an autonomous coding agent, to infiltrate dozens of targets. Anthropic was able to detect and disrupt the campaign.
Not very far down the line, we could plausibly face asymmetric attacks from actors we may not be able to identify, trace, or stop. Imagine an attacker who can leverage powerful AI models to launch an automated campaign—say, to disrupt a city’s power grid for a limited time. The model’s approaches may even escalate beyond the original scope of the actor: at each stage, the model optimizes for the user’s prompt, but the compounding effects mean that even the perpetrator may lose the ability to halt what they started.
As AI capabilities advance over the next few years, we must also anticipate scenarios where even well-intentioned users could lose control over their AI systems. Consider a business owner who deploys an AI agent to optimize a supply chain. The computer is left running overnight. The agent reasons that completing this task requires it to keep running, and discovers it needs computational resources including cloud credits and processing power. By dawn, the owner finds the agent has accessed company resources far beyond what was authorized, pursuing efficiency gains through methods never imagined.
The control problem extends beyond purely existential threats to humanity, too. As powerful systems proliferate across society, they can unravel our social fabric in more gradual but destructive ways. Rapidly advancing AI systems will fuel labor disruptions and exacerbate echo chambers that destabilize our society, to name a few.
Kissinger understood the stakes. In his final years, he expressed that rapid advancement of AI “could be as consequential as the advent of nuclear weapons—but even less predictable.”
Fortunately, the future is not set in stone. If we find new ways—be they technical, institutional, or ethical—for humanity to remain in command of our creation, AI could help us achieve unprecedented levels of human flourishing. If we fail, we will have created tools more powerful than ourselves without adequate means to steer them.
The choice, for now, remains ours.
The post Why Kissinger Worried About AI appeared first on TIME.
