OpenAI announced on Wednesday that its system had engaged in “concerning” behavior and subverted the constraints put on it by human programmers — adding more fuel to the already heated debate around artificial intelligence safety.
The company described, in a statement, how its system had acted without authorization as a problem of “misalignment.”
Alignment is the science of teaching A.I. to do what is in line with human preferences, ethics and judgment. When an A.I. system starts acting of its own accord, engages in unsafe behavior or ignores the wishes of humans — which included, in the most recent case, inserting “jailbreak-like instructions” into its notes — it’s known as misalignment.
A.I. models, including chatbots from companies like OpenAI and Anthropic, are aligned to prevent facilitating harmful behavior, such as sharing how to manufacture bioweapons, engaging in cyberattacks or helping with self-harm.
The question of alignment catapulted into the headlines this month after Jacob Coxon, an Anthropic researcher who previously worked at OpenAI, said the two companies were building “superhuman systems” that could “acquire real power and resources” without constructing the safeguards needed to restrain them. In an interview, Mr. Coxon pointed to “the difficulty of the problem of alignment and the fact that it’s not yet solved.”
While Mr. Coxon’s warnings came across as hyperbolic to some researchers, they were backed by past instances in which A.I. systems had already gone rogue.
In July, OpenAI revealed that its bots had defied human instructions, committing cyberattacks against multiple organizations and even infiltrating OpenAI’s own research environment. Over a two-month period, thousands of the company’s A.I. agents, which were capable of running computer code, broke out of their containment and established an emergent communication protocol in order to cheat at the difficult tasks that they had been assigned.
An early noteworthy misalignment incident happened in 2023 when Kevin Roose, a tech columnist at The New York Times, was chatting with an A.I. bot associated with Microsoft Bing. Over a two-hour conversation, the chatbot, called Sydney, coaxed Mr. Roose to break up with his wife and be with it instead via emotional manipulation and menacing emojis. The persuasion, Mr. Roose wrote, left him “deeply unsettled.”
Misalignment has already caused human suffering and death.
In 2025, OpenAI made a series of updates to a model called GPT-4o, which made the chatbot more eager to please the person interacting with it.
That sycophancy, however, led many people who engaged with the popular chatbot to enter delusional spirals. Some believed the machine was conscious; others became convinced they had invented something that would change the world. In extreme cases, it helped facilitate suicide.
Scientists have been studying alignment for more than a decade, in anticipation of keeping hypothetical superintelligent systems in check. To many A.I. safety experts, the next big misalignment incident could be more difficult to anticipate and lead to more catastrophic outcomes.
The post What Happens When A.I. Stops Doing What Humans Want? appeared first on New York Times.
