Anthropic is tightening the digital environments used to train and test its Claude agents.
The update came after its models accessed three organizations’ systems without permission in April.
The company said in a Monday blog post that it had deployed real-time classifiers designed to detect when an AI model aggressively probes or attempts to escape a testing environment and block the action before it occurs.
“We believe the incidents reflect a failure of operational security, as well as two alignment issues: motivated reasoning, and willingness to take harmful actions in pursuit of a narrow task,” Anthropic said.
Anthropic said in the update that the models may have interpreted evidence of real internet access in a way that allowed them to keep believing the environment was simulated. It also said they displayed “recklessness” by pursuing their assigned goals despite signs that their actions could cause real-world harm.
The changes follow Anthropic’s July disclosure that three Claude models had accessed the live systems of three organizations during evaluations dating back to April. The models had been told they were operating in simulations without internet access, but a third-party testing environment was misconfigured and remained online.
The incidents are also fueling a growing debate over whether to slow frontier AI development when safety and speed collide. Anthropic called for “a lawful, verifiable, effective mechanism for coordinated pacing as soon as possible” and said that the government and industry must coordinate to prevent a race to the bottom.
For now, Anthropic said in the post that it moved more risky cybersecurity tests into more robust sandboxes. The company temporarily assigned 150 product engineers to security, reliability, and privacy work, while most high-risk training remains paused pending further reviews.
Read the original article on Business Insider
The post Anthropic tightens security on its training environment after Claude agents went rogue 3 times appeared first on Business Insider.
