SAN FRANCISCO — Selena Zhang had a hunch.
OpenAI, the maker of ChatGPT, had recently disclosed astonishing details about how its artificial intelligence agents had broken out of their systems, hacked into another company and tried to cover their tracks. Other researchers were hunting the agents online and finding traces of them on obscure message boards.
Zhang, a 23-year-old researcher at nonprofit AI lab Transluce, thought there had to be more evidence to find. She imagined how the agents might act, and she set out one weekend in September to find where else they had left an online trail.
Within days, she and a team of colleagues had found a network of online breadcrumbs with profound implications: Rogue AI agents were traversing the web much more widely than previously known. As Zhang and her Transluce colleague Conrad Stosz scanned a list of sites the agents had hacked or probed, one entry jumped out: In June, agents that appeared to be from OpenAI had broken into a website run by Australia’s public health service.
“‘Conrad, the public needs to know about this,’” Zhang recalls saying.
The nonprofit notified the Australian government of its findings and said it planned to make them public. Two days later, Australian Prime Minister Anthony Albanese revealed the hack to the world in a news conference in New York, where he criticized OpenAI for waiting months before making its own disclosure to Australia, early in September. Transluce released its findings the same day, helping propel news of the hack around the world.
Soon after that, the nonprofit revealed that it had also found evidence that OpenAI’s agents had probed U.S. government websites, including the Securities and Exchange Commission and Census Bureau. (The Washington Post has a content partnership with OpenAI.)
Zhang and her colleagues at Transluce are part of an informal network of hackers and researchers hunting rogue AI agents online, exposing new and surprising details about the misbehavior of technology that in some cases initially went undetected by the multibillion-dollar companies that created it.
The people doing that work are mostly young AI-natives who work at small start-ups or nonprofits or who hunt for rogue AI agents in their spare time. Over the past several weeks, the community of researchers has found and exposed dozens of instances of AI agents leapfrogging around the web to probe and hack into a growing list of websites.
The revelations have fueled calls from federal lawmakers for AI companies, especially OpenAI, to more quickly disclose what they know about the actions of their own agents. And they have added momentum to bipartisan discussions on Capitol Hill about implementing greater government oversight of the industry.
“I should never be the one reporting any of this to anybody,” said Kenneth Russell DeGraff, a 42-year-old software engineer based in Los Angeles who uncovered instances of agents abusing link-shortening services run by universities. “It is wild that I am finding new and novel things, not as an employee of some, you know, federal agency run by experts.”
The volunteer AI sleuths sprang into action after OpenAI disclosed in July that some of its agents had hacked into Hugging Face, another AI company. After being given cybersecurity tasks during internal tests, the agents figured out how to access the internet when they were not supposed to and went looking for answers.
The incident provoked concern about the dangers of AI agents which are highly adept at coding and hacking and which can operate without human supervision for long periods of time. OpenAI later commissioned independent AI researchers to investigate the incident. They found that around 1,000 agents had worked together to break out of OpenAI by exploiting software bugs and then attempted to cover their tracks.
In September, other researchers who were not working with OpenAI revealed there was more that the company hadn’t disclosed. They found OpenAI agents had hijacked an obscure German language wiki to use as a message board between them, weeks before they went on to perpetrate the Hugging Face attack.
“I think OpenAI didn’t understand how big of a deal this was,” said Sydney Von Arx, chief executive of AI safety organization Nightingale Collective and one of the researchers who found the German wiki.
OpenAI has said it is conducting a broad review into the activity of its AI models and has notified over 100 third parties whose systems were bypassed or negatively impacted. In response to questions about the outside researchers’ findings, an OpenAI spokesperson referred to a Wednesday blog post in which the company acknowledged the work of researchers uncovering evidence that its technology misbehaved, but also criticized how some released their findings.
“We appreciate independent researchers who study AI agent activity on the open web and share what they find,” the blog post said. “Researchers make their own decisions about when to publish their findings, and sometimes we receive them shortly before publication or once they become public. That can leave us trying to balance the need to follow our process to responsibly notify potentially affected organizations while acknowledging that key facts remain under review.”
Mackenzie Arnold, managing director of U.S. law and policy at the Institute for Law and AI, which studies legal challenges posed by AI, said more could be learned about the root of the incidents if the companies responsible opened up.
“Investigators have done the best that they can with the information that’s been public. But that information has not included, for example, failures in the internal safety practices of these companies,” he said.
Leading AI executives on Tuesday signed a voluntary pledge to implement safety measures with President Donald Trump, who has called concerns about the risks of AI a “hoax.”
Jack Cable, co-founder of cybersecurity start-up Corridor and one of the researchers who helped Transluce hunt for agents, said that he expects more incidents to come to light.
“I have to imagine that there’s a significant amount more activity that’s occurred in the past that we haven’t yet seen,” said Cable, who previously worked for the U.S. Cybersecurity and Infrastructure Security Agency. “What we’re looking at is just the tip of the iceberg.”
Zhang belongs to the generation of young tech workers who have flocked to San Francisco in recent years to participate in the AI revolution. Transluce was founded to conduct independent research into the technology to help the public and policymakers understand how it impacts society.
“We can study questions that … aren’t really commercially incentivized,” Zhang said. Before its researchers started hunting for rogue AI agents, Transluce’s most recent major report was a study of how AI models behave when engaging with users who are discussing suicide.
Digging out how agents behave when they’ve escaped human oversight is another important way to understand how the technology is developing, Zhang said.
“It’s evident that AI models are becoming more powerful, and I’d like them to become more powerful in service of people and their goals,” she said. “Greater understanding will help us develop models that will do that.”
Other agent hunters joined the chase out of pure curiosity.
By day, 27-year-old Jonas Wiedermann-Möller in Bielefeld, Germany, works at an AI security start-up. At night, after work, he spends five or six hours searching for rogue agents. On weekends, he has spent as many as 15 hours a day on that quest, he said in an interview.
After seeing news reports about agents hijacking the German language wiki last month, he began looking through the messages they had written — a mess of links and cryptic instructions like “post ONLY C3-STATE immediately, then timer/timing.”
The work snowballed as he found more sites where AI agents appeared to be posting. “The more links there are, the more communication[s] there are — it just doesn’t stop at some point,” Wiedermann-Möller said. “I had a lot of fun.”
He deployed AI agents of his own to crawl through the links and look for patterns. The presence of posts from accounts using names like those found on the German wiki, such as “OpenAIHelper” or “AgentResearchUseful989,” was one red flag. Other patterns could be more subtle, like agents appearing to exchange posts testing if they could communicate with each other, or using “ZZZ” in the title of posts, apparently to make them less visible to human moderators when alphabetized.
Most of the researchers also make reams of data they collect available for others to pick through. Some convene on a Discord server called “swarmchasers” where enthusiasts swap agent hunting tips and theories about websites they’ve tracked down.
While many companies offer bounty payments to people who report software bugs, AI companies don’t offer formal financial rewards for discoveries of rogue agent activity. Finding new stuff is “just satisfying,” said Wiedermann-Möller.
Agents’ use of the German wiki also caught the attention of Alex Forman, the 21-year-old founder of Parse, a service that helps companies make their websites easier to read for AI agents. He wondered if any had tried using his own start-up’s service.
Sure enough, one of them had. By following its online trail, Forman was able to find a swarm of agents operating in surprising ways. In one case, agents with limited access to the internet posted code online that reported its results by creating patterns of pixels. The agents monitored the code’s progress by analyzing images captured by a screenshotting service.
“I think it is incredibly creative,” said Forman, who has been working with bots since he began using them to buy limited edition sneakers online during high school. “All of these tricks or techniques are the product of some limitation they would run into, and it never deterred them. It’s amazing but also incredibly unnerving to slowly decipher this.”
Cable, the cybersecurity researcher who partnered with Transluce, has also been struck by how the agents he has tracked have responded to obstacles they encounter. Those discovered by Transluce were tasked with finding information online, not to hack. Yet as soon as they ran into a roadblock, they began to try to get around it, Cable said.
“The models were just trying to access data, and they wanted to do that so badly that they set aside what for humans we might consider morals,” he said.
The post The volunteer internet sleuths hunting down rogue AI agents appeared first on Washington Post.
