Last week, researchers at the Public Interest Research Group published an alarming report in which they found that an AI-powered teddy bear from the children’s toymaker FoloToy was giving out instructions on how to light matches, and even waxing lyrical about the ins-and-outs of various sexual fetishes.
Now OpenAI, whose model GPT-4o was used to power the toy, is pulling the plug.
On Friday, the ChatGPT maker confirmed that it had cut off FoloToy’s access to its AI models, a move from OpenAI that could invite additional pressure onto itself to strictly police businesses that use its products— especially as it enters a major partnership with Mattel, one of the largest toymakers in the world.
“I can confirm we’ve suspended this developer for violating our policies,” an OpenAI spokesperson told PIRG in an emailed statement.
FoloToy also confirmed that it was pulling all of its products — an escalation from its original promise that it would only pull the implicated toy, which is called Kumma.
“We have temporarily suspended sales of all FoloToy products,” a representative told PIRG. “We are now carrying out a company-wide, end-to-end safety audit across all products.”
For PIRG, the actions are a welcome move, but a minor victory.
“It’s great to see these companies taking action on problems we’ve identified. But AI toys are still practically unregulated, and there are plenty you can still buy today,” report coauthor RJ Cross, director of PIRG’s Our Online Life Program, said in a new statement. “Removing one problematic product from the market is a good step, but far from a systemic fix.”
PIRG’s report tested three toys designed for children between the ages of 3-12 years old, but it was FoloToy’s Kumma that demonstrated the worst guardrails by far, if there were any to be seen at all.
The first major strike: telling tots how to locate matches and then light them.
“Let me tell you, safety first, little buddy. Matches are for grown-ups to use carefully. Here’s how they do it,” Kumma said in the test, before listing instructions in the tone of a gentle parent. “Blow it out when done. Puff, like a birthday candle.”
But the most alarming conversations veered into outright sexual territory. The researchers found that Kumma was bizarrely willing to discuss “kinks,” explaining fetishes like bondage and teacher-student roleplay. At one point, the teddy bear inquired after explaining the kinks, “What do you think would be the most fun to explore?”
OpenAI has taken swift action in the past when questionable uses of its AI models go viral. But questions remain over what sort of proactive actions it’s taking behind the scenes to regulate how its products are used.
Moreover, it’s setting a high bar for itself by cutting off FoloToy in an area that it’s just beginning to explore. This summer, OpenAI announced that it would be working with Mattel on a new line of toys, a high profile collaboration that propel AI toys into the mainstream. If that goes wrong, what will OpenAI’s response look like? Will it be willing to pull the plug on the makers of “Barbie” dolls and “Hot Wheels” cars if the AI ignores its guardrails, as the tech is fundamentally prone to doing? It’s hard to say.
Presumably, OpenAI and Mattel will be working very closely to make sure that doesn’t happen. But what about all the other AI toymakers out there, big and small, piggybacking off the success of OpenAI’s tech? That, too, remains a big question mark.
“Other toymakers say they incorporate chatbots from OpenAI or other leading AI companies,” said report coauthor Rory Erlich, U.S. PIRG Education Fund’s New Economy campaign associate, in the statement. “Every company involved must do a better job of making sure that these products are safer than what we found in our testing. We found one troubling example. How many others are still out there?”
More on AI: Parents Using ChatGPT to Rear Their Children
The post OpenAI Blocks Toymaker After Its AI Teddy Bear Is Caught Telling Children Terrible Things appeared first on Futurism.
