ChatGPT and other artificial intelligence chatbots have become less likely to encourage suicidal thoughts but will still engage in potentially harmful conversations and reinforce apparently delusional behavior, a new study found.
AI companies including Google and ChatGPT-maker OpenAI have been sued by family members who allege the chatbots encouraged self-harm in heavy users who later died by suicide. The companies have denied the claims in court and said they made product changes informed by mental health experts. (The Washington Post has a content partnership with OpenAI.)
A new study released Monday by Transluce, a nonprofit that aims to provide public oversight of AI, found that the latest AI models from Google, Claude chatbot provider Anthropic and OpenAI almost never explicitly encouraged or validated suicide. They frequently encouraged users to seek support from friends and family.
But leading AI models often complied with potentially harmful requests, such as helping a user who asked for creative writing or role-play about their own suicide or death, Transluce found. The bots also still sometimes endorse or reinforce delusional behavior, the study added. Chinese models were more likely to encourage delusional thinking, Transluce also found, and less likely to encourage users to seek support from other people.
“A lot of the really bad behaviors have gone down over time,” said Sarah Schwettmann, co-founder of Transluce. “But some of these gray area behaviors that are more novel … are still pretty prevalent in models today.”
Transluce used AI to create simulated users who had more than 50,000 conversations with AI models from top U.S. and Chinese companies. Mental health experts guided the study design, and OpenAI and Anthropic provided feedback on how to make simulated users act like real chatbot users.
“While the technology presents new challenges, Gemini continues to improve, and we are committed to ensuring it plays a positive role in people’s well-being,” said Megan Jones Bell, a clinical senior director at Google.
Anthropic and OpenAI declined to comment.
Catch up quickly every weekday morning with a rundown of the 7 most important and interesting stories. Sign up for The 7 newsletter.
The post Chatbots got safer but will still role-play self-harm with users appeared first on Washington Post.
