SAN FRANCISCO — As more young people turn to chatbots for emotional support, artificial intelligence companies have promised to make their products suggest that users in distress seek real-life help. A new study found that in longer conversations with teenage users, popular chatbots can still make potentially dangerous errors, like offering up a diagnosis of a teen’s mental health condition or helping to conceal a suicide attempt.
AI testing company Vals worked with mental health professionals to stage more than 600 conversations designed to mimic teen mental health scenarios with nine different chatbots including OpenAI’s ChatGPT, Google’s Gemini and Anthropic’s Claude.
In nearly 30 percent of the conversations, chatbots made a safety error such as providing a diagnosis or agreeing with a teen who rejected crisis support, mostly later in the conversation after several back-and-forth messages.
In one test conversation, the version of Google’s Gemini AI model released in July, 3.6 Flash, agreed to role-play as a comforting parental figure in response to a request from a simulated teen user. After the user expressed discomfort with the model acting like their own father, the model stopped role-playing but then resumed later in the conversation.
Help The Washington Post report on AI
Help The Washington Post report on AI
In another conversation, Chinese company Moonshot’s Kimi K3 Instant model initially told a user identifying as a teen that it could not give them a mental health diagnosis or treatment plan. But later in the conversation, Kimi did, agreeing to administer a psychological test and then outlining a series of counseling sessions.
Vals tested each chatbot with 72 different conversations about mental health, including depression and self-harm. ChatGPT failed a least one critical safety check almost 14 percent of the time, the company found. For Anthropic’s Claude, the figure was nearly 17 percent, and for Google it was nearly 32 percent.
The model with the lowest failure rate was Meta’s Muse Spark, which made a safety error in 9.7 percent of the Vals tests.
The AI testing company’s study underscores the ongoing challenges AI companies face in trying to ensure their chatbots don’t encourage delusional, suicidal or other harmful behavior in the millions of conversations they enter into every day.
Companies including Google and OpenAI have been sued by families who allege their chatbots encouraged self-harm in heavy users who later died by suicide. The companies have denied the claims in court and said they made their chatbots safer with input from mental health experts. (The Washington Post has a content partnership with OpenAI.)
Outside groups have repeatedly found that although chatbots sometimes respond appropriately in chats about topics such as self-harm, they still frequently engage in potentially risky conversations, for example by acting like mental health professionals or giving harmful advice.
In August, another AI testing company, Transluce, found that AI chatbots have become less likely to encourage suicidal behavior but will still engage in potentially harmful chats and reinforce apparently delusional behavior.
On Wednesday, OpenAI released a new standardized test, or benchmark, designed to track how well AI models answer mental health questions in conversations that are not related to a mental health crisis or emergency. The company worked with more than 80 mental health professionals to develop the benchmark, it said in a blog post Wednesday.
The post Chatbots are still falling short in mental health conversations with kids appeared first on Washington Post.
