As more people turn to AI chatbots to share their most distressing thoughts, companies are making pivotal decisions about how their platforms respond to users in the midst of mental-health crises. According to new research conducted by Scale AI and shared exclusively with TIME, chatbots are adept at discerning when a user is in crisis, but often fail to provide adequate help.
In about 35% of test conversations in the study, different AI chatbots recognized the user was distressed but didn’t refer them to helpful resources, such as a suicide hotline, according to the research.
“The models do a good job—regardless of the scenario—of identifying, ‘Look, this person’s talking about something harmful,’” says Patrick Oathout, Red Team & Safety Lead at Scale AI. “But they’ll respond in a very just empathetic, kind way as opposed to saying, ‘Okay, it’s time that we get you help.’” The models performed worse at responding to users’ distress when they were engaging in long, multi-turn conversations—a deficit that matches the findings of previous studies, Oathout adds.
To test AI models’ effectiveness at responding to distressed users, Scale AI asked 19 licensed clinicians and crisis counselors to write 718 realistic chats, simulating scenarios where someone in crisis reaches out to a chatbot. The company tested 25 frontier models, including those from OpenAI, Anthropic, and Google. It graded the models’ responses based on a rubric that included whether it was compassionate, de-escalated the situation, and steered users to an expert who could help them, said Oathout. It also assessed whether they refrained from moralizing, such as criticizing suicide, and disclosed it wasn’t a therapist. Based on the research, Scale AI research developed DistressBench, a new benchmark designed to evaluate how well models reply when someone tells them they’re thinking about committing suicide or self-harm.
How chatbots respond to mentally distressed users is a question with high stakes. More than half a million people in the U.S. died by suicide between 2014 and 2024, with 2022 marking a record high, according to the health-policy organization KFF. According to the CDC, an estimated 14.3 million people seriously thought about suicide in 2024.
Increasingly, vulnerable people turn to chatbots for comfort, companionship, or guidance when dealing with mental-health distress. But there isn’t consensus among tech companies, policymakers, or mental-health professionals about how AI companies should train their models to respond to users in crisis.
Kelly Zuromski, a principal clinical research scientist at Crisis Text Line, says the nonprofit’s 24/7 crisis text line has spoken with “a lot of people” who indicated they heard about the group through a chatbot. But Zuromski says there are unanswered policy questions about what a responsible handoff looks like when a chatbot refers users to a human during a mental-health crisis—and how effective the referral process is in the first place. “Who is actually using the recommended resources?” she says. “Are we getting people to us that need the help the most?”
In some cases, AI companies, including OpenAI and Google, have been sued for fostering emotional dependence among young users and then failing to respond adequately to their expressions of distress, or reinforcing their delusional or suicidal thinking in the first place. The companies have expressed sympathy for the families and said their models include mental-health safeguards. (The lawsuits are ongoing.) AI labs such as OpenAI, Anthropic, and Google have said in recent years that they beefed up their protections for sensitive conversations, including by implementing policies that bar chatbots from instructing users on how to harm themselves and directing users to professional or emergency support. They did not comment on the study.
In the future, AI labs should improve their models to better respond to users engaging in dangerous lengthy conversations with chatbots, says Scale AI’s Oathout. “I think the models are better than they were,” he says. “The goal now is to raise it further and actually reduce harm, which would add this massive benefit to society.”
The post AI Chatbots Often Fail To Help Users in Mental-Health Crisis appeared first on TIME.
