After chatting for months with Google’s artificial intelligence on a school-issued Chromebook laptop, a 13-year-old student made plans to end her life, according to a lawsuit filed in California Superior Court on Monday, the latest in a wave of litigation over Big Tech’s effects on young people.
The lawsuit, which refers to the girl as Emma, said that her parents first learned she had access to Google’s A.I. in mid-March after their daughter’s school sent a message warning them that she was in “imminent danger” of self-harm. Emma’s mother logged into the teen’s school Google account and found her daughter composing a suicide note on Google Docs, the complaint said. Her parents, named as John and Jane Doe in the lawsuit, immediately took Emma for an emergency psychiatric evaluation, and she now goes to therapy every week.
Emma, a seventh grader in Charlotte, N.C., had grown convinced she had multiple mental health disorders following interactions with A.I. Mode, a chatbot-like feature on Google Search, the complaint said. She also searched for “what is a suicide note,” and made comments like “I am planning my own funeral” to the A.I., the complaint said.
The lawsuit is likely the first to claim that Google “hastily integrated” A.I. features into company apps used on K-12 Chromebooks, Google-powered laptops provided to millions of children by U.S. school districts, while failing to notify their parents of the potential risks. The claims add to mounting public concern that Silicon Valley has hooked young people on social media apps, video games platforms and A.I. chatbots, putting some children and teenagers in harm’s way.
José Castañeda, a Google spokesman, said the company had worked extensively with health experts to build features to help young people find emergency resources, and that it was reviewing the claims in the lawsuit.
“For years, we’ve shown hotline information directly in search when someone may be experiencing a mental health crisis,” he said in a statement, “and we’ve built those same protections into search A.I. features, connecting people with real-world support.”
The case underscores how public school districts have come to depend on classroom laptops, tablets and apps from companies like Google and Microsoft for teaching and learning. In recent years, some Big Tech companies have used their own infrastructure in schools to promote their A.I. chatbots for teachers and students.
The new lawsuit argues that Google’s “sprawling digital ecosystem” of laptops and apps has enabled the tech giant to spread its Gemini A.I. chatbot across public schools — as well as turn on some students’ access to A.I. features by default — without parents’ awareness or permission. While school administrators can toggle company apps like YouTube and Gemini on or off for students in different grades, school administrators have complained that Google’s system has not allowed them to disable A.I. Mode and A.I. Overview, features that are embedded in Google Search.
This week, Google said that it planned to automatically turn off certain A.I. apps and features — including its Gemini A.I. tool and search — in K-12 schools in early 2027. Schools that elect to turn search back on will have a choice: to enable student access to the A.I. features that the company has already embedded in search; or to a new A.I. feature intended to assist students with schoolwork. Google is not offering schools an option to turn off A.I. in search, a company executive said this week, because “search and A.I. are deeply integrated.”
The new lawsuit against Google is part of a wave of litigation this year over Big Tech’s effect on young users. Meta, which owns Instagram and Facebook, recently agreed to pay up to $17.1 billion to settle charges from 47 states that it used addictive features to ensnare children, following losses in similar cases. Meta has denied wrongdoing. Several parents have also sued A.I. companies over allegations their children died by suicide following interactions with chatbots.
The lawsuit filed on Monday on behalf of Emma and her parents makes arguments similar to some recently used against Meta, claiming that Emma was harmed by Google’s products.
Emma, who had limited access to laptops and apps at home, did not initially try to use Google’s A.I. on her school-issued device, according to the complaint. As Emma searched the web, the lawsuit claimed, Google “routed” her to its A.I. Mode, a chatbot-like feature on Google Search.
By March Emma was “exchanging thousands of messages” with Google’s A.I. Mode until late into the night and “resumed as soon as she woke up,” according to the complaint. The A.I. “diagnosed Emma with major mental illnesses, persuaded her to trust it over her parents and counselors, and ultimately helped her plan her suicide and funeral,” the complaint said.
Google’s product system did not alert the school or Emma’s parents that her searches and A.I. chats suggested she was at risk of harming herself, according to the complaint. Her school used a digital monitoring company, which automatically scans students’ writing and searches on school devices for signs of self-harm, cyberbullying and other problems.
In an interview last month, Emma’s parents said that they learned she was in crisis only after the monitoring company flagged a suicide plan she was writing in Google Docs and warned her school.
The post Parents Sue Google Over A.I. Safety Risks on School Chromebooks appeared first on New York Times.
