California lawmakers want Gov. Gavin Newsom to approve bills they passed that aim to make artificial intelligence chatbots safer. But as the governor weighs whether to sign the legislation into law, he faces a familiar hurdle: objections from tech companies that say new restrictions would hinder innovation.
Californian companies are world leaders in AI and have spent hundreds of billions of dollars to stay ahead in the race to create the most powerful chatbots. The rapid pace has alarmed parents and lawmakers worried that chatbots are harming the mental health of children by exposing them to self-harm content and other risks.
Parents who allege chatbots encouraged their teens to harm themselves before they died by suicide have sued tech companies such as OpenAI, Character Technologies and Google. They’ve also pushed for more guardrails.
Calls for more AI regulation have reverberated throughout the nation’s capital and various states. Even as the Trump administration’s “AI Action Plan” proposes to cut red tape to encourage AI development, lawmakers and regulators from both parties are tackling child safety concerns surrounding chatbots that answer questions or act as digital companions.
California lawmakers this month passed two AI chatbot safety bills that the tech industry lobbied against. Newsom has until mid-October to approve or reject them.
The high-stakes decision puts the governor in a tricky spot. Politicians and tech companies alike want to assure the public they’re protecting young people. At the same time, tech companies are trying to expand the use of chatbots in classrooms and have opposed new restrictions they say go too far.
Meanwhile, if Newsom runs for president in 2028, he might need more financial support from wealthy tech entrepreneurs. On Sept. 22, Newsom promoted the state’s partnerships with tech companies on AI efforts and touted how the tech industry has fueled California’s economy, calling the state the “epicenter of American innovation.”
He has vetoed AI safety legislation in the past, including a bill last year that divided Silicon Valley’s tech industry because the governor thought it gave the public a “false sense of security.” But he also signaled that he’s trying to strike a balance between addressing safety concerns and ensuring California tech companies continue to dominate in AI.
“We have a sense of responsibility and accountability to lead, so we support risk-taking, but not recklessness,” Newsom said at a discussion with former President Clinton at a Clinton Global Initiative event on Wednesday.
Two bills sent to the governor — Assembly Bill 1064 and Senate Bill 243 — aim to make AI chatbots safer but face stiff opposition from the tech industry. It’s unclear if the governor will sign both bills. His office declined to comment.
AB 1064 bars a person, business and other entity from making companion chatbots available to a California resident under the age of 18 unless the chatbot isn’t “foreseeably capable” of harmful conduct such as encouraging a child to engage in self-harm, violence or disordered eating.
SB 243 requires operators of companion chatbots to notify certain users that the virtual assistants aren’t human.
Under the bill, chatbot operators would have to have procedures to prevent the production of suicide or self-harm content and put in guardrails, such as referring users to a suicide hotline or crisis text line.
They would be required to notify minor users at least every three hours to take a break, and that the chatbot is not human. Operators would also be required to implement “reasonable measures” to prevent companion chatbots from generating sexually explicit content.
Tech lobbying group TechNet, whose members include OpenAI, Meta, Google and others, said in a statement that it “agrees with the intent of the bills” but remains opposed to them.
AB 1064 “imposes vague and unworkable restrictions that create sweeping legal risks, while cutting students off from valuable AI learning tools,” said Robert Boykin, TechNet’s executive director for California and the Southwest, in a statement. “SB 243 establishes clearer rules without blocking access, but we continue to have concerns with its approach.”
A spokesperson for Meta said the company has “concerns about the unintended consequences that measures like AB 1064 would have.” The tech company launched a new Super PAC to combat state AI regulation that the company thinks is too burdensome, and is pushing for more parental control over how kids use AI, Axios reported on Tuesday.
Opponents led by the Computer & Communications Industry Assn. lobbied aggressively against AB 1064, stating it would threaten innovation and disadvantage California companies that would face more lawsuits and have to decide if they wanted to continue operating in the state.
Advocacy groups, including Common Sense Media, a nonprofit that sponsored AB 1064 and recommends that minors shouldn’t use AI companions, are urging Newsom to sign the bill into law. California Atty. Gen. Rob Bonta also supports the bill.
The Electronic Frontier Foundation said SB 243 is too broad and would run into free-speech issues.
Several groups, including Common Sense Media and Tech Oversight California, removed their support for SB 243 after changes were made to the bill, which they said weakened protections. Some of the changes limited who receives certain notifications and included exemptions for certain chatbots in video games and virtual assistants used in smart speakers.
Lawmakers who introduced chatbot safety legislation want the governor to sign both bills, arguing that they can both “work in harmony.”
Sen. Steve Padilla (D-Chula Vista), who introduced SB 243, said that even with the changes he still thinks the new rules will make AI safer.
“We’ve got a technology that has great potential for good, is incredibly powerful, but is evolving incredibly rapidly, and we can’t miss a window to provide commonsense guardrails here to protect folks,” he said. “I’m happy with where the bill is at.”
Assemblymember Rebecca Bauer-Kahan (D-Orinda), who co-wrote AB 1064, said her bill balances the benefits of AI while safeguarding against the dangers.
“We want to make sure that when kids are engaging with any chatbot that it is not creating an unhealthy emotional attachment, guiding them towards suicide, disordered eating, any of the things that we know are harmful for children,” she said.
During the legislative session, lawmakers heard from grieving parents who lost their children. AB 1064 highlights two high-profile lawsuits: one against San Francisco ChatGPT maker OpenAI and another against Character Technologies, the developer of chatbot platform Character.AI.
Character.AI is a platform where people can create and interact with digital characters that mimic real and fictional people. Last year, Florida mom Megan Garcia alleged in a federal lawsuit that Character.AI’s chatbots harmed the mental health of her son Sewell Setzer III and accused the company of failing to notify her or offer help when he expressed suicidal thoughts to virtual characters.
More families sued the company this year. A Character.AI spokesperson said they care very deeply about user safety and “encourage lawmakers to appropriately craft laws that promote user safety while also allowing sufficient space for innovation and free expression.”
In August, the California parents of Adam Raine sued OpenAI, alleging that ChatGPT provided the teen information about suicide methods, including the one the teen used to kill himself.
OpenAI said it’s strengthening safeguards and plans to release parental controls. Its chief executive, Sam Altman, wrote in a September blog post that the company believes minors need “significant protections” and the company prioritizes “safety ahead of privacy and freedom for teens.” The company declined to comment on the California AI chatbot bills.
To California lawmakers, the clock is ticking.
“We’re doing our best,” Bauer-Kahan said. “The fact that we’ve already seen kids lose their lives to AI tells me we’re not moving fast enough.”
The post AI chatbot safety bills under threat as Newsom ponders restrictions tech groups say would hurt California appeared first on Los Angeles Times.
