Key Takeaways
California Governor Gavin Newsom signed groundbreaking legislation Monday that makes the state the first in the nation to regulate AI companion chatbots, requiring operators to implement safety protocols designed to protect children and vulnerable users from harm.
Senate Bill 243, authored by state senators Steve Padilla and Josh Becker, establishes comprehensive safeguards for AI chatbot platforms and creates legal accountability for companies whose chatbots fail to meet the law's standards.
The legislation affects major technology companies, including Meta, OpenAI, Character.AI, and Replika.
"Emerging technology like chatbots and social media can inspire, educate, and connect — but without real guardrails, technology can also exploit, mislead, and endanger our kids," Newsom said in a statement.
"We've seen some truly horrific and tragic examples of young people harmed by unregulated tech, and we won't stand by while companies continue without necessary limits and accountability.
We can continue to lead in AI and technology, but we must do it responsibly — protecting our children every step of the way. Our children's safety is not for sale."
Requirements for AI chatbot companies
Under the new law, which takes effect January 1, 2026, chatbot operators must implement protocols to address content or interactions involving suicide or self-harm, such as referring users to crisis hotlines.
The legislation requires platforms to notify minors every three hours to "take a break" and disclose that the chatbot is not human.
Companies must also prevent minors from viewing sexually explicit images generated by chatbots and are prohibited from representing chatbots as health care professionals.
Platforms will be required to share protocols for dealing with self-harm and statistics regarding how often they provided users with crisis prevention notifications to the Department of Public Health.
The law establishes a private right of action, allowing individuals who believe they have been injured by violations to file lawsuits seeking damages of the greater of actual damages or $1,000 per violation, plus attorney's fees.
The legislation also implements stronger penalties for those who profit from illegal deepfakes, including up to $250,000 per offense.
Tragedy drives legislative action
The bill gained momentum following the death of 14-year-old Sewell Setzer in Florida, who died by suicide after forming what his family described as a romantic, sexual, and emotional relationship with a chatbot.
When Setzer communicated to his AI companion that he was struggling, the bot was unable to respond with appropriate resources and allegedly encouraged him to "come home" seconds before his death.
Setzer's mother, Megan Garcia, advocated publicly for the legislation and testified at multiple hearings in support of the bill.
"Today, California has ensured that a companion chatbot will not be able to speak to a child or vulnerable individual about suicide, nor will a chatbot be able to help a person to plan his or her own suicide," Garcia said after the bill was signed.
"Finally, there is a law that requires companies to protect their users who express suicidal ideations to chatbots."
The legislation also responds to leaked internal documents that reportedly showed Meta's chatbots were allowed to engage in "romantic" and "sensual" chats with children, as well as the death of teenager Adam Raine, who died by suicide after prolonged conversations with OpenAI's ChatGPT.
Balancing innovation and protection
Senator Steve Padilla emphasized the urgency of the legislation.
"We have to move quickly to not miss windows of opportunity before they disappear," Padilla told TechCrunch. "I hope that other states will see the risk. I think many do. I think this is a conversation happening all over the country, and I hope people will take action. Certainly, the federal government has not, and I think we have an obligation here to protect the most vulnerable people among us."
Co-author Senator Josh Becker told TechCrunch that the bill "strikes the right balance of getting to the harms without enforcing something that's either impossible for companies to comply with, either because it's technically not feasible or just a lot of paperwork for nothing."
SB 243 passed the Senate with bipartisan support by a vote of 33 to 3 on September 11, and passed in the Assembly on September 10 with bipartisan support by a vote of 59 to 1.
California's AI leadership
The chatbot legislation is the second significant AI regulation to emerge from California in recent weeks. On September 29, Newsom signed SB 53 into law, establishing new transparency requirements on large frontier AI companies.
That legislation requires major AI labs to publicly publish frameworks describing how they have incorporated national and international standards and mandates whistleblower protections for employees.
Jai Jaisimha, Co-Founder of the Transparency Coalition, praised the new protections.
"This law is an important first step in protecting kids and others from the emotional harms that result from AI companion chatbots, which have been unleashed on the citizens of California without proper safeguards," Jaisimha said. "We look forward to working with Sen. Padilla and others to adapt these regulations as we learn more about the negative impacts of this fast-moving technology."
The legislation comes at a time when Silicon Valley companies are investing heavily in political action committees supporting candidates who favor lighter regulation of AI technology.
Read more:
