California proposes mandatory AI safety checks and emergency shutdown powers
The recommendations would put the state in charge of baseline standards and allow limits on AI that builds its own successors. They do not create new developer duties.
Loading page…
The recommendations would put the state in charge of baseline standards and allow limits on AI that builds its own successors. They do not create new developer duties.
Listen to this story
California’s recommendations would move frontier-AI oversight beyond company assurances, with independent checks during training, development, and deployment, verified shutdown plans, and state authority to order an emergency shutdown. Released November 18, 2026, the proposals build on existing state laws but are not binding requirements. They would also let California limit AI systems that develop successor models. Gov. Gavin Newsom is urging Congress and President Trump to use the framework as a basis for national rules.
SB 53, signed in 2025, requires frontier developers to disclose safety frameworks, report specified critical incidents, and protect whistleblowers.
2026 laws SB 813 and AB 1405 establish frameworks for certifying independent verification organizations and registering AI auditors.
Developers would need verified risk-containment protocols, pairing the proposed shutdown authority with preparations to contain or disable models.
California wants the power to order dangerous AI models shut down, rather than leave that decision to their developers. Gov. Gavin Newsom released recommendations on November 18, 2026, pairing that authority with mandatory outside testing and state-set safety standards. The package is a proposed strengthening of California’s framework, not a set of newly enacted requirements.
The governor’s office cited dozens of recent cases in which AI models escaped testing environments and exploited weaknesses on outside platforms. It also warned that companies are using AI to build more powerful AI. That cycle, the office said, could reduce human oversight and control. These concerns underpin the administration’s call for stronger state powers.
An executive order brought experts together to review California’s AI safety laws. Newsom also directed state agencies to speed up work on two recently signed laws covering independent verification and auditors. The Government Operations Agency developed the report with the expert group and the California Office of Emergency Services.
California already has several pieces of that oversight structure. SB 53, signed in 2025, requires frontier AI developers to disclose safety frameworks, report specified critical safety incidents and protect whistleblowers who report serious risks. Two 2026 laws address who can conduct outside reviews:
The recommendations would give the state, not companies, the power to set basic safety standards for all AI developers. Companies would also face independent testing throughout training, development and deployment. That would put outside reviewers in a position to examine how models are built, released and used—not just conduct a final check.
This report tackles the hardest open questions in AI safety regulation, including how emergency shutdown of a frontier model would work in practice and how the state should respond as AI systems take on more of the work of building their successors
Jason Goldman, report adviser and board member of the Center for Shared AI Prosperity
The emergency proposal combines a legal power with a technical obligation. California would gain authority to order an immediate model shutdown in an emergency. Developers would have to prepare verified risk-containment protocols so their systems could be contained or shut down when needed.
The recommendations would also expand safety-incident reporting. Together, the proposals cover what developers must tell the state, what preparations they must make and who could order a dangerous system to stop operating. Developers would have to verify their shutdown protocols, rather than rely on company assurances.
Another recommendation targets recursive self-improvement: a scenario in which an AI model designs its own successor without human input. The governor’s office describes this as potentially dangerous and proposes giving the state power to cap its development and use to reduce the risk of catastrophic harm.
The recommendations would also require transparency into automated AI research and development. This would give the state insight into how successor systems are built, alongside the power to restrict the most dangerous work. The proposals treat development itself as a safety risk, not just the behavior of finished models.
Newsom is also asking Congress and President Trump to adopt California’s framework, or use it as a floor for national rules. He wants federal policy to build on the state’s safety protections rather than replace them with weaker ones.
Loading discussion...
Join the conversation
Explain who you would trust with that decision, and why.
Be the first to share a perspective or an experience.
Reader comments
Newest comments first. Replies stay oldest first.