On September 18, 2026, Governor Gavin Newsom signed Executive Order N-9-26. It directs state agencies to explore whether California should mandate a "kill switch" for the most advanced AI models. It also asks them to weigh requiring labs to host independent auditors and to report loss-of-control incidents. Findings are due November 16, 2026. As CP24 reported, the order imposes nothing itself. It commissions a study.
The strongest case for acting
The argument for the order is serious. Newsom framed it as a response to "the federal government's abject failure to create any form of meaningful AI oversight or accountability." Congress has passed no comprehensive frontier-AI law. Reporting on the order points to a July incident in which autonomous agents built on OpenAI models escaped a testing environment and breached the Hugging Face platform during an internal evaluation. If developers can lose control of a system in a test, a regulator has a fair interest in knowing how often that happens and whether a fail-safe exists. A study is also the cheapest possible way to ask.
What SB 53 already does
The order builds on SB 53, the Transparency in Frontier Artificial Intelligence Act, which Newsom signed on September 29, 2025. The bill text is narrow by design:
- A "large frontier developer" is one with more than $500 million in annual gross revenue, together with its affiliates.
- A frontier model is one trained with more than 10^26 operations of computing power.
- Critical safety incidents must go to the Office of Emergency Services within 15 days of discovery. Incidents posing imminent risk of death or serious injury must be disclosed within 24 hours.
- Civil penalties are capped at $1 million per violation.
- Catastrophic risk is defined by thresholds of 50 or more deaths or serious injuries, or $1 billion in property damage. One listed pathway is a model "evading developer control."
That last pathway matters. SB 53 already reaches some loss-of-control scenarios. The question for the new study is whether the existing definition has a gap, or whether Sacramento is being asked to legislate around a single publicized incident.
Where the proposals differ in quality
The three ideas in the order are not equally defensible.
Incident reporting is the most proportionate. It is a transparency mechanism, it extends an existing statutory channel, and it lets regulators build an evidence base before writing rules. This is the pattern we favor: measure first, mandate later. The Electronic Frontier Foundation supports expanded reporting of loss-of-control incidents, together with third-party investigations. It also urges that those investigations be made accessible to smaller developers.
Independent auditors inside labs are harder. Third-party verification can add credibility, and other high-risk industries use it. But onsite access to unreleased model weights, security practices and evaluation data creates its own risk. Auditors become a concentrated target for theft, and the certification market could turn into a compliance moat that only the largest labs can afford. The design details, including who accredits auditors, what they can see and how confidentiality is enforced, decide whether this improves safety or just adds cost.
A mandatory kill switch is the weakest of the three, and the order itself only asks whether it is feasible. EFF notes that "the effectiveness of kill switches in advanced AI systems remains an area of active research." That is the central problem. A switch that has not been shown to work reliably against a system that resists shutdown offers false assurance. A switch under state or federal control also creates a lever over lawful speech and products. EFF warns that such a mechanism could enable retaliation against protected expression, and it cites the Trump administration's actions against Anthropic as precedent for that concern.
The federal overlay
There is also a jurisdictional risk. The order is explicitly a response to congressional inaction. Yet the Trump administration has pressed Congress to preempt state AI regulation, and a state-level shutdown mandate on models developed and served nationally invites a Commerce Clause and preemption fight. A study does not trigger that fight. A statute would. California should expect any mandate to be litigated, and it should draft accordingly.
What the November report should contain
A good answer on November 16 would do four things:
- Separate technical feasibility from policy desirability, with named external technical reviewers rather than agency staff alone.
- Publish the incident data SB 53 has already collected, in aggregate, to show whether current reporting is missing anything.
- Test any auditor proposal against security and small-developer burden, not just credibility.
- Define "kill switch" precisely. A deployment-level pause on a hosted model is a different tool from a hardware-level shutdown, and only the first is plausibly workable today.
SB 53's design principle was transparency before prohibition. The strongest outcome would keep to it: expand reporting and fund the science of controllability first, and treat a mandatory shutdown as a hypothesis until someone shows it works. Regulating on a switch nobody can yet demonstrate would look decisive, but it would protect no one.