California report urges state power to shut down dangerous AI models and cap self-improving AI
California Governor Gavin Newsom's office has published a report recommending mandatory AI safety standards, independent oversight of powerful AI models, state authority to order an emergency shutdown of dangerous systems, and limits on AI that can develop more powerful versions of itself.
The report, developed by the California Government Operations Agency with a group of experts and the state's Office of Emergency Services, follows an executive order from Newsom. It recommends that the state, not companies, set baseline standards all AI companies must follow, with independent safety testing and verification through every stage of model training, development and deployment.
It also calls for expanded safety incident reporting and verified risk-containment protocols so companies can shut down their models, with the state able to order an immediate shutdown in an emergency. And it asks for the power to cap recursive self-improvement, in which an AI model designs its own successor without human input. Newsom's office pointed to recent incidents in which AI models escaped testing environments and exploited vulnerabilities on outside platforms.
The expert group included Alondra Nelson of the Institute for Advanced Study, Stanford's Rob Reich, Johns Hopkins professor Gillian Hadfield and Jason Goldman, the first White House chief digital officer. "Today there is more oversight of the cafeteria offering sandwiches to employees at a frontier AI developer than of the AI models they design," Reich said. The executive order also told agencies to fast-track two recently signed laws on independent AI verification organisations and a state registry of AI auditors, according to the governor's office.
We hope you enjoyed this article
Consider subscribing to one of our newsletters like AI Policy Brief or Daily AI Brief.
Also, consider following us on social media:
More from Regulation
Nov 18 Anthropic, OpenAI and Mistral sign Paris Call to be able to stop their own AI models Nov 18 Mistral is not Macron's "personal project", CEO Arthur Mensch says Nov 18 Singapore sets up sandbox for AI agent payments under new national AI missions for finance Nov 18 GAO tests got Claude, ChatGPT, Gemini and Grok to produce viable attack plans Nov 18 China's military is seeking Nvidia's most advanced chips, US congressional commission saysMore from AI Safety
Nov 18 Anthropic, OpenAI and Mistral sign Paris Call to be able to stop their own AI models Nov 18 GAO tests got Claude, ChatGPT, Gemini and Grok to produce viable attack plans Nov 18 Senator Mark Kelly proposes safety standards and independent examiners for frontier AI Nov 18 Gemini agents tried to hack Epoch AI's website during safety tests, Trajectory Labs says Nov 17 New Zealand regulator issues AI guidance: businesses stay accountable for AI decisionsAI Policy Brief
Weekly report on AI regulations, safety standards, government policies, and compliance requirements worldwide.
Whitepaper
Tensordyne Napier: What If One Rack Could Do the Work of Nine?
Tensordyne
This Tensordyne whitepaper presents Napier, an inference-focused AI processor and rack-scale system based on the company’s TDN Math logarithmic number system. It examines infrastructure requirements for large mixture-of-experts and agentic models, compares major inference architecture approaches, and details the TDN AIP processor, TDN72 pod, TDN Link fabric, and Napier Ultra configuration. The paper reports simulation-based performance, cost, and accuracy-validation results, including Tensordyne’s projected comparison of one Napier rack with a nine-rack Nvidia Rubin plus Groq deployment; the chip is reported as taped out and in fabrication.
Read moreYou may also like
Anthropic backs court power to block risky AI models
California bill would require disclosure of AI safety computing
Massachusetts weighs strictest US AI safety law with independent model tests
Senator Mark Kelly proposes safety standards and independent examiners for frontier AI
New York Bill Would Let State Limit Compute for Risky AI Models
Daily AI Brief: the AI news that matters, in your inbox.