The California State Assembly and Senate have passed the Safe and Secure Innovation for Frontier Artificial Intelligence Models Act (SB 1047), one of the first significant regulations of artificial intelligence in the US.

The bill, which has been a flashpoint for debate in Silicon Valley and beyond, would obligate AI companies operating in California to implement a number of precautions before they train a sophisticated foundation model. Those include making it possible to quickly and fully shut the model down, ensuring the model is protected against “unsafe post-training modifications,” and maintaining a testing procedure to evaluate whether a model or its derivatives is especially at risk of “causing or enabling a critical harm.”

Senator Scott Wiener, the bill’s main author, said SB 1047 is a highly reasonable bill that asks large AI labs to do what they’ve already committed to doing: test their large models for catastrophic safety risk. “We’ve worked hard all year, with open source advocates, Anthropic, and others, to refine and improve the bill. SB 1047 is well calibrated to what we know about forseeable AI risks, and it deserves to be enacted.”

Critics of SB 1047 — including OpenAI and Anthropic, politicians Zoe Lofgren and Nancy Pelosi, and California’s Chamber of Commerce — have argued, among other issues, that it’s overly focused on catastrophic harms and could unduly harm small, open-source AI developers. The bill was amended in response, replacing potential criminal penalties with civil ones, narrowing enforcement powers granted to California’s attorney general, and adjusting requirements to join a “Board of Frontier Models” created by the bill.

The AI safety bill will now head to Governor Gavin Newsom, who will have until the end of September to decide its fate, according to The New York Times.

Anthropic declined to comment beyond pointing to a letter sent by Anthropic CEO Dario Amodei to Governor Newsom last week. In that letter, Amodei said the bill was “substantially improved, to the point where we believe its benefits likely outweigh its costs” after the amendments. OpenAI declined to comment and pointed to a letter by OpenAI chief strategy officer Jason Kwon sent to Senator Wiener, also last week.

Update, August 28th: OpenAI declined to comment.

Update, August 29th: Added text from Anthropic’s letter on the amended version of SB 1047. Noted that the bill passed California’s Senate following the Assembly vote.