California Issues New Generative AI Procurement Guidelines

Yesterday, California issued new guidelines for the state’s procurement, use, and deployment of generative AI technologies. These guidelines, which build on California Governor Newsom’s Executive Order on Generative AI and a recent California government report on generative AI risks and benefits, are some of the first state efforts to formalize specific rules for government’s procurement and use of AI.

The new guidelines impose state government-wide responsibilities to monitor generative AI tools incidentally purchased, including continuous evaluation and additional staff trainings, but also add several specific procurement requirements for agencies seeking out generative AI technologies. These requirements include (1) a pre-procurement assessment of what agency problems a generative AI tool is meant to address, (2) an AI risk assessment, (3) pre-deployment testing and evaluation, and (4) ongoing monitoring that may include additional human review of the AI model. California’s guidelines also require generative AI procurement and risk assessment training for state employees, covering everything from general AI education to risk identification and technical skill-building.

While California’s guidelines are limited to generative AI technologies and their risks, they echo many of EPIC’s own recommendations for government procurement and use of AI generally, including pre-deployment testing, ongoing monitoring, and robust training. EPIC has also advocated for strong protections from the risks of generative AI, publishing Generating Harms: Generative AI’s Impact & Paths Forward and regularly filing formal comments to agencies tasked with regulating generative AI.