Government Use of AI

Government agency AI adoption is now widespread, covering a broad range of functions from surveillance monitoring to screening, scoring, transcription, and more. For example, the OMB’s 2023 AI inventory reported roughly 700 AI use cases, while the 2024 inventory reported 2,133 use cases. Even with ongoing attempts to regulate government use and procurement of AI, though, many of the AI tools procured by federal, state, and local agencies have proven to be deeply flawed. Despite the known problems, these AI systems are used in law enforcement and the broader criminal legal cycle, in public benefit administration, in housing processes, and more, often without the public’s knowledge.

AI and the Criminal Legal Cycle

In the criminal justice system, the deployment of AI and other algorithmic decision-making tools tends to exacerbate discriminatory policing patterns that already disadvantage minorities. AI systems trained on historic policing data recognize patterns of which races have been targeted most by police, charged with the most serious crimes, and sentenced to the longest terms and will replicate and perpetuate those pattern unless AI developers explicitly address the context and weights of this data. Racial and gender bias trends occur time and time again across AI and automated decision-making tools used in pretrial dispositions, sentencing, and prison settings. These tools often yield inaccurate or biased results that perpetuate existing inequalities.

Many of the AI tools used in the criminal legal system also exacerbate passive surveillance and racialized policing trends. In 2023, for example, EPIC urged the Department of Justice to investigate the discriminatory effects of automated, acoustic gunshot detection systems. Extensive research—including one 2021 study by the Inspector General of Chicago—has shown that these systems, which cities have procured and placed disproportionately in majority-minority neighborhoods, produce tens of thousands of false positives that increase police activity and lead to false arrests.

AI and Public Benefits

EPIC has long used state Freedom of Information Act (FOIA) requests to obtain detailed records about state and local use of AI, including AI systems developed by private companies. The results of our research are concerning: across the country, state and local governments are experimenting with AI tools that outsource important government decisions to private companies, all without public input or oversight. In D.C., for example, 20 different agencies use AI and automated decision-making for decisions like who should receive public benefits or access public housing.

While some form of automation is nothing new to public benefits programs, the scope and sophistication of AI tools used in public benefits programs has steadily increased, in part due to increased demand for benefits during the COVID-19 pandemic. These tools are designed to automate, assist, or replace human decision-making for a variety of tasks, including eligibility determinations, fraud detection, and identity verification. And when these tools produce errors, they can dramatically impact peoples’ lives: automated public benefits decisions have incorrectly rejected eligible applicants, spurred on improper fraud allegations and overpayment recollection proceedings, and cost state governments millions. Individuals who have had public benefits claims wrongly rejected face a challenging battle to correct AI errors and obtain a human appeal of the decision and lack critical benefits while trying to right this wrong.

Unfortunately, many of the AI systems on which state and local government rely do produce errors and biases. The accuracy, reliability, and effectiveness of an AI system depends entirely on the data used to train and operate the system, the analytic technique used to produce system outputs, and the system’s programmed risk tolerance. Without proper safeguards and oversight, AI systems can produce outputs that are flawed, biased, or overly simplistic. For example, EPIC filed an FTC complaint against a Thomson Reuters-backed fraud detection system which purported to help identify and reject fraudulent public benefits claims. However, an audit of the system found that it incorrectly flagged 600,000 eligible claimants as fraudulent. The system was only 46% accurate.

Agencies have begun to publish guidance for AI in public benefits programs, but without more transparency and safeguards in place, these AI tools will continue to produce errors and biases that disproportionately harm marginalized communities.

AI and Housing

In housing, there are several automated decision-making systems used: facial recognition and other biometric collection and analysis; algorithms deciding if someone is financially able to pay a mortgage or rent; profiles collected from opaque sources and combined into “trustworthiness” reports, and other instances of scoring and screening.

For lending, a rule that was struck down in 2020 created a defense to a discrimination claim under the Fair Housing Act where the “predictive analysis” tools used for lending decisions were not “overly restrictive on a protected class” or where they “accurately assessed risk.” The Judge explained that this regulation would “run the risk of effectively neutering disparate impact liability under the Fair Housing Act” in granting a preliminary injunction. EPIC and several others have warned the federal housing agency that providing such a safe harbor for the use of algorithms in housing without imposing transparency, accountability, or data protection regulations would exacerbate harms to individuals subject to discrimination. The Alliance for Housing Justice called the rule “a vague, ambiguous exemption for predictive models that appears to confuse the concepts of disparate impact and intentional discrimination.”

For biometric identification, The No Biometric Barriers Housing Act has been repeatedly introduced in Congress by Senator Booker (D-NJ), and Congresswomen Yvette D. Clarke (D-NY), Ayanna Pressley (D-MA), and Rashida Tlaib (D-MI). If passed, this act would prohibit the usage of facial and biometric recognition in most federally funded public housing and direct the Department of Housing and Urban Development (HUD) to submit a report to Congress about the impact of the technology on its tenants.

AI screening tools used by landlords and housing authorities to help make decisions about whether to accept a tenant’s rent application are frequently biased or riddled with errors. Companies who offer tenant screening tools collect, store, and select records for housing providers to use in evaluating tenants. These tenant screening reports often contain errors and misleading information, and there is little oversight of the companies’ record collection or matching practices. These errors may be due to lack of information in the records, record matching errors, and failure to update records databases. Regardless of reason, the consequences of these AI system errors are devastating for those impacted.

In 2024, EPIC filed a lawsuit on behalf of the National Association of Consumer Advocates (NACA) challenging one automated tenant screening company, RentGrow, over unfair trade practices tied to inaccurate records and insufficient oversight over its automated systems. This suit is still ongoing.

AI Procurement

With government agencies using a larger and more sophisticated array of AI tools, AI procurement has become a popular path toward more oversight into government AI use. In 2024, for example, the federal Office of Management and Budget released Memo M-24-18, outlining federal government-wide policies for procuring AI systems, and California released its own state procurement guidelines for generative AI systems.

Procurement is a particularly effective avenue for regulating government AI systems because companies bidding for government contracts must (1) disclose information they may be unwilling to disclose publicly and (2) agree to contractual terms that can include strong oversight requirements. For example, after Michigan scrapped a faulty unemployment insurance fraud detection system, it required the replacement vendor to agree to a “source code escrow” provision, wherein an independent auditor could monitor whether the new system was accurate.

For more information on the challenges surrounding government use of procured AI systems and the potential for AI procurement reform, see EPIC’s 2023 report, Outsourced & Automated.

AI and Regulatory Enforcement Assistance

Many U.S. agencies such as the Securities Exchange Commission, Internal Revenue Service, and the Department of the Treasury use algorithms to help direct enforcement resources for potential fraud cases. These programs are detailed further in Government by Algorithm, a 2020 report from ACUS.