Algorithms in the Criminal Justice System
Evidence-based risk assessments predict future behavior by analyzing statistical data. In the criminal justice system, risk-assessment algorithms use data about defendants including their criminal history (e.g. previous offenses, failure to appear in court, violent offenses, etc.) or socio-demographic characteristics (e.g. age, sex, employment status, drug history) to then predict the person’s risk of recidivism or risk of failing to appear when on bail. Such predictions are based on average recidivism rates for the group of offenders that share the defendant’s characteristics. The algorithms are becoming increasingly prevalent and the Justice Department’s National Institute of Corrections encourages the use of the assessments at every stage of the criminal justice process. But many have questioned the underlying data, the reliability of the outcomes as well as defendants’ lack of opportunity to challenge the results.
Risk assessment tools are algorithms that use variables including socioeconomic status, family background, level of neighborhood crime, and employment status to predict an individual’s recidivism risk, either on a scale from “low” to “high” or with specific percentages. See Wisconsin’s COMPAS risk assessment questionnaire, from ProPublica. In 2014, then-U.S. Attorney General Eric Holder called for the U.S. Sentencing Commission to study the use of algorithms in courts, concerned that the scores may be a source of bias. At the same time, the Justice Department expressed concern about the use of factors such as education levels, employment history, family circumstances, and demographic information. While the Sentencing Commission has studied the recidivism risk for federal offenders, it has not commissioned a study of risk scores.
Predictive algorithms are used across the country, but the specific tools differ by state or even county. In addition, because such algorithms are proprietary, they are not subject to state or federal open government laws. Jurisdictions have generally used one of three main systems, or adapted their own version of each: Correctional Offender Management Profiling for Alternative Sanctions (COMPAS), Public Safety Assessment (PSA) and Level of Service Inventory Revised (LSI-R). COMPAS, created by the for-profit company Northpointe, assesses variables under five main areas: criminal involvement, relationships/lifestyles, personality/attitudes, family, and social exclusion. The LSI-R, developed by Canadian company Multi-Health Systems, also pulls information from a wide set of factors, ranging from criminal history to personality patterns. Using a narrower set of parameters, the Public Safety Assessment, developed by the Laura and John Arnold Foundation, only considers variables that relate to a defendant’s age and criminal history.
A recent investigation by ProPublica tested the COMPAS system used by the state of Florida, using the same benchmark as COMPAS, a likelihood of re-offending in two years. ProPublica found that the formula was particularly likely to flag black defendants as future criminals, labeling them as such at almost twice the rate as white defendants. In addition, white defendants were labeled as low risk more often than black defendants. But the investigators also found that the scores were unreliable in forecasting violent crime: only 20 percent of the people predicted to commit violent crimes actually went on to do so. When considering a full range of crimes, including misdemeanors, the correlation was found to be higher but not exceedingly accurate. Sixty-one percent of the candidates deemed liked to reoffend were arrested for any subsequent crimes within two years. According to ProPublica, some miscalculations of risk stemmed from inaccurate inputs (for example, failing to include one’s prison record from another state), while other results were attributed to the way factors are weighed (for example, someone who has molested a child may be categorized as low risk because he has a job, while someone who was convicted of public intoxication would be considered high risk because he is homeless).
Prediction Fails Differently for Black Defendants
|Labeled Higher Risk, But Didn't Re-Offend||23.5%||44.9%|
|Labeled Lower Risk, Yet Didn't Re-Offend||47.7%||28.0%|