ProPublica publishes the COMPAS investigation

On May 23, 2016, ProPublica published an investigation of COMPAS, Northpointe’s criminal-risk assessment software. Comparing scores assigned in Broward County, Florida, with subsequent arrest records, the reporters found that Black defendants without a recorded new arrest were more often classified as higher risk than their white counterparts. Among those with a recorded new arrest, white defendants were more often classified as low risk. Northpointe disputed the interpretation of these differences as evidence of bias.^[1] This date marks the investigation’s publication, rather than a software launch or judicial finding.

What the reporters measured

Broward used COMPAS to inform pretrial release decisions. ProPublica obtained scores for 18,610 people assessed in 2013–2014, then restricted attention to 11,757 assessed before trial. The reporters matched scores with criminal cases using names and dates of birth and collected court records through April 1, 2016. Their operational outcome was generally a new arrest within two years, involving a qualifying charge and jail booking; they excluded failures to appear and charges for crimes preceding assessment. A recorded arrest is neither a conviction nor a direct measure of all offending.^[2]

The general-recidivism error comparison used 7,214 people, a different sample from the 6,172 used in the general-score regression. It grouped scores 5–10, originally medium or high, as higher risk, against low scores 1–4. The published counts give these comparisons, with percentages rounded directly from the counts:^[2]

Reported general-score measure	Black defendants	White defendants
Higher-risk classification among those without a recorded new arrest (false-positive rate)	805 / 1,795 ≈ 44.8%	349 / 1,488 ≈ 23.5%
Low-risk classification among those with a recorded new arrest (false-negative rate)	532 / 1,901 ≈ 28.0%	461 / 966 ≈ 47.7%
Recorded new arrest among those classified higher risk (positive predictive value)	1,369 / 2,174 ≈ 63.0%	505 / 854 ≈ 59.1%

The racial categories came from sheriff’s-office records; these two groups do not exhaust the full sample. The first row conditions on people without a recorded new arrest, whereas the last conditions on people receiving a higher-risk classification. Confusing those denominators changes the question. These are ProPublica’s reported general-score results, not the separate violent-risk analysis or a replication of the underlying records.^[2]

Why the findings were contested

In a report dated July 8, Northpointe’s William Dieterich, Christina Mendoza and Tim Brennan rejected ProPublica’s bias conclusion. They emphasized similar predictive values among higher-scoring defendants and similar ability to rank defendants by subsequent arrest risk, criticized combining medium and high scores, and argued that different recorded outcome rates affect the error comparison.^[3] ProPublica’s July 29 technical reply defended comparing error rates and said the false-positive disparity persisted when only scores 8–10 counted as high risk.^[4]

Alexandra Chouldechova’s later analysis helps explain how these observations can coexist. Calibration compares recorded outcome frequencies among people receiving the same score; predictive parity compares outcome frequencies among people above a threshold. Neither is the same as equal error rates. Her reanalysis found approximate calibration and predictive parity at some thresholds alongside unequal errors. Her algebra relates predictive values, error rates and group outcome prevalence under explicit conditions; her penalty examples model possible consequences rather than measuring actual sentences.^[5]

Kleinberg and colleagues prove a separate result about probability scores: group calibration and equal average scores within both outcome classes require perfect prediction or equal base rates in their model. Those average-score conditions differ from threshold error-rate balance. The theorem does not declare one metric universally fair or establish what caused COMPAS’s observed disparities.^[6]

Scores and decisions

The reporting gives these comparisons human stakes. It contrasted Brisha Borden’s score of 8 with Vernon Prater’s score of 3 after separate petty-theft arrests, then described their different subsequent records. Borden and her friend Sade Jones spent two nights in jail after their bonds were set at $1,000 each. The judge could not recall whether scores influenced that decision. Broward officials described pretrial use, not use in sentencing; the article’s sentencing examples came from Wisconsin. These accounts should not be collapsed into a measured causal effect on incarceration.^[1]

Fairness Tradeoffs develops the metric distinctions, while Julia Angwin and Alexandra Chouldechova provide paths into the reporting and research. The incidents guide places the dispute alongside other institutional uses of automated systems. The case makes both the statistical comparison and the decision policy inspectable: predicting a recorded outcome, distributing errors and justifying a restriction on someone’s liberty are different questions.