Actionable Auditing: Investigating the Impact of Publicly Naming Biased Performance Results of Commercial AI Products

Inioluwa Deborah Raji and Joy Buolamwini investigate what followed the public naming of companies in Gender Shades. Their follow-up measures commercial gender-classification APIs again and examines disclosure practices and corporate responses. It joins a technical question—did subgroup errors change?—to an institutional question about how external audits can prompt action.^[1]

From private notice to public comparison

The paper reconstructs a coordinated disclosure process. The original auditors sent Microsoft, IBM and Face++ an anonymized paper on December 19, 2017, giving the companies time to respond before public release. Public reporting and the named study followed in February 2018. Updated results and company communications could accompany that release even when a company had already responded. The authors adapt this sequence from coordinated vulnerability disclosure in computer security.^[1]

The audit acts like a developer using a commercial API, rather than inspecting its training data or source code. The authors call this a carrier-puppet audit: the developer can pass an API’s errors into products used by other people. Testing that intermediary helps reveal a problem before considering every downstream application, but does not measure every downstream consequence.^[1]

What the follow-up measures

On August 21, 2018, the researchers re-evaluated the three original targets on the Pilot Parliaments Benchmark (PPB), adding Amazon and Kairos as non-targeted comparisons. These additional companies were selected for their relevance and public positions, not randomly assigned to an untreated control group. PPB supports comparisons by binary gender labels, lighter or darker skin type, and their intersections. This is gender classification from images, rather than matching a face to an identity.^[1]

The subgroup error rate counts incorrect classifications among processed faces; a face that is not detected is omitted. Consequently, these rates do not include every failure to provide a usable result. The benchmark’s purpose is representation across groups, rather than reproducing the demographic proportions of a particular customer population.^[1]

Improvement and remaining disparity

All three targeted companies released new API versions. Table 2 reports overall error falling by 8.30 percentage points for Face++, 5.72 for Microsoft and 7.69 for IBM relative to the May 2017 audit. For the darker-female subgroup, the corresponding reductions were 30.40, 19.28 and 17.73 percentage points. These are changes in error rates, not relative percentage reductions.^[1]

The August 2018 results still differ substantially between companies. Table 1 reports darker-female error of 4.1% for Face++, 1.52% for Microsoft and 16.97% for IBM. Amazon and Kairos have 31.37% and 22.50% error for that subgroup, versus 0% for lighter-male faces in this test. Lower overall error can therefore coexist with a large intersectional gap.^[1]

The authors also analyze corporate blog posts, releases and statements. They interpret the changes as evidence that a targeted, public comparison can make disparities a priority. However, prior performance for the two non-target companies is unknown, and the companies’ changes were not experimentally assigned. The follow-up documents improvement after disclosure without independently isolating naming as its cause.^[1]

Accuracy is only part of accountability

The paper identifies a concrete measurement discrepancy: IBM reported 3.46% error on darker-female faces when retaining predictions above a 99% confidence threshold, while the follow-up’s higher-confidence-label rule produced 16.97%. A confidence cutoff changes which cases enter the reported metric. Auditors need to compare those operating conditions before treating two figures as contradictory measurements of the same thing.^[1]

Training changes and new data composition remain unclear. Because PPB is public, improved performance might also reflect adaptation to that benchmark rather than transfer to unfamiliar images. The communications analysis mainly covers English-language sources, limiting its account of Face++'s response. These are reasons to test additional data and inspect institutional behavior, rather than to equate a better benchmark score with a complete fairness assessment.^[1]

The paper’s 2019 proceedings record supplies the chronology year. Its concluding distinction is between reducing classification disparities and justifying deployment: more accurate systems can still threaten privacy or civil liberties. An actionable audit can expose a remediable performance problem while leaving the legitimacy of the application open.^[1]