Face Recognition Vendor Test (FRVT), Part 3: Demographic Effects

Patrick Grother, Mei Ngan and Kayee Hanaoka measure how face-recognition errors vary by demographic group, algorithm and image data. NISTIR 8280 extends the Face Recognition Vendor Test to identity verification, which checks a claimed identity, and identification, which searches for an identity in a gallery. Both differ from the gender-classification task studied in Gender Shades.^[1]

What the test compares

The evaluation processes 18.27 million photographs of 8.49 million people through 189 algorithms from 99 developers. Submitted prototypes are not necessarily mature products. Four government datasets supply domestic mugshots, immigration-benefit application photos, visa photos and border-crossing images. The first three generally follow image-capture standards; the last has less controlled capture conditions (executive summary).^[1]

Sex and age metadata accompany the images. Mugshots also supply race metadata; the other sets use country of birth. For the latter comparison, the report selects 24 countries in seven regions with less long-distance immigration. Country of birth is consequently a study variable with a particular selection rule, rather than a universally interchangeable measurement of race.^[1]

Errors depend on the task and threshold

One-to-one verification asks whether two images depict the same person. A false positive associates different people; a false negative fails to associate the same person. The algorithm produces a similarity score, and the operator chooses a threshold. One-to-many identification searches an enrolled gallery and returns candidates or scores above a threshold. Its false-positive identification measure concerns a search producing an incorrect candidate, rather than just an isolated pair comparison (printed pp.4–6).^[1]

The report emphasizes comparing groups at a common operating threshold. Selecting a different threshold for each group to force the same false-positive rate can hide the disparity that an actual fixed-threshold system would produce. Both false-positive and false-negative rates matter: changing the threshold trades off their consequences, which also differ between an access check and an investigation.^[1]

Findings are algorithm- and dataset-specific

Across many, but not all, algorithms, false-positive rates vary between demographic groups by factors of ten to more than one hundred. False-negative differences are generally smaller and more algorithm-specific. These are relative differences under the tested conditions, not a statement that a fixed percentage of every population is misidentified.^[1]

In the higher-quality application images, many algorithms have higher false-positive rates for people from West or East Africa and East Asia than for those from Eastern Europe. Some algorithms developed in China reverse the East Asian pattern. In domestic mugshots, the ordering differs again. The report also finds higher false positives for women and the youngest and oldest groups. False-negative patterns change with the image source: the border-crossing results do not simply reproduce the standardized mugshot results (executive summary).^[1]

Some highly accurate one-to-many algorithms show no detectable false-positive demographic differential in these tests. The variation prevents treating a single result as a property of every recognition system. NIST recommends that operators understand their particular algorithm and conditions; the report does not experimentally isolate training-data composition as the explanation for every difference.^[1]

What does an investigative candidate establish?

An investigation can request a fixed number of candidates even when the searched person is absent from the gallery. The report’s zero-threshold example therefore always returns candidates for an unenrolled person. That is a consequence of the search policy, not a finding that every returned candidate was accepted as a match. Human adjudication, its interaction with machine ranking and reviewer efficacy are outside the report’s scope (printed p.5).^[1]

The Williams incident raises the subsequent operational question of what evidence follows a lead. This algorithm benchmark cannot diagnose that arrest. The distinction is also why findings from Joy Buolamwini and Timnit Gebru on classification need their own task labels: a comparison becomes useful only after identifying what output is produced, how errors are counted and who acts on it.

Historical context

NISTIR 8280 is dated December 2019. Its enduring contribution is a task-specific measurement of demographic performance, paired with a warning that algorithm choice, image quality and operating rules shape both the measured disparity and its practical consequences.^[1]