Grading gaps in summer 2020: who was affected by differences between centre assessment grades and calculated grades?

Philip Noden, Eleanor Rutherford, Nadir Zanini, Tim Stratton and James Bowsher-Murray investigate who experienced large differences between teacher assessments and calculated A level grades in England in 2020. Their retrospective study asks how these gaps relate to candidates’ characteristics, schools and subjects. It compares two grading approaches after the reversal, rather than observing the exams students would have taken.^[1]

What is a grading gap?

Teachers supplied a centre assessment grade (CAG) and a rank order for each subject within a centre. Calculated grades retained those rankings while adjusting grade distributions using historical centre results and prior attainment. After the reversal, candidates received whichever was higher for each qualification. A positive grading gap in this report means the CAG exceeded the calculated grade.^[1]

The candidate-level outcome aggregates across subjects. A three-grade gap could mean three subjects each calculated one grade below the CAG, or another combination totaling at least three. It is not necessarily a three-grade difference in any one qualification. Nor does the size of that sum measure personal consequence: one grade could determine admission to a course, while a larger sum need not identify who experienced the greatest distress.^[1]

Which candidates enter the models?

The underlying data link Ofqual results to socioeconomic and demographic variables from the National Pupil Database, focusing on candidates in England aged 18. Linking fails for 16.8% of candidates and disproportionately for independent schools and sixth-form colleges. Missing characteristics are retained as explicit categories, rather than simply dropping those candidates (Data section).^[1]

The main candidate analysis uses 82,830 candidates taking exactly three A levels, with every subject taken in a subject-within-centre cohort larger than fifteen. The smaller cohorts used CAG-only or tapered allocation rules and are excluded from these models. Restricting the number of subjects also makes the aggregate three-grade threshold more comparable (Table 3; Analytical approach).^[1]

The authors first fit a logistic model treating candidates as independent, then account for clustering within centres and add the subjects studied. They explicitly warn that the initial model understates uncertainty. A second approach examines a gap of at least one grade per entry, cross-classifying entries within candidates and subject-within-centre groups. Its larger model covers 450,810 entries in cohorts above fifteen. Candidate-level and entry-level findings therefore have different units and inclusion rules.^[1]

Small conditional differences, larger centre variation

Once centre clustering and subject choices are included, the report finds no evidence of an association between the three-grade outcome and socioeconomic background, SEND status or home language. Female candidates and candidates in the Asian ethnic category retain small differences. Model 3 predicts a 15.7% probability for a white female candidate at the median centre with other variables at their reference values, versus 14.5% for the corresponding male candidate. These are conditional predictions for specified profiles, not each group’s observed national rate.^[1]

An apparent difference for Black candidates in the unclustered model is no longer statistically significant after accounting for centres. The authors interpret this as suggesting that which centre a candidate attended matters for the observed pattern. Demographic differences are much smaller than differences between centres, but the study does not explain why centres had different incidences of grading gaps. Controlling for centres consequently answers a narrower question than whether students experienced unequal outcomes across the whole system.^[1]

The entry-level models produce broadly similar conclusions. Changing the candidate outcome to a four-grade gap also changes which demographic associations are statistically significant. The report’s findings depend on a defined discrepancy, selected population and model structure, rather than establishing that every possible measure of grading disadvantage is equal.^[1]

What the comparison leaves unresolved

Neither grade is an observed examination result. The analysis cannot determine which approach more accurately represented students’ potential performance, assign causes to the small conditional differences, or assess all admissions and personal consequences. Fairness Tradeoffs provides a useful question about what outcome is being compared, without importing its theorem assumptions into this different study.^[1]

Published July 29, 2021, the report is complemented by the teaching-staff interviews. Those accounts address how teachers formed judgments; this statistical comparison addresses how two outputs differed. Understanding the reversal requires both the process that produced the grades and the consequences of selecting one output as the result.