Categorizing Variants of Goodhart's Law

David Manheim and Scott Garrabrant ask why a useful metric can become harmful under stronger optimization. Their conceptual taxonomy uses mathematical examples to distinguish four mechanisms.^[1]

On this page: Mechanisms · Selection · Worked example · Limits · Context

Method and contribution

Regressional failures select noise in an imperfect proxy. Extremal failures push a relationship beyond the region where it holds. Causal failures change the system through intervention. Adversarial failures involve other agents exploiting a metric. These mechanisms can overlap; the distinctions help identify what a proposed mitigation would need to address.^[1]

Selection and intervention

The regressional model adds noise to the goal to obtain the metric. A high metric selects both genuine quality and favorable error; expected goal quality can still rise. Sections 2–3 distinguish selecting states outside a model’s useful range from intervening on the causal system. The latter includes changing a shared cause, an intermediary or the metric itself. Section 4 adds agents responding to incentives and shows how their actions can reproduce the earlier mechanisms.^[1]

A higher score can still select real quality

Here is a discrete teaching example of the selection distinction, not the paper’s Gaussian model or measured data. Let score = quality + error. Quality is equally likely to be 0 or 1; error is independently equally likely to be −1 or +1. Each row therefore has probability one quarter.

Quality	Error	Score
0	−1	−1
0	+1	1
1	−1	0
1	+1	2

Before selection, average quality and average score are both 0.5. Selecting only score 2 leaves quality 1 and error +1. Real quality improves by 0.5, while the score improves by 1.5. The metric has selected both a better outcome and favorable error. Calling that divergence a Goodhart effect does not imply that selection made quality worse.

No actor changed the scoring process in this example. Directly altering a recorded score would introduce an intervention; moving beyond a region where the relationship holds would raise the extremal question; another actor optimizing against the evaluator would raise the adversarial question. Those are editorial contrasts with the taxonomy, not a claim that each observed failure has one unique cause.

Alignment relevance and limits

The examples and equations are explanatory models, not a benchmark of deployed systems. The colonial cobra story is presented as a supposed historical account; this article does not treat it as a verified incident. The historical illustration does not supply empirical evidence for how frequently each mechanism occurs.

Goodhart’s Law connects Outer Alignment to Reward Hacking: maximizing a measured objective can diverge from the intended outcome. The taxonomy offers explanations, not an empirical estimate of every failure or a universal cure. It does not imply that all metrics are useless.

The objective-failure journey in Key Papers: A Reading Guide uses this taxonomy before learned optimization. Carry the distinction forward: a proxy’s disappointing performance identifies a question about the score–goal relationship, while a claim about an internal optimizer needs evidence about the learned mechanism.

See David Manheim and Scott Garrabrant.

Historical context

arXiv records the first submission of Categorizing Variants of Goodhart’s Law on 13 March 2018. The reviewed version is v4, revised 24 February 2019; submission and revision are different events.^[1]