The Off-Switch Game

Dylan Hadfield-Menell, Anca Dragan, Pieter Abbeel and Stuart Russell study when a robot benefits from allowing a human to stop its proposed action. The chronology marks the November 24, 2016 first arXiv submission; the reviewed original is the IJCAI 2017 proceedings paper, pp.220–227.^[1]

On this page: Choices · Worked comparison · Findings · Context · Limits

Three choices

The robot can act directly, stop itself, or wait for the human’s decision. Both actors share the human’s utility; the robot is uncertain about the proposed action’s value. Shutdown has utility zero in the model. Waiting lets the human permit the action or stop it (Figure 1 and §2).^[1]

The robot evaluates waiting as E[π(U)U], where π(U) is the probability the human permits the action at utility U. Its gain over the better direct choice is E[π(U)U] − max(E[U], 0). If permission is independent of U, waiting cannot beat the best direct choice; a human decision correlated with value can change that comparison (§2).^[1]

A worked comparison

For a hypothetical illustration, suppose the robot assigns equal probability to an action giving the human utility +10 or −4. Both actors value the human’s utility, stopping gives zero, and waiting has no extra cost. Assume the human knows which value applies and allows the action exactly when its value is positive. These are teaching numbers, not the paper’s experimental data.

Robot’s choice	Expected utility under these assumptions
Act directly	0.5 × 10 + 0.5 × (−4) = 3
Stop directly	0
Wait for the informed human	0.5 × 10 + 0.5 × 0 = 5
Wait for an independent fair coin to permit or stop	0.5 × 3 = 1.5

The informed decision removes the harmful branch while preserving the beneficial one. The coin discards half the opportunities without discriminating between them. This illustrates why uncertainty and a decision connected to value work together: installing a stop mechanism does not itself make its use informative. It also makes the scope visible. The favorable comparison assumes the shared utility, the human’s information and decision rule, zero shutdown utility and the small decision tree; it does not establish those conditions in an actual oversight process.

Conditional findings

If the human permits the action when U ≥ 0 and stops it when U < 0, waiting is never worse than the best direct choice. It is strictly better when the robot assigns positive probability to both beneficial and harmful outcomes (Theorem 1, p.222). Uncertainty alone therefore does not imply a strictly positive incentive: the support of the robot’s belief matters.^[1]

With a noisy human, waiting can lose its advantage. Section 4 models the robot’s belief as Gaussian and the human’s permission probability with a logistic (Boltzmann) rule: higher-value actions are more likely to be permitted, with a parameter controlling decision noise. Within this setup, the analysis relates the robot’s uncertainty, confidence in the action’s value and how closely human decisions track that value. The designer analysis also warns against arbitrarily inflating uncertainty: the robot may then choose poorer actions (§5, pp.225–226). These are conditional analytical results and model illustrations, rather than a deployed shutdown test.^[1]

Historical context

The model builds on Cooperative Inverse Reinforcement Learning and offers a different approach from the utility-indifference model in Corrigibility. It makes human intervention informative about value under its assumptions.^[1][2][3]

Limits

The theorem concerns one human, one robot, a shared human utility, a single proposed action and shutdown assigned zero utility. The human’s policy is an assumption, not a guarantee about real oversight. When choices are noisy, the gain can disappear; the paper analyzes how uncertainty and decision quality interact. The authors also warn that artificially increasing uncertainty may lead the robot to choose worse actions (§§4–5). The model does not establish safe behavior in extended interactions, conflicts among people, or systems that manipulate their overseers.^[1]

For the next question—how people can evaluate actions whose consequences are difficult to judge—continue to Scalable Oversight. Meaningful Human Control asks whether the people assigned to supervise can understand and discharge that role. These are complementary questions to the game’s shutdown incentive.