Scalable Oversight

Scalable oversight develops ways for people to evaluate AI work when direct review is too costly or too difficult. The central problem is that producing an impressive answer and recognizing whether it is correct are different abilities. More feedback helps only if the feedback measures what matters.

A supervisor may know how to judge an answer but lack time to inspect every example. Alternatively, the supervisor may not recognize a subtle error even after reading the answer. These bottlenecks motivate different methods: reuse reliable judgments, enlist AI assistance, or make decisive evidence easier to inspect. Concrete Problems in AI Safety introduces costly supervision as a research problem; Who Supplies Alignment Feedback? separates feedback availability, reliability and representation.

Reuse human judgments

A reward model learns to predict people’s assessments and supplies scores for many subsequent training updates. In Deep reinforcement learning from human preferences, people compare short behavior clips; a learned reward guides simulated control and Atari policies. Continuing comparisons help keep the reward model relevant as behavior changes. The same paper’s offline-feedback Pong failure illustrates the risk of optimizing an imperfect learned score.^[1]

This can reduce the number of direct judgments without making a difficult task easier to judge. Learning to summarize from human feedback and InstructGPT extend the feedback-learning approach to language tasks. Their dedicated entries explain the comparisons, evaluator work and experimental limits. A reward model’s agreement with raters does not establish that those raters can detect every important error, or that their preferences represent everyone affected.

Use AI assistance to improve supervision

AI assistance can supply critiques or comparisons before a person makes a judgment. Constitutional AI uses human-written principles to guide model critiques, revisions and harmlessness comparisons. Its 2022 experiment replaces human harmlessness training labels with AI labels while retaining human helpfulness feedback and human outcome evaluation.^[2]

The distinction is between generating judgments more cheaply and making them more reliable. Principles identify intended criteria; the model must still apply them to unfamiliar cases. The study’s evaluated improvements do not establish that an AI supervisor can recognize all mistakes by a more capable system.^[2]

Make decisive evidence checkable

AI Safety via Debate proposes competing arguments that bring a disagreement within a judge’s ability to assess. Its human protocol work exposes a crucial condition: being able to check a small mistake is insufficient if neither debater can locate it in a long argument.^[3]

Evidence access also matters for automated monitors. How does information access affect LLM monitors’ ability to detect sabotage? compares ways of selecting information for evaluating agent trajectories. Its controlled sabotage tasks distinguish selecting relevant excerpts from judging them; extraction can omit evidence and evaluation can rationalize suspicious excerpts. Giving a judge more text is therefore not interchangeable with giving it useful evidence.^[4]

Evaluate the whole interaction

A monitor scores behavior; an oversight system also decides whether to audit, interrupt or permit it. Its usefulness depends on missed failures, false alarms and the available review budget. Attack selection against trusted monitoring shows in a coding game that an attacker can withhold conspicuous attacks and submit subtler candidates. Detection measured against indiscriminate attacks can overstate performance against a selective adversary. It assumes perfect human audits and treats executing a valid inserted backdoor as a successful attack, without a separate exploitation step.^[5]

For a Safety Case, the relevant question is what evidence supports supervision for the intended use: which errors the evaluator can see, which it can recognize, what happens after its judgment, and whether the actor can adapt. The dedicated studies of SLEIGHT-Bench: A Benchmark of Evasion Attacks Against Agent Monitors, reasoning monitorability and ensemble monitoring explore particular parts of this problem. They do not turn a detection score into a general alignment guarantee.

Scalable oversight connects training feedback with evaluation and intervention, while leaving whose judgments should govern as a separate question. The reading guide offers routes through the underlying research; the debate entry explains one proposed mechanism in more depth.