Deep reinforcement learning from human preferences

Paul Christiano, Jan Leike, Tom B. Brown, Miljan Martic, Shane Legg and Dario Amodei investigate how to communicate a task when writing its reward function or demonstrating the desired behavior is difficult. They learn a reward model from human comparisons, then use that model to train deep reinforcement-learning agents in Atari games and simulated robotics. Their contribution is scaling an existing family of preference-learning methods, rather than inventing human feedback from scratch.^[1]

On this page: Comparisons · Feedback loop · Experiments · Changing behavior · History

Learning from comparisons

People compare two clips lasting one to two seconds. They can prefer either clip, mark a tie or decline to compare them; incomparable pairs are excluded from training. A neural reward model predicts preferences using the total predicted reward over each clip. An ensemble supplies both an averaged reward and a rough disagreement signal for selecting queries. Disagreement-based selection is a heuristic and can impair performance on some tasks.^[1]

Three processes run asynchronously: the policy generates behavior and learns from predicted rewards, selected clips go to a human, and the reward model learns from the accumulated comparisons. The policy uses advantage actor-critic for Atari and trust-region policy optimization for simulated robotics. People do not label every environment interaction.^[1]

This differs from Apprenticeship Learning via Inverse Reinforcement Learning: the supervisory data consists of comparisons of the agent’s behavior, rather than demonstrations from an expert. A person may recognize a desired movement without being able to perform it.^[1]

From a clip choice to a changing policy

The reward model sums its predictions across each clip, then uses the difference between those totals to predict the human’s choice. Fitting the accumulated comparisons trains this predictor; reinforcement learning seeks behavior with a high predicted total reward. This assumes that summed rewards explain comparisons of short segments, rather than giving direct access to the intended reward. The implemented likelihood includes a 10 percent chance of a uniformly random response, alongside ensemble averaging, normalization and regularization.^[1]

The study’s Ant task illustrates why matching the benchmark reward and satisfying the evaluator can differ. Raters were instructed to favor the simulated robot standing upright. Human feedback outperformed synthetic feedback on this task, which the authors attribute to useful reward shaping: their hand-written upright bonus was less effective. A person’s recorded comparison, the learned reward and the resulting policy therefore remain distinct objects; a higher environment score need not mean more accurate recovery of its reward function.^[1]

What the experiments show

Benchmark experiments hide the environment’s reward from the learning agent but use it to measure performance. The study compares real human feedback, synthetic preferences from an oracle with access to that reward, and ordinary reinforcement learning trained directly on it. Eight simulated robotics tasks show broadly comparable mean performance with 700 human comparisons, with less stable learned-reward training. Results across seven Atari games vary: learning remains substantially below the direct-reward baseline on some tasks, and human-feedback Qbert fails to beat the first level.^[1]

The main figures average five runs for simulated-robotics baselines and synthetic feedback, and three for Atari baselines and synthetic feedback; each human-feedback curve represents a single run. Some robotics feedback comes from authors. Human feedback can also shape the target differently from the environment reward, so a higher score does not isolate more accurate recovery of that reward.^[1]

Separate demonstrations train novel behaviors with author feedback, including a simulated Hopper performing repeated backflips using 900 comparisons in under an hour. These are simulated behaviors, not physical-robot deployments. Sparse feedback reduces the labeling burden in these settings; it does not establish that people can judge every important consequence of a harder task.^[1]

Feedback must follow changing behavior

An ablation gathers comparisons only at the beginning of training. The resulting reward predictor can miss behaviors later produced by the policy. In Pong, the agent sometimes avoids losing points without trying to score, sustaining repetitive volleys. The authors use this as evidence for continuing feedback as behavior changes.^[1]

This connects to Reward Hacking: a learned reward remains a proxy that optimization can exploit. For Scalable Oversight, the editorial implication is to examine both the cost of feedback and the evaluator’s ability to see the relevant outcome. The experiments concern short, recognizable behaviors; they do not validate supervision of systems whose important errors people cannot detect.

The oversight route in Key Papers: A Reading Guide moves from reusable comparison judgments to methods intended to make evidence checkable. Who Supplies Alignment Feedback? asks a complementary question: does the supply of feedback keep pace with what optimization newly produces? This study’s static-feedback ablation helps motivate that question without validating a general oversight protocol.

Historical context

The first arXiv version was submitted on June 12, 2017, which supplies this article’s single chronology entry. Later revisions are distinct bibliographic stages. InstructGPT subsequently applies preference-based reward learning to broad language tasks, with demonstrations and different policy-training choices. The shared reward-learning pattern should not erase the differences in task, feedback or evaluation.^[1]