Training language models to follow instructions with human feedback

Long Ouyang and colleagues train GPT-3 models to respond more closely to users’ intended tasks. Their InstructGPT study compares ordinary language-model training with fine-tuning on demonstrations and human preferences across a broad set of language tasks. It treats instruction following as an empirical alignment problem, while explicitly limiting the target to the selected labelers and researchers.^[2]

On this page: Method · Comparisons · Evaluation · Whose intent · Limits · History

From demonstrations to a learned reward

The procedure has three stages. Contractors first write desired responses, which provide supervised fine-tuning data. They then rank candidate responses to the same prompt; a reward model learns to predict these preferences. Finally, proximal policy optimization (PPO) updates the language model to obtain higher reward-model scores. Most comparison data comes from supervised policies, with some from PPO policies; this study does not continuously obtain a human judgment for every policy update.^[2]

The data requirements differ at each stage: about 13,000 training prompts have demonstrations, 33,000 have response rankings for reward modeling, and 31,000 supply prompts for PPO without new human labels. Labelers rank four to nine responses at once; all pairwise comparisons from one ranking are grouped during reward training because treating them as independent examples caused overfitting. The study uses a 6-billion-parameter reward model even when the policy is larger.^[2]

A penalty for divergence from the supervised policy is intended to limit reward-model overoptimization. The main InstructGPT variant, PPO-ptx, also mixes in gradients from the original pretraining distribution to reduce losses on other language tasks. These are optimization choices for a learned proxy, not a guarantee that the reward measures all desired outcomes. The distinction connects to Outer Alignment and Reward Hacking.^[2]

A shared training pattern with different judgments

Deep reinforcement learning from human preferences supplies a 2017 comparison point: a reward model learned from short behavior clips guides agents in games and simulated robotics, with continuing feedback as behavior changes. InstructGPT combines demonstrations and response rankings for broad language tasks. The common pattern is learning a reward from judgments; the supervisory data, policy optimization and evaluation conditions differ.^[3]

Learning to summarize from human feedback studies a narrower 2020 task: a learned reward guides summaries of Reddit posts, with transfer to news articles. Its optimization experiment shows predicted reward diverging from human preference. Read alongside InstructGPT, it separates a reusable training pattern from the task-specific judgments that define success.^[1]

When comparing these studies, follow what a person actually supplies: a behavior comparison, a summary comparison or a demonstration and response ranking. Then identify the behavior the policy can change and the evaluation that would reveal a mistake. A common training loop does not make evidence from one task sufficient for another. These are editorial reading questions, not a controlled comparison among the papers.

What the evaluations establish

On the study’s held-out customer prompt distribution, human evaluators prefer the 1.3-billion-parameter InstructGPT model’s outputs to those of the 175-billion-parameter GPT-3 baseline. For the 175-billion-parameter comparison, outputs are preferred to ordinary GPT-3 85 ± 3 percent of the time and to few-shot-prompted GPT-3 71 ± 4 percent of the time; the paper reports 95 percent confidence intervals. On closed-domain tasks, where responses should use only information supplied in the prompt, unsupported additions fall from 41 to 21 percent. This last comparison is pooled across model sizes, rather than a measured universal factual-error rate. The paper also reports improvements on TruthfulQA. These results concern response preference and task behavior under the stated evaluation conditions; they do not show that the smaller model is more capable on every task.^[2]

Safety results are more qualified. In the RealToxicityPrompts evaluations, InstructGPT produces less toxic text when instructed to respond respectfully; the advantage disappears without that instruction. Explicitly asking for toxic text can make its outputs more toxic than GPT-3’s. The modified Winogender and CrowS-Pairs evaluations do not show reduced bias. Pretraining updates mitigate the regressions caused by PPO fine-tuning, but performance still trails GPT-3 on some tested datasets.^[2]

Whose intent supplies the target?

Training uses labeler-written prompts and prompts submitted to early InstructGPT models in the API Playground, not production API customer data in this paper. Splits separate customers by user ID; the tasks are over 96 percent English. About forty contractors provide the demonstrations, comparisons and main evaluations. The authors identify labeler selection, researchers’ instructions and the customer prompt distribution as choices shaping the target behavior.^[2]

Training prioritizes helpfulness, whereas final evaluations ask labelers to prioritize truthfulness and harmlessness. The paper explicitly warns that the resulting models can follow harmful instructions, fabricate facts and produce biased or toxic content. Preference improvement therefore leaves a question central to Preference Aggregation: whose judgments count, and how should conflicts between users, affected people and designers be resolved? The study describes that problem rather than settling it.^[2]

Alignment relevance and limits

InstructGPT supplies empirical evidence that demonstration learning plus preference-based reinforcement learning can change broad language-task behavior without another large increase in model size. It also shows why Scalable Oversight depends on what humans can reliably judge: optimizing a favorable assessment is insufficient when mistakes are difficult to detect or the target group omits affected stakeholders. Generalization to tasks beyond the training distribution requires further study; this is not a demonstration of reliable supervision of superhuman systems.^[2]

Constitutional AI: Harmlessness from AI Feedback subsequently tests replacing human harmlessness comparisons with model judgments guided by written principles, while retaining human helpfulness feedback. Read together, the two papers distinguish improving instruction following from choosing and enforcing limits on which requests an assistant should fulfill. Their reported evaluations use different setups and should not be treated as a direct performance comparison.

The oversight reading sequence returns to the distinction between reusing judgments and helping people assess difficult outputs. Use Who Supplies Alignment Feedback? to compare availability, reliability and representation after tracing this article’s training target.

Historical context

The original arXiv v1 was submitted on March 4, 2022. This article uses that supported submission date for its single chronology entry. The paper applies earlier human-feedback methods to a broad instruction distribution; it does not introduce reinforcement learning from human feedback from scratch. Its discussion places the work in an iterative alignment research program focused on existing systems.^[2]