STA-RLHF: Stackelberg Aligned Reinforcement Learning with Human Feedback

How should a language model learn when its preference model also responds to its behavior? Jacob Makar-Limanov, Arjun Prakash and coauthors formulate feedback training as a Stackelberg Game.^[1]

Method and contribution

The language model is the leader and the preference model the follower. A nested gradient-based procedure accounts for their coupled optimization. The authors compare approaches on synthetic tasks and report improvements over selected PPO and DPO baselines.^[1]

What responds to what?

In the Brown manuscript’s formulation, a preference model predicts which of two continuations a designer would prefer. The language model generates the continuations; the preference model then learns from labels on those current outputs. The nested algorithm collects new pairs, ranks them with an oracle, updates the preference model in an inner loop, and updates the language model using that refreshed signal. Its simultaneous comparison instead updates each model using the other model’s preceding iteration. This addresses feedback becoming unrepresentative as the policy changes; it does not remove the need for a preference-labeling oracle.^[1]

The formal setup assumes a finite string space and complete, transitive preferences. Its strong Stackelberg equilibrium selects a follower best response with ties favorable to the leader. That solution concept describes the specified utilities; the implementation’s finite inner updates are not proof that an equilibrium has been reached.^[1]

What the synthetic rewards measure

The Brown experiments use a sentiment classifier to score positive movie-review completions and a word-count objective to reward inclusion of distinct target words. These supply inspectable task rewards rather than a broad assessment of human values. The sentiment experiment uses GPT-2 Large; word collection uses OPT-125m. Reward and KL divergence from the reference policy are reported together, so faster reward improvement alone is an incomplete comparison. The manuscript describes its findings as preliminary and leaves larger models and preference aggregation for future work.^[1]

Two implementations in the RLC-headed text

The RLC-headed version distinguishes Naive STA-RLHF, which updates the language model against the refreshed reward model, from Total STA-RLHF, which additionally differentiates through the follower’s response. Total approximates an inverse-Hessian-vector product with ten conjugate-gradient iterations. That extra computation does not convincingly outperform the naive variant in the reported experiments.^[1]

Its oracle ranks four generated continuations and selects the best and worst as a training pair. The reward model receives five inner updates, except in the simultaneous comparison. PPO is given the ground-truth task reward directly rather than a learned reward model. These design differences matter when interpreting comparisons between algorithms.^[1]

Alignment relevance and limits

The formulation concerns optimizing learned feedback, a concrete part of Outer Alignment. It does not settle whose preferences should govern the system; Preference Aggregation addresses a different aspect of that question.

The Brown manuscript reports preliminary synthetic-task results. It supplies no general convergence guarantee, and larger models and additional tasks need validation. Higher model reward does not by itself establish faithful human-value representation or deployment safety.^[1]

Version and evidence scope

Two available versions differ. The Brown manuscript evaluates sentiment generation and word collection. An original-document text headed RLJ / RLC 2024 uses reward-model terminology, distinguishes Naive and Total STA-RLHF, and reports word collection, constrained word collection, and unique-noun tasks. They share the title and five authors; this does not establish identical results or revision order.^[1]

In the RLC-headed text, higher rewards on the two word-collection tasks accompany greater divergence from the reference policy: the authors describe those methods as Pareto dominated. Its results therefore support a narrower claim than uniformly better reward–divergence trade-offs. The experiments use small models, five seeds, and no hyperparameter optimization.^[1]

The central contribution is a way to keep feedback training responsive to the policy’s changing outputs. Its usefulness still depends on the labeling objective: efficiently learning sentiment or word-count preferences cannot establish that the objective represents the people affected by deployment.^[1]