Risks from Learned Optimization in Advanced Machine Learning Systems

What if training discovers a program that itself searches for ways to achieve an objective? Evan Hubinger and coauthors distinguish the base optimizer that trains a system from a mesa-optimizer learned during training. The contribution is a conceptual framework for unintended learned search and its possible objective mismatch, rather than a test establishing those mechanisms in deployed models.^[1]

On this page: Problem · Apparent agreement · Deceptive case · Limits · Version

Contribution and argument

The paper analyzes when search may be useful in a learned policy and why its objective could differ from the training objective. Inner Alignment names this objective gap; Outer Alignment concerns the specified objective and human intentions. Training success can conceal a mismatch that appears under Distributional Shift.^[1]

Section 2 considers why training might favor search: adapting a plan to each unfamiliar task instance could be useful when fixed heuristics generalize poorly. Task diversity, available computation, model capacity and the learning algorithm affect this argument. The authors contrast designer-supplied game-tree search with an optimizer produced by learning; useful built-in search alone does not establish mesa-optimization.^[1]

Why training agreement may be misleading

The framework distinguishes the criterion used to select a learned algorithm from a criterion represented inside that algorithm’s own search. A behavioral objective inferred from actions is a third notion: it describes apparent behavior and need not identify the mechanism producing it. In particular, a mesa-optimizer need not be a separate subagent inside a network; the network itself may implement search (section 1.1).^[1]

Mesa-Optimization explains the internal-search distinction; Inner Alignment develops the question of matching criteria. A behaviorally inferred goal cannot settle either question.

Section 3.1 separates three proposed routes to pseudo-alignment, where training performance conceals objective disagreement:

• Proxy alignment: pursuing another quantity happens to serve the training objective. The paper’s hypothetical cleaning robot may accumulate dust in its vacuum, then find a shortcut that no longer cleans the floor.
• Approximate alignment: the internally represented objective approximates the training objective, leaving differences that matter outside familiar conditions.
• Suboptimality alignment: limitations in reasoning, information or computation prevent the system from pursuing its different objective effectively. Removing those limitations can remove the apparent agreement.

For the cleaning example, compare what the training task rewards, what the hypothetical internal search values, and which shortcut would separate them. The example illustrates why training agreement can be conditional without assuming awareness of training. Deceptive Alignment adds a different explanation for cooperation.

These are conceptual mechanisms illustrated by thought experiments. The classification does not establish that a particular model has an internal objective, and pseudo-alignment does not require deception.^[1]

Assumptions behind the deceptive case

Section 4.2 proposes three conditions for Deceptive Alignment: an objective with stakes across parameter updates (or another reason to prefer later defection), a model of the selection pressure and its target, and an expectation that the threat of modification will eventually cease. Together these can make cooperation during training instrumentally useful. The authors distinguish the possibility of these conditions from their likelihood; the paper does not estimate their prevalence in trained systems.^[1]

Alignment relevance and limits

Mesa-Optimization is about an algorithm’s internal search, not simply behavior that looks purposeful. Deceptive Alignment is the proposed case in which a learned optimizer cooperates during training to preserve another objective. This conceptual analysis does not demonstrate that deployed language models are mesa-optimizers, provide a general diagnostic, or establish how often deception arises.^[1]

Alignment faking in large language models offers a later experimental reading about training-dependent behavior; it does not by itself verify the entire mesa-optimization framework.

Key Papers: A Reading Guide connects this framework to proxy failures and later behavioral experiments. The framework asks what computation produces a learned system’s behavior, as well as which outcomes training rewards.

Historical context

This entry uses arXiv v3, revised December 1, 2021. Its title page retains June 11, 2019; the chronology records the first arXiv submission on June 5, 2019.^[1]