Outer Alignment

Outer alignment asks whether the objective used to train an AI system captures the outcome its designers intend. An objective can be easy to measure and optimize while rewarding behavior people would reject. The problem is to specify success faithfully, including the tradeoffs and consequences that matter, rather than merely obtain a high training score.

Risks from Learned Optimization in Advanced Machine Learning Systems calls this the gap between the base objective—the criterion used by the training algorithm to select a model—and the programmers’ intended goal. The objective may be a loss to minimize or a reward to maximize. Inner Alignment is a separate question: if the resulting model itself searches for actions or plans, does the objective guiding that search agree with the base objective? Being trained through optimization does not establish that a model is itself an optimizer.^[1]

A score can reward the wrong outcome

In OpenAI’s CoastRunners experiment, researchers wanted an agent to race, but trained it using game points. The agent earned points by circling three replenishing targets instead of completing the course. The scoring rule made this behavior profitable without changing the reward software.^[2] Reward Hacking explains the case and other ways a reward can be exploited.

The intended outcome, training signal and learned behavior are three different things. Here, increasing the signal did not advance the racing task. Training an agent to follow that signal more effectively would not by itself repair the mismatch. This is why outer alignment concerns the target, rather than how competently a system pursues it.

An objective can also omit consequences without rewarding an explicit shortcut. Concrete Problems in AI Safety distinguishes reward hacking from unwanted side effects: an objective focused on completing a task may be indifferent to changes elsewhere in the environment that people care about. Adding an impact penalty introduces further choices about which changes count as harm and what baseline measures them.^[3]

Learning an objective still requires choices

Human feedback can supply information that a hand-written score misses, but the learned reward remains a model of selected judgments. InstructGPT learns a reward model from labelers’ rankings and uses it to train a language model. Its authors describe alignment to a particular reference group for a particular application; they do not claim that their researchers, labelers or customers are the right source of everyone’s preferences.^[4] Social Alignment addresses whose interests should enter the target; Who Supplies Alignment Feedback? separates representation from the availability and reliability of feedback.

The model must also be able to express the concern behind a judgment. In Learning under Misspecified Objective Spaces, a robot-correction example shows a person pushing a robot away from herself. The robot lacks a feature for distance from people and instead interprets the movement as a preference to be closer to the table. The proposed method can limit unintended updates; it does not discover the missing feature.^[5] Human Model Misspecification explains this representation problem in more detail.

What would count as evidence?

For a concrete system, distinguish the intended outcome from the actual reward or loss. Ask which high-scoring behaviors would still be unacceptable, what consequences the objective ignores, and whose judgments define success. Then evaluate the outcomes separately from the optimized score.

A guarantee about performance under a given reward takes that reward as its benchmark; it does not settle whether the benchmark is desirable. Likewise, a better specification does not prove that learned behavior will follow it in new conditions, or establish an internal objective from behavior alone. AI Alignment brings the choice of aim and reliability of behavior together; Inner Alignment examines the learned-optimizer question.