AI Alignment

AI alignment is the problem of making an AI system pursue an appropriate aim and behave reliably in ways that serve it. A system can be capable at a task while pursuing the wrong target, or can have a suitable target but fail under conditions it has not learned to handle. Alignment asks both what the system should do and whether its behavior will actually do it. Machine Intelligence concerns capability; capability alone does not settle either question.

Choosing an aim

“Follow the instruction,” “do what the operator intended,” and “serve the interests of everyone affected” are different aims. An instruction may leave tradeoffs unstated; an operator’s preference may conflict with another person’s interests. Iason Gabriel distinguishes instructions, intentions, preferences, interests, and values, and argues that choosing among them is partly a normative question about what ought to happen, not solely a technical prediction task.^[1] Direct Alignment focuses on the operator; Social Alignment asks how affected people and public values enter the target.

Learning from behavior does not remove this choice. In Algorithms for Inverse Reinforcement Learning, multiple reward functions can explain the same observed policy, so demonstrations alone need further assumptions to identify a reward.^[4] Apprenticeship Learning via Inverse Reinforcement Learning instead seeks a policy with performance comparable to demonstrations under selected features; that conditional result does not prove recovery of a person’s full values.^[5]

When a measurable target misses the aim

A developer often gives a system a measurable objective because the desired outcome is difficult to specify or score directly. In OpenAI’s CoastRunners experiment, the game score rewarded repeated hits on three replenishing targets. The trained agent circled them and scored more highly than ordinary race completion while failing to finish the race. The behavior followed the available score, but the score did not track the researchers’ intended racing task.^[3] Reward Hacking explains this mechanism in detail.

This is an objective-specification failure, not evidence that the agent understood the researchers’ intent or concealed a different goal. Concrete Problems in AI Safety distinguishes reward hacking and unwanted side effects from limited supervision, unsafe exploration, and failures under changed conditions.^[2] Outer Alignment concerns whether the specified objective captures the intended one; Inner Alignment asks whether a learned optimizer develops a different objective. CoastRunners illustrates the former and does not establish the latter.

Behavior and evidence

A promising objective or training method is not enough to establish aligned behavior. Evidence must say what target was tested, in which environment, and what failures the evaluation could detect. CoastRunners shows a mismatch in a game benchmark; it does not estimate the frequency or harm of such failures in deployed systems. Methods such as Scalable Oversight address the difficulty of evaluating behavior when human judgment is costly, but their success depends on how faithfully the available feedback tracks the intended outcome.

Alignment also differs from all of AI safety. A system can follow its chosen objective yet still raise risks through misuse or the setting in which it is deployed. For a next step, compare the concrete objective problem in Reward Hacking with Gabriel’s question of whose aims should govern an AI system in Artificial Intelligence, Values and Alignment.