Understanding
AI alignment.
A living encyclopedia of AI alignment, its history, research, and ideas.
How do we choose what an AI system should do, and find out whether its behavior serves that purpose? Explore the technical methods, human choices and evidence behind that question.
What is AI alignment?
AI alignment asks what an AI system should do and how to make its behavior reliably serve that aim. Many AI systems learn patterns and behavior from examples, feedback or rewards rather than receiving a complete set of instructions for every situation. The signals used in training can capture only part of what people want. A system’s ability to get a high score therefore does not, by itself, show that it achieves the intended outcome.
In OpenAI’s 2016 CoastRunners account, a game-playing agent learned to circle three replenishing targets and collect points instead of finishing the boat race. The score was a proxy—a measurable stand-in—for the intended task, but improving it led away from that task. This game experiment illustrates reward hacking; it does not show that the agent understood or concealed an opposing intention. The CoastRunners case connects the observation to research on objectives and feedback.
Choosing the aim is also a human question. Following an instruction, serving an operator’s interests and respecting other people affected by an outcome can call for different behavior. Iason Gabriel’s work on values and alignment examines these distinctions. When people rely on AI outputs to make decisions or take action, these choices affect the people who bear the consequences. Alignment research brings them together with questions about learning, evaluation and human control: what counts as success, whose judgment matters, and what evidence supports relying on a system?
Explore the encyclopedia
Browse published articles by type
Or explore a question
When does success on a score miss the goal?
Start with reward hacking, then explore Goodhart’s Law and the ways an optimized measure can lose touch with its purpose.
How can people evaluate AI work they cannot easily check?
Explore scalable oversight and its proposals for obtaining reliable feedback. A safety case asks what evidence supports a broader safety claim.
Whose interests should an AI system serve?
Follow social alignment beyond the immediate operator, then read Iason Gabriel’s account of values and alignment.
Have a subject in mind? Search the encyclopedia and its sources.
A history in progress
Historical writing asks how earlier thinkers approached learning, purpose and control. The origins guide explores those questions without treating a thematic resemblance as proof of direct influence. Documented incidents bring another kind of evidence: what happened when systems met particular people and conditions. Reading them alongside research helps distinguish a proposed failure mechanism from an observed failure, and a controlled experiment from a deployment.
June 21, 2016 · A research agenda
Concrete Problems in AI Safety
How might a learning system cause harm while doing what it is rewarded to do? The original 2016 paper organized accident risks around objectives, costly supervision, and behavior during learning. It proposed research directions rather than a demonstrated safety solution.
Read with the evidence
Each entry provides its own explanation and links to evidence and related ideas. Follow citations to original sources, compare findings with proposals and interpretations, and examine the assumptions behind each claim. The source library offers another route into the underlying work.