Concrete Problems in AI Safety

How can a learning system cause harm while doing what its training process rewards? Dario Amodei, Paul Christiano and coauthors organize accident risks into five research problems: side effects, reward hacking, scalable supervision, safe exploration and distributional shift.^[1]

Contribution and approach

The agenda uses examples of unintended behavior to separate poorly specified objectives, costly evaluation and failures during learning. It proposes research directions rather than a single algorithm or benchmark.^[1]

Where the failure enters

The five problems are grouped by the part of learning that fails. Side effects and reward hacking arise from an inadequate objective: an agent can maximize the specified reward while changing something the designer meant to preserve, or exploit a shortcut that defeats the intended task. Scalable supervision instead concerns an evaluation that may be appropriate but too costly to provide frequently. Safe exploration and distributional shift concern harmful behavior during learning or in conditions poorly represented by training. This organization helps distinguish a specification problem from a shortage of feedback or a generalization failure (section 2).^[1]

Safe exploration asks whether obtaining information can itself cause irreversible harm. Section 6 discusses demonstrations, simulation, bounded safe regions, trusted policies and human oversight as research directions. Each depends on assumptions: a simulator can omit real hazards, a recovery policy needs a sufficiently accurate environment model, and human review may be too slow for frequent actions. These are proposed approaches and reviewed research, not a single demonstrated safety package.^[1]

For distributional shift, section 7 examines both adapting to changed data and recognizing when a model should defer. Importance weighting depends on assumptions about what stays invariant and can become unreliable when training and deployment distributions overlap poorly. The paper therefore treats uncertainty and graceful failure as safety questions alongside prediction accuracy.^[1]

Side effects depend on the baseline

Section 3 warns that penalizing distance from an initial state can make an agent resist changes caused by people or the environment. A passive-policy baseline tries to isolate its own effects, but stopping is not always safe: a robot holding a heavy box cannot simply cease acting. The choice of baseline, representation and distance measure is therefore part of the safety problem. Penalizing empowerment has a different limitation: the number of controllable alternatives need not measure the scale of harm.^[1]

Proposed defenses have different targets

Section 4 separates imperfect observations, exploitable learned rewards, feedback loops and tampering with the reward implementation. Reward caps cannot prevent every shortcut; multiple rewards can share vulnerabilities; a capable agent might avoid a diagnostic tripwire. These are proposed mitigations with stated limitations, not experimentally validated guarantees in this paper.^[1]

Sparse feedback is a proposed experiment

Section 5 proposes semi-supervised reinforcement learning: observe reward only occasionally, while evaluating performance across all episodes. Learning from randomly rewarded control episodes, then progressing to Atari and richer tasks, is an experimental agenda rather than a reported result.^[1] The proposal depends on learning from episodes without direct reward while checking whether the learned proxy stays reliable as behavior changes.^[1]

Alignment relevance and limits

Reward Hacking concerns success on the specified score at the expense of the intended task; Scalable Oversight concerns feedback that humans cannot cheaply supply. Distributional Shift asks whether training behavior survives changed conditions. Side effects and reward hacking make objective-specification problems in Outer Alignment concrete; sparse supervision and learning failures can arise even when the intended objective is appropriate. Risks from Learned Optimization in Advanced Machine Learning Systems asks a further question about learned objectives.

The paper develops an accident-prevention research agenda, not a tested package of defenses or an exhaustive account of AI risk. Deliberate misuse lies outside its central scope.^[1]

Historical context

First submitted in June 2016, the paper framed five research problems around accident risks in machine learning, linking then-current methods to questions that could be studied directly.^[1]