Who Supplies Alignment Feedback?

A system can receive many judgments about its answers and still be poorly supervised. A judge may miss an error; the questions put to that judge may leave affected people out. Who supplies alignment feedback, and what can that feedback establish? Comparing human preferences, debate, model-written critiques and social participation reveals three separate bottlenecks: obtaining judgments, making them informative, and deciding whose judgments count.

More judgments do not guarantee better scrutiny

In Learning to summarize from human feedback, people compare summaries, a reward model learns from those comparisons, and optimization uses its predictions. This supplies a trainable signal for a task where evaluating outputs is easier than writing them. The study reports gains in judged summary quality, but also finds that optimizing the reward model too far can reduce quality. More use of a learned judgment is therefore not the same as more reliable judgment of the resulting summaries.^[4]

Deep reinforcement learning from human preferences shows a related problem in simulated control tasks. When comparisons were collected only at the start, the reward predictor could become uninformative about later behavior; in the Pong example, the agent sustained volleys instead of trying to score. The lesson is narrower than a claim about language-model training: feedback must keep contact with the behavior being optimized. A large initial label set cannot by itself assess behavior it never covered.^[7]

Constitutional AI: Harmlessness from AI Feedback shifts some of the labeling work to a model. Written principles guide critiques and revisions, and AI-generated harmlessness comparisons train a preference model; the experiment still uses human helpfulness comparisons and human evaluation of trained assistants. This changes the cost and origin of feedback. It does not show that the written principles cover every important case or that a model applying them will identify every consequential omission.^[6]

Can the judge see the decisive evidence?

Imagine a fluent summary that reverses one qualification in a long report. A reader may prefer it without noticing the reversal. Collecting additional preferences from readers with the same blind spot would reinforce the mistake. This is a hypothetical example of an information problem, not a reported result from the summarization study.

AI Safety via Debate offers a different route: two agents argue before a judge, with an opponent rewarded for exposing a flaw. In principle, an opponent who can locate the qualification could make it easier to evaluate than the full report. But the 2020 debate writeup describes a practical obstacle: a debater may believe a long answer is wrong yet fail to isolate a short, judgeable error. Debate helps only when the relevant evidence can be found, presented and assessed; competition alone does not supply those abilities.^[1][5]

These approaches address different steps. Human comparisons record preference over presented outputs. A learned reward extends those comparisons to additional outputs. Constitutional critique adds a rule-guided assessment. Debate tries to make hidden evidence visible to a judge. Each can increase the amount of feedback available, while leaving a distinct route for errors to pass through. The useful question is not simply how many judgments were made, but what information each judge could use and how later behavior was checked.

Who sets the standard?

Even a judge who catches every factual error may apply a narrow standard. A summary might accurately describe benefits to an operator while giving little weight to costs borne by other people. That is a choice about whose interests count, not a failure to read the report closely.

Stable Alignment uses simulated ratings, critiques and revisions to create training data. Its authors identify the risk that mainstream judgments and underrepresented groups will be poorly represented. Benchmark improvement can demonstrate performance under the study’s evaluation, but cannot establish that the simulated population speaks for affected people.^[2] Society-in-the-Loop instead proposes stakeholder negotiation and ongoing monitoring as part of aligning systems with social values. It sets out a governance direction, not an experiment showing that a particular training objective has public legitimacy.^[3]

Constitutional AI makes selected principles inspectable and its authors discuss wider participation in refining them. That visibility is useful: readers can question the instruction rather than infer it only from outputs. Yet transparency about a principle and authority to choose it are different things. The comparison with Society-in-the-Loop suggests that the source of a rule and the process for contesting it belong alongside technical tests of whether a model follows it.^[6][3]

The synthesis is a sequence of questions for any feedback scheme: Can it obtain a judgment about new behavior? Can the judge inspect the evidence needed to distinguish success from a convincing error? Can affected people contest the standard being applied? A method may answer one question well without answering the others. Combining methods could help, but the studies and proposals here do not establish a common score on which to rank them.

Scalable Oversight develops the problem of reliable judgment; Social Alignment develops whose values are represented. Preference Aggregation follows the step from separate judgments to a collective objective; Meaningful Human Control follows the relationship between judgment, intervention and informed human responsibility. The oversight reading sequence links the original studies and proposals behind this comparison.