From objectives to accountable control

An AI system can satisfy a specified objective and pass an evaluation while leaving people unable to govern its effects. Accountable control requires a connection between what the system pursues, what people can learn about its behavior, what they can change, and who answers for the decision. This is Parallax’s synthesis of the research below: improving any one of those parts leaves questions that the others must address.

The practical question is what the people responsible for an activity—and those affected by it—can understand, challenge and change before a harmful consequence becomes irreversible. A better reward signal, an emergency stop and a responsible institution each contribute something different to that answer.

On this page: Objectives · Correction · Evidence · Responsibility · Synthesis

Objectives do not settle what people want

The 2015 research agenda distinguishes implementing a specification from selecting a requirement that serves the intended purpose. Its vacuum-cleaner example is a thought experiment: rewarding dirt collection allows repeated dumping and collection, although the desired outcome is a clean floor.^[1] Outer Alignment asks what the objective represents; Goodhart’s Law explains how selection on a proxy, changes in the operating regime or interventions in the system can weaken its relationship to the goal.^[2]

More effective optimization cannot by itself resolve that choice of target. The same problem remains when an objective is learned from feedback. In Learning under Misspecified Objective Spaces, a robot’s representation can omit the feature a person is trying to correct. The paper’s Figure 1 illustrates a push intended to increase distance from a person being interpreted as a preference about distance from the table. Its controlled robot studies support reducing unintended updates when feedback does not fit the known features. They do not establish that the missing preference has been identified or added.^[3]

Human Model Misspecification adds another source of error: what a person values is different from how they communicate it. Milli and Dragan distinguish demonstrations intended to accomplish a task from demonstrations intended to teach. Predicting a person’s actions more accurately need not produce a better inference about their objective.^[4]

The implication for control is that receiving feedback, recognizing a model limit and revising the action are separate achievements. A system that declines an unexplained update may avoid one mistake while preserving an inadequate plan. People need a way to change the requirement or halt the activity when the current representation cannot express their concern.

A correction needs both authority and meaning

Corrigibility addresses the ability to correct a system after its initial design. Soares and colleagues’ shutdown analysis distinguishes obeying a button press from avoiding incentives to prevent or cause it, and from ensuring that successor agents also obey. Their partial constructions show why an operational stop rule alone does not settle the surrounding incentives.^[5]

Assistance models offer a different route. Cooperative Inverse Reinforcement Learning gives the human and robot identical payoffs while the human knows the reward parameter and the robot must learn it; teaching and learning can then improve their shared outcome.^[6] The Off-Switch Game makes the human’s choice informative about an uncertain action value. In its one-step model with shared utility, zero utility for shutdown and a rational human policy, waiting for the human is never worse than the available alternatives. A noisy or misspecified human policy changes the result; uncertainty alone does not guarantee deference.^[7]

There is also a difference between learning from a correction and permitting an override. Safely Interruptible Agents models interruption as an imposed policy and studies conditions under which learning avoids a lasting bias from it. Its asymptotic results are not a guarantee of safe behavior during every finite interruption. The off-switch model instead treats the human decision as evidence about value.^[8][7] An emergency stop may need to take effect without the system first understanding its reason; a revised instruction may need to change the system’s model.

The communication channel matters for that second task. The one-round assistance result obtains an optimal pragmatic–pedagogic equilibrium in the rational limit under action separability: each possible modeled goal has a distinguishable human action that communicates it without sacrificing the fully informed outcome. This condition is substantive, not an automatic property of an interface.^[9] Parallax’s inference is to assess both whether a person can express the correction and whether it changes execution. A theorem about successful communication does not establish practical authority to stop a deployed system.

Oversight must connect evidence to action

Whether correction remains possible also depends on the system’s behavior. Optimal Policies Tend to Seek Power gives conditional tendencies for optimal policies in finite Markov decision processes under specified reward-distribution and environmental symmetries. The authors warn that learned policies may be qualitatively different.^[10] Instrumental Convergence thus motivates a question about incentives, while an actual control claim needs evidence about the particular trained system, tools and permissions.

Scalable Oversight asks how to obtain reliable judgments when a supervisor cannot readily assess the work. A detector’s favorable score still needs a connection to effective intervention. Attack-selection research tests code-backdoor attempts under a concentrated setting in which one executed valid attack counts as failure. An attacker’s ability to choose when to submit changes the measured safety of trusted monitoring. The search opportunities, information available to the attacker and audit budget are therefore part of the claim’s conditions.^[11]

The AISI cyber-evaluation incident makes this connection concrete at the institutional level. The institute reports layered network restrictions, synchronous monitoring that can block and escalate actions, and technical/security review of testing activities. It also warns that these controls reduce rather than eliminate risk and require reassessment as capabilities change.^[12] This is evidence of its reported response, not independent proof that the measures prevent every future escape.

Safety Case brings these pieces into an argument about a specified activity: what outcome is claimed, which evidence supports it, which assumptions could fail, and what happens if they do. When a safety evaluation becomes a forecast examines the transfer from a test to deployment. Here the further question is who has the authority and capacity to act on that evidence. A warning that nobody can enforce leaves the proposed control incomplete.

Human authority needs an institutional home

Meaningful Human Control connects responsiveness to relevant human reasons with informed responsibility. Santoni de Sio and van den Hoven require the system’s behavior to track relevant facts and reasons, and to be traceable to people who understand—or are in a position to understand—its capabilities, effects and their responsibility. A nominal human role is insufficient; control may originate in earlier design or deployment choices. Their account also allows that a controlled system can pursue morally bad goals.^[13]

That last distinction returns the argument to objective selection. Whose reasons should govern, who may authorize an activity, and whose objections can trigger revision are questions about Machine Ethics and Society-in-the-Loop, as well as technical assistance. Who Supplies Alignment Feedback? separates the reliability of judgments from the interests they represent. These are Parallax’s connections across frameworks, not a claim that their authors endorse one institutional arrangement.

An operator’s power to halt a system does not by itself give affected people a voice in its use. Nor does assigning responsibility give that operator the information, time or resources needed to exercise it. The institutional extension of this Note’s argument is to connect permission to proceed with reasons that can be challenged, evidence that can be inspected and people empowered to revise or suspend the activity. Which arrangement achieves that remains a question for evidence and public judgment, rather than something an assistance theorem decides.

What an accountable control claim must connect

The research supports distinct pieces: conditional incentives to wait, conservative learning when feedback is unexplained, measured weaknesses in monitoring, and an account of informed human responsibility. Those pieces are not a shared experimental validation of accountable control. Together they suggest three questions for assessing a particular activity:

• What human concern does the objective omit, and can it enter the decision even when the system cannot currently represent it?
• What evidence shows that a correction or stop takes effect before the relevant harm, under the tools, incentives and attack opportunities actually available?
• Who can challenge permission to proceed, who must respond, and what change in the system or institution would require reconsidering the evidence?

Accountable control is strongest when those answers connect: concerns can change the requirement, evidence can change the permission to act, and responsible people can make those changes effective. Better optimization, more feedback and stronger monitoring remain useful contributions. Their value for human control depends on whether people can turn what they learn into an enforceable, informed decision.