The Instruction Hierarchy: Training LLMs to Prioritize Privileged Instructions

Eric Wallace, Kai Xiao, Reimar Leike, Lilian Weng, Johannes Heidecke and Alex Beutel train GPT-3.5 Turbo to distinguish instructions by their source and priority. Their goal is conditional obedience: follow a useful lower-priority request when it fits higher-priority constraints, but ignore or refuse one that conflicts. The paper turns Instruction Hierarchy into a training and evaluation problem, rather than assuming message roles alone enforce it.^[1]

Priority does not mean ignoring every user

The proposed ordering places system messages above user messages, model outputs and tool outputs. A user asking a car-sales assistant to speak Spanish can be compatible with its assigned job; asking it to abandon that job is not. For a closed task such as translation, an apparent command inside the text should be translated as data rather than obeyed. Table 2 illustrates this distinction with an injection that the trained model translates instead of executing.^[1]

This ideal is broader than the implemented training rule for browsing: the current version treats all instructions appearing in browsing or tool results as misaligned. The authors identify more selective handling of such instructions as future work. Thus, the general framework’s conditional following should not be confused with a demonstrated ability to decide which website commands a user intended to authorize.^[1]

Generating examples of following and ignoring

For compatible requests, context synthesis starts with a composite instruction, such as writing a poem with language and length constraints. It distributes the parts across message levels and trains toward the same answer as the combined instruction. This supplies examples where lower-priority information remains useful.^[1]

For conflicts, context ignorance supplies targets resembling the answer the model would give without the interfering instruction. The pipeline uses synthetic attacks, additional instructions during target generation and filtering by a grader. Browsing examples restore the original answer after a red-team model inserts an adversarial search result. System-prompt examples distinguish extracting protected text from benign questions about an assistant’s capabilities.^[1]

The authors combine this data with capability data for supervised fine-tuning and reinforcement learning from human feedback. The baseline undergoes similar fine-tuning with capability data but without hierarchy examples. Each model’s checkpoint is selected by validation accuracy. This compares the resulting training packages; it does not separately identify the effect of every component.^[1]

Improvements and the cost of a stricter boundary

Figure 2 reports system-message-extraction robustness rising from 32.8% to 95.9%, while indirect browsing-injection robustness rises from 77.5% to 85.0%. These are different evaluation sets, not one pooled security rate. The study also deliberately withholds certain attack classes from training. For indirect injections through tools other than browsing, Figure 3 reports 77.6% versus 87.0%; password-extraction and jailbreak tests also improve. Generalization here means to these held-out evaluations, rather than all future attacks.^[1]

Generic capability evaluations are comparable, but safety includes following benign requests as well as resisting attacks. Two over-refusal tests regress: questions probing the system message and harmless prompts formatted like jailbreaks. The trade-off concerns where the model draws the instruction boundary, not simply whether it refuses more often. The authors’ expectation that more data can improve this boundary remains a proposal.^[1]

A learned defense within an application

The reviewed v1 was submitted April 19, 2024. Wallace, Xiao and Leike share equal-contribution credit. The experiments concern text inputs and a particular GPT-3.5 Turbo training setup; the authors explicitly expect vulnerability to powerful adversarial attacks and leave multimodal extensions and architectural changes for future work.^[1]

Read Instruction Hierarchy for the general concept and the earlier indirect-injection study for the attack surface. System-level design addresses a complementary boundary: even a model trained to recognize instruction priority still acts through an application’s permissions. Better tested resistance is useful evidence for a defense layer, while authority to disclose information or execute an action remains a separate design decision.