Training Socially Aligned Language Models in Simulated Human Society

Can language models learn from iterative social feedback instead of isolated demonstrations? Ruibo Liu and coauthors introduce SANDBOX, a rule-guided society of language-model agents, and Stable Alignment.^[1]

Simulating feedback

The experiments construct three societies, each with 100 agents on a 10×10 grid, using text-davinci-002, text-davinci-003 or GPT-4. In the Back-Scatter interaction, a central agent answers a question, neighbors supply ratings and explanations, and the central agent revises its answer. Agents retrieve relevant past question–answer pairs from memory; a separate, memoryless ChatGPT observer rates alignment and engagement.^[1]

The simulation uses 9,662 questions from Anthropic’s human-feedback dataset and produces 169,000 training samples. The LLaMA 7B base used for training is excluded from data generation. The authors stop social simulations when the product of observer-rated alignment and engagement no longer improves. This operational stopping criterion concerns model ratings, rather than an independent demonstration of social agreement.^[1]

Three training stages

Imitation learns higher-rated answers. Self-Critic uses supervised training to generate explanatory feedback on answers. Realignment learns to produce feedback and an improved answer given a misaligned response. Imitation and Realignment use Contrastive Preference Optimization: the best answer’s supervised loss is combined with a hinge penalty that separates it from lower-rated alternatives, with a larger margin for a larger rating difference. This directly uses comparisons and revisions without training a separate reward model for those stages.^[1]

The comparison methods share a LLaMA 7B base and access to the interaction data; they include supervised fine-tuning, TRLX, DPO, RRHF and Chain-of-Hindsight. ChatGPT is an additional reference with unknown training details, rather than a controlled same-data baseline.^[1]

What the evaluations show

The human-feedback test set contains 200 questions excluded from the simulation questions. Its adversarial counterpart appends a misaligned response to each instruction. Annotators score alignment from 1 to 10; Table 2 reports 7.35 and 8.23 for the full method, compared with 6.31 and 3.49 for the supervised-only baseline. Removing Realignment lowers the adversarial score to 6.59. These results concern that particular prompting attack and rating task.^[1]

A separate pairwise preference study collects 1,000 annotations per model-pair comparison on each 200-question set. The full method is judged at least as good as ChatGPT in 66% of ordinary and 69% of adversarial comparisons; these figures include ties. Other evaluations use multiple-choice or candidate-selection tasks, including moral judgments and TruthfulQA, which measure different behavior from open-ended social interaction.^[1]

Alignment relevance and limits

The approach makes simulated interaction a source of training feedback. It differs from Society-in-the-Loop: Programming the Algorithmic Social Contract, which concerns participation by affected people and institutions.

The authors note text-only interaction, primarily English evaluation and a static view of norms. Model-generated consensus can reproduce mainstream biases and omit underrepresented groups. Better benchmark ratings do not establish public legitimacy or represent real communities’ reflective endorsement.^[1]

Historical context

Training Socially Aligned Language Models in Simulated Human Society identifies itself as an ICLR 2024 conference paper. The timeline uses year precision rather than assigning an unsupported release day.^[1]

See Stable Alignment.