RLHF and Constitutional AI: how raw models become helpful assistants
TL;DR
RLHF teaches models through human rankings; Constitutional AI through written principles and AI feedback. Both explain why assistants are helpful, and the sycophancy and verbosity users need to actively steer against.
A model that's only pretrained to predict the next word is great at imitating the internet, but not at following instructions. Ask it to "write a leave request email" and it might continue with a list of similar questions. The step that turns a raw model into a helpful assistant is called alignment, with two big milestones: RLHF in InstructGPT (OpenAI, 2022) and Constitutional AI (Anthropic, 2022).
RLHF: InstructGPT's three steps
- Supervised fine-tuning (SFT): labellers write example answers to prompts; the model is fine-tuned on this data.
- Reward model: the model generates several answers to the same prompt, labellers rank them, and a reward model learns to predict those rankings.
- RL optimisation (PPO): the model is optimised for high reward, with a constraint keeping it close to the SFT model.
The headline result: evaluators preferred answers from the 1.3-billion-parameter InstructGPT over the 175-billion GPT-3. Good alignment can matter more than size.
Constitutional AI: principles instead of human labels
Labelling harmful content at scale is expensive and harmful to the people doing it. Anthropic proposed using a set of principles (a "constitution") so the model supervises itself:
| Phase | How it works |
|---|---|
| Supervised | The model critiques its own answers against the principles, then revises them; it's fine-tuned on the revisions |
| RL from AI Feedback (RLAIF) | The model compares pairs of answers against the principles to create preference data, trains a preference model, then optimises with RL |
The paper's stated goal is an assistant that is harmless but not evasive: when it declines, it explains why, rather than giving a vague non-answer.
Quick comparison
| RLHF | Constitutional AI | |
|---|---|---|
| Feedback source | Human rankers | A model judging against written principles |
| Transparency | Implicit in thousands of labels | Principles you can read and edit |
| Cost to scale | Grows with human labels | Mostly compute |
Today there are also simpler alternatives to PPO such as DPO, which optimises directly on preference pairs without a separate reward model.
Side effects users can see
- Sycophancy: the model learns to please the grader, so it sometimes agrees with your wrong assumptions.
- Verbosity: long, thorough answers tend to score well, so models tend to write long.
- Over- or under-refusal: depends on how helpfulness and harmlessness are balanced.
Risks and caveats
- Reward hacking: the model can maximise reward in unintended ways; hence constraints and independent evals.
- Principles are only as good as their authors: a constitution reflects the values of whoever wrote it.
How to apply it
- Dev: When you want genuine pushback, ask neutrally ("evaluate the pros and cons of this approach") rather than "is this right?", to reduce sycophancy.
- Tech Lead: Borrow the Constitutional AI idea for your product: write clear principles and use a model to grade outputs against them.
- PM: When users complain the model "refuses for no reason", check the system prompt first; aligned behaviour can be steered with clear context.
Sources & references
- Training language models to follow instructions with human feedback — Ouyang et al. (arXiv)
- Constitutional AI: Harmlessness from AI Feedback — Bai et al. (arXiv)