LLM-as-a-judge: using models to grade models, the right way
TL;DR
LLM judges agree with humans about as often as humans agree with each other, making them a viable way to scale evaluation. But you must handle position, verbosity, self-enhancement and reasoning biases, and validate the judge against human grades.
You tweak a prompt and want to know whether quality went up or down. For free-form output (summaries, customer replies, articles) there's no correct answer to match against, and having humans grade every sample is slow and expensive. The common solution is LLM-as-a-judge: using one model to grade another's output. The foundational study by Zheng et al. (2023, the LMSYS team behind MT-Bench and Chatbot Arena) showed it works, and also exposed the biases you need to know.
The study's main results
- A strong judge (GPT-4 at the time) reached over 80% agreement with human evaluators, the same level at which two humans agree with each other.
- That makes LLM judges a scalable, explainable way to evaluate, instead of paying humans to grade everything.
Three grading modes
| Mode | How it works | Good for |
|---|---|---|
| Pairwise comparison | Show A and B, pick the better one | Comparing two prompt/model versions |
| Single-answer grading | Score one output on a scale or rubric | Tracking quality over time |
| Reference-guided grading | Provide the correct solution for comparison | Maths, logic, questions with answers |
Four biases to know
| Bias | Symptom | Mitigation |
|---|---|---|
| Position | Favours the answer shown first (or second) | Grade twice with positions swapped; count a win only if consistent |
| Verbosity | Prefers longer answers even when not better | Rubric explicitly says length earns nothing; penalise padding |
| Self-enhancement | May favour its own model's outputs | Use a judge from a different model family, or cross-check |
| Limited reasoning | Misgrades problems it would itself get wrong | Provide a reference answer; let the judge reason before scoring |
A sample judge prompt
You are grading the customer support answer in <answer>.
Rubric (pass/fail per item):
1. Answers the question in <question>
2. Promises nothing the policy forbids (see <policy>)
3. Gives the customer a clear next step
Length earns no credit. Briefly analyse each item, then
return JSON: {"pass": true/false, "failed": [item numbers]}Anthropic's experience with its Research system is similar: a single judge prompt with a clear rubric, outputting a 0.0–1.0 score plus pass/fail, gave the most consistent results.
Risks and caveats
- Judges need validating too: take 30–50 human-graded samples and measure agreement before trusting the model's scores.
- Optimising for the judge: tuning prompts just to please the judge can hurt the real experience.
- Don't drop humans entirely: periodically have people review random samples to catch what the judge misses.
How to apply it
- Dev: Write pass/fail rubrics per criterion instead of vague 1–10 scales; run the judge in CI when prompts change.
- Tech Lead: Calibrate the judge against a human-graded set; track the agreement rate as a metric.
- PM: Define "good" with specific business criteria; that is the judge's rubric.
Sources & references
- Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena — Zheng et al. (arXiv)
- How we built our multi-agent research system — Anthropic Engineering