Models Grading Models: LLM Judges, Reward Models and How They Get Fooled
Most of what a chat model produces has no single right answer: an essay, a summary, advice, a plan. Grading millions of those by hand is impossible, so labs let one model grade another. "LLM-as-a-judge" now scores leaderboards, ranks post-training candidates and, in the form of reward models, steers training itself. This part explains how automated judges work, how often they agree with people, the biases researchers have measured, how easily they can be gamed, and the safeguards that keep them useful.
1. Case file
2. Why models grade models
A human expert might grade a few dozen long answers an hour, at a cost of dollars per answer. A model judge grades thousands per minute for fractions of a cent. During post-training, labs compare many candidate models on thousands of prompts after every change (ML Layer Part 5); without automated judges that loop would take weeks instead of hours. Judges also score the open-ended halves of public leaderboards and grade agent transcripts (Part 5). Humans remain the reference point (Part 4), but judges do the volume.
3. Judging formats
Four ways to ask a model for a grade
Tap one.
4. Do judges agree with people?
The study that made the method mainstream, Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena (Zheng et al., NeurIPS 2023), compared GPT-4's verdicts with expert and crowd votes on 80 multi-turn questions and thousands of Arena conversations. Strong judges agreed with human preferences more than 80% of the time, roughly the rate at which humans agree with each other. That headline is why automated judging spread so fast. It holds best for general chat quality; agreement is lower on specialist topics, on hard reasoning the judge cannot do itself, and on subtle safety questions.
5. The biases
6. Bias lab
Two answers, one judge
Answer A is correct and concise; answer B is long, confident and subtly wrong. Turn biases up or down and switch the order. Our illustrative model, not a real judge.
7. Gaming the judge
If a judge can be swayed, a model can be tuned to sway it. The sharpest demonstration came from researchers at Sea AI Lab and Singapore Management University (ICLR 2025 oral). They built a "null model" that returns the same fixed response to every question, crafted to manipulate the judge's instructions. It scored an 86.5% length-controlled win rate on AlpacaEval 2.0, 83.0 on Arena-Hard-Auto and 9.55 on MT-Bench, top-tier numbers for a system that does not read the question. Length and style controls did not stop it.
A null model's scores vs what it actually does
Scores from the null-model paper; "useful answers" is zero by construction.
8. Reward models: the judge inside training
A reward model is a judge turned into a training signal: a model trained on human preference pairs to output a score, which reinforcement learning then pushes the policy to maximise (how post-training works). That makes judge errors more dangerous, because training actively searches for them: the policy learns whatever the reward model over-rewards, known as reward hacking. Evaluating the evaluator became its own field. Ai2's RewardBench (2024) was the first public benchmark and leaderboard for reward models; RewardBench 2 (2025) used new, unseen human prompts, and models scored about 20 points lower on average, while scores still tracked downstream performance in best-of-N sampling and RLHF. For tasks with checkable answers, labs increasingly replace learned reward models with verifiers (tests, proof checkers, exact answers): the shift behind reasoning models such as DeepSeek-R1.
Same machinery, three jobs
9. Safeguards
| Safeguard | Against |
|---|---|
| Swap the order and average | Position bias |
| Length or style control (regression adjustment) | Verbosity and formatting bias |
| Panels of judges from different model families | Self-preference, single-judge quirks |
| Reference answers and rubrics | Limited reasoning, vague criteria |
| Execution or verifier checks where possible (tests, proofs, exact answers) | Judging what can be checked |
| Regular human audits of a sample of verdicts | Drift and blind spots (Part 4) |
| Adversarial checks (null responses, prompt-injection attempts in answers) | Deliberate gaming |
10. The series
Inside the Evaluation Layer: seven parts
11. Reading the numbers
- Primary / widely confirmed
- MT-Bench (arXiv 2306.05685): >80% agreement with human preferences; 80 questions, 3K expert votes, 30K Arena conversations released. Length-Controlled AlpacaEval (arXiv 2404.04475): Spearman 0.94 โ 0.98 (repository says 0.93 โ 0.98). Null models (arXiv 2410.07137): 86.5% LC win rate AlpacaEval 2.0, 83.0 Arena-Hard-Auto, 9.55 MT-Bench. RewardBench (arXiv 2403.13787); RewardBench 2 (arXiv 2506.01937): ~20 points lower on average. Self-preference in LLM-as-a-judge (arXiv 2410.21819).
- Contested
- How much measured self-preference is bias versus judge error (2026 critique).
- Our illustrations
- Bias lab; "three jobs" framing.
- Sources
- Zheng et al.: Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena
- Dubois et al.: Length-Controlled AlpacaEval
- Zheng et al.: Cheating Automatic LLM Benchmarks: Null Models Achieve High Win Rates
- Lambert et al.: RewardBench ยท RewardBench 2
- Wataoka et al.: Self-Preference Bias in LLM-as-a-Judge