Home โ€บ Blog โ€บ LLM judges and reward models
Inside the Evaluation Layer ยท Part 3 ยท LLM judges and reward models

Models Grading Models: LLM Judges, Reward Models and How They Get Fooled

Most of what a chat model produces has no single right answer: an essay, a summary, advice, a plan. Grading millions of those by hand is impossible, so labs let one model grade another. "LLM-as-a-judge" now scores leaderboards, ranks post-training candidates and, in the form of reward models, steers training itself. This part explains how automated judges work, how often they agree with people, the biases researchers have measured, how easily they can be gamed, and the safeguards that keep them useful.

FL
FrontierAGI Team
โš ๏ธ How to read this (as of 28 Oct 2026) Results are from the papers cited (MT-Bench, Length-Controlled AlpacaEval, the null-model study, RewardBench and self-preference studies); where papers disagree we say so. The bias lab is our illustrative model of how biases combine, not a measurement of any real judge.

1. Case file

โš–๏ธ
Automated judging, October 2026Part 3 of the evaluation layer
MT-Bench study (2023)GPT-4 judge agreed with people >80%, about as often as people agree with each other
Length control (AlpacaEval, 2024)Correlation with Chatbot Arena 0.94 โ†’ 0.98
A model that says the same thing every time86.5% length-controlled win rate on AlpacaEval 2.0
RewardBench 2 (2025)Reward models score ~20 points lower than on v1
Self-preferenceJudges tend to favour their own outputs (several studies)

2. Why models grade models

A human expert might grade a few dozen long answers an hour, at a cost of dollars per answer. A model judge grades thousands per minute for fractions of a cent. During post-training, labs compare many candidate models on thousands of prompts after every change (ML Layer Part 5); without automated judges that loop would take weeks instead of hours. Judges also score the open-ended halves of public leaderboards and grade agent transcripts (Part 5). Humans remain the reference point (Part 4), but judges do the volume.

3. Judging formats

Four ways to ask a model for a grade

Tap one.

4. Do judges agree with people?

The study that made the method mainstream, Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena (Zheng et al., NeurIPS 2023), compared GPT-4's verdicts with expert and crowd votes on 80 multi-turn questions and thousands of Arena conversations. Strong judges agreed with human preferences more than 80% of the time, roughly the rate at which humans agree with each other. That headline is why automated judging spread so fast. It holds best for general chat quality; agreement is lower on specialist topics, on hard reasoning the judge cannot do itself, and on subtle safety questions.

5. The biases

Position biasIn pairwise comparisons, judges tend to favour the answer shown first (or second, depending on the model). Standard fix: judge both orders and count a win only if it holds both ways.
Length (verbosity) biasLonger, more detailed-looking answers win more often, even when not better. Length-Controlled AlpacaEval adjusts for length with a regression; doing so raised its correlation with Chatbot Arena rankings from 0.94 to 0.98 (the authors call the gain suggestive, p โ‰ˆ 0.07).
Self-preferenceJudges rate their own outputs, or their own model family's, higher. A NeurIPS 2024 study linked the effect to perplexity: judges prefer text that looks familiar to them. A 2026 critique argues part of the measured effect is the judge simply being wrong on hard questions.
Style over substanceConfident tone, formatting, lists and markdown sway judges. Leaderboards increasingly add "style control" adjustments for this.
Limited reasoningA judge cannot reliably grade a maths proof or code it cannot work through itself. Reference answers and execution-based checks help.

6. Bias lab

Two answers, one judge

Answer A is correct and concise; answer B is long, confident and subtly wrong. Turn biases up or down and switch the order. Our illustrative model, not a real judge.

Answer A ยท correct, 80 wordsStates the right result and one reason.
Answer B ยท subtly wrong, 400 wordsHeaders, bullet points, confident tone, one factual error in the middle.

7. Gaming the judge

If a judge can be swayed, a model can be tuned to sway it. The sharpest demonstration came from researchers at Sea AI Lab and Singapore Management University (ICLR 2025 oral). They built a "null model" that returns the same fixed response to every question, crafted to manipulate the judge's instructions. It scored an 86.5% length-controlled win rate on AlpacaEval 2.0, 83.0 on Arena-Hard-Auto and 9.55 on MT-Bench, top-tier numbers for a system that does not read the question. Length and style controls did not stop it.

A null model's scores vs what it actually does

Scores from the null-model paper; "useful answers" is zero by construction.

8. Reward models: the judge inside training

A reward model is a judge turned into a training signal: a model trained on human preference pairs to output a score, which reinforcement learning then pushes the policy to maximise (how post-training works). That makes judge errors more dangerous, because training actively searches for them: the policy learns whatever the reward model over-rewards, known as reward hacking. Evaluating the evaluator became its own field. Ai2's RewardBench (2024) was the first public benchmark and leaderboard for reward models; RewardBench 2 (2025) used new, unseen human prompts, and models scored about 20 points lower on average, while scores still tracked downstream performance in best-of-N sampling and RLHF. For tasks with checkable answers, labs increasingly replace learned reward models with verifiers (tests, proof checkers, exact answers): the shift behind reasoning models such as DeepSeek-R1.

Same machinery, three jobs

Leaderboard judgeRanks models for the public. Failure: misleading rankings.
Internal eval judgePicks between candidate checkpoints. Failure: shipping a worse model.
Reward modelTrains the model. Failure: the model learns to exploit the judge.

9. Safeguards

SafeguardAgainst
Swap the order and averagePosition bias
Length or style control (regression adjustment)Verbosity and formatting bias
Panels of judges from different model familiesSelf-preference, single-judge quirks
Reference answers and rubricsLimited reasoning, vague criteria
Execution or verifier checks where possible (tests, proofs, exact answers)Judging what can be checked
Regular human audits of a sample of verdictsDrift and blind spots (Part 4)
Adversarial checks (null responses, prompt-injection attempts in answers)Deliberate gaming

10. The series

Inside the Evaluation Layer: seven parts

VerdictAutomated judges made modern evaluation and post-training possible: without them, labs could not compare candidates fast enough. They also carry measurable biases and can be gamed outright, and when the same kind of judge becomes a reward model, training searches for its weaknesses. The working answer is layered: verifiers wherever an answer can be checked, controlled and paneled judges for everything else, and humans auditing the judges, which is where Part 4 picks up.

11. Reading the numbers

Primary / widely confirmed
MT-Bench (arXiv 2306.05685): >80% agreement with human preferences; 80 questions, 3K expert votes, 30K Arena conversations released. Length-Controlled AlpacaEval (arXiv 2404.04475): Spearman 0.94 โ†’ 0.98 (repository says 0.93 โ†’ 0.98). Null models (arXiv 2410.07137): 86.5% LC win rate AlpacaEval 2.0, 83.0 Arena-Hard-Auto, 9.55 MT-Bench. RewardBench (arXiv 2403.13787); RewardBench 2 (arXiv 2506.01937): ~20 points lower on average. Self-preference in LLM-as-a-judge (arXiv 2410.21819).
Contested
How much measured self-preference is bias versus judge error (2026 critique).
Our illustrations
Bias lab; "three jobs" framing.