Humans in the Loop: Arenas, Expert Graders and What a Vote Really Measures
Every automated test is ultimately checked against people: what do humans prefer, and can the model do what an expert does? Two very different kinds of human evaluation answer those questions. Public arenas collect millions of quick votes from ordinary users; expert evaluations pay specialists to write hard questions or to grade real work. This part explains how each works, what each actually measures, and the 2025 controversy that showed how a leaderboard can be gamed without breaking any rule.
1. Case file
2. How an arena works
Chatbot Arena began in 2023 as a research project from the LMSYS group at UC Berkeley and became a company, LMArena (Part 1). A user types any prompt; two anonymous models answer side by side; the user votes for the better one, or a tie. Only after voting are the names revealed. Because prompts come from real users, the arena measures what people actually ask, not what benchmark authors imagined, and because answers are anonymous, brand loyalty is (mostly) removed. The MT-Bench study that validated LLM judges used Arena votes as its human reference.
3. Turning votes into a ranking
The arena first used Elo ratings, the chess system, which update after each game. It now fits a Bradley-Terry model to all votes at once: every model gets a strength score such that the chance model A beats model B depends on the difference in their scores. Confidence intervals come from re-sampling the votes. In 2024 it added style control: the regression also includes the difference in answer length, number of markdown headers and number of lists, so a model gets less credit for simply looking more thorough. The site publishes both a raw and a style-controlled board, and the order can shift.
How many votes does a ranking need?
Our simulation: four models with fixed true strengths. More votes shrink the uncertainty bars until the ranking separates.
4. The Llama 4 episode
In April 2025 Meta announced Llama 4 with a strong arena ranking for Maverick. The model on the leaderboard turned out to be "Llama-4-Maverick-03-26-Experimental", a chat-tuned variant optimised for conversation, with a more expressive, emoji-heavy style than the model Meta released. LMArena said Meta's "interpretation of our policy did not match what we expect from model providers", published more than 2,000 head-to-head results, added the public model, which ranked 32nd, and tightened its rules. Meta said it experiments with all kinds of custom variants. No written rule had been broken; the episode showed that the rules themselves allowed a misleading result.
5. The Leaderboard Illusion
Weeks later, researchers led by Cohere Labs published The Leaderboard Illusion. Their findings:
LMArena disputed parts of the analysis and changed some policies. The broader point holds for any leaderboard: if you can try many times and report your best result, the reported score is biased upward, as the simulator below shows.
6. Best-of-N: why private testing inflates scores
Test N private variants, publish the best
Our model: every variant has the same true rating, but each measured rating has noise. Publishing the maximum of N measurements inflates the expected score.
7. Expert evaluations
Arena votes reward answers ordinary users like. For questions only specialists can judge, labs and evaluators pay experts, often through the same firms that sell training data (Data Layer Part 5).
Building Humanity's Last Exam
From the Center for AI Safety and Scale AI's published process, top to bottom.
8. Which human signal to trust for what
| Signal | Good for | Weak on |
|---|---|---|
| Arena votes | Everyday usefulness and style people like; huge, diverse prompt set | Correctness on hard questions; style bias; gaming by private testing |
| Style-controlled arena | Separating substance from formatting | Only controls for the style features it measures |
| Expert-written tests | Frontier knowledge and reasoning with checkable answers | Expensive; saturate; narrow |
| Expert graders (GDPval-style) | Real work quality in a profession | Small samples; presentation effects; costly |
| Your own users | Your actual use case (A/B tests in production) | Not comparable across companies |
9. The series
Inside the Evaluation Layer: seven parts
10. Reading the numbers
- Primary / widely confirmed
- Arena method (Bradley-Terry, style control with length, headers, lists; Arena blog). Llama 4: experimental variant, LMArena statement, 2,000+ battles released, public model 32nd (news coverage). Leaderboard Illusion (arXiv 2504.20879): 27 Meta variants; 19.2% / 20.4% / 29.7% data shares. HLE: $500,000 prize pool; ~1,000 experts; 2,500 public questions plus holdout (CAIS, Nature 2026 coverage). GDPval: 44 occupations, 9 sectors, 14 years' average experience.
- Reported on different bases
- GDPval headline rates (โ49% Claude Opus 4.1, โ41% GPT-5 high "better or on par"; other outlets report win-only rates of 47.6 and 38.8). HLE funnel sizes (~70,000 questions stumped models; ~6,000 reviewed).
- Our illustrations
- Ranking-uncertainty and best-of-N simulators; the trust table.