Home โ€บ Blog โ€บ Humans in the loop
Inside the Evaluation Layer ยท Part 4 ยท Humans in the loop

Humans in the Loop: Arenas, Expert Graders and What a Vote Really Measures

Every automated test is ultimately checked against people: what do humans prefer, and can the model do what an expert does? Two very different kinds of human evaluation answer those questions. Public arenas collect millions of quick votes from ordinary users; expert evaluations pay specialists to write hard questions or to grade real work. This part explains how each works, what each actually measures, and the 2025 controversy that showed how a leaderboard can be gamed without breaking any rule.

FL
FrontierAGI Team
โš ๏ธ How to read this (as of 28 Oct 2026) Arena methods and the Llama 4 episode are reported from the Arena's own posts, news coverage and the Leaderboard Illusion paper; GDPval results are OpenAI's and sources report them on different bases (with or without ties). The ranking and private-testing simulators are our illustrations. Disclosure: Claude, made by Anthropic, helps write this site; an Anthropic model appears in the GDPval results below, reported as published.

1. Case file

๐Ÿ™‹
Human evaluation, October 2026Part 4 of the evaluation layer
Arena methodPairwise anonymous votes โ†’ Bradley-Terry ratings, optional style control
Llama 4 (Apr 2025)Experimental chat variant topped the board; released model ranked 32nd
Leaderboard Illusion (2025)27 private Meta variants tested before Llama 4
Humanity's Last Exam~1,000 experts; 2,500 public questions + private holdout
GDPval (OpenAI, Sep 2025)44 occupations; experts compare AI and human deliverables

2. How an arena works

Chatbot Arena began in 2023 as a research project from the LMSYS group at UC Berkeley and became a company, LMArena (Part 1). A user types any prompt; two anonymous models answer side by side; the user votes for the better one, or a tie. Only after voting are the names revealed. Because prompts come from real users, the arena measures what people actually ask, not what benchmark authors imagined, and because answers are anonymous, brand loyalty is (mostly) removed. The MT-Bench study that validated LLM judges used Arena votes as its human reference.

3. Turning votes into a ranking

The arena first used Elo ratings, the chess system, which update after each game. It now fits a Bradley-Terry model to all votes at once: every model gets a strength score such that the chance model A beats model B depends on the difference in their scores. Confidence intervals come from re-sampling the votes. In 2024 it added style control: the regression also includes the difference in answer length, number of markdown headers and number of lists, so a model gets less credit for simply looking more thorough. The site publishes both a raw and a style-controlled board, and the order can shift.

How many votes does a ranking need?

Our simulation: four models with fixed true strengths. More votes shrink the uncertainty bars until the ranking separates.

4. The Llama 4 episode

In April 2025 Meta announced Llama 4 with a strong arena ranking for Maverick. The model on the leaderboard turned out to be "Llama-4-Maverick-03-26-Experimental", a chat-tuned variant optimised for conversation, with a more expressive, emoji-heavy style than the model Meta released. LMArena said Meta's "interpretation of our policy did not match what we expect from model providers", published more than 2,000 head-to-head results, added the public model, which ranked 32nd, and tightened its rules. Meta said it experiments with all kinds of custom variants. No written rule had been broken; the episode showed that the rules themselves allowed a misleading result.

5. The Leaderboard Illusion

Weeks later, researchers led by Cohere Labs published The Leaderboard Illusion. Their findings:

Private testingA few providers could test many private variants before release and publish only the best; Meta tested 27 private variants before Llama 4.
Data asymmetryClosed models were sampled more often and retired less often. Google and OpenAI received an estimated 19.2% and 20.4% of all arena data; 83 open-weight models together received about 29.7%.
Overfitting to the arenaAccess to arena prompts and votes lets providers tune for arena-style preferences rather than general quality.

LMArena disputed parts of the analysis and changed some policies. The broader point holds for any leaderboard: if you can try many times and report your best result, the reported score is biased upward, as the simulator below shows.

6. Best-of-N: why private testing inflates scores

Test N private variants, publish the best

Our model: every variant has the same true rating, but each measured rating has noise. Publishing the maximum of N measurements inflates the expected score.

7. Expert evaluations

Arena votes reward answers ordinary users like. For questions only specialists can judge, labs and evaluators pay experts, often through the same firms that sell training data (Data Layer Part 5).

Building Humanity's Last Exam

From the Center for AI Safety and Scale AI's published process, top to bottom.

A $500,000 prize pool ($5,000 for each of the top 50 questions, $500 for the next 500) and co-authorship drew nearly 1,000 experts. Questions were kept only if frontier models failed them, then reviewed by experts. A private holdout set lets organisers detect memorisation (Part 2).
GDPval (OpenAI, September 2025)Real work products from 44 occupations in the nine sectors contributing most to US GDP, written by professionals with an average of 14 years' experience. Other experts compared the AI deliverable with the human one, blind. OpenAI reported Claude Opus 4.1 rated as good as or better than the expert in about 49% of tasks and GPT-5 (high) in about 41%; OpenAI attributed part of Claude's lead to more appealing formatting. Results varied strongly with file type.
Expert-written test setsGPQA (graduate science), FrontierMath (research mathematics), HLE: experts write questions with verifiable answers, so grading can be automatic even though writing cannot.
Expert red-teamers and rubric gradersSpecialists probe for dangerous knowledge (Part 6) and grade answers in medicine, law or finance against rubrics they wrote (Part 3).

8. Which human signal to trust for what

SignalGood forWeak on
Arena votesEveryday usefulness and style people like; huge, diverse prompt setCorrectness on hard questions; style bias; gaming by private testing
Style-controlled arenaSeparating substance from formattingOnly controls for the style features it measures
Expert-written testsFrontier knowledge and reasoning with checkable answersExpensive; saturate; narrow
Expert graders (GDPval-style)Real work quality in a professionSmall samples; presentation effects; costly
Your own usersYour actual use case (A/B tests in production)Not comparable across companies

9. The series

Inside the Evaluation Layer: seven parts

VerdictHuman evaluation is the reference point for everything else, but "human" covers very different things: a crowd's quick taste, an expert's hard question, a professional's judgement of real work. Arenas made model comparison public and fast, and the 2025 controversies made them more careful about private testing and style. Expert evaluations are where the frontier is now measured, and the frontier is increasingly about doing tasks, not answering questions: Part 5.

10. Reading the numbers

Primary / widely confirmed
Arena method (Bradley-Terry, style control with length, headers, lists; Arena blog). Llama 4: experimental variant, LMArena statement, 2,000+ battles released, public model 32nd (news coverage). Leaderboard Illusion (arXiv 2504.20879): 27 Meta variants; 19.2% / 20.4% / 29.7% data shares. HLE: $500,000 prize pool; ~1,000 experts; 2,500 public questions plus holdout (CAIS, Nature 2026 coverage). GDPval: 44 occupations, 9 sectors, 14 years' average experience.
Reported on different bases
GDPval headline rates (โ‰ˆ49% Claude Opus 4.1, โ‰ˆ41% GPT-5 high "better or on par"; other outlets report win-only rates of 47.6 and 38.8). HLE funnel sizes (~70,000 questions stumped models; ~6,000 reviewed).
Our illustrations
Ranking-uncertainty and best-of-N simulators; the trust table.