Measuring the Machine: The Evaluation Harnesses That Decide What Ships
Every training run, RL experiment and release decision ends with a number. That number comes from an evaluation harness: software that formats prompts, calls the model, extracts answers, scores them and adds them up. Small choices at each step move scores by several points, benchmarks saturate within a year or two, and agent evaluations now need containers and sandboxes like production systems. This part explains how harnesses work, why the same model gets different scores, and how to read a leaderboard without being fooled.
1. Case file
2. Timeline
Evaluating language models, 2018 โ 2026
Tap a milestone. Colour: benchmarks, harnesses, arenas and leaderboards, agent and safety evals.
3. What happens in one eval run
A multiple-choice question, end to end
Step through the five stages a harness runs for every question.
4. Same model, different scores
Each stage involves a choice, and each choice moves the score. A base model scored by comparing log-probabilities of the options can look very different from the same model asked to write its answer. A strict answer-extraction regex can mark correct answers wrong. Few-shot examples help base models and can hurt chat models. This is why papers that report "MMLU 85" often cannot be compared.
Move the knobs
An illustration of a hypothetical chat model whose "true" ability is 80%. Effect sizes are invented but in the range harness papers report. Not data about any real model.
5. The harnesses
The same task, two harnesses
6. Feature matrix
Our reading of project documentation. โ built in ยท โ partial ยท โ not a focus. Tap a cell.
7. Benchmarks saturate
From launch to ceiling
Best reported score near each benchmark's launch, and a recent top score. Launch figures from the benchmark papers; recent figures from third-party trackers that disagree (hatched, โ ๏ธ).
Benchmarks follow a life cycle: launched with headroom, adopted, optimised against, saturated, replaced. MMLU (2020) gave way to MMLU-Pro and GPQA; Hugging Face rebuilt its Open LLM Leaderboard around harder tests in 2024 and then retired it in March 2025, saying it was "slowly becoming obsolete" and could encourage people "to hill climb in irrelevant directions". Two forces speed saturation: genuine progress, and contamination, when test questions leak into training data. Harnesses respond with private held-out sets, canary strings that let labs detect leaked data, and regularly refreshed question pools.
8. Statistics: is it really better?
Confidence interval calculator
95% intervals using the normal approximation to a binomial. Questions are treated as independent; clustered questions widen the interval further.
A 3-point gain on a 200-question benchmark is usually inside the noise. Anthropic's 2024 paper "Adding Error Bars to Evals" (Miller) argued that labs should report standard errors, cluster them when questions share a source passage, and compare models with paired tests on the same questions, which is more sensitive than comparing two separate intervals.
9. Agent evals are infrastructure
Agent benchmarks such as SWE-bench (real GitHub issues, judged by the repository's own tests), OSWorld (computer use) and terminal tasks need a fresh container per attempt, tool access, time limits and logs of every step. A single run can take minutes to hours and cost real money in tokens; variance between runs is large. Harnesses like Inspect ship sandboxing (Docker and Kubernetes) and agent scaffolds, and SWE-bench runs each task in its own Docker image. In 2024 OpenAI released SWE-bench Verified, a 500-task subset screened by human engineers because some original tasks were under-specified or had faulty tests.
pass@k vs pass^k
If an agent solves a task with probability p per attempt: pass@k = chance at least one of k tries succeeds (research benchmarks); pass^k = chance all k succeed (reliability, as ฯ-bench reports). Independent attempts assumed.
10. Human preference and arenas
Chatbot Arena (May 2023, UC Berkeley's LMSYS) asks users to compare two anonymous models and fits a BradleyโTerry model (an Elo-style rating) to millions of votes. It became the most-watched leaderboard, spun out as LMArena, raised $100 million at a $600 million valuation in May 2025 and $150 million at a $1.7 billion valuation in January 2026, and sells evaluation services to labs. Its critics matter for anyone reading the rankings:
- The Leaderboard Illusion (April 2025, researchers including Cohere Labs, preprint) found that a few providers privately tested many variants and published only the best; Meta tested 27 private variants before Llama 4. It estimated Google and OpenAI received about 19.2% and 20.4% of arena data, against 29.7% for 83 open-weight models combined. LMArena disputed several points and said its pre-release testing policy was public.
- Style effects: longer, more formatted answers win votes; the arena added "style control" to adjust for this.
- Who votes: arena users are not a random sample of real use.
11. Inside the labs
12. Choosing a harness
Which harness fits?
An editorial guide.
13. Reading the numbers
- Primary
- Open LLM Leaderboard retired 13 Mar 2025, 13K+ models evaluated (Hugging Face). Leaderboard Illusion: 27 Llama 4 private variants; Google 19.2%, OpenAI 20.4%, 83 open models 29.7% of data (arXiv 2504.20879, preprint, disputed by LMArena). GPQA Diamond: 198 questions. SWE-bench Verified: 500 tasks (OpenAI, 2024). Launch scores: MMLU GPT-3 43.9% (2020); GPQA GPT-4 ~39% (2023); HLE frontier models under 10% (Jan 2025); SWE-bench Claude 2 1.96% (2023).
- Reported / secondary โ ๏ธ
- LMArena: $100M at $600M (May 2025), $150M at $1.7B post-money (Jan 2026), ~$30M revenue run-rate (Dec 2025) per press. Recent tops: GPQA Diamond ~94%, HLE ~45โ55% (trackers disagree); MMLU ~90%+.
- Our illustrations
- Score-sensitivity knobs (invented effect sizes), intervals (independent questions, normal approximation), pass@k (independent attempts) and the chooser.
- Harnesses
- EleutherAI lm-evaluation-harness ยท Lessons from the trenches (2024)
- Stanford HELM
- Inspect (UK AI Security Institute)
- Hugging Face lighteval ยท OpenCompass
- Benchmarks and critiques
- MMLU ยท GPQA ยท Humanity's Last Exam ยท SWE-bench
- OpenAI: SWE-bench Verified
- The Leaderboard Illusion
- Adding Error Bars to Evals (Miller, 2024)
- Hugging Face: Open LLM Leaderboard retired
- SiliconANGLE: LMArena raises $100M
- LMArena $150M at $1.7B
- Part 5: post-training & RL ยท Part 4: kernels