The Evaluation Layer, Mapped: Who Tests a Frontier Model, When, and What It Costs
Before a frontier model reaches anyone, it is tested hundreds of times: on training checkpoints, after post-training, by safety teams, by government institutes and outside labs, and then, once released, by leaderboards and millions of people voting in arenas. This first part of the series maps that whole evaluation layer: the stages, the organisations that run them, how fast the tests wear out, what a full evaluation costs, and why measuring AI has become an industry of its own.
1. Case file
Evaluation used to mean one number in a paper: accuracy on a benchmark. In 2026 it is a layer of its own in the AI factory, with its own clusters, tools, companies and job titles. It decides which checkpoint becomes the release, whether a model is safe enough to ship, how it is priced against rivals and what the public believes it can do. This series follows that layer from the inside, the way Inside the ML Layer and Inside the Data Layer followed theirs. It goes deeper than our earlier harness overview and the benchmark landscape.
2. The pipeline: six moments a model is tested
From training checkpoint to public leaderboard
Tap a stage. A composite of how labs describe their process in model cards, system cards and technical reports.
The stages ask different questions. Early checkpoints answer "is training on track?" with cheap, fast tests. Pre-release evaluation asks "is this the best candidate, and is it safe?" with expensive, careful ones. Outside testers ask "is the lab's own assessment right?" And after launch, the public asks "is it better than the alternatives for what I do?" Our case study of how Meta tested Llama 3 405B walks through one lab's version end to end.
3. Who evaluates
Six kinds of evaluators
Tap one for what they test, their tools and their incentives.
The most important change since 2024 is that labs no longer grade only themselves. Government institutes in Britain and the United States test some models before release; non-profits such as METR and Epoch AI publish their own measurements; companies sell evaluation as a product; and academic leaderboards such as HELM and the Holistic Agent Leaderboard try to make comparisons fair. Each has different access, budgets and incentives, which is why the same model can get very different reviews in its first week.
4. Kinds of evaluation
What gets measured, and where this series covers it
โ = commonly run at that stage (our summary). Parts link to the rest of the series.
5. Tests wear out
A benchmark is useful while models score between "can't do it" and "can always do it". The space between those two points keeps shrinking. Stanford's 2025 AI Index found that on three demanding benchmarks introduced in 2023, scores rose sharply in a single year:
One-year gains on three 2023 benchmarks
Percentage-point increase from 2023 to 2024, Stanford AI Index 2025.
6. What a full evaluation costs
Static multiple-choice tests are cheap. Agent evaluations, where a model works for many steps with tools, are not. The Holistic Agent Leaderboard (HAL), published at ICLR 2026, ran 21,730 agent rollouts across 9 models and 9 benchmarks for roughly $40,000 of compute. A follow-up analysis of HAL data found a single SWE-bench Verified run had a median cost of $163, and one agent across eight HAL benchmarks about $800 in API fees, with some frontier runs far higher.
Price an evaluation suite
Our model: tasks ร tokens per task ร price ร repeats. Change the inputs.
Add the hidden costs: engineers to build and maintain harnesses, human graders (Part 4), sandboxes for agent tasks, and compute reserved for evaluation inside the factory. On our site plan of an AI factory, evaluation clusters share Data Hall B with inference and agents.
7. The evaluation economy
8. The series
Inside the Evaluation Layer: seven parts
9. Reading the numbers
- Primary / widely confirmed
- AI Index 2025: MMMU +18.8, GPQA +48.9, SWE-bench +67.3 percentage points; SWE-bench 4.4% (2023) โ 71.7% (2024). HAL (ICLR 2026): 21,730 rollouts, 9 models, 9 benchmarks, ~$40,000. UK AISI: Inspect open-sourced 2024; joint pre-deployment tests of o1 and upgraded Claude 3.5 Sonnet; renamed AI Security Institute. Epoch AI Benchmarking Hub launched Nov 2024. LMArena funding rounds (May 2025, Jan 2026).
- Company or secondary figures
- LMArena ARR (>$30M) and user numbers are company-reported. HAL per-run medians ($163, ~$800) are from a follow-up paper using HAL data. Artificial Analysis index composition is from its own pages.
- Our illustrations
- The six-stage pipeline, the kinds-of-evaluation matrix and the cost calculator.
- Sources
- Stanford HAI: 2025 AI Index press release
- Kapoor et al.: Holistic Agent Leaderboard (ICLR 2026) ยท Efficient Benchmarking of AI Agents
- UK AI Security Institute: research
- Epoch AI: Benchmarking Hub update
- LMArena (funding history)
- Artificial Analysis: model comparisons and Intelligence Index
- On this site
- Measuring the Machine (ML Layer Part 6) ยท Is It Ready? How Meta Tested Llama 3 405B ยท The Frontier AI Benchmark Landscape