Home โ€บ Blog โ€บ The evaluation layer, mapped
Inside the Evaluation Layer ยท Part 1 ยท The map

The Evaluation Layer, Mapped: Who Tests a Frontier Model, When, and What It Costs

Before a frontier model reaches anyone, it is tested hundreds of times: on training checkpoints, after post-training, by safety teams, by government institutes and outside labs, and then, once released, by leaderboards and millions of people voting in arenas. This first part of the series maps that whole evaluation layer: the stages, the organisations that run them, how fast the tests wear out, what a full evaluation costs, and why measuring AI has become an industry of its own.

FL
FrontierAGI Team
โš ๏ธ How to read this (as of 28 Oct 2026) Figures come from the Stanford AI Index, published papers, government and company announcements; company-reported numbers (funding, revenue, users) are labelled as such, and anything we could not confirm against a primary source is marked โš ๏ธ. The pipeline diagram is a simplified composite of how labs describe their process, not one lab's exact process, and the cost calculator is our own model. Disclosure: Claude, made by Anthropic, helps write this site; Anthropic models appear below as examples of external pre-deployment testing.

1. Case file

๐ŸŽฏ
The evaluation layer, October 2026The testing machinery around every model the ML layer produces
Benchmark gains in one year (AI Index 2025)MMMU +18.8 pts ยท GPQA +48.9 ยท SWE-bench +67.3
SWE-bench, 2023 โ†’ 20244.4% โ†’ 71.7% of problems solved
One academic agent leaderboard (HAL)21,730 rollouts, ~$40,000 of compute
Government pre-deployment testingUK AI Security Institute and US partners since 2024
Largest public arena (LMArena)$1.7B valuation, Jan 2026 (company figure)
Open-source eval frameworkslm-evaluation-harness, HELM, Inspect and others

Evaluation used to mean one number in a paper: accuracy on a benchmark. In 2026 it is a layer of its own in the AI factory, with its own clusters, tools, companies and job titles. It decides which checkpoint becomes the release, whether a model is safe enough to ship, how it is priced against rivals and what the public believes it can do. This series follows that layer from the inside, the way Inside the ML Layer and Inside the Data Layer followed theirs. It goes deeper than our earlier harness overview and the benchmark landscape.

2. The pipeline: six moments a model is tested

From training checkpoint to public leaderboard

Tap a stage. A composite of how labs describe their process in model cards, system cards and technical reports.

The stages ask different questions. Early checkpoints answer "is training on track?" with cheap, fast tests. Pre-release evaluation asks "is this the best candidate, and is it safe?" with expensive, careful ones. Outside testers ask "is the lab's own assessment right?" And after launch, the public asks "is it better than the alternatives for what I do?" Our case study of how Meta tested Llama 3 405B walks through one lab's version end to end.

3. Who evaluates

Six kinds of evaluators

Tap one for what they test, their tools and their incentives.

The most important change since 2024 is that labs no longer grade only themselves. Government institutes in Britain and the United States test some models before release; non-profits such as METR and Epoch AI publish their own measurements; companies sell evaluation as a product; and academic leaderboards such as HELM and the Holistic Agent Leaderboard try to make comparisons fair. Each has different access, budgets and incentives, which is why the same model can get very different reviews in its first week.

4. Kinds of evaluation

What gets measured, and where this series covers it

โœ“ = commonly run at that stage (our summary). Parts link to the rest of the series.

5. Tests wear out

A benchmark is useful while models score between "can't do it" and "can always do it". The space between those two points keeps shrinking. Stanford's 2025 AI Index found that on three demanding benchmarks introduced in 2023, scores rose sharply in a single year:

One-year gains on three 2023 benchmarks

Percentage-point increase from 2023 to 2024, Stanford AI Index 2025.

On SWE-bench, AI systems solved 4.4% of the coding problems in 2023 and 71.7% in 2024. When a test saturates it stops separating models, and the field needs a harder one: the reason AGI benchmarks keep being invented. Part 2 covers saturation and the related problem of contamination.

6. What a full evaluation costs

Static multiple-choice tests are cheap. Agent evaluations, where a model works for many steps with tools, are not. The Holistic Agent Leaderboard (HAL), published at ICLR 2026, ran 21,730 agent rollouts across 9 models and 9 benchmarks for roughly $40,000 of compute. A follow-up analysis of HAL data found a single SWE-bench Verified run had a median cost of $163, and one agent across eight HAL benchmarks about $800 in API fees, with some frontier runs far higher.

Price an evaluation suite

Our model: tasks ร— tokens per task ร— price ร— repeats. Change the inputs.

thousand tokens; agent tasks often use hundreds of thousands

Add the hidden costs: engineers to build and maintain harnesses, human graders (Part 4), sandboxes for agent tasks, and compute reserved for evaluation inside the factory. On our site plan of an AI factory, evaluation clusters share Data Hall B with inference and agents.

7. The evaluation economy

ArenasLMArena, which grew out of the academic Chatbot Arena, raised a $100M seed round in May 2025 at a $600M valuation and a $150M Series A in January 2026 at about $1.7B, and says it passed $30M of annual recurring revenue (company figures). Labs pay for private pre-release testing; the public sees the leaderboard.
Independent benchmarkersArtificial Analysis combines nine evaluations into an "Intelligence Index" and publishes the cost of running it per model; Epoch AI runs its own Benchmarking Hub on the open-source Inspect framework and publishes prompts, answers and scores.
Government institutesThe UK AI Security Institute (renamed from the AI Safety Institute in 2025) open-sourced Inspect in 2024 and has run joint pre-deployment tests with its US counterpart, including of OpenAI's o1 and Anthropic's upgraded Claude 3.5 Sonnet.
Data and gradersExpert-written test questions and human graders are bought from the same data-labelling industry described in Inside the Data Layer, Part 5.

8. The series

Inside the Evaluation Layer: seven parts

VerdictEvaluation is no longer a final exam; it runs through the whole life of a model, and it is done by many hands with different motives. The hard problems in the rest of this series follow from that: tests wear out or leak into training data, automated judges have biases, human votes measure taste as much as ability, agent tasks are slow and expensive to score, safety thresholds must be judged under uncertainty, and someone still has to turn all of it into a launch decision.

9. Reading the numbers

Primary / widely confirmed
AI Index 2025: MMMU +18.8, GPQA +48.9, SWE-bench +67.3 percentage points; SWE-bench 4.4% (2023) โ†’ 71.7% (2024). HAL (ICLR 2026): 21,730 rollouts, 9 models, 9 benchmarks, ~$40,000. UK AISI: Inspect open-sourced 2024; joint pre-deployment tests of o1 and upgraded Claude 3.5 Sonnet; renamed AI Security Institute. Epoch AI Benchmarking Hub launched Nov 2024. LMArena funding rounds (May 2025, Jan 2026).
Company or secondary figures
LMArena ARR (>$30M) and user numbers are company-reported. HAL per-run medians ($163, ~$800) are from a follow-up paper using HAL data. Artificial Analysis index composition is from its own pages.
Our illustrations
The six-stage pipeline, the kinds-of-evaluation matrix and the cost calculator.