Tests That Leak and Tests That Wear Out: Benchmark Contamination and Saturation
A benchmark is a promise: these questions are new to the model, and they are hard enough to tell models apart. Both halves of the promise break. Test questions leak into the web and then into training data, so a model can score well by remembering rather than reasoning. And even clean tests wear out as models improve, often within a year or two. This part shows how both failures happen, how researchers catch them, and how the field is building tests that last longer.
1. Case file
2. How test questions leak
Nobody needs to cheat for contamination to happen. Pre-training data is a large slice of the public web (Data Layer Part 2), and the web contains benchmark questions: in the original GitHub repositories, in papers that quote examples, in blog posts and tutorials, in forum threads where people discuss answers, and in other datasets that copied them. Coding benchmarks built from public GitHub issues are especially exposed, because the issue, the fix and the discussion all sit in public repositories.
3. The evidence
These are not accusations of cheating. Contamination is mostly an accident of training on the web, and the same models often still reason well on new problems, as the GSM1k authors stress. The issue is what a score means: without clean tests, a benchmark number mixes skill with memory in unknown proportions.
4. Remembering vs reasoning: a simulator
How much of a score is memory?
Our model: a model really solves a share of problems; on contaminated problems it can also recall the answer. Change the inputs and compare the reported score with the score on a fresh test.
5. Catching it
Five detection methods
Tap one. They need different levels of access to the model and its training data.
6. Tests wear out
Even a perfectly clean test stops being useful once the best models answer nearly every question correctly; remaining errors are often mislabelled questions. The useful life of a benchmark has shortened as models improve faster.
Benchmark lifetimes: top score from launch
Selected data points from the sources below; hover or tap a dot. Lines join points; they are not measured curves.
When a test saturates, the field builds a harder one: MMLU gave way to MMLU-Pro and GPQA; GSM8K to competition maths; SWE-bench to SWE-bench Verified and harder agent suites; and in January 2025 a consortium released Humanity's Last Exam, written by experts specifically to resist saturation. By mid-2026 secondary trackers reported top text-only scores in the mid-40s per cent without tools and above 60% with tools โ ๏ธ, so even that test is moving fast. Our benchmark landscape lists the current generation.
7. Defences
| Defence | How it works | Example | Weakness |
|---|---|---|---|
| Decontamination | Remove training documents that overlap with test sets before training | 13-gram overlap filters reported for GPT-3 and later models | Misses paraphrases and translations |
| Canary strings | A unique marker string in benchmark files so crawlers and filters can drop them | BIG-bench's canary GUID | Only works if training pipelines respect it |
| Private holdouts | Keep a share of questions secret, release later or never | GSM1k; FrontierMath's 50-problem holdout | Trust in whoever holds the secret set; less reproducible |
| Live, dated tests | Add new questions continually and score models only on items after their training cutoff | LiveBench (refreshed monthly); LiveCodeBench (contest problems with release dates) | Costly to maintain; question difficulty drifts over time |
| Functional variants | Generate new versions of each problem (new numbers, new code) at test time | Templated maths and code variants | Variants may be easier or harder than originals |
| Agent tasks in fresh environments | Score what a model does in a new environment rather than an answer it could recall | See Part 5 | Expensive and slow to run |
8. The series
Inside the Evaluation Layer: seven parts
9. Reading the numbers
- Primary / widely confirmed
- GSM1k (arXiv 2405.00332): drops up to 13% (v1) / 8% (NeurIPS 2024 version); frontier families minimal overfitting; correlation with likelihood of generating GSM8K. SWE-Bench Illusion (arXiv 2506.12286): up to 76% vs up to 53%; 5-gram overlap up to 35% vs 18%. FrontierMath: OpenAI funding disclosed Dec 2024; 50-problem holdout (Epoch AI). LiveBench (2024, monthly updates); LiveCodeBench (problems dated May 2023โMay 2024 in its first release). MMLU: GPT-3 43.9% (2020), GPT-4 86.4% (2023). SWE-bench 4.4% โ 71.7% (AI Index 2025).
- Reported โ ๏ธ
- Humanity's Last Exam scores in 2026 (secondary trackers; tool and no-tool settings differ); HLE launch-time top score of roughly 9% (early 2025).
- Our illustrations
- Memorisation simulator; lifetime chart lines between data points.
- Sources
- Zhang et al.: A Careful Examination of LLM Performance on Grade School Arithmetic (GSM1k)
- The SWE-Bench Illusion: When State-of-the-Art LLMs Remember Instead of Reason
- Epoch AI: Clarifying the creation and use of FrontierMath
- LiveBench ยท LiveCodeBench
- Evaluation data contamination in LLMs: how do we measure it and (when) does it matter?
- Hendrycks et al.: MMLU ยท GPT-4 Technical Report