Home โ€บ Blog โ€บ Contamination and saturation
Inside the Evaluation Layer ยท Part 2 ยท Contamination and saturation

Tests That Leak and Tests That Wear Out: Benchmark Contamination and Saturation

A benchmark is a promise: these questions are new to the model, and they are hard enough to tell models apart. Both halves of the promise break. Test questions leak into the web and then into training data, so a model can score well by remembering rather than reasoning. And even clean tests wear out as models improve, often within a year or two. This part shows how both failures happen, how researchers catch them, and how the field is building tests that last longer.

FL
FrontierAGI Team
โš ๏ธ How to read this (as of 28 Oct 2026) Study results come from the papers cited; where versions of a paper report different numbers we say so. Leaderboard figures for 2026 come from secondary trackers and are marked โš ๏ธ. The memorisation simulator and benchmark-lifetime chart are our illustrations built from the cited data points, not a model of any specific system. Disclosure: Claude, made by Anthropic, helps write this site; Anthropic's models appear in one of the studies discussed.

1. Case file

๐Ÿงช
Contamination and saturation, October 2026Part 2 of the evaluation layer
GSM1k (Scale AI, 2024)Some models dropped up to 8โ€“13% on a fresh copy of GSM8K
SWE-Bench Illusion (2025)Up to 76% name the buggy file from the issue alone; โ‰ค53% elsewhere
Classic decontamination13-gram overlap checks (GPT-3 era)
Live benchmarksLiveBench (monthly), LiveCodeBench (dated problems)
MMLU (2020)GPT-3 43.9% โ†’ GPT-4 86.4% in under three years
FrontierMath (2024)50-problem holdout kept from the funder

2. How test questions leak

Nobody needs to cheat for contamination to happen. Pre-training data is a large slice of the public web (Data Layer Part 2), and the web contains benchmark questions: in the original GitHub repositories, in papers that quote examples, in blog posts and tutorials, in forum threads where people discuss answers, and in other datasets that copied them. Coding benchmarks built from public GitHub issues are especially exposed, because the issue, the fix and the discussion all sit in public repositories.

VerbatimThe exact question and answer are in the training data. Easiest to detect with text overlap.
Paraphrased or translatedThe same problem in different words or another language. Overlap checks miss it.
Solution leakageThe question is unseen, but the answer (a patch, a proof, a worked solution) circulated publicly.
Indirect, via tuningSynthetic data or preference data generated by other models that were themselves exposed to the test (Data Layer Part 4).

3. The evidence

GSM1k: a fresh copy of a maths testScale AI wrote 1,000 new grade-school maths problems matched in style and difficulty to the widely used GSM8K. Some model families scored noticeably worse on the new problems: up to 13% lower in the first version of the paper and up to 8% in the NeurIPS 2024 version. Frontier models from Google, OpenAI and Anthropic showed little sign of overfitting. Models that were more likely to reproduce GSM8K examples tended to have bigger gaps. GSM1k itself was kept private so it would not leak in turn.
The SWE-Bench IllusionResearchers asked models to name the buggy file in a GitHub issue from the issue text alone, with no access to the code. State-of-the-art models did so up to 76% of the time on SWE-Bench, but at most 53% on comparable repositories outside the benchmark. They also found up to 35% consecutive 5-gram overlap with reference fixes on SWE-Bench Verified, against up to 18% on other coding benchmarks. Their conclusion: some of the gains may come from memory rather than problem-solving.
FrontierMath and who sees the answersOpenAI commissioned and funded FrontierMath, a set of very hard maths problems built by Epoch AI, and has access to most problems and solutions; the funding was disclosed only in December 2024, which drew criticism. Epoch keeps a 50-problem holdout whose solutions OpenAI does not receive, so scores can be checked on problems the developer has never seen.

These are not accusations of cheating. Contamination is mostly an accident of training on the web, and the same models often still reason well on new problems, as the GSM1k authors stress. The issue is what a score means: without clean tests, a benchmark number mixes skill with memory in unknown proportions.

4. Remembering vs reasoning: a simulator

How much of a score is memory?

Our model: a model really solves a share of problems; on contaminated problems it can also recall the answer. Change the inputs and compare the reported score with the score on a fresh test.

score on the public benchmark
score on a fresh copy (like GSM1k)

5. Catching it

Five detection methods

Tap one. They need different levels of access to the model and its training data.

6. Tests wear out

Even a perfectly clean test stops being useful once the best models answer nearly every question correctly; remaining errors are often mislabelled questions. The useful life of a benchmark has shortened as models improve faster.

Benchmark lifetimes: top score from launch

Selected data points from the sources below; hover or tap a dot. Lines join points; they are not measured curves.

MMLU took about three years to go from GPT-3's 43.9% to GPT-4's 86.4%; SWE-bench went from 4.4% to 71.7% in one year.

When a test saturates, the field builds a harder one: MMLU gave way to MMLU-Pro and GPQA; GSM8K to competition maths; SWE-bench to SWE-bench Verified and harder agent suites; and in January 2025 a consortium released Humanity's Last Exam, written by experts specifically to resist saturation. By mid-2026 secondary trackers reported top text-only scores in the mid-40s per cent without tools and above 60% with tools โš ๏ธ, so even that test is moving fast. Our benchmark landscape lists the current generation.

7. Defences

DefenceHow it worksExampleWeakness
DecontaminationRemove training documents that overlap with test sets before training13-gram overlap filters reported for GPT-3 and later modelsMisses paraphrases and translations
Canary stringsA unique marker string in benchmark files so crawlers and filters can drop themBIG-bench's canary GUIDOnly works if training pipelines respect it
Private holdoutsKeep a share of questions secret, release later or neverGSM1k; FrontierMath's 50-problem holdoutTrust in whoever holds the secret set; less reproducible
Live, dated testsAdd new questions continually and score models only on items after their training cutoffLiveBench (refreshed monthly); LiveCodeBench (contest problems with release dates)Costly to maintain; question difficulty drifts over time
Functional variantsGenerate new versions of each problem (new numbers, new code) at test timeTemplated maths and code variantsVariants may be easier or harder than originals
Agent tasks in fresh environmentsScore what a model does in a new environment rather than an answer it could recallSee Part 5Expensive and slow to run

8. The series

Inside the Evaluation Layer: seven parts

VerdictBenchmark numbers are a mix of skill, memory and how long the test has been around. The best current practice treats any public static benchmark as provisional: check it against a fresh copy or a dated subset, prefer tests where answers are hard to recall (live problems, private holdouts, agent tasks in new environments), and expect any test to need replacing within a couple of years. The next problem is what to do when there is no fixed answer at all and another model has to do the grading: Part 3.

9. Reading the numbers

Primary / widely confirmed
GSM1k (arXiv 2405.00332): drops up to 13% (v1) / 8% (NeurIPS 2024 version); frontier families minimal overfitting; correlation with likelihood of generating GSM8K. SWE-Bench Illusion (arXiv 2506.12286): up to 76% vs up to 53%; 5-gram overlap up to 35% vs 18%. FrontierMath: OpenAI funding disclosed Dec 2024; 50-problem holdout (Epoch AI). LiveBench (2024, monthly updates); LiveCodeBench (problems dated May 2023โ€“May 2024 in its first release). MMLU: GPT-3 43.9% (2020), GPT-4 86.4% (2023). SWE-bench 4.4% โ†’ 71.7% (AI Index 2025).
Reported โš ๏ธ
Humanity's Last Exam scores in 2026 (secondary trackers; tool and no-tool settings differ); HLE launch-time top score of roughly 9% (early 2025).
Our illustrations
Memorisation simulator; lifetime chart lines between data points.