Reference Series · Capability Benchmarks 📊

The Frontier AI Benchmark Landscape: How Every Capability Gets Measured

Every model this project has covered — from GPT-1 to Mistral 3 — gets judged against a sprawling, fragmented set of benchmarks, no two labs reporting quite the same ones. This article maps that landscape: what each major benchmark actually measures, which have become saturated and stopped discriminating between top models, which methodology controversies have shaken confidence in self-reported scores, and the one metric — how long a task an AI can complete unsupervised — that may matter more than any single capability score. It is written as a reference other articles on this site, including a forthcoming piece on "computer use," can cite back to.

FrontierAGI Team

September 17, 2026 · 55 min read

There Is No IQ Test for AI

Every lineage article on this site quotes benchmark numbers — GPT-6 Astra's classification, Claude's ASL activations, Gemini's Frontier Safety Framework thresholds, DeepSeek's disputed cost-to-capability ratio. None of those numbers come from one shared test. They come from a fragmented ecosystem of dozens of benchmarks, each measuring a narrow slice of capability, each with its own saturation point, contamination risk, and incentive for labs to report selectively. This article is the map underneath those numbers — what each major benchmark actually tests, which ones stopped being useful once every frontier model started scoring near-perfectly, and which recent controversies should make any reader skeptical of a single self-reported score, including the ones this site's own articles have cited.

How to Read a Score

Before Any Number: Four Questions

1. Is it contaminated?

Public benchmarks eventually leak into training data. A model can score well on a benchmark by having memorized it, not by having the capability the benchmark claims to measure. This is why held-out benchmarks like FrontierMath and Humanity's Last Exam exist — and why even they have run into disputes over who gets early access.

2. Is it saturated?

A benchmark stops being useful once every frontier model scores within a few points of the ceiling — differences at that point are statistical noise, not real capability gaps. MMLU, GSM8K, and HumanEval have all reached this state; each was replaced by a harder successor.

3. Who reported it?

A lab's own blog post claiming a state-of-the-art score is a marketing claim until an independent party (the benchmark's own maintainers, or a third-party tracker) reproduces it. This article distinguishes primary-sourced claims from aggregator-sourced ones throughout, and where the two disagree, says so rather than picking one.

4. Does it predict real-world reliability?

A benchmark score is a controlled, single-shot measurement. Real deployment involves ambiguous instructions, adversarial inputs, and long task chains where small per-step error rates compound. The gap between benchmark performance and production reliability is a recurring theme across every category below.

Reasoning & Knowledge

From MMLU's Saturation to ARC-AGI's Resistance

MMLU / MMLU-Pro2020 / 2024

What It Measures: Broad academic knowledge across roughly 57 subjects (original MMLU), later a harder successor (MMLU-Pro) with more reasoning-heavy questions and more answer choices per question.
Saturation Status: MMLU is the most-cited example of a saturated benchmark — frontier models now consistently score above 88%, with top-of-leaderboard differences (90–93%) small enough to be treated as statistical noise rather than a real capability ranking, per Stanford HAI's 2026 AI Index.

2023GPQA Diamond

198 PhD-level science questions deliberately written to be resistant to simple web lookup — "Google-proof" by design. Still meaningfully discriminates between models, unlike MMLU, though this article does not cite specific 2026 point-scores here, since the figures found in research came only from aggregator sites with no primary-lab confirmation.

Dec 2024 / 2025ARC-AGI-1 and ARC-AGI-2

Designed by François Chollet specifically to test abstraction and generalization rather than memorized knowledge — puzzle-like visual grid problems a model has never seen a similar version of. OpenAI's o3 scored a widely reported ~87.5% on the ARC-AGI-1 semi-private evaluation set in December 2024, a genuine milestone at the time. ARC-AGI-2, a deliberately harder successor, followed in 2025 specifically because ARC-AGI-1 was approaching saturation for top reasoning models.

Why This Article Avoids Specific 2026 Point-Scores Research for this article turned up numerous 2026-dated GPQA and ARC-AGI-2 leaderboard figures attributed to model names (including several this site's own writers could not confirm exist via any primary lab announcement). Rather than repeat unverifiable numbers, this article names the benchmarks and their design intent, and directs readers to each benchmark's own official leaderboard for current standings.

Mathematics

A Difficulty Ladder That Keeps Climbing

GSM8K (grade-school word problems) and MATH (competition-style problems) are both now considered saturated at the frontier level, superseded by AIME-tier competition problems and, above that, Epoch AI's FrontierMath — a held-out benchmark built specifically to resist the fate of every math benchmark before it.

FrontierMathNov 2024, revised Jun 2026

Structure: Four difficulty tiers — Tiers 1 through 3 contain 295 problems in total, Tier 4 contains 43 exceptionally hard research-level problems. Scoring is all-or-nothing per problem, built and maintained by Epoch AI as a genuinely held-out set specifically to resist contamination.
Quality Control: In June 2026, Epoch AI released a corrected version after discovering errors in roughly 42% of the original problem set — a significant, self-disclosed data-quality finding about how hard it is to build a contamination-resistant benchmark correctly the first time.

The FrontierMath–OpenAI Access Controversy OpenAI funded FrontierMath's creation and, per Epoch AI's own later disclosures, retained ownership of and access to the original 300 problems and their solutions — a fact Epoch AI did not disclose until the benchmark's final published paper, not in earlier versions. Several contributing mathematicians said afterward they were unaware OpenAI would have exclusive access and would not have contributed under those terms had they known. Epoch AI's remediation was to withhold a specific subset of Tier 4 problems from OpenAI going forward, to preserve at least one uncontaminated evaluation slice. This is one of the clearest documented cases of a benchmark's independence being compromised by the same lab it was built to evaluate.

Coding

From "Write a Function" to "Fix a Real GitHub Issue"

HumanEval and MBPP — early function-level code generation benchmarks — are both considered saturated. The current standard for agentic coding capability is SWE-bench Verified: 500 real GitHub issues drawn from popular open-source Python repositories, requiring a model to understand an existing codebase, locate the relevant files, and produce a patch that passes the project's actual test suite.

A Contamination Concern Serious Enough to Change Reporting Practice Reporting in early 2026 indicated OpenAI stopped publishing SWE-bench Verified scores and began recommending a successor, SWE-bench Pro, over concern that some of the 500 tasks' "gold patches" existed verbatim in pretraining data before the benchmark's release — meaning a model could reproduce the correct fix from memorization rather than genuine code reasoning. This article treats the underlying concern as credible and worth naming, while flagging that it comes from secondary reporting rather than a direct OpenAI statement.

Away from agentic coding, competitive programming has its own well-documented and independently verifiable milestone: OpenAI's own published research (arXiv:2502.06807, February 2025) reports its o3 model achieving a Codeforces rating of 2724 — the 99.8th percentile of ranked human competitive programmers — up from a predecessor model's 1673 rating a few months earlier, alongside a gold-medal-level score at the 2024 International Olympiad in Informatics. This is one of the more rigorously documented capability jumps in this entire landscape, because Codeforces ratings are an external, human-calibrated scale a lab cannot simply redefine in its own favor.

Agentic Tool-Use

Measuring Chains of Actions, Not Single Answers

GAIA

450 questions requiring reasoning, multimodal understanding, web browsing, and tool use in combination, at varying levels of required autonomy. Current leaderboard standings found in research for this article were inconsistent across sources by tens of percentage points — this article does not cite a specific current leader for GAIA and recommends checking the benchmark's own official leaderboard directly.

τ-bench (tau-bench)

Simulated business tool-use scenarios (e.g., customer-service workflows), testing whether an agent can follow policy constraints correctly across a multi-turn interaction with simulated tools, not just produce a plausible-sounding single response.

BrowseComp

OpenAI's deep-research benchmark: 1,266 questions whose answers require persistent, multi-step web navigation to locate deliberately hard-to-find, entangled information — designed to evaluate genuine browsing agents rather than single-search lookup.

AgentBench

A broader multi-environment suite (databases, operating systems, web shopping, and more) testing an LLM's capacity to act as an agent across diverse simulated environments rather than one specific domain.

Computer Use

The Screen-and-Mouse Benchmarks

A distinct family of benchmarks evaluates models that perceive a screen and act on it directly with mouse and keyboard, rather than calling structured APIs — the capability this site covers in a dedicated companion article. OSWorld runs agents against real Ubuntu, Windows, and macOS environments across roughly 369 tasks spanning applications, the web, and OS-level operations. ScreenSpot and its harder variant ScreenSpot-Pro measure raw click-grounding accuracy — can a model correctly identify where to click for a described UI element, especially in dense, professional software. WebVoyager and Mind2Web test end-to-end web-navigation task completion. AndroidWorld extends the same evaluation philosophy to 116 tasks across 20 real Android apps, judged by inspecting the device's final state rather than the agent's own report of success.

Long Context

Advertised Context vs. Effective Context

Needle-in-a-haystack testing — inserting a single fact deep inside a long document and checking whether a model retrieves it — was the original long-context evaluation method, and it is now considered too easy on its own: a model can find a literal, isolated fact while still failing to reason across information spread throughout a long context. RULER and BABILong extend the same idea to multi-hop, reasoning-under-length tasks that better approximate real long-document use.

The Advertised-vs-Effective Gap Several labs advertise context windows in the millions of tokens — Google's Gemini line has offered up to 2 million tokens via API since 2024, and later 2026 marketing for some models references windows as large as 10 million tokens. Independent long-context evaluation, however, consistently shows that retrieval and reasoning quality degrade well before a model's advertised maximum — a gap this article flags explicitly rather than treating the advertised number as the usable one.

Multimodal

Vision, Video, and Audio Get Their Own Ladders

MMMU / MMMU-Pro

Massive Multi-discipline Multimodal Understanding — college-level questions requiring both image interpretation and subject reasoning together. The harder MMMU-Pro variant has reportedly approached saturation among frontier multimodal models by 2026, following the same trajectory as MMLU before it.

MathVista

Visual mathematical reasoning — reading a diagram, chart, or geometric figure and solving a math problem based on it, combining visual perception with quantitative reasoning.

Video-MME

Evaluates understanding of video content across varying lengths and question types, a newer and less saturated category than static-image multimodal benchmarks.

AIR-Bench / MMAU

Audio-understanding benchmarks, testing comprehension of speech, environmental sound, and music content — a category that has received comparatively less standardization than vision benchmarks.

Instruction & Preference

When Humans, Not Answer Keys, Are the Judge

IFEval tests precise instruction-following — can a model obey exact formatting or constraint instructions ("respond in exactly three bullet points," "do not use the letter e") rather than just produce a generally good answer. MT-Bench and AlpacaEval use automated or human-judged preference scoring across multi-turn conversations. The most consequential preference benchmark, however, is the head-to-head human voting platform this site's Meta and xAI lineage articles have already discussed in the context of controversy.

2023 → Jan 2026Chatbot Arena → LMArena → Arena

Launched in 2023 as Chatbot Arena, the platform became an independent company and rebranded to LMArena, then rebranded again on January 28, 2026 to simply "Arena," migrating its domain — the same underlying team, methodology, and leaderboard throughout each name change. Beyond the Llama 4 Maverick incident covered in this site's Meta lineage article, at least one academic paper has separately demonstrated a "vote-rigging" vulnerability, showing that an attacker controlling only a few hundred votes could meaningfully shift rankings on the platform — a documented methodological weakness in human-preference benchmarking generally, not specific to any one lab's conduct.

Safety & Robustness

The Category With No Shared Leaderboard

HarmBench, JailbreakBench, ToxiGen, TruthfulQA, and bias-focused suites like BBQ and WinoBias remain the standard named academic benchmarks for harm resistance, jailbreak robustness, toxic-content avoidance, truthfulness under pressure to confabulate, and demographic bias respectively. Unlike every capability category above, this project found no evidence of a unified, cross-lab safety leaderboard analogous to OSWorld or SWE-bench Verified — each lab instead runs its own internal red-teaming and evaluation suite, disclosed (if at all) through its own safety framework documents rather than a shared public scoreboard. This asymmetry — rich, competitive, standardized benchmarking for capability, and fragmented, lab-controlled evaluation for safety — is itself worth noting as a structural feature of the current landscape.

Long-Horizon Autonomy

The Metric That May Matter Most

METR's Time-Horizon ResearchMar 2025, updated Jan 2026

What It Measures: Rather than a pass/fail score on a fixed task set, METR measures the length of task (calibrated against how long a skilled human would take) that an AI agent can complete autonomously at 50% reliability — directly answering "how long can you trust this to work unsupervised," which no single-turn benchmark captures.
The Headline Finding: From 2019 to 2025, the length of task AI agents can reliably complete has been doubling roughly every seven months — a trend line, not a single score, and one directly relevant to how labs set autonomous-capability thresholds in their own safety frameworks.
2026 Update: METR released an updated methodology ("Time Horizon 1.1") on January 29, 2026, expanding the task suite from 170 to 228 tasks and roughly doubling the number of 8-hour-plus tasks included, alongside a separate note explicitly discussing the metric's own limitations.

A Widely Discussed but Unconfirmed Acceleration Claim Multiple secondary analyses of METR's data claim the doubling period has shortened considerably since 2023 — some citing figures as short as roughly four months, or even three months using only 2024-onward data. This article treats that acceleration narrative as a plausible, widely discussed interpretation of METR's published data rather than a headline claim METR itself has made in exactly those terms — readers should consult METR's own Time Horizon 1.1 publication directly for the underlying numbers before citing a specific doubling-time figure elsewhere.

Specialized Domains

Medicine, Law, and Beyond

MedQA and the broader MultiMedQA suite test medical licensing-exam-style knowledge and clinical reasoning. LegalBench covers a range of legal reasoning tasks, from issue-spotting to rule application. Finance-specific evaluation remains comparatively less standardized across labs than medicine or law — no single finance benchmark has achieved the same cross-lab adoption as SWE-bench Verified has for coding. These specialized domains matter disproportionately for enterprise adoption claims, since a lab citing a high MedQA or LegalBench score is implicitly making a claim about real-world professional deployment readiness that a general reasoning benchmark does not.

Multilingual

Beyond English-First Evaluation

MGSM extends grade-school math evaluation across multiple languages, testing whether reasoning capability transfers or degrades outside English. Belebele and FLORES evaluate reading comprehension and machine translation quality respectively across a wide range of languages, including many with limited training data availability — directly relevant to labs like Mistral (this site's Mistral lineage article covers its French/multilingual design emphasis) and DeepSeek that compete partly on non-English performance rather than English-benchmark leadership alone.

Cross-Cutting Patterns

Four Patterns That Repeat Across Every Category

Pattern 1 — Saturation Is the Default End State Every benchmark category above shows the same lifecycle: introduced as discriminating, adopted broadly, then saturated within a few generations of frontier models, forcing a harder successor. MMLU→MMLU-Pro, GSM8K→AIME→FrontierMath, and HumanEval→SWE-bench Verified all follow this exact pattern.

Pattern 2 — Self-Reported Scores Need Independent Verification The FrontierMath–OpenAI access controversy, the SWE-bench Verified contamination concern, and the Arena vote-rigging vulnerability are three separate, well-documented cases where a benchmark's integrity was compromised or disputed — not hypothetical risks, but events that already happened across three different benchmark categories.

Pattern 3 — No Single Number Captures "Capability" A model can lead on GPQA while trailing on SWE-bench Verified, lead on MMMU while trailing on OSWorld, or score well on a single-turn benchmark while failing at the long-horizon autonomy METR measures. This project's lineage articles have already shown labs emphasizing whichever benchmarks flatter their latest release — a pattern this landscape article makes structurally visible rather than benchmark-by-benchmark.

Pattern 4 — Safety Evaluation Lags Capability Evaluation The capability side of this landscape has converged on shared, competitive, third-party-tracked benchmarks. The safety side has not — each lab still largely grades its own homework, disclosed through its own framework rather than a shared scoreboard, a structural asymmetry this article treats as a genuine gap rather than a settled state of affairs.

Snapshot Table

Fourteen Categories, at a Glance

Category	Standard Benchmark(s)	Saturation Status	Notable Fact
General reasoning	MMLU-Pro, GPQA Diamond, ARC-AGI-2	MMLU saturated; successors not yet	o3 scored ~87.5% on ARC-AGI-1 in Dec 2024, prompting a harder ARC-AGI-2
Mathematics	AIME, FrontierMath	GSM8K/MATH saturated	FrontierMath found 42% of its own original problems contained errors
Coding	SWE-bench Verified, Codeforces rating	HumanEval saturated	o3 reached a 2724 Codeforces rating (99.8th percentile), per OpenAI's own paper
Agentic tool-use	GAIA, τ-bench, BrowseComp	Not saturated; leaderboards inconsistent	Reported GAIA standings vary by tens of points across trackers
Computer use / GUI	OSWorld, ScreenSpot-Pro, AndroidWorld	Not saturated	Google shut down its own Project Mariner in 2026, citing reliability limits
Long context	RULER, BABILong, needle-in-haystack	Advertised windows outpace verified effective ones	Some 2026 marketing cites 10M-token windows with no benchmark confirming quality at that length
Multimodal	MMMU-Pro, MathVista, Video-MME	MMMU-Pro reportedly nearing saturation	Video and audio evaluation remain less standardized than static-image tasks
Instruction & preference	IFEval, Arena (formerly LMArena)	Ongoing, human-judged	An academic paper demonstrated a vote-rigging vulnerability in Arena rankings
Safety & robustness	HarmBench, JailbreakBench, TruthfulQA	No shared cross-lab leaderboard exists	Each lab evaluates safety largely on its own terms, unlike capability benchmarks
Long-horizon autonomy	METR time-horizon methodology	An actively evolving trend line, not a fixed test	Autonomous task length has roughly doubled every ~7 months since 2019
Specialized domains	MedQA, LegalBench	Varies by domain; finance least standardized	Directly relevant to enterprise deployment claims, not just research prestige
Multilingual	MGSM, Belebele, FLORES	Under-benchmarked relative to English	Central to Mistral's and DeepSeek's competitive positioning specifically

⚠️ What's Missing or Uncertain

This article deliberately omits specific 2026 leaderboard point-scores for most benchmarks. Research conducted for this piece surfaced numerous numeric claims from aggregator sites that could not be verified against any primary lab or benchmark-maintainer source — several attributed to model names this site's writers could not confirm exist. Rather than risk publishing fabricated or hallucinated figures as fact, this article names benchmarks, their design intent, and well-documented structural facts (methodology controversies, saturation status, the METR trend line), and directs readers to each benchmark's own official leaderboard for current standings. The safety-benchmark section in particular reflects a genuine research gap — no unified cross-lab safety leaderboard was found to exist, but this absence itself was not exhaustively confirmed and may reflect incomplete research rather than certainty that no such effort exists anywhere.

🔗 Reference Links

🎥 Recommended Videos

But What Is a Neural Network?

3Blue1Brown — the underlying architecture every benchmark in this article is ultimately testing

Attention Is All You Need — Paper Explained

Yannic Kilcher — foundational context for why every model on every leaderboard in this article shares one architectural lineage

🧭 Closing — Read the Benchmark, Not Just the Score

🎯 The Bottom Line

There is no single "AI IQ test," and there will not be one — capability is genuinely multi-dimensional, and every attempt to compress it into one leaderboard has produced a benchmark that saturates, gets gamed, or gets disputed within a couple of years. The right habit is not memorizing today's leading score, but knowing which benchmark is being cited, what it actually measures, whether it has saturated, and who reported the number. Applied consistently, that habit is also the best defense against exactly the kind of benchmark-gaming this project has already documented once, in the Llama 4 LMArena incident — and it is the standard this article has tried to hold itself to throughout.