AGI Benchmarks: How the Field Tries to Measure the Thing It Can't Yet Define

You Can't Benchmark What You Haven't Defined

Every capability benchmark in this project's earlier Benchmark Landscape article measures something concrete: can this model solve a competition math problem, patch a real GitHub issue, click the right pixel on a screenshot. AGI benchmarks are a different animal entirely — they are attempts to measure progress toward a destination that the field has never agreed how to define. OpenAI defines AGI in economic terms. Google DeepMind defines it as a six-level spectrum of performance and generality. Anthropic largely avoids the term. A 2025 paper with 33 co-authors proposed yet another definition grounded in human cognitive-science theory. Every AGI benchmark covered in this article is, underneath its methodology, a bet on which of these competing definitions matters most — and that is worth understanding before trusting any single score.

Four Labs, Four Definitions

The Last Exam Before Knowledge Testing Runs Out

HLE's own creators are explicit that a perfect score would not itself indicate AGI — it measures structured, closed-form academic knowledge recall and reasoning, not autonomy, creativity, or real-world task competence. This self-imposed caveat is unusually candid for a benchmark and is worth taking at face value: HLE answers "how much expert knowledge does this model have," not "is this AGI."

From Static Puzzles to Interactive Games

Of every benchmark in this article, ARC-AGI is the one most explicitly built around a specific theory of intelligence. François Chollet's 2019 paper "On the Measure of Intelligence" argues that intelligence should be measured as skill-acquisition efficiency relative to prior experience — not the possession of any particular skill, however impressive, but the rate at which a system can acquire a new skill from limited exposure. ARC-AGI's puzzle-grid tasks were designed so that each one is trivial for a human seeing it for the first time, yet resistant to the kind of large-scale pattern memorization that lets language models ace ordinary knowledge benchmarks.

"AGI" Redefined as a Duration, Not a Score

This site's Benchmark Landscape article already covered METR's headline finding — the length of task an AI agent can complete autonomously at 50% reliability has roughly doubled every seven months since 2019, with evidence of acceleration to roughly four months more recently. What makes this research distinctly AGI-relevant, rather than just another capability metric, is METR's own explicit framing of a "one-month AGI" threshold: a system that can outperform humans given roughly 167 working hours — about one month of full-time work — chosen specifically because that is roughly the period a new human hire needs to onboard into a role and begin producing independent value. A system clearing that bar would, by this framing, be matching a genuinely human capacity for sustained, high-context, unsupervised contribution — an operational definition of general capability that sidesteps the exam-question approach entirely in favor of a duration.

Can the AI Do AI Research?

A distinct and safety-relevant category tests whether AI systems can perform the actual work of machine learning research and engineering — directly bearing on recursive self-improvement discourse, since a system that can meaningfully accelerate its own successor's development crosses a threshold most AGI definitions treat as significant regardless of how it performs on exam-style questions.

MLE-bench, published by OpenAI, takes a complementary approach: 75 real Kaggle machine-learning competitions with established human leaderboard baselines. The best-performing setup found (an o1-preview model paired with an agent scaffold called AIDE) achieved a Kaggle bronze-medal-equivalent result in roughly 17% of the competitions tested — genuine competence, but still a minority of cases, not the dominant outcome.

One Score to Rule Them All?

Epoch AI's Capabilities Index attempts to solve the fragmentation problem directly: it combines scores across more than 50 individual benchmarks onto a single difficulty-anchored scale, using a statistical approach related to item-response theory, where models tested across many benchmarks help establish each benchmark's relative difficulty against the others. Epoch reports a strong internal fit (R² around 0.86–0.87) for the resulting composite scale.

Passed, and Mostly Retired Anyway

A pre-registered, randomized controlled study from UC San Diego tested a modern three-party Turing Test setup: five-minute conversations, with a human judge trying to distinguish an AI from a real human interlocutor. With a persona-style system prompt, GPT-4.5 was judged to be the human party 73% of the time — exceeding chance, and even exceeding the rate at which the actual human confederate was correctly identified as human. Without persona prompting, the same model's rate dropped sharply to 36%, and GPT-4o (no persona) scored only 21% — showing the result depends heavily on prompting technique, not a fixed underlying capability.

The Papers That Critique the Whole Ecosystem

Beyond individual benchmarks, a growing body of academic meta-research examines whether the AI benchmarking ecosystem itself is trustworthy. Several papers explicitly invoke Goodhart's Law — the principle that once a measure becomes a target, it stops being a good measure — as the central organizing critique of how labs report and compete on benchmark scores.

The Fragmentation Is Structural, Not Temporary

This project's Benchmark Landscape article identified saturation as the default lifecycle of every capability benchmark — introduced, adopted, saturated, replaced. AGI benchmarks show the same pattern at a higher level of abstraction: each new attempt (HLE, ARC-AGI-2, ARC-AGI-3) exists specifically because its predecessor stopped discriminating between top models, and each new attempt will very likely face the same fate eventually. The deeper reason is not a temporary measurement gap that better engineering will close — it is that "AGI" itself has never been reduced to a single operational definition the field agrees on, so no benchmark built around any one definition can claim to have captured the whole concept. A benchmark built around Chollet's skill-acquisition-efficiency theory (ARC-AGI) will always look incomplete to someone who thinks AGI is fundamentally about economic value (OpenAI's definition), and vice versa.

Four Recurring Criteria From the 2025–2026 Literature

⚠️ What's Missing or Uncertain

🔗 Reference Links

🎥 Recommended Videos

🧭 Closing — Track the Trend Lines, Not the Finish Line