Grading Agents: Long Tasks, Live Environments and Why Reliability Beats a Single Score
A chatbot answers a question. An agent does a job: it reads a codebase, runs commands, clicks through a desktop, calls tools, talks to a customer, and only at the end produces a result. Testing that is harder in every way. Each task needs a working environment, a check that the job was really done, many steps of compute, and repeated runs, because an agent that succeeds once may fail the next time. This part covers the main agent benchmarks, how they score, the bugs that made some of them misleading, and METR's measure of how long a task an AI can complete.
1. Case file
2. Anatomy of an agent evaluation
Each of these can go wrong independently, which is why agent scores need more context than a single percentage: which harness, how many attempts, at what cost, and checked how.
3. The main agent benchmarks
Six tests, six kinds of job
Tap one.
4. Reliability: pass@k vs pass^k
Most leaderboards report pass@1: the share of tasks solved on one attempt, averaged. Coding papers often report pass@k: solved in at least one of k attempts, which is generous. The ฯ-bench authors (Sierra, 2024) argued that a customer-service agent must succeed every time, and introduced pass^k: the chance an agent solves the same task in all k independent attempts. GPT-4o solved fewer than half of ฯ-bench tasks, and its pass^8 in the retail domain was under 25%.
Same agent, three numbers
If an agent succeeds on a task with probability p each time, independently. Our calculator.
5. Time horizons: how long a task can an AI finish?
METR's 2025 paper Measuring AI Ability to Complete Long Tasks timed skilled people on a set of software and research tasks, from seconds to many hours, then asked at what human task length a model succeeds 50% of the time. That "50% time horizon" doubled roughly every seven months from 2019 to early 2025, possibly faster in 2024. Secondary summaries of METR's 2026 reports put the public frontier at around 12 hours in early 2026 and the strongest models at 16 hours or more, the upper limit METR says its current task suite can measure reliably โ ๏ธ.
Extrapolate the trend
Our extrapolation from a ~50-minute horizon in early 2025 (reported for a model of that time) at a chosen doubling time. Dots: early 2025 (~50 min), ~12 h public frontier and โฅ16 h strongest models in 2026 (reported โ ๏ธ).
Two cautions from METR itself: the horizon is measured on its particular task suite (mostly software and research), and a 50% horizon says nothing about reliability; the 80% horizon is several times shorter. We cover what the trend implies for automating AI research in AI automating AI research and the RSI Evolution Map.
6. When the benchmark is wrong
Agent benchmarks are software, and software has bugs. A 2025 study that proposed an Agentic Benchmark Checklist (NeurIPS 2025, Datasets and Benchmarks) audited popular benchmarks and found that flawed task setups and checkers could push estimated scores off by up to 100% in relative terms:
Applying the checklist to one cybersecurity benchmark reduced overestimation by 33%. Benchmark maintainers now re-release cleaned versions (Terminal-Bench 2.0 dropped tasks that could not be reproduced; HAL reports "clean" and "original" ฯ-bench variants).
7. Cost and efficiency
Agent evaluation is the most expensive kind (Part 1): HAL spent about $40,000 on 21,730 rollouts. Leaderboards increasingly plot accuracy against cost per task, because a slightly more accurate agent that costs ten times more is often the worse choice. Efficiency matters too: an OSWorld follow-up found even the best computer-use agents took 1.4โ2.7ร more steps than necessary. Cost and step counts belong next to every agent score.
8. The series
Inside the Evaluation Layer: seven parts
9. Reading the numbers
- Primary / widely confirmed
- SWE-bench Verified: 500 tasks, 93 developers (OpenAI, Aug 2024). Terminal-Bench 2.0: 89 tasks, top <65%, small models ~15% (Stanford and Laude Institute, Nov 2025). OSWorld (arXiv 2404.07972): 369 tasks, humans 72.36%, best model 12.24% at launch. ฯ-bench (arXiv 2406.12045): GPT-4o <50% of tasks; retail pass^8 <25%. METR (arXiv 2503.14499): doubling ~7 months. ABC (arXiv 2507.02825): up to 100% relative misestimation; empty response 38% on ฯ-bench airline; 33% reduction on CVE-Bench. HAL: ~$40,000. OSWorld-Human: 1.4โ2.7ร extra steps.
- Reported โ ๏ธ
- METR 2026 horizons (~12 h public frontier early 2026; โฅ16 h strongest models) from secondary summaries; the ~50-minute early-2025 base figure; environment-rot example figures.
- Our illustrations
- Reliability calculator (assumes independent attempts); horizon extrapolation.
- Sources
- OpenAI: Introducing SWE-bench Verified
- Terminal-Bench 2.0 ยท Snorkel: Terminal-Bench 2.0
- OSWorld ยท OSWorld-Human
- ฯ-bench
- METR: Measuring AI Ability to Complete Long Tasks ยท METR (2026 summary)
- Establishing Best Practices for Building Rigorous Agentic Benchmarks (ABC)
- Holistic Agent Leaderboard ยท Evaluating Agents (on this site)