Home โ€บ Blog โ€บ Agent and long-task evaluations
Inside the Evaluation Layer ยท Part 5 ยท Agents and long tasks

Grading Agents: Long Tasks, Live Environments and Why Reliability Beats a Single Score

A chatbot answers a question. An agent does a job: it reads a codebase, runs commands, clicks through a desktop, calls tools, talks to a customer, and only at the end produces a result. Testing that is harder in every way. Each task needs a working environment, a check that the job was really done, many steps of compute, and repeated runs, because an agent that succeeds once may fail the next time. This part covers the main agent benchmarks, how they score, the bugs that made some of them misleading, and METR's measure of how long a task an AI can complete.

FL
FrontierAGI Team
โš ๏ธ How to read this (as of 28 Oct 2026) Benchmark designs and launch results come from the original papers; leaderboards move weekly, so we quote launch-time or clearly dated figures. 2026 time-horizon figures are from secondary summaries of METR's reports and are marked โš ๏ธ. The reliability calculator and horizon extrapolation are our illustrations. Disclosure: Claude, made by Anthropic, helps write this site; an Anthropic model is among those cited in METR's 2026 figures.

1. Case file

๐Ÿค–
Agent evaluation, October 2026Part 5 of the evaluation layer
SWE-bench Verified (Aug 2024)500 tasks checked by 93 developers
Terminal-Bench 2.0 (Nov 2025)89 hard, containerised terminal tasks; top agents <65% at launch
OSWorld (2024)369 desktop tasks; humans 72.4%, best model 12.2% at launch
ฯ„-bench (2024)GPT-4o pass^8 in retail <25%
METR time horizons (2025)Task length doubling about every 7 months since 2019
Benchmark bugs (ABC, 2025)An empty answer "passed" 38% of ฯ„-bench airline tasks

2. Anatomy of an agent evaluation

1 ยท EnvironmentA container, virtual machine or simulated service with the starting state: a repository at a commit, a desktop with files, a mock airline database. Must be reproducible and isolated (sandboxing).
2 ยท Task and harnessThe instruction, the tools the agent may use, and the scaffold that runs the loop (harness engineering). The same model scores differently in different harnesses.
3 ยท CheckerHidden unit tests, a script that inspects the final state, or a rubric graded by a model or person. The checker defines what "done" means.
4 ยท Repeats and costsSeveral runs per task to measure consistency, with tokens, time and steps logged.

Each of these can go wrong independently, which is why agent scores need more context than a single percentage: which harness, how many attempts, at what cost, and checked how.

3. The main agent benchmarks

Six tests, six kinds of job

Tap one.

4. Reliability: pass@k vs pass^k

Most leaderboards report pass@1: the share of tasks solved on one attempt, averaged. Coding papers often report pass@k: solved in at least one of k attempts, which is generous. The ฯ„-bench authors (Sierra, 2024) argued that a customer-service agent must succeed every time, and introduced pass^k: the chance an agent solves the same task in all k independent attempts. GPT-4o solved fewer than half of ฯ„-bench tasks, and its pass^8 in the retail domain was under 25%.

Same agent, three numbers

If an agent succeeds on a task with probability p each time, independently. Our calculator.

pass@1 (one try)
pass@k (any of k)
pass^k (all k)

5. Time horizons: how long a task can an AI finish?

METR's 2025 paper Measuring AI Ability to Complete Long Tasks timed skilled people on a set of software and research tasks, from seconds to many hours, then asked at what human task length a model succeeds 50% of the time. That "50% time horizon" doubled roughly every seven months from 2019 to early 2025, possibly faster in 2024. Secondary summaries of METR's 2026 reports put the public frontier at around 12 hours in early 2026 and the strongest models at 16 hours or more, the upper limit METR says its current task suite can measure reliably โš ๏ธ.

Extrapolate the trend

Our extrapolation from a ~50-minute horizon in early 2025 (reported for a model of that time) at a chosen doubling time. Dots: early 2025 (~50 min), ~12 h public frontier and โ‰ฅ16 h strongest models in 2026 (reported โš ๏ธ).

Two cautions from METR itself: the horizon is measured on its particular task suite (mostly software and research), and a 50% horizon says nothing about reliability; the 80% horizon is several times shorter. We cover what the trend implies for automating AI research in AI automating AI research and the RSI Evolution Map.

6. When the benchmark is wrong

Agent benchmarks are software, and software has bugs. A 2025 study that proposed an Agentic Benchmark Checklist (NeurIPS 2025, Datasets and Benchmarks) audited popular benchmarks and found that flawed task setups and checkers could push estimated scores off by up to 100% in relative terms:

Impossible tasks that reward doing nothingOn ฯ„-bench airline, some tasks are impossible (for example, refunding a non-refundable ticket), so the correct outcome is no change. A trivial agent that returns an empty response was marked successful on 38% of tasks, beating a GPT-4o agent.
Tests too weak to catch wrong fixesOn SWE-bench Verified, passing the provided tests does not always mean the issue is fixed; some patches pass while being wrong.
Answers reachable from inside the sandboxWhere agents are not fully isolated from the ground truth, they can find and copy the answer.
Environments that rotLive websites change; checkers that rely on page structure break and under-count success โš ๏ธ.

Applying the checklist to one cybersecurity benchmark reduced overestimation by 33%. Benchmark maintainers now re-release cleaned versions (Terminal-Bench 2.0 dropped tasks that could not be reproduced; HAL reports "clean" and "original" ฯ„-bench variants).

7. Cost and efficiency

Agent evaluation is the most expensive kind (Part 1): HAL spent about $40,000 on 21,730 rollouts. Leaderboards increasingly plot accuracy against cost per task, because a slightly more accurate agent that costs ten times more is often the worse choice. Efficiency matters too: an OSWorld follow-up found even the best computer-use agents took 1.4โ€“2.7ร— more steps than necessary. Cost and step counts belong next to every agent score.

8. The series

Inside the Evaluation Layer: seven parts

VerdictAgent evaluation is where the frontier is measured now, and where measurement is weakest: environments are costly, checkers can be wrong, and one-shot scores hide unreliability. Read agent numbers with four questions: which harness, how many attempts (pass@1, pass@k or pass^k), how was success checked, and at what cost. The same environments are used to test dangerous capabilities, such as autonomous cyber attacks or self-replication: Part 6.

9. Reading the numbers

Primary / widely confirmed
SWE-bench Verified: 500 tasks, 93 developers (OpenAI, Aug 2024). Terminal-Bench 2.0: 89 tasks, top <65%, small models ~15% (Stanford and Laude Institute, Nov 2025). OSWorld (arXiv 2404.07972): 369 tasks, humans 72.36%, best model 12.24% at launch. ฯ„-bench (arXiv 2406.12045): GPT-4o <50% of tasks; retail pass^8 <25%. METR (arXiv 2503.14499): doubling ~7 months. ABC (arXiv 2507.02825): up to 100% relative misestimation; empty response 38% on ฯ„-bench airline; 33% reduction on CVE-Bench. HAL: ~$40,000. OSWorld-Human: 1.4โ€“2.7ร— extra steps.
Reported โš ๏ธ
METR 2026 horizons (~12 h public frontier early 2026; โ‰ฅ16 h strongest models) from secondary summaries; the ~50-minute early-2025 base figure; environment-rot example figures.
Our illustrations
Reliability calculator (assumes independent attempts); horizon extrapolation.