The Evaluation Harness
Phase 3 asks the uncomfortable question: does Maestro-1 actually work, and how would anyone know? Gopala swaps the builder's hat for the evaluator's and builds a harness whose job is to say no when the answer is no. This part covers what to measure for an engineering-agent product, the four task pools, pass@k done correctly, contamination checks, why 500 tasks may not be enough to see a real improvement, the harness software, and the report that decides whether Maestro-1 ships.
pass_at_k.py, paired_bootstrap.py and contamination_check.py were run (real output below) on made-up counts, simulated results and sample texts. eval_task.py uses an external framework and was not executed for this article. Maestro-1's results are the scenario. Disclosure: Claude, made by Anthropic, helps write this site.
1. Mission log: weeks 12โ56
- The harness started in month 3 (Part 3's plan): small evaluations on Maestro-0 and every ablation (Part 9).
- Intermediate checkpoints scored every ~300B tokens (Part 10).
- Full harness on Maestro-1 vs Maestro-0 vs two open models; contamination audit; report written for the release gate (Part 14).
2. What to measure
- The product is an engineering agent, so the main question is task success.
- Given a real repository and an issue, does the agent produce a change that passes the project's hidden tests?
- This is the design of SWE-bench (Jimenez et al., 2023): real GitHub issues from Python projects, judged by the project's own tests.
- Public benchmarks are a common language, not the decision.
- SWE-bench Verified is a 500-task subset that OpenAI and the SWE-bench authors released in August 2024 after 93 developers screened 1,699 samples for problems.
- Useful for comparing with others; but popular benchmarks leak into training data and saturate (see the site's contamination and saturation article).
- Five things he measures.
- Task success on partners' private held-out tasks (the decision metric).
- Task success on public benchmarks (comparison).
- Regression suite: old skills still work (Part 11's gate).
- Cost and latency per task (Phase 4 economics).
- Safety behaviour (Part 13).
Evaluation questions, from cheapest to most decisive
Each level costs more and tells more.
3. Four task pools, kept apart
| Pool | Size (story) | Who sees it | Purpose |
|---|---|---|---|
| Training tasks | thousands | The loop (Part 11) | Rewards and training data |
| Development set | ~600 | Researchers | Daily iteration; may be overfit over time |
| Hidden test pool | ~1,000 | Only the verifier | The loop's reward and gate; rotated quarterly |
| Private release set | ~1,200 | Only the evaluator hat | Release decisions; never published, never trained on |
- The release set is the one that matters.
- It is built from partner tasks collected after the training data was frozen, so it cannot be in the training data.
- Access is logged; nobody on the training side sees individual tasks.
4. pass@k, done correctly
- Agents are random; one attempt per task gives a noisy number.
- Generate n attempts per task, count how many pass, and estimate the chance that one of k attempts passes.
- The obvious formula 1 โ (1 โ c/n)^k is biased; the Codex paper (Chen et al., 2021) gives an unbiased estimator.
"""pass_at_k.py โ the unbiased pass@k estimator (Chen et al., 2021, Codex paper).
Generate n samples per task, count c that pass the tests; the chance that
at least one of k random samples passes is 1 - C(n-c, k) / C(n, k).
Naively computing 1-(1-c/n)^k is biased. Counts below are made up.
"""
from math import comb
def pass_at_k(n, c, k):
if n - c < k:
return 1.0
return 1 - comb(n - c, k) / comb(n, k)
TASKS = [(20, 0), (20, 1), (20, 5), (20, 12), (20, 20)] # (samples, correct) per task
def main():
print(f"{'n':>3s} {'c':>3s} {'pass@1':>7s} {'pass@5':>7s} {'pass@10':>8s} {'naive@10':>9s}")
for n, c in TASKS:
naive = 1 - (1 - c / n) ** 10
print(f"{n:3d} {c:3d} {pass_at_k(n,c,1):7.3f} {pass_at_k(n,c,5):7.3f} {pass_at_k(n,c,10):8.3f} {naive:9.3f}")
for k in (1, 5, 10):
print(f"benchmark pass@{k}: {sum(pass_at_k(n, c, k) for n, c in TASKS) / len(TASKS):.3f}")
if __name__ == "__main__":
main()- Reading it.
- For a task solved once in 20 attempts, the naive formula gives 0.401 for pass@10 where the unbiased estimate is 0.500.
- Report pass@1 as the headline: a client gets one answer, not ten. pass@k is useful for RL (Part 11), where many attempts are drawn.
5. Contamination: did the answers leak?
- If test items appear in training data, scores measure memory, not skill.
- The GPT-3 paper checked 13-word overlaps between training data and benchmarks; long n-gram overlap is still the standard first check.
- Part 8 already removed known benchmarks from the blend; this check runs again on the final model's full data manifest.
"""contamination_check.py โ did benchmark items leak into training data?
Long word n-gram overlap (13-grams, as in the GPT-3 paper's check)
between each benchmark item and a training shard. Texts are written for
this article. Real runs index the 10T-token blend with hashed n-grams.
"""
N = 13
BENCH = {
"task-017": "write a function that returns the longest strictly increasing contiguous subsequence of a list of integers and its starting index",
"task-042": "given a log file with timestamps in mixed time zones convert every entry to utc and report the gaps longer than five minutes",
"task-108": "implement a rate limiter that allows at most n requests per rolling window of t seconds for each api key",
}
SHARD = ("forum post: homework help please. write a function that returns the longest strictly increasing contiguous "
"subsequence of a list of integers and its starting index, my code fails on empty lists. "
"another thread discusses how to implement a rate limiter for an api using a token bucket.")
def grams(t):
w = t.lower().replace(",", "").replace(".", "").split()
return {" ".join(w[i:i + N]) for i in range(len(w) - N + 1)}
def main():
sg = grams(SHARD)
for k, v in BENCH.items():
g = grams(v)
hit = len(g & sg)
print(f"{k}: {hit}/{len(g)} 13-grams found in training shard -> {'CONTAMINATED: remove from eval or from data' if hit else 'clean'}")
if __name__ == "__main__":
main()- What happens to a contaminated item.
- It is dropped from the release set, and the score is reported both ways for public benchmarks.
- Overlap checks miss paraphrases and translations, so the private release set (collected after the data freeze) remains the real protection.
6. Is the difference real?
- Two models on the same tasks: compare them task by task.
- A paired bootstrap resamples tasks and recomputes the difference each time; the middle 95% is the confidence interval.
- Pairing removes the noise from tasks that are simply easy or hard for both.
"""paired_bootstrap.py โ is Maestro-1 really better than Maestro-0?
Both models attempt the SAME tasks; a paired bootstrap resamples tasks
(not answers) to get a confidence interval for the difference. Then:
how many tasks are needed to detect a given improvement? Per-task results
are SIMULATED with a fixed seed, not real model results.
"""
import random
def simulate(n, p0, gain, rho_seed=11):
rng = random.Random(rho_seed)
out = []
for _ in range(n):
hard = rng.random() # shared task difficulty makes results correlated
a = 1 if rng.random() < p0 * (1.6 - hard) else 0
b = 1 if rng.random() < (p0 + gain) * (1.6 - hard) else 0
out.append((a, b))
return out
def paired_ci(pairs, reps=4000, seed=3):
rng = random.Random(seed)
n = len(pairs)
diffs = []
for _ in range(reps):
s = [pairs[rng.randrange(n)] for _ in range(n)]
diffs.append(sum(b - a for a, b in s) / n)
diffs.sort()
return diffs[int(0.025 * reps)], diffs[int(0.975 * reps)]
def main():
pairs = simulate(500, 0.40, 0.04)
a = sum(p[0] for p in pairs) / 500; b = sum(p[1] for p in pairs) / 500
lo, hi = paired_ci(pairs)
print(f"500 tasks: Maestro-0 {a:.1%}, Maestro-1 {b:.1%}, difference {b-a:+.1%}, 95% CI [{lo:+.1%}, {hi:+.1%}]")
print("tasks needed so the 95% CI of a +4-point gain excludes zero (simulated):")
for n in (100, 250, 500, 1000, 2000):
lo, hi = paired_ci(simulate(n, 0.40, 0.04, rho_seed=n), reps=1500)
print(f" {n:5d} tasks: CI [{lo:+.1%}, {hi:+.1%}] -> {'detected' if lo > 0 else 'not detected'}")
if __name__ == "__main__":
main()A real +4-point improvement: detected or not?
Real output of paired_bootstrap.py on simulated results (true gain +4 points). Bar length = number of tasks.
- The finding that changed the plan.
- With 500 tasks, the size of SWE-bench Verified, a real 4-point gain was not reliably distinguishable from noise in the simulation; the 500-task run even observed only +2.0 points.
- So the private release set was grown to about 1,200 tasks, and every reported number carries its interval.
- The site's from score to ship article covers error bars on evals in more depth.
7. How many tasks do you need?
Detectable difference calculator (our model)
8. The harness software
- He does not write an evaluation framework from scratch.
- Open frameworks such as the UK AI Security Institute's Inspect provide tasks, sandboxes, agents, scorers and logs.
- His code is the tasks, the hidden-test scorer and the report.
- Every run is reproducible.
- Model version, harness version, task-set version, sampling settings, random seeds and sandbox image are logged with the results.
# eval_task.py โ one harness task in the style of the UK AI Security Institute's
# open-source Inspect framework (names follow its documentation; check them
# against the version you install).
from inspect_ai import Task, task
from inspect_ai.dataset import json_dataset
from inspect_ai.solver import basic_agent, system_message
from inspect_ai.tool import bash, python
from inspect_ai.scorer import scorer, accuracy, stderr, Score, CORRECT, INCORRECT
@scorer(metrics=[accuracy(), stderr()])
def hidden_tests():
async def score(state, target):
ok = await run_hidden_tests(state.metadata["task_id"], state.output.completion) # sandboxed
return Score(value=CORRECT if ok else INCORRECT)
return score
@task
def partner_heldout():
return Task(
dataset=json_dataset("private/partner_heldout_v3.jsonl"), # never published, never trained on
solver=basic_agent(init=system_message(SYSTEM), tools=[bash(timeout=120), python()], max_attempts=1),
scorer=hidden_tests(),
sandbox="docker",
epochs=5, # 5 samples per task -> pass@1 and pass@5 (pass_at_k.py)
)9. Evaluating an agent, not just a model
- Agent results depend on the scaffold as much as the model.
- Tools, step limits, prompts and retries can move scores by many points.
- So he fixes the scaffold for the comparison and reports it with the score.
- Long tasks need different measures.
- Time and steps per task, cost in tokens, and how often the agent asks for help appropriately.
- The site's agent and long-task evaluation article covers these.
- Watch for test-gaming here too.
- The same reward-hacking behaviour Part 11 guarded against can appear at evaluation time: edits to test files, special-casing inputs. The scorer rejects any change that touches tests.
10. Where people still decide
- Tests miss qualities engineers care about.
- Readability, sensible scope, not adding unrequested changes.
- A blind review panel.
- Partner engineers rate 200 paired changes from Maestro-0 and Maestro-1 without knowing which is which.
- Arena-style comparisons work for chat; for code he keeps them task-based (see humans in evaluation).
11. The release report
From raw runs to a release decision
The evaluator writes the report; the release gate reads it.
- In the story, Maestro-1 beats Maestro-0 on the private release set with a 95% interval that excludes zero, matches it on the regression suite, and costs less per task. It still trails the best frontier models on public benchmarks.
- No specific scores are given on purpose: this is a teaching story, and real numbers would need a real model.
12. The regression suite
- Every skill the product ever had is a test it must keep passing.
- Tasks from past partner work that Maestro-0 solved, plus every bug a client reported and the fix confirmed.
- It grows with every release; nothing is removed without a written reason.
- It feeds two gates.
- The loop gate (Part 11) rejects updates that drop any skill by more than half a point.
- The release gate (Part 14) needs the full suite green.
- Flaky tests are fixed, not ignored.
- A test that passes and fails at random hides real regressions; each flaky test is either made deterministic or removed with a note.
What a release must not break
Our rules for the story.
13. Cost and speed are evaluation results too
| Measure | Why it matters | How it is reported |
|---|---|---|
| Tokens per solved task | Drives the price clients pay (Part 16) | Median and 90th percentile |
| Wall-clock time per task | Engineers wait for it | Median and 90th percentile |
| Tool calls per task | More calls, more risk and cost | Median |
| Cost per solved task | Combines success and spend | Dollars at the serving price (Part 15) |
- A model that solves 2% more tasks at three times the cost may be the wrong release.
- The report puts success and cost side by side, so the release decision is a business decision with evidence.
14. Reading other people's numbers
- Leaderboards compare systems, not just models.
- Scaffolds, retries, test-time compute and even task subsets differ between submissions.
- Questions he asks before believing a claim.
- Which tasks exactly, and were any excluded?
- One attempt or best of many?
- Which scaffold and tools, and is it public?
- Is there a confidence interval?
- Could the benchmark be in the training data?
- He applies the same questions to his own marketing.
- Every number on the website links to the evaluation report and states the scaffold and the number of tasks.
15. The evaluation that happens after release
- Benchmarks predict; pilots measure.
- During the design-partner stage (Part 14), every change the agent proposes is tracked from proposal to merge.
- Pilot metrics.
- Acceptance rate: share of proposed changes merged.
- Edit distance: how much engineers changed before merging.
- Reverts: merged changes later undone.
- Time saved, reported by engineers, treated as soft evidence.
- Pilot data closes the loop.
- Rejected and reverted changes become new tasks for the hidden pool and new regression tests, with the partner's agreement.
From benchmark to production evidence
Production failures feed the next evaluation.
16. Protecting the release set
- A leaked test set is worthless, and leaks are usually accidental.
- Copied into a prompt while debugging; pasted into a chat tool; used as training data by a script with a wrong path.
- Controls.
- The set lives in a separate storage bucket; only the evaluation service account can read it.
- Every read is logged; training jobs fail if they reference its path.
- Canary strings are embedded in tasks, so a model that reproduces one reveals the leak.
17. Eight evaluation mistakes
- Deciding on public benchmarks alone.
- Reporting pass@k with the biased formula.
- Single numbers without confidence intervals.
- Too few tasks to see the improvement you claim.
- Letting the training side see the release set.
- Changing the scaffold between models.
- Trusting a scorer nobody audited.
- Building the harness after the model.
18. Delegation box
- The release set and its access rules
- The analysis and the report
- Saying no when the numbers say no
- Running sweeps and collecting logs
- First-pass audits of scorer failures
- Drafting report sections for review
- Open evaluation frameworks
- Partner engineers for blind review
- Public benchmarks for comparison
19. Budget and risk tracker
| Item | Status after Part 12 | Note |
|---|---|---|
| Spend to date | ~$6.8M (scenario) | Evaluation sweeps on the reserved block |
| Evidence | Release report written (scenario) | Private set grown to ~1,200 tasks |
| Time | Week 56 (month ~13) | Release review next |
| Measurement risk | Lower | Paired CIs, larger set, contamination audit |
| Overfitting risk | Watch | Dev set reused for months |
| Generalisation risk | Open | Partner pilots in Phase 4 |
20. The series
The Maestro: 18 parts in five phases
- The harness turns "it seems better" into a number with an interval, on tasks the model cannot have seen.
- Its biggest contribution was negative: it showed 500 tasks were too few to see the improvement the team hoped for.
- Next, Part 13: making sure a capable agent is also a safe one.
21. Reading the numbers
- Primary / standard
- SWE-bench (Jimenez et al., 2023). SWE-bench Verified: 500 tasks, 93 developers screened 1,699 samples, released 13 Aug 2024 (OpenAI with the SWE-bench authors). Unbiased pass@k (Chen et al., 2021). 13-gram contamination check (Brown et al., 2020). Inspect evaluation framework (UK AI Security Institute).
- Secondary โ ๏ธ
- None in this part.
- Our assumptions
- Pool sizes, the simulated results, sample counts, the release-set growth and all Maestro-1 outcomes.