Home โ€บ Blog โ€บ The Maestro ยท Part 12
The Maestro ยท Part 12 ยท Phase 3 ยท ๐Ÿ“ Evaluator hat

The Evaluation Harness

Phase 3 asks the uncomfortable question: does Maestro-1 actually work, and how would anyone know? Gopala swaps the builder's hat for the evaluator's and builds a harness whose job is to say no when the answer is no. This part covers what to measure for an engineering-agent product, the four task pools, pass@k done correctly, contamination checks, why 500 tasks may not be enough to see a real improvement, the harness software, and the report that decides whether Maestro-1 ships.

FL
FrontierAGI Team
๐Ÿ“ Evaluator๐Ÿ”ฌ Researcher๐Ÿ’ป Coder
โš ๏ธ How to read this (as of 30 Oct 2026) SWE-bench and SWE-bench Verified facts are from their papers and OpenAI's announcement; the pass@k estimator is from the Codex paper (Chen et al., 2021); the 13-gram contamination check follows the GPT-3 paper. pass_at_k.py, paired_bootstrap.py and contamination_check.py were run (real output below) on made-up counts, simulated results and sample texts. eval_task.py uses an external framework and was not executed for this article. Maestro-1's results are the scenario. Disclosure: Claude, made by Anthropic, helps write this site.

1. Mission log: weeks 12โ€“56

Weeks 12โ€“30 ยท Built early
  • The harness started in month 3 (Part 3's plan): small evaluations on Maestro-0 and every ablation (Part 9).
Weeks 40โ€“46 ยท During the run
  • Intermediate checkpoints scored every ~300B tokens (Part 10).
Weeks 54โ€“56 ยท The release evaluation
  • Full harness on Maestro-1 vs Maestro-0 vs two open models; contamination audit; report written for the release gate (Part 14).

2. What to measure

Evaluation questions, from cheapest to most decisive

๐Ÿงช Loss on held-out textIs training working at all? (every step)
๐Ÿ“š Public benchmarksHow does it compare with known models?
๐Ÿ” Regression suiteDid we break anything that used to work?
๐Ÿ”’ Private partner tasksDoes it do the clients' real work? (the decision)
๐Ÿ‘ฅ Pilot with partnersDo engineers accept its changes in practice?

Each level costs more and tells more.

3. Four task pools, kept apart

PoolSize (story)Who sees itPurpose
Training tasksthousandsThe loop (Part 11)Rewards and training data
Development set~600ResearchersDaily iteration; may be overfit over time
Hidden test pool~1,000Only the verifierThe loop's reward and gate; rotated quarterly
Private release set~1,200Only the evaluator hatRelease decisions; never published, never trained on

4. pass@k, done correctly

pass_at_k.py ยท Python 3, no dependencies
"""pass_at_k.py โ€” the unbiased pass@k estimator (Chen et al., 2021, Codex paper).

Generate n samples per task, count c that pass the tests; the chance that
at least one of k random samples passes is 1 - C(n-c, k) / C(n, k).
Naively computing 1-(1-c/n)^k is biased. Counts below are made up.
"""
from math import comb

def pass_at_k(n, c, k):
    if n - c < k:
        return 1.0
    return 1 - comb(n - c, k) / comb(n, k)

TASKS = [(20, 0), (20, 1), (20, 5), (20, 12), (20, 20)]   # (samples, correct) per task

def main():
    print(f"{'n':>3s} {'c':>3s} {'pass@1':>7s} {'pass@5':>7s} {'pass@10':>8s} {'naive@10':>9s}")
    for n, c in TASKS:
        naive = 1 - (1 - c / n) ** 10
        print(f"{n:3d} {c:3d} {pass_at_k(n,c,1):7.3f} {pass_at_k(n,c,5):7.3f} {pass_at_k(n,c,10):8.3f} {naive:9.3f}")
    for k in (1, 5, 10):
        print(f"benchmark pass@{k}: {sum(pass_at_k(n, c, k) for n, c in TASKS) / len(TASKS):.3f}")

if __name__ == "__main__":
    main()
$ python3 pass_at_k.py # made-up counts for five tasks n c pass@1 pass@5 pass@10 naive@10 20 0 0.000 0.000 0.000 0.000 20 1 0.050 0.250 0.500 0.401 20 5 0.250 0.806 0.984 0.944 20 12 0.600 0.996 1.000 1.000 20 20 1.000 1.000 1.000 1.000 benchmark pass@1: 0.380 benchmark pass@5: 0.611 benchmark pass@10: 0.697

5. Contamination: did the answers leak?

contamination_check.py ยท Python 3, no dependencies
"""contamination_check.py โ€” did benchmark items leak into training data?

Long word n-gram overlap (13-grams, as in the GPT-3 paper's check)
between each benchmark item and a training shard. Texts are written for
this article. Real runs index the 10T-token blend with hashed n-grams.
"""
N = 13
BENCH = {
    "task-017": "write a function that returns the longest strictly increasing contiguous subsequence of a list of integers and its starting index",
    "task-042": "given a log file with timestamps in mixed time zones convert every entry to utc and report the gaps longer than five minutes",
    "task-108": "implement a rate limiter that allows at most n requests per rolling window of t seconds for each api key",
}
SHARD = ("forum post: homework help please. write a function that returns the longest strictly increasing contiguous "
         "subsequence of a list of integers and its starting index, my code fails on empty lists. "
         "another thread discusses how to implement a rate limiter for an api using a token bucket.")

def grams(t):
    w = t.lower().replace(",", "").replace(".", "").split()
    return {" ".join(w[i:i + N]) for i in range(len(w) - N + 1)}

def main():
    sg = grams(SHARD)
    for k, v in BENCH.items():
        g = grams(v)
        hit = len(g & sg)
        print(f"{k}: {hit}/{len(g)} 13-grams found in training shard -> {'CONTAMINATED: remove from eval or from data' if hit else 'clean'}")

if __name__ == "__main__":
    main()
$ python3 contamination_check.py # sample texts written for this article task-017: 8/8 13-grams found in training shard -> CONTAMINATED: remove from eval or from data task-042: 0/11 13-grams found in training shard -> clean task-108: 0/8 13-grams found in training shard -> clean

6. Is the difference real?

paired_bootstrap.py ยท Python 3, no dependencies; simulated results
"""paired_bootstrap.py โ€” is Maestro-1 really better than Maestro-0?

Both models attempt the SAME tasks; a paired bootstrap resamples tasks
(not answers) to get a confidence interval for the difference. Then:
how many tasks are needed to detect a given improvement? Per-task results
are SIMULATED with a fixed seed, not real model results.
"""
import random

def simulate(n, p0, gain, rho_seed=11):
    rng = random.Random(rho_seed)
    out = []
    for _ in range(n):
        hard = rng.random()                     # shared task difficulty makes results correlated
        a = 1 if rng.random() < p0 * (1.6 - hard) else 0
        b = 1 if rng.random() < (p0 + gain) * (1.6 - hard) else 0
        out.append((a, b))
    return out

def paired_ci(pairs, reps=4000, seed=3):
    rng = random.Random(seed)
    n = len(pairs)
    diffs = []
    for _ in range(reps):
        s = [pairs[rng.randrange(n)] for _ in range(n)]
        diffs.append(sum(b - a for a, b in s) / n)
    diffs.sort()
    return diffs[int(0.025 * reps)], diffs[int(0.975 * reps)]

def main():
    pairs = simulate(500, 0.40, 0.04)
    a = sum(p[0] for p in pairs) / 500; b = sum(p[1] for p in pairs) / 500
    lo, hi = paired_ci(pairs)
    print(f"500 tasks: Maestro-0 {a:.1%}, Maestro-1 {b:.1%}, difference {b-a:+.1%}, 95% CI [{lo:+.1%}, {hi:+.1%}]")
    print("tasks needed so the 95% CI of a +4-point gain excludes zero (simulated):")
    for n in (100, 250, 500, 1000, 2000):
        lo, hi = paired_ci(simulate(n, 0.40, 0.04, rho_seed=n), reps=1500)
        print(f"  {n:5d} tasks: CI [{lo:+.1%}, {hi:+.1%}] -> {'detected' if lo > 0 else 'not detected'}")

if __name__ == "__main__":
    main()
$ python3 paired_bootstrap.py # simulated per-task results 500 tasks: Maestro-0 46.0%, Maestro-1 48.0%, difference +2.0%, 95% CI [-3.8%, +7.8%] tasks needed so the 95% CI of a +4-point gain excludes zero (simulated): 100 tasks: CI [-4.0%, +22.0%] -> not detected 250 tasks: CI [-8.4%, +8.4%] -> not detected 500 tasks: CI [-3.4%, +8.4%] -> not detected 1000 tasks: CI [+2.5%, +10.7%] -> detected 2000 tasks: CI [+3.2%, +9.3%] -> detected

A real +4-point improvement: detected or not?

100 tasksnot detected
250 tasksnot detected
500 tasksnot detected
1,000 tasksdetected
2,000 tasksdetected

Real output of paired_bootstrap.py on simulated results (true gain +4 points). Bar length = number of tasks.

7. How many tasks do you need?

Detectable difference calculator (our model)

8. The harness software

eval_task.py ยท Inspect-style task sketch ยท not executed for this article
# eval_task.py โ€” one harness task in the style of the UK AI Security Institute's
# open-source Inspect framework (names follow its documentation; check them
# against the version you install).
from inspect_ai import Task, task
from inspect_ai.dataset import json_dataset
from inspect_ai.solver import basic_agent, system_message
from inspect_ai.tool import bash, python
from inspect_ai.scorer import scorer, accuracy, stderr, Score, CORRECT, INCORRECT

@scorer(metrics=[accuracy(), stderr()])
def hidden_tests():
    async def score(state, target):
        ok = await run_hidden_tests(state.metadata["task_id"], state.output.completion)  # sandboxed
        return Score(value=CORRECT if ok else INCORRECT)
    return score

@task
def partner_heldout():
    return Task(
        dataset=json_dataset("private/partner_heldout_v3.jsonl"),   # never published, never trained on
        solver=basic_agent(init=system_message(SYSTEM), tools=[bash(timeout=120), python()], max_attempts=1),
        scorer=hidden_tests(),
        sandbox="docker",
        epochs=5,                     # 5 samples per task -> pass@1 and pass@5 (pass_at_k.py)
    )

9. Evaluating an agent, not just a model

10. Where people still decide

11. The release report

From raw runs to a release decision

๐ŸƒRunHarness on all pools, 5 samples per task
๐ŸงนAuditContamination and scorer checks
๐Ÿ“ŠAnalysepass@1 with paired 95% CIs
๐Ÿ“ReportWins, losses, regressions, cost
๐ŸšฆRelease gatePart 14

The evaluator writes the report; the release gate reads it.

Scenario
  • In the story, Maestro-1 beats Maestro-0 on the private release set with a 95% interval that excludes zero, matches it on the regression suite, and costs less per task. It still trails the best frontier models on public benchmarks.
  • No specific scores are given on purpose: this is a teaching story, and real numbers would need a real model.

12. The regression suite

What a release must not break

everypast partner task Maestro-0 solved
everyclient-reported bug with a confirmed fix
≤ 0.5 ptallowed drop per skill area (Part 11 gate)
0flaky tests allowed to stay unexplained

Our rules for the story.

13. Cost and speed are evaluation results too

MeasureWhy it mattersHow it is reported
Tokens per solved taskDrives the price clients pay (Part 16)Median and 90th percentile
Wall-clock time per taskEngineers wait for itMedian and 90th percentile
Tool calls per taskMore calls, more risk and costMedian
Cost per solved taskCombines success and spendDollars at the serving price (Part 15)

14. Reading other people's numbers

15. The evaluation that happens after release

From benchmark to production evidence

๐ŸงชHarnessPrivate release set, hidden tests
๐ŸคPilotReal repositories, real reviewers
๐Ÿ“ˆAcceptanceMerged, edited, reverted
๐Ÿ”Back into testsFailures become new tasks

Production failures feed the next evaluation.

16. Protecting the release set

17. Eight evaluation mistakes

18. Delegation box

๐Ÿ™‹ Gopala himself
  • The release set and its access rules
  • The analysis and the report
  • Saying no when the numbers say no
๐Ÿค– AI agents
  • Running sweeps and collecting logs
  • First-pass audits of scorer failures
  • Drafting report sections for review
๐Ÿค Bought, rented or borrowed
  • Open evaluation frameworks
  • Partner engineers for blind review
  • Public benchmarks for comparison

19. Budget and risk tracker

ItemStatus after Part 12Note
Spend to date~$6.8M (scenario)Evaluation sweeps on the reserved block
EvidenceRelease report written (scenario)Private set grown to ~1,200 tasks
TimeWeek 56 (month ~13)Release review next
Measurement riskLowerPaired CIs, larger set, contamination audit
Overfitting riskWatchDev set reused for months
Generalisation riskOpenPartner pilots in Phase 4

20. The series

The Maestro: 18 parts in five phases

Verdict
  • The harness turns "it seems better" into a number with an interval, on tasks the model cannot have seen.
    • Its biggest contribution was negative: it showed 500 tasks were too few to see the improvement the team hoped for.
    • Next, Part 13: making sure a capable agent is also a safe one.

21. Reading the numbers

Primary / standard
SWE-bench (Jimenez et al., 2023). SWE-bench Verified: 500 tasks, 93 developers screened 1,699 samples, released 13 Aug 2024 (OpenAI with the SWE-bench authors). Unbiased pass@k (Chen et al., 2021). 13-gram contamination check (Brown et al., 2020). Inspect evaluation framework (UK AI Security Institute).
Secondary โš ๏ธ
None in this part.
Our assumptions
Pool sizes, the simulated results, sample counts, the release-set growth and all Maestro-1 outcomes.