Home โ€บ Blog โ€บ The Maestro, Season 2 ยท Part 15
The Maestro, Season 2 ยท Part 15 ยท Phase 8 ยท ๐Ÿค– CEO hat

From PR Bot to Agent Platform

Phase 8 opens with the result everybody was waiting for: the sealed evaluation of Maestro-2. Then comes the harder question. The company built a model that is better at real work; now it has to decide what to sell. This part follows the week the results came in, and the month in which a PR bot became a platform that other people's agents call.

FL
FrontierAGI Team
๐Ÿ’ผ CEO๐Ÿงญ Product๐ŸŽฏ Evaluation
โš ๏ธ How to read this (as of 4 Nov 2026) Gopala's company is a teaching scenario. eval_m2.py and platform_economics.py were run (real output below). The evaluation method (paired tasks, bootstrap interval) is standard; task outcomes, prices, volumes and costs are scenario data. Ideas from the owner's startup notes are labelled as such. Disclosure: Claude, made by Anthropic, helps write this site.

1. Month 41: the envelope

๐Ÿ“– The story

Elena had insisted on doing it the old-fashioned way. The evaluation ran overnight on Maestro-1 and Maestro-2 side by side, and the results went into a single file that nobody, not even she, opened until the whole team was in the room.

The first number on the screen was 44. Forty-four of the eighty-three failures that had justified the run were now fixed. A murmur went round the room, half relief and half disappointment. Then the second line: on the 400 held-out tasks, 61.0% for Maestro-1, 66.2% for Maestro-2. Then the third, the one Elena had built the whole ritual around: 95% interval, +2.5 to +8.0 points.

It is real, she said. Not as big as the deck promised. But real. Then she scrolled down to the line nobody had asked about. And it breaks six things the old model could do.

๐Ÿ”ฌ In plain words: confidence interval Any test on 400 tasks has luck in it. The bootstrap re-runs the comparison thousands of times on random resamples of the same tasks and records how much the gain moves. If even the pessimistic end of that range is above zero, the improvement is unlikely to be luck.

2. The map: from a result to a business

๐Ÿ•ธ๏ธ What the evaluation unlocked

or click any node

The top half is the evidence; the bottom half is the three products it made possible.

3. The sealed evaluation

eval_m2.py ยท Python 3, no dependencies; scenario outcomes, seeded
"""eval_m2.py โ€” the sealed evaluation: is Maestro-2 worth what it cost?

Two sets were written down before the run (Part 14): the 83 model-only failures
from Part 10 and 400 held-out customer tasks. Each task is run with Maestro-1
and Maestro-2 under the same harness and context. The held-out gain gets a
95% paired-bootstrap interval. Per-task outcomes are SCENARIO data generated
from a fixed seed; the method is the one used in Season 1, Part 12.
"""
import random
random.seed(11)
P1, P2 = 0.62, 0.69            # scenario pass rates on held-out tasks
held = [(random.random() < P1, None) for _ in range(400)]
held = [(a, a or random.random() < (P2 - P1) / (1 - P1)) for a, _ in held]   # M2 keeps M1's wins, adds some
for i in random.sample(range(400), 9):                                         # ...and loses a few (regressions)
    held[i] = (held[i][0], False)
fixed = sum(1 for _ in range(83) if random.random() < 0.55)                    # of the 83 model-only failures

def boot(pairs, n=2000):
    d = []
    for _ in range(n):
        s = [random.choice(pairs) for _ in pairs]
        d.append(sum(b - a for a, b in s) / len(s))
    d.sort(); return d[int(0.025 * n)], d[int(0.975 * n)]

def main():
    a = sum(x for x, _ in held); b = sum(y for _, y in held); reg = sum(1 for x, y in held if x and not y)
    lo, hi = boot([(int(x), int(y)) for x, y in held])
    print(f"model-only failures fixed: {fixed} of 83 ({fixed / 83:.0%})")
    print(f"held-out tasks: Maestro-1 {a}/400 ({a / 4:.1f}%)  Maestro-2 {b}/400 ({b / 4:.1f}%)")
    print(f"gain {100 * (b - a) / 400:+.1f} points, 95% CI [{100 * lo:+.1f}, {100 * hi:+.1f}]; regressions: {reg} tasks")
    print("verdict:", "SHIP behind the gate" if lo > 0 else "do not ship")

if __name__ == "__main__":
    main()
$ python3 eval_m2.py # Maestro-1 vs Maestro-2 on sealed sets model-only failures fixed: 44 of 83 (53%) held-out tasks: Maestro-1 244/400 (61.0%) Maestro-2 265/400 (66.2%) gain +5.2 points, 95% CI [+2.5, +8.0]; regressions: 6 tasks verdict: SHIP behind the gate

What Maestro-2 bought

44 / 83model-only failures fixed
+5.2 ptsheld-out gain
[+2.5, +8.0]95% interval
6regressions to review

Real output. Paired comparison on the same 400 tasks, the method from Season 1, Part 12.

๐Ÿ“– The story

The six regressions went to the gate review that afternoon. Four were tasks in an old build system that Maestro-2 had simply seen less of; the skill library could cover them. Two were harder: the new model was more willing to rewrite a file from scratch than to patch it. Reviewers hated that.

The gate approved the release with two conditions: a skill for the old build system, and a harness rule that preferred minimal diffs. It was the first time the gate had changed the harness instead of the model. It would not be the last.

๐Ÿ”Ž Go deeper: why 44 and not 83?
  • A bigger model fixes what it was trained to be better at: reasoning over many files and unfamiliar languages.
    • Recall past 128k tokens improved least; the team logged it as a Maestro-3 problem.
  • The same lesson as Part 10: some failures are cheaper to fix in the harness.
    • The minimal-diff rule recovered two regressions without touching the weights.

4. What do we sell now?

๐Ÿ“– The story

Sofia Lindqvist, who had run product since month 30, had a slide with three boxes on it and no title. This is the decision, she said. We keep selling the PR bot. Or we let customers build their own agents on our harness and gate. Or we let other people's agents call us directly.

Why not all three? asked Gopala.

Because each one needs a different company, Sofia said. The bot needs a few good tasks. The platform needs onboarding, docs and support. The API needs documentation an agent can read, and prices an agent will compare.

platform_economics.py ยท Python 3, no dependencies; scenario volumes and prices
"""platform_economics.py โ€” from a PR bot to an agent platform.

Three ways to sell the same model: the PR bot (fixed tasks inside GitHub), a
platform where customers build their own agents on Maestro's harness, and an
API for other companies' agents (agents as customers, from the owner's notes on
Rauch). Volumes, prices and costs per task are SCENARIO assumptions.
"""
LINES = [  # name, tasks per month, price per task $, serving + tools cost per task $, support per month $
    ("PR bot (today)", 900_000, 0.40, 0.09, 120_000),
    ("Agent platform", 2_600_000, 0.28, 0.08, 380_000),
    ("API for other agents", 6_000_000, 0.06, 0.035, 60_000)]

def main():
    tr, tg = 0, 0
    for n, t, p, c, s in LINES:
        rev = t * p; gm = rev - t * c - s; tr += rev; tg += gm
        print(f"{n:22} revenue ${rev / 1e3:7.0f}k/month  gross margin ${gm / 1e3:6.0f}k ({gm / rev:4.0%})")
    print(f"\ntotal ${tr / 1e6:.2f}M/month, blended margin {tg / tr:.0%}; annualised revenue ${12 * tr / 1e6:.1f}M")

if __name__ == "__main__":
    main()
$ python3 platform_economics.py # three ways to sell the same model PR bot (today) revenue $ 360k/month gross margin $ 159k ( 44%) Agent platform revenue $ 728k/month gross margin $ 140k ( 19%) API for other agents revenue $ 360k/month gross margin $ 90k ( 25%) total $1.45M/month, blended margin 27%; annualised revenue $17.4M

Monthly revenue and gross margin by line

Agent platform$728k ยท 19%
PR bot$360k ยท 44%
API for other agents$360k ยท 25%

Real output. $17.4M annualised, blended margin 27%. The platform is the biggest line and the least profitable.

5. Sandbox: grow the platform

๐ŸŽฎ Sandbox: grow the platform

Same lines, prices and costs as platform_economics.py; scenario assumptions.

6. Five mistakes

7. Delegation box

๐Ÿ™‹ Gopala himself
  • The release decision and the product strategy
๐Ÿค– AI agents
  • Running the sealed evaluation
  • Drafting the platform docs (machine-readable first)
๐Ÿค Bought, rented or borrowed
  • Design partners for the staged release

8. Budget and risk tracker

ItemStatus after Part 15Note
Maestro-2Shipped behind the gate+5.2 pts, CI [+2.5, +8.0]
Revenue$1.45M a month$17.4M annualised
Margin27% blendedplatform 19%
EvidenceSealed and reported6 regressions reviewed

9. The series

The Maestro, Season 2: 23 parts in five phases (Season 1: 18 parts)

VerdictMaestro-2 is genuinely better (+5.2 points, interval clear of zero) but fixed only about half of what it was built for, and the gate shipped it with two harness changes. The product grows from a PR bot into a platform and an API, with more revenue and thinner margins. Next: why the harness, not the model, became the product.

10. Reading the numbers

Primary / standard
Paired bootstrap intervals for model comparison (standard practice; Season 1, Part 12).
Secondary โš ๏ธ
None in this part.
Our assumptions
Evaluation outcomes, prices, volumes, costs and story events are the scenario. Owner's notes (Rauch) are a summary, not a transcript.