From PR Bot to Agent Platform
Phase 8 opens with the result everybody was waiting for: the sealed evaluation of Maestro-2. Then comes the harder question. The company built a model that is better at real work; now it has to decide what to sell. This part follows the week the results came in, and the month in which a PR bot became a platform that other people's agents call.
eval_m2.py and platform_economics.py were run (real output below). The evaluation method (paired tasks, bootstrap interval) is standard; task outcomes, prices, volumes and costs are scenario data. Ideas from the owner's startup notes are labelled as such. Disclosure: Claude, made by Anthropic, helps write this site.
1. Month 41: the envelope
Elena had insisted on doing it the old-fashioned way. The evaluation ran overnight on Maestro-1 and Maestro-2 side by side, and the results went into a single file that nobody, not even she, opened until the whole team was in the room.
The first number on the screen was 44. Forty-four of the eighty-three failures that had justified the run were now fixed. A murmur went round the room, half relief and half disappointment. Then the second line: on the 400 held-out tasks, 61.0% for Maestro-1, 66.2% for Maestro-2. Then the third, the one Elena had built the whole ritual around: 95% interval, +2.5 to +8.0 points.
It is real, she said. Not as big as the deck promised. But real. Then she scrolled down to the line nobody had asked about. And it breaks six things the old model could do.
2. The map: from a result to a business
๐ธ๏ธ What the evaluation unlocked
The top half is the evidence; the bottom half is the three products it made possible.
- Two decisions in one month.
- Ship Maestro-2? Yes, behind the gate.
- Sell what? A PR bot, a platform and an API, each with different economics.
3. The sealed evaluation
"""eval_m2.py โ the sealed evaluation: is Maestro-2 worth what it cost?
Two sets were written down before the run (Part 14): the 83 model-only failures
from Part 10 and 400 held-out customer tasks. Each task is run with Maestro-1
and Maestro-2 under the same harness and context. The held-out gain gets a
95% paired-bootstrap interval. Per-task outcomes are SCENARIO data generated
from a fixed seed; the method is the one used in Season 1, Part 12.
"""
import random
random.seed(11)
P1, P2 = 0.62, 0.69 # scenario pass rates on held-out tasks
held = [(random.random() < P1, None) for _ in range(400)]
held = [(a, a or random.random() < (P2 - P1) / (1 - P1)) for a, _ in held] # M2 keeps M1's wins, adds some
for i in random.sample(range(400), 9): # ...and loses a few (regressions)
held[i] = (held[i][0], False)
fixed = sum(1 for _ in range(83) if random.random() < 0.55) # of the 83 model-only failures
def boot(pairs, n=2000):
d = []
for _ in range(n):
s = [random.choice(pairs) for _ in pairs]
d.append(sum(b - a for a, b in s) / len(s))
d.sort(); return d[int(0.025 * n)], d[int(0.975 * n)]
def main():
a = sum(x for x, _ in held); b = sum(y for _, y in held); reg = sum(1 for x, y in held if x and not y)
lo, hi = boot([(int(x), int(y)) for x, y in held])
print(f"model-only failures fixed: {fixed} of 83 ({fixed / 83:.0%})")
print(f"held-out tasks: Maestro-1 {a}/400 ({a / 4:.1f}%) Maestro-2 {b}/400 ({b / 4:.1f}%)")
print(f"gain {100 * (b - a) / 400:+.1f} points, 95% CI [{100 * lo:+.1f}, {100 * hi:+.1f}]; regressions: {reg} tasks")
print("verdict:", "SHIP behind the gate" if lo > 0 else "do not ship")
if __name__ == "__main__":
main()What Maestro-2 bought
Real output. Paired comparison on the same 400 tasks, the method from Season 1, Part 12.
The six regressions went to the gate review that afternoon. Four were tasks in an old build system that Maestro-2 had simply seen less of; the skill library could cover them. Two were harder: the new model was more willing to rewrite a file from scratch than to patch it. Reviewers hated that.
The gate approved the release with two conditions: a skill for the old build system, and a harness rule that preferred minimal diffs. It was the first time the gate had changed the harness instead of the model. It would not be the last.
๐ Go deeper: why 44 and not 83?
- A bigger model fixes what it was trained to be better at: reasoning over many files and unfamiliar languages.
- Recall past 128k tokens improved least; the team logged it as a Maestro-3 problem.
- The same lesson as Part 10: some failures are cheaper to fix in the harness.
- The minimal-diff rule recovered two regressions without touching the weights.
4. What do we sell now?
Sofia Lindqvist, who had run product since month 30, had a slide with three boxes on it and no title. This is the decision, she said. We keep selling the PR bot. Or we let customers build their own agents on our harness and gate. Or we let other people's agents call us directly.
Why not all three? asked Gopala.
Because each one needs a different company, Sofia said. The bot needs a few good tasks. The platform needs onboarding, docs and support. The API needs documentation an agent can read, and prices an agent will compare.
"""platform_economics.py โ from a PR bot to an agent platform.
Three ways to sell the same model: the PR bot (fixed tasks inside GitHub), a
platform where customers build their own agents on Maestro's harness, and an
API for other companies' agents (agents as customers, from the owner's notes on
Rauch). Volumes, prices and costs per task are SCENARIO assumptions.
"""
LINES = [ # name, tasks per month, price per task $, serving + tools cost per task $, support per month $
("PR bot (today)", 900_000, 0.40, 0.09, 120_000),
("Agent platform", 2_600_000, 0.28, 0.08, 380_000),
("API for other agents", 6_000_000, 0.06, 0.035, 60_000)]
def main():
tr, tg = 0, 0
for n, t, p, c, s in LINES:
rev = t * p; gm = rev - t * c - s; tr += rev; tg += gm
print(f"{n:22} revenue ${rev / 1e3:7.0f}k/month gross margin ${gm / 1e3:6.0f}k ({gm / rev:4.0%})")
print(f"\ntotal ${tr / 1e6:.2f}M/month, blended margin {tg / tr:.0%}; annualised revenue ${12 * tr / 1e6:.1f}M")
if __name__ == "__main__":
main()Monthly revenue and gross margin by line
Real output. $17.4M annualised, blended margin 27%. The platform is the biggest line and the least profitable.
- The owner's notes behind the choice.
- Agents as a new customer class (Rauch): products now have two audiences, humans and agents, and agents prefer clean APIs and machine-readable docs.
- Trust as the moat (Rauch): the gate record is what a buyer, human or agent, is really paying for.
- What they decided.
- All three, in that order of priority: platform first, PR bot kept as the entry product, API with a small team.
- Margin work deferred to Part 19, on purpose: first prove customers build on it.
5. Sandbox: grow the platform
๐ฎ Sandbox: grow the platform
Same lines, prices and costs as platform_economics.py; scenario assumptions.
- Try this.
- Triple platform volume with support at 100%: margin improves, because support is a fixed monthly cost.
- Halve platform support cost: the blended margin rises by about 13 points.
- Set platform volume to 25%: the platform loses money and the PR bot is again the most profitable line.
6. Five mistakes
- Opening the evaluation before the team agreed what would count as success.
- Reporting the gain and hiding the regressions.
- Fixing every regression with more training instead of the harness or a skill.
- Assuming the best model makes the best product.
- Building an API for agents with documentation written only for humans.
7. Delegation box
- The release decision and the product strategy
- Running the sealed evaluation
- Drafting the platform docs (machine-readable first)
- Design partners for the staged release
8. Budget and risk tracker
| Item | Status after Part 15 | Note |
|---|---|---|
| Maestro-2 | Shipped behind the gate | +5.2 pts, CI [+2.5, +8.0] |
| Revenue | $1.45M a month | $17.4M annualised |
| Margin | 27% blended | platform 19% |
| Evidence | Sealed and reported | 6 regressions reviewed |
9. The series
The Maestro, Season 2: 23 parts in five phases (Season 1: 18 parts)
10. Reading the numbers
- Primary / standard
- Paired bootstrap intervals for model comparison (standard practice; Season 1, Part 12).
- Secondary โ ๏ธ
- None in this part.
- Our assumptions
- Evaluation outcomes, prices, volumes, costs and story events are the scenario. Owner's notes (Rauch) are a summary, not a transcript.