Model-Agnostic or Not?
If the harness is the product, why insist on running only your own model? This part follows an argument that splits the company: the researchers who built Maestro-2 against the product team, who want to route each task to whichever model does it best, including a rival's. The answer turns out to be a router, a rule, and a plan for the day the other provider changes its terms.
router.py and lockin_risk.py were run (real output below). Quality rates and costs are scenario numbers; "frontier API" stands for any external frontier model, and its prices are placeholders, not a real vendor's. Ideas from the owner's startup notes (Sosin, Chase) are labelled. Disclosure: Claude, made by Anthropic, helps write this site.
1. Month 43: the argument
Tomรกs took it personally, and everyone could see it. He had spent nine months on Maestro-2. Now Sofia was proposing that fifteen percent of the company's traffic go to a model built by a frontier lab that was, in some markets, a direct competitor.
We built a model so we would not depend on them, he said.
We built a model so we would be good at our customers' work, Sofia said. On Rust and Kotlin, they are better. Our customers do not care whose weights did it. They care whether it merges.
Gopala let them argue for twenty minutes. Then he asked Arjun for one number: what happens to the business if the other company doubles its price, or turns us off?
2. The map: one task, three possible models
๐ธ๏ธ How a task is routed, and what could go wrong
Models in the middle row; the rule that chooses; the shock and the fallback; and what the company owns whichever model it uses.
- Model + harness + context (owner's notes on Chase).
- Routing makes the model layer swappable; the harness and context stay the same.
- That is only true if every route is evaluated with the same sealed tests.
3. The router
"""router.py โ model-agnostic or not?
Each task type can go to Maestro-2 (own model), an external frontier API, or a
small open model. Quality (merge rate) and cost per task are SCENARIO numbers;
external prices are placeholders, not any vendor's list price. The router picks,
per task type, the cheapest option within 2 points of the best quality.
"""
OPT = {"Maestro-2": {}, "frontier API": {}, "small open": {}}
TASKS = [ # type, share of volume, {option: (merge rate, $ per task)}
("repo-specific fixes", 0.45, {"Maestro-2": (0.61, 0.08), "frontier API": (0.55, 0.22), "small open": (0.38, 0.02)}),
("dependency upgrades", 0.20, {"Maestro-2": (0.72, 0.06), "frontier API": (0.73, 0.18), "small open": (0.66, 0.02)}),
("new-language code", 0.15, {"Maestro-2": (0.48, 0.10), "frontier API": (0.63, 0.25), "small open": (0.30, 0.03)}),
("docs and tests", 0.20, {"Maestro-2": (0.80, 0.05), "frontier API": (0.81, 0.15), "small open": (0.79, 0.01)})]
def main():
q = c = 0; share = {k: 0 for k in OPT}
for t, w, o in TASKS:
best = max(v[0] for v in o.values())
pick = min((k for k, v in o.items() if v[0] >= best - 0.02), key=lambda k: o[k][1])
q += w * o[pick][0]; c += w * o[pick][1]; share[pick] += w
print(f"{t:22} -> {pick:12} merge {o[pick][0]:.0%} ${o[pick][1]:.2f}")
m2q = sum(w * o['Maestro-2'][0] for _, w, o in TASKS); m2c = sum(w * o['Maestro-2'][1] for _, w, o in TASKS)
print(f"\nrouted: merge {q:.1%} ${c:.3f}/task volume " + ", ".join(f"{k} {v:.0%}" for k, v in share.items()))
print(f"Maestro-2 only: merge {m2q:.1%} ${m2c:.3f}/task")
if __name__ == "__main__":
main()Where tasks go
Real output. Routed: 67.1% merged at $0.087 per task; Maestro-2 alone: 65.1% at $0.073.
- What routing buys and costs.
- +2.0 points of merged changes across all traffic.
- +$0.014 per task, almost all of it on the 15% sent to the external API.
- A dependency on a company that can change its prices, its models or its mind.
4. The day the other provider changes its terms
"""lockin_risk.py โ what if the external provider changes the terms?
Stress-test the routed plan: the frontier API raises prices 2x, deprecates the
model (quality drops while re-tuning), or is cut off. Gross margin per task is
compared with a plan that keeps a fallback route warm. All numbers SCENARIO.
"""
PRICE, BASE_COST, EXT_SHARE, EXT_COST = 0.28, 0.087, 0.15, 0.25 # platform price/task; routed cost (router.py); share on API
def margin(cost): return (PRICE - cost) / PRICE
def main():
print(f"{'shock':36} {'cost/task':>9} {'margin':>7}")
for name, extra in (("none", 0), ("API price x2", EXT_SHARE * EXT_COST), ("API cut off, fallback to Maestro-2", EXT_SHARE * (0.10 - EXT_COST))):
c = BASE_COST + extra; print(f"{name:36} {c:9.3f} {margin(c):7.1%}")
print("\ncut-off also costs quality on new-language tasks (63% -> 48% merge) until Maestro-3")
if __name__ == "__main__":
main()Gross margin per platform task
Real output. A cut-off is cheaper but costs quality on new-language tasks (63% โ 48% merged) until Maestro-3.
Arjun's answer settled it. A doubled price would cost thirteen points of margin; a cut-off would cost quality on one task type, not the business. Both were survivable, as long as the fallback was real: tested every week, not just written in a document.
They shipped the router with three rules. Every route evaluated on the same sealed sets. Every external route with a warm fallback. And no customer data sent to any external model without that customer opting in.
- Owner's notes behind the decision.
- Intelligence commoditises; value moves to its complements (Sosin): the gate, skills, environments and traces are Maestro's, whichever model runs.
- Agents erode price stickiness (Sosin): the same is true of Maestro's own suppliers. Routing is how a buyer of intelligence stays a buyer, not a hostage.
5. Sandbox: build the router
๐ฎ Sandbox: build the router
Same table and rule as router.py. External prices are placeholders, not any provider's list price.
- Try this.
- Set the tolerance to 0: the frontier API also wins dependency upgrades and docs by a single point, at three to fifteen times the cost.
- Set the API price to 3x: new-language code still goes to the API, because a 15-point quality gap outweighs the price.
- Raise the tolerance to 20 points: dependency upgrades move to the small model and new-language code falls back to Maestro-2. Cheaper, and about 3.5 points worse overall.
6. Five mistakes
- Refusing a better model for pride in your own.
- Routing without evaluating every route on the same tests.
- Writing a fallback plan and never running it.
- Sending customer data to an external model without consent.
- Treating the router as plumbing instead of a product decision.
7. Delegation box
- Which external providers are allowed at all
- The customer opt-in policy
- Weekly route evaluations
- Fallback drills
- An external frontier API for one task type
- Hosting for the small open model
8. Budget and risk tracker
| Item | Status after Part 17 | Note |
|---|---|---|
| Routing | Live | 65% / 20% / 15% |
| Quality | 67.1% merged | +2.0 points vs Maestro-2 only |
| Provider risk | Managed | margin 55.5% if the API price doubles |
| Fallback | Drilled weekly | cut-off survivable |
9. The series
The Maestro, Season 2: 23 parts in five phases (Season 1: 18 parts)
10. Reading the numbers
- Primary / standard
- None beyond the methods.
- Secondary โ ๏ธ
- None in this part.
- Our assumptions
- Quality rates, prices, shocks and story events are the scenario. Owner's notes (Sosin, Chase) are summaries; Sosin's views are an investor's opinion.