Adoption Metrics That Matter
Every chart on the company dashboard was going up and to the right. Then a design partner asked a question nobody could answer: how many hours did Maestro actually save us this quarter? This part closes Phase 8 with the uncomfortable discovery that the company had been measuring activity, and the one number it chose instead.
metrics.py and cohort_retention.py were run (real output below). All product and revenue data are scenario numbers; net revenue retention is a standard SaaS measure. Ideas from the owner's startup notes are labelled. Disclosure: Claude, made by Anthropic, helps write this site.
1. Month 46: up and to the right
The quarterly review opened with Sofia's favourite slide. Sessions up sixty percent. Tokens processed up eighty-seven. A new logo wall with eleven enterprise customers on it.
Ines, the lead investor, waited until the end. Then she read out an email from the CTO of their largest design partner. It was polite and very short: Can you tell us how many engineering hours Maestro saved us this quarter? Our CFO is asking.
Nobody in the room could answer. Mira went back to her desk and did not leave until she could.
2. The map: activity vs value
๐ธ๏ธ Which numbers measure value
Red: activity that can rise while value falls. Green: value. The north star at the bottom.
- The pattern to look for.
- When activity grows much faster than value, the product is being used more and helping less per use.
3. What the dashboard did not show
"""metrics.py โ vanity metrics vs value metrics.
Two quarters of product data. Sessions and tokens went up; the metrics tied to
customer value tell a different story. Data are SCENARIO numbers.
"""
Q = {"Q1": dict(sessions=410_000, tokens_b=38.0, tasks=2_100_000, merged=1_090_000, reverted=41_000, hours_saved=520_000),
"Q2": dict(sessions=655_000, tokens_b=71.0, tasks=2_900_000, merged=1_310_000, reverted=88_000, hours_saved=560_000)}
def main():
a, b = Q["Q1"], Q["Q2"]
for k in ("sessions", "tokens_b"): print(f"vanity {k:12} {a[k]:>12,} -> {b[k]:>12,} ({b[k] / a[k] - 1:+.0%})")
acc = lambda q: q["merged"] / q["tasks"]; kept = lambda q: (q["merged"] - q["reverted"]) / q["merged"]
print(f"value merge rate {acc(a):12.1%} -> {acc(b):12.1%}")
print(f"value kept 30 days {kept(a):12.1%} -> {kept(b):12.1%}")
print(f"value hours saved {a['hours_saved']:>12,} -> {b['hours_saved']:>12,} ({b['hours_saved'] / a['hours_saved'] - 1:+.0%})")
print(f"\nhours saved per 1k sessions: {a['hours_saved'] / a['sessions'] * 1000:.0f} -> {b['hours_saved'] / b['sessions'] * 1000:.0f}")
if __name__ == "__main__":
main()Q1 โ Q2
Real output (merge rate and value per session shown as relative change). Negative bars are drawn at their size; red means down.
Mira's answer arrived at two in the morning, in a document titled We are busier, not better. New platform customers were throwing harder, messier tasks at Maestro. The agent tried more, flailed more, and burned more tokens doing it. Merged work still grew, but only by eight percent in hours saved, against sixty percent more sessions.
The good news, she wrote, is that every one of these failures is a trace. We know exactly where to look.
- Owner's notes behind the fix.
- Traces as ground truth and evals as the new PRD (Chase): the failing sessions became next quarter's evaluation set.
- Ambient intelligence (Matias): the best agent is noticed least. A CI fixer that quietly merges overnight produces no sessions at all, so sessions cannot be the goal.
4. The number investors trust
"""cohort_retention.py โ net revenue retention by customer cohort.
Monthly recurring revenue ($k) per cohort at month 0, 6 and 12 after signing.
Net revenue retention (NRR) = revenue a cohort pays now / what it paid at
start, including expansion and churn. Data are SCENARIO numbers.
"""
COH = {"PR-bot cohort (month 14)": (300, 318, 345), "Platform early (month 30)": (520, 690, 860), "Enterprise (month 38)": (900, 1150, None)}
def main():
for n, (m0, m6, m12) in COH.items():
line = f"{n:28} start ${m0}k m6 {m6 / m0:5.0%}"
if m12: line += f" m12 {m12 / m0:5.0%}"
print(line)
print("\nNRR above 100% means existing customers grow faster than any churn: the platform cohort is the business")
if __name__ == "__main__":
main()Revenue from each cohort vs its first month
Real output. Platform customers expand fastest: they build more agents once the first ones work.
The cohort table did what the dashboard could not. The PR-bot customers from Season 1 were steady. The platform customers were growing their spend by two-thirds a year. The enterprise cohort was too young to judge, and already above 125%.
The company changed its north star that week: merged changes still in place after thirty days, per customer, per month. Sessions came off the board slide. Nobody missed them.
5. Sandbox: grow the right number
๐ฎ Sandbox: grow the right number
Scales Q2 from metrics.py: tasks per session, 93.3% kept and about 0.46 hours saved per kept change are held fixed. Scenario numbers.
- Try this.
- Double sessions with the merge rate fixed: everything scales, and value per session stays the same.
- Hold sessions and raise the merge rate from 45% to 55%: about 22% more kept changes, with no growth in usage at all.
- Halve sessions and push the merge rate to 70%: fewer sessions, value per session up by more than half. That is the ambient-agent future.
6. Five mistakes
- Reporting sessions and tokens to the board as success.
- Letting a falling merge rate hide behind rising volume.
- Waiting for a customer to ask how much time you saved them.
- Averaging retention across very different cohorts.
- Choosing a north star that the product can raise by flailing.
7. Delegation box
- The north-star metric and the board dashboard
- Turning failing traces into evaluation sets
- Sampling hours-saved estimates with customers
- Customer finance teams for hours-saved validation
8. Budget and risk tracker
| Item | Status after Part 19 | Note |
|---|---|---|
| North star | Merged changes kept 30 days | per customer, per month |
| NRR | Platform 165% | enterprise 128% at 6 months |
| Merge rate | 45.2%, falling | harder tasks from new users |
| Phase 8 | Complete | product grown up |
9. The series
The Maestro, Season 2: 23 parts in five phases (Season 1: 18 parts)
10. Reading the numbers
- Primary / standard
- Net revenue retention as a standard SaaS metric.
- Secondary โ ๏ธ
- None in this part.
- Our assumptions
- All product, revenue and cohort data and all story events are the scenario. Owner's notes (Chase, Matias) are summaries, not transcripts.