Operations and the Flywheel
Shipping is a beginning. From now on Maestro runs every day for paying clients: models are updated every two weeks by the loop, the service must stay up, costs must stay down, incidents must be handled, and the verified-skill flywheel must keep turning without drifting. This part covers service-level objectives and error budgets (with a burn-rate script), progressive rollouts and rollback, the weekly operating rhythm, a simulation of whether the flywheel compounds or stalls, cost operations, and how one maestro hands parts of the job to a small team.
burn_rate.py and flywheel.py were run (real output below) on simulated error rates and scenario parameters. rollout.yaml was not executed for this article. Operating results are the scenario. Disclosure: Claude, made by Anthropic, helps write this site.
1. Mission log: months 15โ18
- Defines service-level objectives with partners; writes
burn_rate.py; sets alerting.
- Weekly operations review; two-week loop cycles shipping through canaries; one severity-1 incident (section 6).
- First two hires take on-call and the release pipeline; Gopala keeps the gate and the data agreements.
2. Promises in numbers: SLOs
- A service-level objective says what "working" means, measurably.
- Availability: 99.5% of agent requests succeed over 30 days.
- Latency: interactive first token within 1.5 s for 95% of requests.
- Task quality: accepted-change rate per client does not fall below its baseline interval (Part 12).
- The error budget is what the SLO allows to fail.
- At 99.5%, about 216 minutes of full outage a month (or many small errors adding up to the same).
- Inside the budget, the team ships changes; when it is spent, changes stop and reliability work starts.
Maestro's service-level objectives
Our objectives for the story; the first two are standard SRE practice, the last two product-specific.
3. Alerting on the budget, not on every error
- Burn rate: how fast the budget is being used.
- A burn rate of 1 uses exactly the monthly budget in a month; 14.4 uses 2% of it in an hour.
- The Google SRE Workbook recommends paging only when the burn is fast in both a long window and a short one, so alerts are both significant and current.
"""burn_rate.py โ SLO error budgets and multi-window burn-rate alerts.
SLO: 99.5% of agent requests succeed (non-5xx, under the latency limit)
over 30 days. Burn rate = error rate / allowed error rate. The Google SRE
Workbook suggests paging when the budget burns fast in both a long and a
short window (e.g. 14.4x over 1 h and 5 min). Error rates are SIMULATED.
"""
SLO = 0.995
ALLOWED = 1 - SLO
RULES = [("page", 14.4, "1h", "5m"), ("page", 6.0, "6h", "30m"), ("ticket", 1.0, "3d", "6h")]
WINDOWS = { # scenario error rates per window during an incident
"5m": 0.09, "30m": 0.05, "1h": 0.08, "6h": 0.02, "3d": 0.004,
}
def main():
print(f"30-day budget: {ALLOWED:.1%} of requests = {ALLOWED*30*24*60:.0f} minutes of full outage")
for sev, factor, long_w, short_w in RULES:
bl = WINDOWS[long_w] / ALLOWED; bs = WINDOWS[short_w] / ALLOWED
fire = bl >= factor and bs >= factor
print(f"{sev:6s} if burn >= {factor:4.1f}x over {long_w:>3s} and {short_w:>3s}: "
f"{bl:5.1f}x / {bs:5.1f}x -> {'FIRE' if fire else 'quiet'}")
hours_left = 30 * 24 * ALLOWED / WINDOWS["1h"]
print(f"at the current 1 h error rate the whole month's budget lasts {hours_left:.1f} hours")
if __name__ == "__main__":
main()- Reading it.
- At an 8% error rate the monthly budget would last 45 hours; the fast rule fires and pages.
- The slower rules stay quiet because the incident is recent: they catch slow leaks, not sudden outages.
4. Try the burn rate
Error budget calculator
5. Shipping a new model safely
Progressive rollout of a loop update
Any failed analysis rolls back automatically.
- Two kinds of signal decide.
- Operational: errors and latency (the SLOs).
- Quality: accepted-change rate and reverts for canary traffic vs the old model, with intervals.
- Tools do the mechanics.
- A canary controller shifts traffic and runs the analysis; the sketch below follows the style of Argo Rollouts. It was written for this article and not run.
# rollout.yaml โ progressive delivery of a new Maestro-1 version (sketch in the
# style of a Kubernetes canary controller such as Argo Rollouts; field names
# follow its documentation, check them for your version).
apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata: {name: maestro-1}
spec:
strategy:
canary:
steps:
- setWeight: 5 # 5% of conversations, design partners who opted in
- pause: {duration: 24h}
- analysis: {templates: [{templateName: maestro-slo-and-acceptance}]}
- setWeight: 25
- pause: {duration: 48h}
- analysis: {templates: [{templateName: maestro-slo-and-acceptance}]}
- setWeight: 100
abortScaleDownDelaySeconds: 3600 # keep the old version warm for instant rollback6. An incident, start to finish
- Month 16, 14:05: a dependency update in the sandbox image breaks test runs for Python 3.9 repositories; error rate jumps to 8%; the fast burn-rate rule pages at 14:11.
- 14:20: the canary controller is not involved (no model change), so on-call rolls back the sandbox image; errors fall by 14:32.
- Next day: post-incident review adds an image canary and a test against the oldest supported Python version. About 10% of the month's error budget was used.
- These events are the story.
- What made it short.
- An alert that fired within minutes because it watched the budget, not individual errors.
- Everything versioned, so "roll back the last change" was one command.
- A blameless review that produced a test, not a culprit.
7. The weekly operating rhythm
| Day | Meeting or task | Inputs |
|---|---|---|
| Monday | Operations review (30 min) | SLOs, budget burn, incidents, cost per solved task |
| Tuesday | Loop cycle planning | New partner tasks, last gate decisions |
| Wednesday | Client reviews (rotating) | Acceptance, edits, reverts per client |
| Thursday | Safety review | Monitoring samples, red-team updates (Part 13) |
| Friday | Release and gate sign-off (every other week) | Gate report, canary results |
- Measure the team too.
- DORA's four key metrics (deployment frequency, lead time for changes, change failure rate, time to restore service) are tracked monthly for the product code and the model pipeline alike.
8. Is the flywheel compounding?
- Part 1's bet was that verified skills compound.
- More use brings more tasks; more tasks bring more verified skills; approved updates raise success; higher success brings more use.
- But each new skill overlaps more with the library, so its value falls; the question is whether growth beats the overlap.
"""flywheel.py โ does the verified-skill flywheel compound or stall?
Each two-week cycle: tasks attempted -> verified skills -> (some) gate
approvals -> success rate rises a little, which brings more tasks.
Diminishing returns: each new skill helps less as the library grows.
All parameters are SCENARIO assumptions; the point is the shape.
"""
import math
def run(cycles=12, tasks=400, success=0.48, approve=0.5, gain_per_k=0.012, overlap=0.0004):
lib = 0
print(f"{'cycle':>5s} {'tasks':>6s} {'success':>8s} {'new skills':>10s} {'library':>8s}")
for c in range(1, cycles + 1):
solved = tasks * success
new = solved * math.exp(-overlap * lib) # more overlap with existing skills later
lib += new
if c % 2 == 0: # gate approves an update every other cycle on average
success = min(0.85, success + approve * gain_per_k * (new / 100))
tasks = int(tasks * (1.04 + (success - 0.48))) # better product -> more usage
if c in (1, 3, 6, 9, 12):
print(f"{c:5d} {tasks:6d} {success:8.1%} {new:10.0f} {lib:8.0f}")
if __name__ == "__main__":
run()Flywheel over 12 two-week cycles (scenario)
Real output of flywheel.py on scenario parameters.
- The shape, not the numbers.
- Success rises slowly (48% to about 55% over 12 cycles in this scenario) because each approved update is small and new skills overlap more with the library.
- Usage grows faster than success, which is what keeps new skills arriving despite the overlap.
- The lesson for operations: watch new skills per cycle; if it falls while usage grows, the library is saturating and the loop needs new kinds of tasks.
9. Cost operations
- Three numbers reviewed every week.
- Serving cost per solved task (Part 16).
- Idle share of reserved GPUs, day and night (Part 15).
- Loop cost per approved update (attempts, training, review time).
- Typical actions.
- Move background work into night hours; tighten step limits on task types that rarely succeed; raise cache hit rates by improving routing.
10. One maestro, then a small team
Hats handed over by month 18 (scenario)
He gives away operations first and keeps what defines the company.
- What he keeps and why.
- The gate, the data agreements and the research direction are where a mistake would damage the moat or the trust behind it.
11. Watching for drift
- The world changes under a deployed model.
- Clients adopt new frameworks and language versions; their repositories grow; task mixes shift.
- Signals of drift.
- Acceptance rate falling for one client or one language while others are steady.
- More stop-rule exits (Part 16) for a task type.
- New error types in sandbox logs.
- Responses.
- New tasks of the drifting type for the loop (Part 11); new regression tests; sometimes a client-specific skill set.
12. Operating the data agreements
- Every skill in the library is traceable to a client agreement.
- Removal requests are executed within the agreed time: the skill leaves the library at once and future training runs exclude it (Part 8's manifest).
- Audits.
- Quarterly, a sample of skills is checked for leaked identifiers and for a valid agreement; failures trigger a wider review.
13. Learning from incidents
- Every severity-1 and severity-2 incident gets a written review within a week.
- Timeline, impact, what went well, what went badly, and actions with owners and dates.
- Blameless: the question is what in the system allowed the mistake, not who made it.
- Actions become tests.
- Most reviews end with a new check: an alert, a canary step, a regression test or a guard rule. That is how operations improve without growing headcount.
From incident to prevention
Each incident should make the next one less likely.
14. Capacity planning, monthly
- Three curves are compared every month.
- Demand: solved tasks and interactive requests, by hour of day.
- Capacity: serving replicas at the load-tested limit (Part 15).
- Commitments: reserved GPUs and their renewal dates (Part 4).
- Decisions it drives.
- Whether to add reserved serving GPUs before the next renewal.
- How much of the training block to keep after Maestro-1, given Maestro-1.1 loop work and Maestro-2 plans.
- When a second region is justified.
15. Security in daily operations
- Security is a routine, not a project.
- Weekly: review guard blocks and confirmations (Part 13); patch the sandbox image through its canary.
- Monthly: rotate credentials; review access lists for the training data, release set and production.
- Quarterly: a safety drill (Part 13) and an access review with each partner.
- The same on-call handles both reliability and security incidents, with separate runbooks.
16. Agents helping run the company
- Gopala uses AI agents for operations work, under the same rules as the product.
- Read-only access to logs and dashboards; drafting incident timelines and weekly reports; triaging alerts into tickets.
- They cannot change production, approve releases or touch the gate.
- The test for delegating a task to an agent.
- Is its output reviewed by a person before anything changes? If yes, delegate; if no, keep it human.
17. Seven operations mistakes
- Paging on every error instead of on budget burn.
- Shipping model updates without a canary.
- Rolling back models but not the sandbox, tools and prompts around them.
- Measuring uptime but not task quality.
- Assuming the flywheel compounds without watching new skills per cycle.
- Blame-driven incident reviews.
- Handing over the gate before handing over on-call.
18. Delegation box
- SLOs and their trade-offs with clients
- The gate and sign-off
- What to hand over and when
- Incident timelines and first-draft reviews
- Weekly operations reports
- Cost anomaly detection
- Open-source monitoring and rollout tools
- Two hires: SRE and product engineer
- The GPU provider's support
19. Budget and risk tracker
| Item | Status after Part 17 | Note |
|---|---|---|
| Spend to date | ~$11.4M (scenario) | Serving, operations, first hires |
| Operations | SLOs met 2 of 3 months (scenario) | One severity-1 incident |
| Time | Month ~18 | Scorecard next |
| Reliability risk | Managed | Burn-rate alerts, canaries |
| Flywheel risk | Watch | Gains per cycle are small (scenario) |
| Key-person risk | Reduced | On-call and pipeline handed over |
20. The series
The Maestro: 18 parts in five phases
- Operations turn a launch into a business: objectives in numbers, alerts that fire on what matters, rollouts that can undo themselves, and a weekly rhythm that watches both the service and the flywheel.
- The simulation was sobering: the loop compounds, but slowly.
- Next, Part 18: the scorecard, and what Gopala would do differently.
21. Reading the numbers
- Primary / standard
- Error budgets and multi-window burn-rate alerting (14.4ร over 1 h and 5 min, 6ร over 6 h and 30 min, 1ร over 3 days and 6 h) (Google SRE Workbook). DORA four key metrics (DORA research programme).
- Secondary โ ๏ธ
- None in this part.
- Our assumptions
- SLO targets, simulated error rates, flywheel parameters, incident details, hires and costs.