The Maestro-2 Run
Sixty-two days, 4,096 GPUs and about fifteen million dollars, all spent on one file of numbers that will either be the company's next product or its most expensive lesson. This part follows the Maestro-2 run from launch to the last step: the failures, the night the loss jumped three times in a row, and the evaluation that waits at the end.
run_sim.py (a seeded simulation, not a real run) and loss_watch.py (on a synthetic loss curve) were run; real output below. The failure rate is from Meta's Llama 3 report; the spike-handling rules mirror published practice (PaLM, Llama 3). Everything that happens in the story is the scenario. Disclosure: Claude, made by Anthropic, helps write this site.
1. Day 0: freezing the recipe
At 9 a.m. on the first Monday of month 38, Tomรกs read the launch checklist aloud in the operations room, the way he had for Maestro-1. Recipe frozen. Data mixture frozen and hashed. Cluster burn-in passed on 4,032 GPUs with 64 hot spares. Checkpoint interval 54 minutes. Evaluation set written down and sealed: the 83 model-only failures from Part 10, plus four hundred held-out tasks nobody on the training team had seen.
Gopala signed the last line. Mira pressed the key. On the big screen the step counter went from 0 to 1, and for a few seconds nobody said anything.
Sixty-two days, said Arjun. If we are lucky.
2. The map: sixty-two days in one picture
๐ธ๏ธ What happens during a long run
Steps move forward, failures push back, checkpoints limit the damage, and the loss watcher decides when to skip and when to roll back.
- Two loops run at once.
- The reliability loop: fail, restart from a checkpoint, carry on.
- The stability loop: watch the loss, skip a bad batch, or roll back.
3. Days 1โ62: the reliability loop (scenario)
- The day-by-day log is a seeded simulation of the plan, not a real training run.
"""run_sim.py โ 4,096 GPUs, one long run, simulated day by day.
Interruptions arrive at the Llama 3 rate (419 in 54 days on 16,384 GPUs).
Each costs a restart plus, on average, half a checkpoint interval of lost work.
Seeded random numbers make the output repeatable. Restart time and the planned
step budget are SCENARIO assumptions; this is a simulation, not a real run.
"""
import random
random.seed(7)
GPUS, RATE, CKPT_MIN, RESTART_MIN, IV = 4096, 419 / (16384 * 54), 2.0, 15.0, 54
NEED_HOURS = 56.0 * 24 # productive hours needed (FP8 plan: ~56 days of pure compute)
def main():
done, day, intr, lost = 0.0, 0, 0, 0.0
while done < NEED_HOURS:
day += 1; k = sum(1 for _ in range(GPUS) if random.random() < RATE)
l = (k * (RESTART_MIN + IV / 2) + (1440 / IV) * CKPT_MIN) / 60
intr += k; lost += l; done += 24 - l
if day % 10 == 0: print(f"day {day:3}: {intr:4} interruptions so far, {done / NEED_HOURS:5.1%} of the run done")
print(f"\nfinished on day {day}: {intr} interruptions, {lost:.0f} hours lost, goodput {(day * 24 - lost) / (day * 24):.1%}")
if __name__ == "__main__":
main()The run in four numbers
Real output of the simulation. The plan in Part 11 expected 91%.
The first week was the worst. Twenty-one interruptions in ten days: a GPU that threw memory errors, a network switch that dropped links under load, a storage node that slowed every checkpoint to four minutes. Each one paged the on-call engineer, each one cost about forty minutes, and each one taught the team something about a cluster they had only known for six weeks.
By day thirty the rhythm had changed. Failures still came, but the replacement clause from Part 11 worked: a dead node was swapped for a hot spare in minutes, not days. Days 20 to 30 were the quietest of the run: nine interruptions.
4. Night 23: the stability loop (scenario)
- The loss curve and its spikes are synthetic; the watcher's rules are the kind large runs use.
"""loss_watch.py โ spotting a loss spike before it becomes a divergence.
A synthetic loss curve (falling, with noise) has three injected spikes. The
watcher compares each step with a rolling mean and standard deviation of the
last 50 steps: z > 4 once = skip that batch; z > 4 three steps in a row =
roll back to the last checkpoint and lower the learning rate. The curve is
SYNTHETIC; the rules mirror common practice.
"""
import math, random, statistics as st
random.seed(3)
SPIKES = {1200: [1.2], 2600: [0.9, 1.1, 1.3, 1.2], 4100: [0.7]}
def curve(n=5000):
out, i = [], 0
while i < n:
base = 1.85 + 2.2 * math.exp(-i / 900) + random.gauss(0, 0.02)
if i in SPIKES:
for s in SPIKES[i]: out.append(base + s); i += 1
continue
out.append(base); i += 1
return out
def main():
ys, hot = curve(), 0
for i in range(50, len(ys)):
w = ys[i - 50:i]; z = (ys[i] - st.mean(w)) / (st.stdev(w) + 1e-9)
hot = hot + 1 if z > 4 else 0
if hot == 1: print(f"step {i}: z={z:5.1f} skip batch")
if hot == 3: print(f"step {i}: z={z:5.1f} 3 in a row -> ROLL BACK to last checkpoint, lower LR 20%")
print(f"\nfinal loss {st.mean(ys[-50:]):.3f}; spikes handled without a restart from scratch")
if __name__ == "__main__":
main()At 2:40 a.m. on night twenty-three the alert went off three times in thirty seconds. Mira was already awake; she had been on call for the whole week the FP8 kernels were being watched.
The first spike could have been a bad batch. The second, two steps later, could not. By the third the watcher had made the decision for her: roll back to the checkpoint from 41 minutes earlier, lower the learning rate by a fifth, and skip the stretch of data that had caused it. At 3:20 the loss was back on its curve. The team found the cause two days later: a block of machine-generated documents that had slipped past the filters.
An hour lost, Mira wrote in the run log. A divergence would have cost us a week.
What the watcher did
Real output of loss_watch.py on the synthetic curve.
5. Day 62: the last step, and the real test
The run ended on a Tuesday afternoon, which felt wrong for something that had taken two months. The step counter stopped. The final checkpoint was copied to three locations, the third an encrypted vault with access rules written for the Part 12 obligations.
Nobody opened champagne. Elena's team started the sealed evaluation within the hour. Gopala went home and, for the first time in sixty-two days, turned his phone off.
The results would take a week. Whether Maestro-2 was a product, or the company's most expensive lesson, was now a question for Phase 8.
- What the run proved, and what it did not.
- Proved: the plan held. 62 days, 90.8% goodput, one rollback, no restart from scratch.
- Not yet proved: that the model fixes the 21% of failures it was built for. That is the sealed evaluation, and the start of Phase 8.
6. Sandbox: run it your way
๐ฎ Sandbox: run it your way
Expected values with the same rates as run_sim.py (Llama 3 failure rate, 2-minute checkpoints, 56 days of compute on 4,096 GPUs). The script adds random day-to-day variation.
- Try this.
- Set restarts to 60 minutes: about four more days and over $1M more.
- Checkpoint every 4 hours instead of every 54 minutes: you save on saving and lose much more on each failure.
- Find the best checkpoint interval for a 5-minute restart: it is still close to Young's 54 minutes, because the restart time does not enter the formula.
7. Five mistakes
- Writing the evaluation set after seeing the results.
- Treating every loss spike as a bad batch.
- Restarting from scratch after a divergence that a rollback would have fixed.
- Storing the only copy of the final checkpoint on the training cluster.
- Declaring success when the run finishes, before the evaluation.
8. Delegation box
- The launch and the go/no-go
- The final copy of the weights
- Loss and failure monitoring
- Run log and nightly summaries
- 4,096 GPUs and on-call support from the provider
9. Budget and risk tracker
| Item | Status after Part 14 | Note |
|---|---|---|
| Run | Complete ยท day 62 | simulated plan |
| Compute | ~$14.5M | FP8 plan from Part 13 |
| Reliability | 90.8% goodput | 117 interruptions |
| Quality | Pending | sealed evaluation, Phase 8 |
10. The series
The Maestro, Season 2: 23 parts in five phases (Season 1: 18 parts)
11. Reading the numbers
- Primary / standard
- Llama 3 reliability statistics (Meta, 2024); loss-spike handling by skipping batches and restarting from earlier checkpoints (PaLM, Chowdhery et al., 2022).
- Secondary โ ๏ธ
- None in this part.
- Our assumptions
- The simulation, the synthetic loss curve, the failures and all story events are the scenario.