Scaling the Machine
Maestro-1 trained on 1,024 GPUs. Maestro-2 needs four times as many for three months, and at that scale the problem stops being "how fast" and becomes "how often does it break, and who fixes it". This part follows the infrastructure lead through the cluster decision, the arithmetic of failure and the contract that turned out to matter more than the price.
cluster_plan.py and failure_rate.py were run (real output below). The failure rate comes from Meta's published Llama 3 report; H100 peak throughput is NVIDIA's figure; MFU, prices and restart times are scenario assumptions. Disclosure: Claude, made by Anthropic, helps write this site.
1. Month 35: the call from the provider
The call came on a Sunday, which Arjun Mehta took as a bad sign. The provider that had run Maestro-1's 1,024 GPUs had a block of 4,096 coming free in six weeks. Another customer had first refusal until Wednesday.
Price is fine, Arjun told Gopala on Monday morning. The price is not the problem. The problem is that at four thousand GPUs something breaks every twelve hours, and their contract says they will replace a failed node within three business days.
Gopala did the arithmetic in his head and did not like it. Three business days of a missing node, twice a day, for three months.
- What is being decided.
- How many GPUs to rent, for how long, and under which terms.
- At this scale, the terms that matter most are the ones about failure.
2. The map: size, speed and failure
๐ธ๏ธ Why 4,096 GPUs, and why the contract matters
Bigger clusters finish sooner but each GPU works less efficiently, and failures grow with every GPU you add.
- Three forces pull against each other.
- Speed: more GPUs, fewer days.
- Efficiency: more GPUs, more communication, lower MFU.
- Reliability: more GPUs, more interruptions, more lost work.
3. Two months, three months or five
"""cluster_plan.py โ how many GPUs for Maestro-2, and for how long?
Days = FLOPs / (GPUs x peak x MFU x goodput x 86,400). Bigger clusters finish
sooner but lose some efficiency (MFU) and more time to failures (goodput).
Peak is the H100's dense BF16 figure; goodput is 1 minus the time lost per day in failure_rate.py;
MFU and prices are SCENARIO
assumptions.
"""
FLOPS, PEAK, PRICE = 9.6e24 * 1.10, 989e12, 2.40 # B2 plus 10% headroom
def main():
print(f"target {FLOPS:.2e} FLOPs (option B2 + 10% headroom)\n")
print(f"{'GPUs':>6} {'MFU':>5} {'goodput':>8} {'days':>6} {'GPU-hours':>10} {'cost $M':>8}")
for g, mfu, gp in ((2048, 0.42, 0.94), (4096, 0.40, 0.91), (8192, 0.37, 0.86)):
days = FLOPS / (g * PEAK * mfu * gp * 86400); gh = g * days * 24
print(f"{g:6} {mfu:5.0%} {gp:8.0%} {days:6.0f} {gh / 1e6:9.2f}M {gh * PRICE / 1e6:8.1f}")
if __name__ == "__main__":
main()Days to train Maestro-2 (BF16)
Real output. Doubling the cluster does not halve the time, and every doubling costs more in total.
- Why 4,096.
- 2,048 is cheapest but five months is too long: a single bad bug discovered in month three costs a quarter.
- 8,192 finishes in seven weeks but costs $2.8M more and fails four times a day.
- 4,096 is the knee: under three months in BF16, about two months with FP8 (Part 13).
What each doubling buys
From the real output of cluster_plan.py above. Each day saved costs over three times more after the knee.
๐ Go deeper: why not just use more GPUs, if the work is the same?
- The job is fixed (1.06e25 operations), but each extra GPU does less useful work.
- 2,048 GPUs need 7.51M GPU-hours; 8,192 need 9.32M, 24% more for the same model.
- Communication: after every step all GPUs exchange results, so more GPUs means more waiting.
- MFU falls from 42% to 37% in the scenario.
- Failures: one failed GPU stops everyone.
- Lost time grows from 6% to 14% of each day (failure_rate.py, Part 11).
- Batch size: more GPUs means bigger batches, and past a "critical batch size" bigger batches stop saving steps.
- Not modelled in our scripts, but a real limit in large runs.
- When bigger is still right: when time is worth more than money.
- A rival launch or a funding milestone can justify $78k per day saved. Frontier labs run far larger clusters with better networks and faster recovery, and still accept lower efficiency for speed.
4. The arithmetic of failure
Arjun put the Llama 3 report on the screen. Meta's team had published something most labs keep private: in a 54-day snapshot on 16,384 GPUs, there were 419 unexpected interruptions. Faulty GPUs, memory errors, network links, host software.
That is about one interruption per two thousand GPUs per day, Arjun said. On four thousand, two a day. Every one of them stops the whole run.
"""failure_rate.py โ interruptions scale with GPU count.
Rate from Meta's Llama 3 report: 419 unexpected interruptions in a 54-day
snapshot on 16,384 H100 GPUs. Checkpoint interval from Young's approximation,
interval = sqrt(2 x checkpoint cost x mean time between failures).
Checkpoint and restart times are SCENARIO assumptions.
"""
import math
RATE = 419 / (16384 * 54) # interruptions per GPU-day (Llama 3 snapshot)
CKPT_MIN, RESTART_MIN = 2.0, 15.0 # minutes to save a checkpoint; to detect, replace and restart
def main():
print(f"rate: {RATE:.2e} interruptions per GPU-day\n")
print(f"{'GPUs':>6} {'per day':>8} {'MTBF h':>7} {'ckpt every':>10} {'lost/day':>9}")
for g in (2048, 4096, 8192, 16384):
per_day = RATE * g; mtbf = 24 * 60 / per_day; iv = math.sqrt(2 * CKPT_MIN * mtbf)
lost = per_day * (RESTART_MIN + iv / 2) + (24 * 60 / iv) * CKPT_MIN # minutes per day
print(f"{g:6} {per_day:8.1f} {mtbf / 60:7.1f} {iv:8.0f} m {lost / 1440:9.1%}")
if __name__ == "__main__":
main()Share of each day lost to failures
Real output. Restarts take 15 minutes and a checkpoint 2 minutes in this scenario; faster restarts are worth real money.
๐ Go deeper: Young's formula in one line
- Save too often and you waste time saving; too rarely and each failure throws away more work.
- The best interval is about โ(2 ร time to save ร mean time between failures): 54 minutes for 4,096 GPUs.
- Derived by John Young in 1974 for mainframes; still the starting point today.
5. The contract, not the price (scenario)
- Terms and negotiation are the story's scenario; the clauses are the kind large GPU contracts contain.
They signed on Wednesday at 4 p.m., two hours before the deadline. The price had not moved. Three other things had.
A four-hour replacement window for failed nodes instead of three business days. A hot-spare pool of 64 GPUs kept idle inside the reservation so the run could restart immediately on healthy machines. And a clause that credited back any day on which the provider's own network delivered less than the agreed bandwidth.
We paid for 4,096, Arjun said afterwards, and we train on 4,032. Those 64 idle GPUs are the cheapest insurance we will ever buy.
What was negotiated
Scenario terms.
6. Sandbox: choose the cluster
๐ฎ Sandbox: choose the cluster
Failure rate from the Llama 3 report, Young's checkpoint formula, MFU falling as the cluster grows. MFU and prices are scenario assumptions; the sandbox interpolates MFU between the script's three points.
- Try this.
- Slide to 16,384 GPUs: about a month, but nearly a quarter of every day lost to failures.
- Set the restart time to 60 minutes on 4,096 GPUs: goodput falls by about six points. Fast recovery is worth engineering.
- Go back to 1,024 (Maestro-1's size): more than nine months.
7. Five mistakes
- Negotiating price and ignoring replacement terms.
- Assuming a bigger cluster is proportionally faster.
- Checkpointing on a fixed schedule copied from a smaller run.
- Planning a run with no headroom for restarts and repeated work.
- Running with zero hot spares to use every GPU you paid for.
8. Delegation box
- Signing the reservation and its risk terms
- Failure-rate and checkpoint modelling
- Drafting the provider comparison
- 4,096 H100-class GPUs from a cloud provider
- Network and storage from the same provider
9. Budget and risk tracker
| Item | Status after Part 11 | Note |
|---|---|---|
| Cluster | 4,096 GPUs reserved | 3 months + option |
| Compute | $19.6M BF16 plan | FP8 may cut it (Part 13) |
| Recovery | 4-hour replacement | 64 hot spares |
| Failures | ~1.9 a day expected | checkpoint every 54 min |
10. The series
The Maestro, Season 2: 23 parts in five phases (Season 1: 18 parts)
11. Reading the numbers
- Primary / standard
- Llama 3 interruption statistics (Meta, 2024); H100 dense BF16 peak (NVIDIA data sheet); Young (1974) checkpoint interval.
- Secondary โ ๏ธ
- None in this part.
- Our assumptions
- MFU, prices, restart and checkpoint times, contract terms and all story events are the scenario.