Home โ€บ Blog โ€บ The Maestro, Season 2 ยท Part 11
The Maestro, Season 2 ยท Part 11 ยท Phase 7 ยท ๐Ÿ–ฅ๏ธ Infrastructure hat

Scaling the Machine

Maestro-1 trained on 1,024 GPUs. Maestro-2 needs four times as many for three months, and at that scale the problem stops being "how fast" and becomes "how often does it break, and who fixes it". This part follows the infrastructure lead through the cluster decision, the arithmetic of failure and the contract that turned out to matter more than the price.

FL
FrontierAGI Team
๐Ÿ’ผ CEO๐Ÿ–ฅ๏ธ Infrastructure๐Ÿ’ฐ Finance
โš ๏ธ How to read this (as of 4 Nov 2026) Gopala's company is a teaching scenario. cluster_plan.py and failure_rate.py were run (real output below). The failure rate comes from Meta's published Llama 3 report; H100 peak throughput is NVIDIA's figure; MFU, prices and restart times are scenario assumptions. Disclosure: Claude, made by Anthropic, helps write this site.

1. Month 35: the call from the provider

๐Ÿ“– The story

The call came on a Sunday, which Arjun Mehta took as a bad sign. The provider that had run Maestro-1's 1,024 GPUs had a block of 4,096 coming free in six weeks. Another customer had first refusal until Wednesday.

Price is fine, Arjun told Gopala on Monday morning. The price is not the problem. The problem is that at four thousand GPUs something breaks every twelve hours, and their contract says they will replace a failed node within three business days.

Gopala did the arithmetic in his head and did not like it. Three business days of a missing node, twice a day, for three months.

2. The map: size, speed and failure

๐Ÿ•ธ๏ธ Why 4,096 GPUs, and why the contract matters

or click any node

Bigger clusters finish sooner but each GPU works less efficiently, and failures grow with every GPU you add.

3. Two months, three months or five

๐Ÿ”ฌ In plain words: MFU A GPU has a peak speed printed on its data sheet (989 trillion operations a second for an H100 in BF16). A real training run never reaches it, because GPUs wait for data, for each other and for the network. Model FLOPs utilisation is the share of the peak you actually get. 40% is a good large-scale result.
cluster_plan.py ยท Python 3, no dependencies; NVIDIA peak figure, scenario MFU and prices
"""cluster_plan.py โ€” how many GPUs for Maestro-2, and for how long?

Days = FLOPs / (GPUs x peak x MFU x goodput x 86,400). Bigger clusters finish
sooner but lose some efficiency (MFU) and more time to failures (goodput).
Peak is the H100's dense BF16 figure; goodput is 1 minus the time lost per day in failure_rate.py;
MFU and prices are SCENARIO
assumptions.
"""
FLOPS, PEAK, PRICE = 9.6e24 * 1.10, 989e12, 2.40   # B2 plus 10% headroom

def main():
    print(f"target {FLOPS:.2e} FLOPs (option B2 + 10% headroom)\n")
    print(f"{'GPUs':>6} {'MFU':>5} {'goodput':>8} {'days':>6} {'GPU-hours':>10} {'cost $M':>8}")
    for g, mfu, gp in ((2048, 0.42, 0.94), (4096, 0.40, 0.91), (8192, 0.37, 0.86)):
        days = FLOPS / (g * PEAK * mfu * gp * 86400); gh = g * days * 24
        print(f"{g:6} {mfu:5.0%} {gp:8.0%} {days:6.0f} {gh / 1e6:9.2f}M {gh * PRICE / 1e6:8.1f}")

if __name__ == "__main__":
    main()
$ python3 cluster_plan.py # three cluster sizes target 1.06e+25 FLOPs (option B2 + 10% headroom) GPUs MFU goodput days GPU-hours cost $M 2048 42% 94% 153 7.51M 18.0 4096 40% 91% 83 8.15M 19.6 8192 37% 86% 47 9.32M 22.4

Days to train Maestro-2 (BF16)

2,048 GPUs ยท $18.0M153 days
4,096 GPUs ยท $19.6M83 days
8,192 GPUs ยท $22.4M47 days

Real output. Doubling the cluster does not halve the time, and every doubling costs more in total.

What each doubling buys

2,048 โ†’ 4,096 GPUs+$1.6M saves 70 days ยท $23k/day
4,096 โ†’ 8,192 GPUs+$2.8M saves 36 days ยท $78k/day

From the real output of cluster_plan.py above. Each day saved costs over three times more after the knee.

๐Ÿ”Ž Go deeper: why not just use more GPUs, if the work is the same?
  • The job is fixed (1.06e25 operations), but each extra GPU does less useful work.
    • 2,048 GPUs need 7.51M GPU-hours; 8,192 need 9.32M, 24% more for the same model.
  • Communication: after every step all GPUs exchange results, so more GPUs means more waiting.
    • MFU falls from 42% to 37% in the scenario.
  • Failures: one failed GPU stops everyone.
    • Lost time grows from 6% to 14% of each day (failure_rate.py, Part 11).
  • Batch size: more GPUs means bigger batches, and past a "critical batch size" bigger batches stop saving steps.
    • Not modelled in our scripts, but a real limit in large runs.
  • When bigger is still right: when time is worth more than money.
    • A rival launch or a funding milestone can justify $78k per day saved. Frontier labs run far larger clusters with better networks and faster recovery, and still accept lower efficiency for speed.

4. The arithmetic of failure

๐Ÿ“– The story

Arjun put the Llama 3 report on the screen. Meta's team had published something most labs keep private: in a 54-day snapshot on 16,384 GPUs, there were 419 unexpected interruptions. Faulty GPUs, memory errors, network links, host software.

That is about one interruption per two thousand GPUs per day, Arjun said. On four thousand, two a day. Every one of them stops the whole run.

๐Ÿ”ฌ In plain words: why one failure stops everything In large-scale training every GPU holds a piece of the model and they all wait for each other at every step. If one GPU dies, the step cannot finish, so the whole job stops, the bad machine is swapped out, and the run restarts from the last saved checkpoint. Everything computed since that checkpoint is lost.
failure_rate.py ยท Python 3, no dependencies; Llama 3 failure rate, Young's formula
"""failure_rate.py โ€” interruptions scale with GPU count.

Rate from Meta's Llama 3 report: 419 unexpected interruptions in a 54-day
snapshot on 16,384 H100 GPUs. Checkpoint interval from Young's approximation,
interval = sqrt(2 x checkpoint cost x mean time between failures).
Checkpoint and restart times are SCENARIO assumptions.
"""
import math
RATE = 419 / (16384 * 54)            # interruptions per GPU-day (Llama 3 snapshot)
CKPT_MIN, RESTART_MIN = 2.0, 15.0    # minutes to save a checkpoint; to detect, replace and restart

def main():
    print(f"rate: {RATE:.2e} interruptions per GPU-day\n")
    print(f"{'GPUs':>6} {'per day':>8} {'MTBF h':>7} {'ckpt every':>10} {'lost/day':>9}")
    for g in (2048, 4096, 8192, 16384):
        per_day = RATE * g; mtbf = 24 * 60 / per_day; iv = math.sqrt(2 * CKPT_MIN * mtbf)
        lost = per_day * (RESTART_MIN + iv / 2) + (24 * 60 / iv) * CKPT_MIN   # minutes per day
        print(f"{g:6} {per_day:8.1f} {mtbf / 60:7.1f} {iv:8.0f} m {lost / 1440:9.1%}")

if __name__ == "__main__":
    main()
$ python3 failure_rate.py # interruptions and checkpointing by cluster size rate: 4.74e-04 interruptions per GPU-day GPUs per day MTBF h ckpt every lost/day 2048 1.0 24.7 77 m 6.2% 4096 1.9 12.4 54 m 9.4% 8192 3.9 6.2 39 m 14.4% 16384 7.8 3.1 27 m 22.8%

Share of each day lost to failures

2,048 GPUs ยท ckpt every 77 min6.2%
4,096 ยท every 54 min9.4%
8,192 ยท every 39 min14.4%
16,384 ยท every 27 min22.8%

Real output. Restarts take 15 minutes and a checkpoint 2 minutes in this scenario; faster restarts are worth real money.

๐Ÿ”Ž Go deeper: Young's formula in one line
  • Save too often and you waste time saving; too rarely and each failure throws away more work.
    • The best interval is about โˆš(2 ร— time to save ร— mean time between failures): 54 minutes for 4,096 GPUs.
    • Derived by John Young in 1974 for mainframes; still the starting point today.

5. The contract, not the price (scenario)

Scenario
  • Terms and negotiation are the story's scenario; the clauses are the kind large GPU contracts contain.
๐Ÿ“– The story

They signed on Wednesday at 4 p.m., two hours before the deadline. The price had not moved. Three other things had.

A four-hour replacement window for failed nodes instead of three business days. A hot-spare pool of 64 GPUs kept idle inside the reservation so the run could restart immediately on healthy machines. And a clause that credited back any day on which the provider's own network delivered less than the agreed bandwidth.

We paid for 4,096, Arjun said afterwards, and we train on 4,032. Those 64 idle GPUs are the cheapest insurance we will ever buy.

What was negotiated

Price$2.40 per GPU-hour, unchanged
Replacement3 business days โ†’ 4 hours
Hot spares64 GPUs held inside the block (1.6%)
Network creditdays below agreed bandwidth are credited
Termthree months + one-month option

Scenario terms.

6. Sandbox: choose the cluster

๐ŸŽฎ Sandbox: choose the cluster

Failure rate from the Llama 3 report, Young's checkpoint formula, MFU falling as the cluster grows. MFU and prices are scenario assumptions; the sandbox interpolates MFU between the script's three points.

7. Five mistakes

8. Delegation box

๐Ÿ™‹ Gopala himself
  • Signing the reservation and its risk terms
๐Ÿค– AI agents
  • Failure-rate and checkpoint modelling
  • Drafting the provider comparison
๐Ÿค Bought, rented or borrowed
  • 4,096 H100-class GPUs from a cloud provider
  • Network and storage from the same provider

9. Budget and risk tracker

ItemStatus after Part 11Note
Cluster4,096 GPUs reserved3 months + option
Compute$19.6M BF16 planFP8 may cut it (Part 13)
Recovery4-hour replacement64 hot spares
Failures~1.9 a day expectedcheckpoint every 54 min

10. The series

The Maestro, Season 2: 23 parts in five phases (Season 1: 18 parts)

VerdictAt four thousand GPUs the cluster is not a speed problem but a reliability problem: two interruptions a day, a checkpoint every 54 minutes and a contract written around failure. Next: the run crosses the EU compute line.

11. Reading the numbers

Primary / standard
Llama 3 interruption statistics (Meta, 2024); H100 dense BF16 peak (NVIDIA data sheet); Young (1974) checkpoint interval.
Secondary โš ๏ธ
None in this part.
Our assumptions
MFU, prices, restart and checkpoint times, contract terms and all story events are the scenario.