Home โ€บ Blog โ€บ The Maestro ยท Part 4
The Maestro ยท Part 4 ยท Phase 1 ยท ๐Ÿ—๏ธ Infrastructure hat

Securing Compute

Phase 0 ended with $20M in the bank and a plan that needs about 1,024 H100-class GPUs for a month of pre-training, 64 more for serving, and a smaller pool for experiments from month 3. Now Gopala has to turn money into reserved machines. Which kind of provider? On what contract? What must the contract say about failures, networking, storage and security? And how does he compare offers whose list prices differ by 75% but whose useful hours may not? This part is the buyer's guide he wished he had: the market as of late 2026, a scoring script that prices every offer per useful GPU-hour, the clauses that matter and the acceptance test he writes into the contract before signing.

FL
FrontierAGI Team
๐Ÿ—๏ธ Infrastructure๐Ÿ’ผ Founderโš–๏ธ Negotiator๐Ÿ’ป Coder
โš ๏ธ How to read this (as of 30 Oct 2026) GPU specifications are NVIDIA's. SemiAnalysis's ClusterMAX ratings are a third-party review (โš ๏ธ their tiers change every few months; check the live list). The four providers P1โ€“P4 in the script are illustrative offers built from typical ranges, not real quotes, and every price, availability and goodput figure in them is our assumption. The script output below is real. Gopala is a teaching scenario; what he signs is the scenario. Disclosure: Claude, made by Anthropic, helps write this site.

1. Mission log: weeks 11โ€“14

Week 11 ยท The long list
  • Writes down the hard requirements from Part 3: 128 nodes ร— 8 H100-class GPUs in one non-blocking network fabric, ~1.5 MW of IT power, shared storage that can absorb a multi-terabyte checkpoint in minutes, and a 12-month term from month 3.
  • Starts from public ratings and peer references, not ads: 14 providers on the long list, 6 that can offer 1,024 GPUs in a single fabric by month 3.
Weeks 12โ€“13 ยท Quotes and trials
  • Requests quotes in a fixed template so offers are comparable (section 5). Asks each finalist for a free or cheap two-node trial and runs Part 6's burn-in and NCCL tests on it.
  • Writes provider_score.py (section 7).
Week 14 ยท Signing
  • Negotiates the contract clauses in section 12, with an outside lawyer, and signs with the provider that wins on cost per useful hour (scenario).
  • Books a separate on-demand pool for experiments and keeps one hyperscaler account for storage and serving burst.

2. What exactly is he buying?

What "1,024 GPUs" really means

๐Ÿ–ฅ๏ธ GPUs128 nodes ร— 8 H100-class, NVSwitch inside each node
๐Ÿ•ธ๏ธ One fabricAll nodes on one non-blocking 400 Gb/s-per-GPU network
๐Ÿ’พ StorageParallel file system that saves a checkpoint in minutes
๐Ÿ”‘ AccessRoot on nodes, BMC or API to power-cycle
๐Ÿ“… TimeDelivery from month 3, 12-month term

Two clusters with the same GPU count can differ two-fold in speed because of the layers around the GPUs.

Gopala's requirements sheet, the first thing every provider receives:

RequirementWhy it matters for Maestro-1What to ask for
GPUThe budget (Part 3) assumes H100-class: 80 GB HBM, ~989 dense BF16 TFLOP/s peak.H100 or H200 SXM, 8 per node on NVSwitch. Not PCIe cards: their GPU-to-GPU bandwidth is far lower.
One fabricTensor and expert parallelism (Part 9) need every GPU to talk to every other at full speed.All 1,024 GPUs on one non-blocking InfiniBand NDR or RoCE fabric, 8 ร— 400 Gb/s NICs per node ("rail-optimised"). A written topology diagram.
StorageCheckpoints of a large MoE model plus optimiser state run to terabytes; a slow save stalls 1,024 GPUs.Parallel file system (e.g. Lustre, WEKA, VAST, DDN) with a stated write bandwidth, plus object storage for data.
AccessHe needs to install his own stack (Part 7).Bare metal or near-bare-metal, root on nodes, out-of-band (BMC) access or a provider API to power-cycle.
OrchestrationSaves weeks if the provider runs it well; costs weeks if they do not.Managed Slurm or Kubernetes offered, but not forced.
LocationData-residency clauses from design partners; latency to his data.Named data centre, no silent migration.
TimeEvery month of delay is ~$318k of burn with no model.Delivery date with penalties; partial early delivery for experiments.

3. The market in late 2026

SellerExamplesStrengthWeakness for a start-up
HyperscalersAWS, Microsoft Azure, Google Cloud, OracleReliability, security reviews done, every other service next door, credits for start-upsHighest list price for GPUs; large reservations often need long commitments
NeocloudsCoreWeave, Nebius, Lambda, Crusoe, Together AI and othersGPU-first, faster delivery, training-grade fabrics, often better priceQuality varies widely; some resell other operators' capacity
Brokers and marketplacesAggregators across many small sitesCheapest per hour, flexible termsMixed hardware, often no single fabric, thin support
Colocation + owned hardwareEquinix, Digital Realty and regional operatorsLowest cost over 3โ€“5 yearsCapital, lead time and staff Gopala does not have (Part 3 chose to rent)
Why not just use credits?
  • Hyperscaler start-up programmes give credits worth tens or hundreds of thousands of dollars.
    • They pay for Maestro-0 experiments and storage nicely, but they do not reserve a 1,024-GPU block in one fabric.
    • Gopala uses credits for the edges of the plan, not its centre.

4. What GPUs actually cost in 2026

The public evidence, all secondary (โš ๏ธ):

Price pointFigureSource type
One-year H100 contract, Oct 2025 low~$1.70 per GPU-hourSemiAnalysis figure reported by a price tracker โš ๏ธ
One-year H100 contract, Mar 2026~$2.35 per GPU-hour (almost +40%)Same โš ๏ธ (the figure Part 3's budget uses)
Neocloud on-demand listings, Oct 2026from ~$1.70, most between ~$2 and ~$4Price aggregators โš ๏ธ
Hyperscaler on-demandoften several times neocloud prices per GPU-hourAggregators and list prices โš ๏ธ
Reserved vs on-demanddiscounts of roughly 20โ€“40% for 1โ€“12 month termsAggregators โš ๏ธ

5. A quote template that makes offers comparable

Gopala sends one template and refuses quotes that do not fill it in:

  1. Price per GPU-hour, for 1, 6 and 12 months, and the upfront payment required for each.
  2. GPU model, memory, node design, NICs per node and link speed; the fabric topology and oversubscription ratio.
  3. Storage: type, usable capacity, guaranteed read/write bandwidth per node and in aggregate, price.
  4. Availability commitment per node, how it is measured, and the credit when it is missed.
  5. Time to replace a failed node or GPU, and whether spares are held on site.
  6. Support: named engineers, response times by severity, and a shared chat channel.
  7. Security: certifications (SOC 2 Type II, ISO 27001), tenant isolation, who has physical and remote access, how disks are wiped.
  8. Delivery date, the penalty for slipping, and whether part of the block can come early.
  9. Egress charges for moving data and checkpoints out.

6. Reading a real SLA

These are normal industry terms, and three lessons hold for any provider:

7. Scoring offers: cost per useful GPU-hour

List price vs price per useful GPU-hour

P1 top-rated ยท list$2.65
P1 ยท useful$2.88
P2 cheap ยท list$2.10
P2 ยท useful$2.57
P3 hyperscaler ยท useful$3.71
P4 broker ยท useful$2.59

Real output of provider_score.py on illustrative offers (our assumptions, not real quotes).

provider_score.py ยท Python 3, no dependencies
"""provider_score.py โ€” compare GPU cloud offers on cost per USEFUL GPU-hour.

The list price is not the price. What matters is how many hours of real
training you get per dollar, after the cluster's own failures, slow
hand-offs and the support you will need at 3 a.m.
Providers P1-P4 are illustrative offers, not real quotes.
"""
GPUS = 1024            # Maestro-1 main run (Part 3)
MONTHS = 12            # reservation length

# name, $/GPU-hr, availability (fraction of hours the reserved nodes are up),
# goodput (fraction of up-hours doing useful training: restarts, slow nodes),
# hours to replace a failed node, network rating 1-5, support rating 1-5,
# security rating 1-5, upfront payment fraction
OFFERS = [
    ("P1 neocloud, top-rated",   2.65, 0.990, 0.93, 4,  5, 5, 4, 0.25),
    ("P2 neocloud, cheap",       2.10, 0.960, 0.85, 48, 3, 2, 3, 0.50),
    ("P3 hyperscaler",           3.40, 0.995, 0.92, 2,  4, 4, 5, 0.00),
    ("P4 broker, mixed sites",   1.95, 0.940, 0.80, 72, 2, 2, 2, 1.00),
]
WEIGHTS = {"cost": 0.40, "network": 0.20, "support": 0.20, "security": 0.20}

def useful_cost(price, avail, goodput):
    # you pay for every reserved hour; you only get avail*goodput of it
    return price / (avail * goodput)

def main():
    hours = MONTHS * 730
    rows = []
    for n, p, a, g, rep, net, sup, sec, up in OFFERS:
        u = useful_cost(p, a, g)
        rows.append((n, p, u, rep, net, sup, sec, up, p * GPUS * hours))
    best = min(r[2] for r in rows)
    print(f"{'offer':26s} {'list':>6s} {'useful':>7s} {'repl h':>6s} {'contract':>9s} {'upfront':>8s} {'score':>6s}")
    scored = []
    for n, p, u, rep, net, sup, sec, up, total in rows:
        s = (WEIGHTS["cost"] * 5 * best / u + WEIGHTS["network"] * net
             + WEIGHTS["support"] * sup + WEIGHTS["security"] * sec)
        scored.append((s, n))
        print(f"{n:26s} ${p:5.2f} ${u:6.2f} {rep:6d} ${total/1e6:7.1f}M ${total*up/1e6:6.1f}M {s:6.2f}")
    scored.sort(reverse=True)
    print("ranking:", " > ".join(n.split()[0] for _, n in scored))
    # what a 10-point goodput gap costs on the 31-day main run
    run_h = 31 * 24 * GPUS
    print(f"main run = {run_h:,} GPU-hours of useful work; at P2 that needs "
          f"{run_h/(0.96*0.85):,.0f} paid hours vs {run_h/(0.99*0.93):,.0f} at P1")

if __name__ == "__main__":
    main()
$ python3 provider_score.py offer list useful repl h contract upfront score P1 neocloud, top-rated $ 2.65 $ 2.88 4 $ 23.8M $ 5.9M 4.59 P2 neocloud, cheap $ 2.10 $ 2.57 48 $ 18.8M $ 9.4M 3.60 P3 hyperscaler $ 3.40 $ 3.71 2 $ 30.5M $ 0.0M 3.99 P4 broker, mixed sites $ 1.95 $ 2.59 72 $ 17.5M $ 17.5M 3.18 ranking: P1 > P3 > P2 > P4 main run = 761,856 GPU-hours of useful work; at P2 that needs 933,647 paid hours vs 827,475 at P1

Three lessons, all from the made-up offers but true of real ones:

8. Try your own offer

Useful price calculator (our model)

9. When does the cheap offer really win?

offer_sweep.py ยท Python 3, no dependencies
"""offer_sweep.py โ€” when does the cheap offer really win?

Two questions the list price hides:
1. How good must the cheap offer's goodput be to match the premium one?
2. What does the whole contract cost once upfront cash, storage, egress
   and the cost of delay are counted?
All inputs are our assumptions; change them to your quotes.
"""
GPUS, MONTHS, H = 1024, 12, 730
P1 = dict(price=2.65, avail=0.990, good=0.93, upfront=0.25, storage=45_000, egress=0)
P2 = dict(price=2.10, avail=0.960, good=0.85, upfront=0.50, storage=60_000, egress=90_000)
CASH_RATE = 0.12        # yearly cost of cash tied up upfront (what else it could fund)
DELAY_WEEKS = {"P1": 0, "P2": 3}   # later delivery in the scenario
BURN_WEEK = 318_000 * 12 / 52      # Part 3 burn

def useful(o):
    return o["price"] / (o["avail"] * o["good"])

def contract(name, o):
    compute = o["price"] * GPUS * H * MONTHS
    cash = compute * o["upfront"] * CASH_RATE * (MONTHS / 12) / 2
    other = o["storage"] * MONTHS + o["egress"]
    delay = DELAY_WEEKS[name] * BURN_WEEK
    useful_h = GPUS * H * MONTHS * o["avail"] * o["good"]
    total = compute + cash + other + delay
    print(f"{name}: compute ${compute/1e6:5.2f}M + cash ${cash/1e6:4.2f}M + storage/egress "
          f"${other/1e6:4.2f}M + delay ${delay/1e6:4.2f}M = ${total/1e6:5.2f}M "
          f"-> ${total/useful_h:4.2f} per useful GPU-hour")
    return total / useful_h

def main():
    print(f"useful price: P1 ${useful(P1):.2f}  P2 ${useful(P2):.2f}")
    # break-even goodput for P2 against P1
    be = P2["price"] / (P2["avail"] * useful(P1))
    print(f"P2 matches P1 on useful price at goodput {be:.1%} (P1 needs {P1['good']:.0%})")
    print("goodput of P2 -> useful price")
    for g in (0.70, 0.75, 0.80, 0.85, 0.90, 0.95):
        u = P2["price"] / (P2["avail"] * g)
        bar = "#" * int((u - 2.0) * 20)
        print(f"  {g:.0%}  ${u:4.2f} {bar}{'  <- cheaper than P1' if u < useful(P1) else ''}")
    a, b = contract("P1", P1), contract("P2", P2)
    print(f"whole-contract winner: {'P1' if a < b else 'P2'} by ${abs(a-b):.2f} per useful GPU-hour")

if __name__ == "__main__":
    main()
$ python3 offer_sweep.py useful price: P1 $2.88 P2 $2.57 P2 matches P1 on useful price at goodput 76.0% (P1 needs 93%) goodput of P2 -> useful price 70% $3.13 ###################### 75% $2.92 ################## 80% $2.73 ############## <- cheaper than P1 85% $2.57 ########### <- cheaper than P1 90% $2.43 ######## <- cheaper than P1 95% $2.30 ###### <- cheaper than P1 P1: compute $23.77M + cash $0.36M + storage/egress $0.54M + delay $0.00M = $24.67M -> $2.99 per useful GPU-hour P2: compute $18.84M + cash $0.57M + storage/egress $0.81M + delay $0.22M = $20.43M -> $2.79 per useful GPU-hour whole-contract winner: P2 by $0.20 per useful GPU-hour

Gopala writes the decision down this way:

Decision rule
  • Take the cheapest offer per useful hour unless its goodput is unproven and the work cannot be repeated.
    • Experiments and evaluation, which can be rerun, go to the cheap pool; the main run goes to the provider whose goodput the trial confirmed.
    • In the scenario he does exactly that: the main block with a P1-like provider, and a smaller on-demand P2-like pool for ablations.

10. The two-node trial

How an offer is tested before signing

๐Ÿ“Quote templateSame nine questions to everyone
๐ŸงชTwo-node trialDiagnostics, NCCL, storage, own job
๐ŸŒ™Night ticketTime a fake 2 a.m. fault
๐Ÿ”ขScoreUseful price + network, support, security
โœ๏ธContractBilling starts at acceptance

Each step can drop a provider.

Trial checkWhat it reveals
All-reduce bus bandwidth between nodesWhether the fabric delivers what the quote says (Part 6 shows how to read the numbers)
Storage write of a 200 GB file from 16 GPUs at onceWhether checkpoint saves will stall the run
Firmware and driver versionsWhether nodes are maintained, and whether they all match
A support ticket at nightWhether "24/7" means a person or a queue
Asking for root and BMC accessWhether he can run his own stack (Part 7)
Scenario
  • In the story, one finalist's inter-node bandwidth comes in at about half of its peers' on the trial (the pattern Part 6's parser flags), and support takes nine hours to answer the night ticket.
    • It is dropped.
    • The winner is a neocloud similar to P1 in the script.
    • This is the scenario, not a review of any real provider.

11. Security due diligence

The security questions he asks every finalist, and the answers he wants:

QuestionWhyGood answer
Are my nodes single-tenant?Shared hosts widen the attack surfaceDedicated bare-metal nodes, no other tenant's VMs
Is my fabric isolated?InfiniBand partitions or separate VLANs stop cross-tenant trafficA dedicated partition key (PKey) or physically separate fabric
Who has root or BMC access?Provider staff can read memory and disksNamed, logged, break-glass only, with notification
Which audits?Evidence of process, needed for partner contractsCurrent SOC 2 Type II report and ISO 27001 certificate, shared under NDA
How are drives handled at the end or on failure?Weights and data on failed disks leave the buildingWipe or destroy on site, with certificates
Can I encrypt storage with my own keys?Limits exposure if storage is copiedCustomer-managed keys for object and file storage

12. The clauses that matter

Gopala's lawyer drafts, but these are the clauses he insists on as the infrastructure owner:

ClauseWeak versionWhat he asks for
Availability"99.9% platform uptime", measured on the provider's control planePer-node availability, measured by his own health checks, with credits per node-hour lost
Replacement"Best effort"Failed node replaced within a stated number of hours from on-site spares; credit for every hour beyond it
AcceptanceBilling starts on hand-overBilling starts when the acceptance test (section 14) passes on the whole block
Delivery"Target date"A date, with credits for each week late and the right to cancel after a long slip
Fabric"High-speed networking"Named topology and minimum all-reduce bandwidth, tested at acceptance
DataSilenceHis data and weights stay his; disks wiped on exit with a certificate; no use for training anyone's models
ExitSilenceEgress fees capped or waived for moving checkpoints out at the end
Security incidentsNotice "without undue delay"Notice within a stated number of hours and access to the investigation

13. What the research says about failures

14. The acceptance test

The acceptance test

๐ŸฉบEvery nodeDCGM level 3 + 24 h burn-in
๐Ÿ”—Every link8 ร— 400 Gb/s active, firmware matches
๐Ÿ“ถEvery pairAll-reduce within 10% of contract
๐Ÿ’พStorageStated write speed for 10 min
๐Ÿ‹๏ธ72 h jobHis own training, no hardware stop

Pass โ†’ billed. Fail โ†’ fixed or replaced by the provider first.

Gopala's version, built from the burn-in in Part 6:

  1. Every node passes the long GPU diagnostic (NVIDIA DCGM level 3) and a 24-hour burn-in with no GPU errors.
  2. Every node reports 8 GPUs and 8 active 400 Gb/s fabric links, with matching firmware and driver versions.
  3. NCCL all-reduce bus bandwidth on every node pair, and on the full 1,024 GPUs, is within 10% of the provider's stated figure.
  4. Storage sustains the stated write bandwidth from all nodes at once for 10 minutes.
  5. A 72-hour run of his own training job on the whole block completes without a hardware stop.

15. Seven buying mistakes

  1. Comparing list prices instead of useful prices.
  2. Accepting "up to 3.2 Tb/s per node" without a topology and an all-reduce number.
  3. Letting billing start at hand-over rather than acceptance.
  4. Forgetting storage: a cluster that cannot save a checkpoint quickly wastes GPU time on every save.
  5. Ignoring egress: moving tens of terabytes of checkpoints out at the end can cost more than a week of compute.
  6. Splitting the main block across two buildings to save money; a training job cannot span them efficiently.
  7. Signing without a trial, on the strength of a rating or a famous customer.

16. Negotiating the money terms

A start-up's leverage is small, but some terms move more easily than price:

17. Planning capacity month by month

The reservation is a fixed block, but demand is not. He maps GPU demand by month from Part 3's plan to see how full the block will be:

MonthsMain useGPUs busy (planned)Idle risk
3โ€“5Maestro-0 post-training, bring-up of the rest of the block256โ€“512Low: block arrives in stages
5โ€“6Maestro-0 iterations, data-refinery jobs (Part 8)~512Medium: half the block may idle
6โ€“9Maestro-1 ablations and small MoE runs512โ€“1,024Low
9โ€“10.5Main run1,024 (all)None; experiments move to the burst pool
10.5โ€“15Post-training, RSI loop, evaluation256โ€“768Medium: resell idle nodes

18. The final compute portfolio

PoolSizeFromUseHow bought
Main block1,024 GPUs (128 nodes), one fabricMonth 3 (part early, all by month 5)Maestro-0 post-training, Maestro-1 experiments and main run12-month reservation (scenario)
Serving64 GPUsMonth 4Maestro-0 for design partners, then Maestro-1Reserved, separate fabric, separate provider for resilience
Burst0โ€“128 GPUsAs neededEvaluation sweeps, extra ablationsOn demand, paid from cloud credits first
Storage and dataObject storageNowRaw data, checkpoints archiveHyperscaler with start-up credits

19. Delegation box

๐Ÿ™‹ Gopala himself
  • The requirements sheet and the scoring weights
  • The trial checks and the night ticket
  • The final decision and signature
๐Ÿค– AI agents
  • Normalising quotes into the template
  • Running the trial scripts and summarising logs
  • A first draft of the contract markup for the lawyer
๐Ÿค Bought, rented or borrowed
  • A start-up lawyer for the contract
  • Two advisers who have run large clusters (references on providers)
  • Public ratings such as ClusterMAX for the long list

20. Budget and risk tracker

ItemStatus after Part 4Note
Spend to date~$40k (scenario)Trials, legal fees, deposits
CommittedMain block for 12 months (scenario)Largest commitment in the plan; billing starts at acceptance
Time14 of ~78 weeksDelivery from month 3
Compute supply riskReducedDelivery date and penalties signed; two providers
Provider quality riskUnknown until acceptanceAcceptance test written into the contract
Cash riskAcceptableUpfront payment negotiated to a quarter (scenario)

21. The series

The Maestro: 18 parts in five phases

Verdict
  • Gopala buys useful hours, not GPU-hours: he prices every offer per useful hour, tests finalists with his own scripts, and makes billing start only when his acceptance test passes.
    • The machine now exists on paper.
    • Next, Part 5: the power and cooling that physical machine needs, even when someone else runs the building.

22. Reading the numbers

Primary / standard
H100 SXM: 80 GB HBM3, ~989 dense BF16 TFLOP/s, 700 W (NVIDIA). ClusterMAX tiers and criteria (SemiAnalysis). Meta's Llama 3 interruption record; 1,024-GPU job MTTF ~7.9 h vs ~47.7 days for 8-GPU jobs in Meta research clusters (Kokolis et al., arXiv:2410.21680). CoreWeave ToS: โ‰ฅ99.9% monthly uptime objective for platform instances.
Secondary โš ๏ธ
One-year H100 contract prices .70 (Oct 2025) and .35 (Mar 2026), SemiAnalysis figures via a price tracker; Oct 2026 listing ranges and reserved discounts from aggregators. ClusterMAX ratings move every few months; CoreWeave as the only Platinum provider is from the published ratings and the providers' own announcements; a newer edition was reported in Sep 2026 (โš ๏ธ). Price ranges for H100 rental.
Our assumptions
Offers P1โ€“P4, all availability and goodput figures, scoring weights, contract outcome, upfront terms and the portfolio.