Home โ€บ Blog โ€บ The Maestro ยท Part 5
The Maestro ยท Part 5 ยท Phase 1 ยท ๐Ÿ—๏ธ Infrastructure hat

Site, Power and Cooling

Gopala does not own a building, a substation or a chiller. He rents a cluster inside someone else's data hall. So why does he need to understand power and cooling? Because they decide which halls can take his cluster at all, how dense his racks can be, how often a node throttles in summer, and a sizeable line in every price he pays. This part sizes Maestro-1's physical footprint from first principles with a small script, explains what changes between air- and liquid-cooled halls, and gives the site-visit checklist a renting start-up should use before the cluster is delivered.

FL
FrontierAGI Team
๐Ÿ—๏ธ Infrastructureโšก Facilities๐Ÿ’ป Coder
โš ๏ธ How to read this (as of 30 Oct 2026) The DGX H100 maximum power (10.2 kW) and GPU TDP (700 W) are NVIDIA's specifications. The ~1.54 global average PUE is from Uptime Institute's 2025 survey as reported by secondary coverage (โš ๏ธ; Uptime's 2026 survey reports 1.52 on a capacity-weighted basis). The sustained-draw fraction, overhead for network and storage, electricity price and PUE of a modern hall are our assumptions. The script output is real; the site visit is the scenario. Disclosure: Claude, made by Anthropic, helps write this site.

1. Mission log: weeks 13โ€“16

Week 13 ยท Sizing
  • Writes power_size.py (section 3) so he can check providers' answers about power density before signing (Part 4).
Week 15 ยท Site visit
  • Visits the chosen provider's data hall with the checklist below. Photographs the power path and the cooling plant; meets the on-site team that will swap his failed nodes.
Week 16 ยท Agreements
  • Agrees the rack layout, power feeds and maintenance windows in writing (maintenance windows), so a planned generator test cannot surprise his main run.

2. Why a renter needs facilities knowledge

The facility affects him in four ways:

3. Sizing Maestro-1 from first principles

Maestro-1 in numbers

1.46 MWdesign IT load
1.17 MWtypical draw
333 tonscooling to remove
55%of node power is the GPUs

Real output of power_size.py (sustained draw and overheads are our assumptions).

Racks needed for 128 nodes, by rack power limit

15 kW racks (legacy)128 racks
40 kW racks (air, AI-ready)43 racks
130 kW racks (liquid)11 racks

Real output of power_size.py.

power_size.py ยท Python 3, no dependencies
"""power_size.py โ€” how much power, cooling and space Maestro-1 needs.

Gopala rents rather than builds, but he still has to know these numbers:
they decide which data halls can take his cluster at all, and they are
a line on every invoice that passes through power.
"""
NODES = 128                 # 1,024 GPUs, 8 per node
NODE_MAX_KW = 10.2          # DGX H100 maximum system power (NVIDIA user guide)
NODE_TYPICAL = 0.80         # our assumption: sustained draw ~80% of max in training
NET_STORAGE_KW = 0.12       # our assumption: switches, storage, management = +12% of IT
PUE = {"modern liquid-assisted hall": 1.25, "2025 global average": 1.54}
PRICE_KWH = 0.08            # our assumption, $ per kWh
RACK_LIMITS_KW = [15, 40, 130]   # legacy, air-cooled AI-ready, liquid (GB200 NVL72 class)

def main():
    it_max = NODES * NODE_MAX_KW * (1 + NET_STORAGE_KW)
    it_avg = NODES * NODE_MAX_KW * NODE_TYPICAL * (1 + NET_STORAGE_KW)
    print(f"IT load: {it_max/1000:.2f} MW design (max), {it_avg/1000:.2f} MW typical")
    for name, pue in PUE.items():
        fac = it_avg * pue
        mwh = fac * 730 / 1000
        print(f"  {name:28s} PUE {pue:.2f}: facility {fac/1000:.2f} MW, "
              f"{mwh:,.0f} MWh/month, ${mwh*1000*PRICE_KWH/1e3:,.0f}k/month")
    print("racks needed:")
    for lim in RACK_LIMITS_KW:
        per = int(lim // NODE_MAX_KW)
        if per == 0:
            print(f"  {lim:3d} kW racks: cannot hold one node"); continue
        racks = -(-NODES // per)
        print(f"  {lim:3d} kW racks: {per} nodes/rack -> {racks} racks")
    heat_btu = it_avg * 3412
    print(f"heat to remove: {it_avg:,.0f} kW = {heat_btu/12000:,.0f} tons of cooling")
    gpu_kw = NODES * 8 * 0.7
    print(f"GPUs alone at 700 W TDP: {gpu_kw:.0f} kW "
          f"({gpu_kw/(NODES*NODE_MAX_KW):.0%} of node max)")

if __name__ == "__main__":
    main()
$ python3 power_size.py IT load: 1.46 MW design (max), 1.17 MW typical modern liquid-assisted hall PUE 1.25: facility 1.46 MW, 1,067 MWh/month, $85k/month 2025 global average PUE 1.54: facility 1.80 MW, 1,315 MWh/month, $105k/month racks needed: 15 kW racks: 1 nodes/rack -> 128 racks 40 kW racks: 3 nodes/rack -> 43 racks 130 kW racks: 12 nodes/rack -> 11 racks heat to remove: 1,170 kW = 333 tons of cooling GPUs alone at 700 W TDP: 717 kW (55% of node max)

What the numbers tell him:

4. Try your own cluster

Facility sizing calculator (our model)

5. Air, liquid, or both

Cooling options by rack density

๐ŸŒฌ๏ธ Airup to ~30โ€“40 kW per rack with containment
๐Ÿšช Rear-door heat exchanger~40โ€“70 kW per rack
๐Ÿ’ง Direct-to-chip liquid100+ kW per rack; needed for rack-scale systems

Industry ranges, not vendor specifications.

Every watt that goes in comes out as heat. There are three common ways to remove it from an AI rack:

ApproachTypical rackHow it worksFor Maestro-1
Airup to ~30โ€“40 kW with hot/cold aisle containmentFans push air through servers; room units (CRAH) cool the air with chilled waterWorks for air-cooled H100 nodes at 3โ€“4 per rack
Rear-door heat exchangers~40โ€“70 kWA water-cooled coil on the back of each rack takes the heat out of exhaust airA common retrofit; denser racks in an old hall
Direct-to-chip liquid100+ kWCold plates on GPUs and CPUs, fed by a coolant distribution unit (CDU)Required by GB200 NVL72-class racks; optional for H100
The throttling check
  • He asks for the hall's supply-air temperature history for last summer, and then watches GPU temperature and clock speed in his own telemetry (Part 7).
    • A node that is 5โ€“10 ยฐC hotter than its neighbours under the same load is usually a rack airflow problem, not a GPU problem, and is cheap to fix if found early.

6. The power path, and where it fails

The power path to one GPU

๐ŸญUtilityGrid feed
โšกSubstationHigh โ†’ medium voltage
๐Ÿ”€SwitchgearTransformers, distribution
๐Ÿ”‹UPSBatteries until generators start
๐Ÿ”Œ3 rack feedsEach feeds 2 of 6 node PSUs
๐Ÿ–ฅ๏ธNodeNeeds 4 of 6 PSUs powered

Every link has its own redundancy design; the last two are the ones a renter can check.

Each link has a redundancy design:

7. Checking the power cabling with a script

Does a node survive losing one feed?

2 feeds: wirings that survive0 of 64
3 feeds: wirings that survive90 of 729
3 feeds, cabled 2+2+2always survives

Real output of psu_feeds.py.

The script checks why:

psu_feeds.py ยท Python 3, no dependencies
"""psu_feeds.py โ€” does a node survive losing one power feed?

A DGX H100 has six power supplies and needs at least four energised.
NVIDIA's SuperPOD design guide therefore asks for THREE independent
sources per rack, each feeding two of the six supplies. This checks
every possible wiring with two and with three feeds.
"""
from itertools import product

NEED = 4

def survives(wiring, lost):
    return sum(1 for f in wiring if f != lost) >= NEED

def ok_all(w, feeds):
    return all(survives(w, f) for f in feeds)

def main():
    for feeds in ("AB", "ABC"):
        ways = list(product(feeds, repeat=6))
        good = [w for w in ways if ok_all(w, feeds)]
        print(f"{len(feeds)} feeds: {len(good)} of {len(ways)} wirings survive any one feed failing")
    cases = {"A/B, 3+3 (enterprise habit)": ("AAABBB", "AB"),
             "A/B/C, 2+2+2 (NVIDIA design)": ("AABBCC", "ABC"),
             "A/B/C, one cord on wrong strip": ("AAABCC", "ABC"),
             "A/B/C, C strip never cabled": ("AAABBB", "ABC")}
    for name, (w, feeds) in cases.items():
        res = "  ".join(f"lose {f}: {'runs' if survives(w, f) else 'STOPS'}" for f in feeds)
        print(f"  {name:32s} {res}")
    print("lesson: with a 4-of-6 minimum, two feeds can never protect the node;")
    print("        three feeds work only if every node is cabled 2+2+2 - check it rack by rack")

if __name__ == "__main__":
    main()
$ python3 psu_feeds.py 2 feeds: 0 of 64 wirings survive any one feed failing 3 feeds: 90 of 729 wirings survive any one feed failing A/B, 3+3 (enterprise habit) lose A: STOPS lose B: STOPS A/B/C, 2+2+2 (NVIDIA design) lose A: runs lose B: runs lose C: runs A/B/C, one cord on wrong strip lose A: STOPS lose B: runs lose C: runs A/B/C, C strip never cabled lose A: STOPS lose B: STOPS lose C: runs lesson: with a 4-of-6 minimum, two feeds can never protect the node; three feeds work only if every node is cabled 2+2+2 - check it rack by rack
Why this matters for a renter
  • A data hall built for 2N enterprise loads can host H100 nodes, but only if the rack power design follows the three-feed pattern.
    • It is a cheap thing to check before delivery and an expensive thing to discover during the main run, when a routine UPS maintenance on one feed would stop every node in the hall at once.

8. How cold is cold enough?

Two industry frameworks set the vocabulary for cooling conversations:

9. Training is a strange electrical load

Two practical consequences:

10. The site-visit checklist

A renter cannot design the building, but he can check it. Gopala's checklist, used on the scenario visit:

AreaQuestionGood answer
PowerRack power limit, and is it contracted per rack?โ‰ฅ 40 kW per rack for air-cooled H100, written into the order
PowerHow many independent feeds per rack, and how are the six node PSUs split?Three feeds from separate upstream PDUs, each node cabled 2+2+2 (the cabling check)
PowerUPS runtime and generator test scheduleMinutes of UPS, monthly generator tests announced in advance
CoolingSupply temperature and the worst week last summerWithin ASHRAE's recommended range all year
CoolingContainment, and blanking panels in empty slotsHot or cold aisle contained; no gaps
SpaceAre all 128 nodes in one hall, close to the spine switches?Yes, with cable lengths that allow the planned fabric
PeopleWho swaps a node at 3 a.m., and where are spares?Staff on site 24/7, spare nodes and GPUs in the building
SecurityWho can enter the cage; how are failed drives handled?Named access list, drives destroyed on site with a record
WaterHow the hall uses water for coolingA stated water-use figure (WUE) and source
Scenario
  • On the visit, the hall meets every item except one: generator tests are monthly but not announced to tenants.
    • Gopala gets a clause requiring 7 days'
      • notice and agrees with the provider that tests will not be scheduled during the main run (maintenance windows).
    • The hall itself is not a real site.

11. Maintenance windows and change freezes

The agreement Gopala signs:

12. If the next model needs liquid cooling

13. Facility data a renter should ask for

SignalWhere it comes fromWhat he uses it for
Per-outlet power on rack PDUsIntelligent rack PDUsSpotting mis-cabled supplies; real power per node
Rack inlet temperatureSensors at the cold aisleSeparating airflow problems from GPU problems
Planned maintenance calendarFacilities teamTaking a checkpoint before work on a feed
Incident reportsFacilities teamMatching a job failure to a power or cooling event
Monthly energy and PUEBuilding management systemCost and emissions reporting

14. Five facility mistakes renters make

  1. Sizing to GPU TDP (700 W ร— GPUs) instead of node maximum power, and running out of rack power.
  2. Accepting 2N A/B feeds for 6-PSU nodes without checking the cabling (the cabling check).
  3. Splitting the cluster across halls to fit low-density racks, then paying in cable lengths and extra switch layers.
  4. Never asking about maintenance windows until a generator test lands in the middle of a run.
  5. Treating hot nodes as bad GPUs and replacing hardware that was fine.

15. The environmental footprint

16. Delegation box

๐Ÿ™‹ Gopala himself
  • The sizing script and the requirements
  • The site visit and the checklist
  • The change-freeze agreement
๐Ÿค– AI agents
  • Reading the provider's facility documents and summarising gaps
  • Pulling temperature history into a chart
  • Drafting the maintenance-window clause
๐Ÿค Bought, rented or borrowed
  • The provider's facilities team (they run the building)
  • A facilities consultant for one day of the visit
  • The provider's existing certifications (e.g. ISO 27001, SOC 2)

17. Budget and risk tracker

ItemStatus after Part 5Note
Spend to date~$45k (scenario)Plus the visit and a consultant day
Facilities costInside the GPU pricePower is passed through; efficiency matters at renewal
Time16 of ~78 weeksDelivery begins month 3
Power and cooling riskLow for H100 air-cooledChange freeze signed for the main run
Upgrade riskMediumLiquid cooling needed for future rack-scale systems
Throttling riskUnknownWatch GPU temperatures and clocks per node from day one

18. The series

The Maestro: 18 parts in five phases

Verdict
  • Renting the building does not remove the physics: Maestro-1 needs about 1.5 MW of IT capacity, dense racks, a power path that will not stop the job, and a change freeze during the main run.
    • Next, Part 6: the cluster arrives, and nothing is trusted until it is tested.

19. Reading the numbers

Primary / standard
DGX H100: 6 ร— 3.3 kW PSUs, 4+2 redundancy, 10.2 kW maximum (NVIDIA user guide); four of six must be energised, three power sources per rack each feeding two PSUs, each path sized for half the rack's peak (NVIDIA SuperPOD H100 data-centre design guide). H100 700 W TDP. Llama 3 trained on 16,384 H100s; power swings across the cluster (Meta).
Secondary โš ๏ธ
ASHRAE liquid-cooling classes W17โ€“W45 and W+ (5th edition naming, via trade press). Global average PUE ~1.54 in 2025 (Uptime Institute survey, as reported by secondary coverage); 1.52 capacity-weighted in Uptime's 2026 survey. Typical rack densities by cooling type are industry ranges.
Our assumptions
Sustained draw at 80% of maximum, +12% for network and storage, PUE 1.25 for a modern hall, $0.08/kWh, the site visit and agreements.