Site, Power and Cooling
Gopala does not own a building, a substation or a chiller. He rents a cluster inside someone else's data hall. So why does he need to understand power and cooling? Because they decide which halls can take his cluster at all, how dense his racks can be, how often a node throttles in summer, and a sizeable line in every price he pays. This part sizes Maestro-1's physical footprint from first principles with a small script, explains what changes between air- and liquid-cooled halls, and gives the site-visit checklist a renting start-up should use before the cluster is delivered.
1. Mission log: weeks 13โ16
- Writes
power_size.py(section 3) so he can check providers' answers about power density before signing (Part 4).
- Visits the chosen provider's data hall with the checklist below. Photographs the power path and the cooling plant; meets the on-site team that will swap his failed nodes.
- Agrees the rack layout, power feeds and maintenance windows in writing (maintenance windows), so a planned generator test cannot surprise his main run.
2. Why a renter needs facilities knowledge
- A 1,024-GPU cluster is small by frontier standards (Meta trained Llama 3 405B on 16,384 H100s), but it is a large customer for most data halls: well over a megawatt in a few dozen racks.
The facility affects him in four ways:
- Can it fit? Many existing halls were built for 5โ15 kW racks. An 8-GPU H100 node alone can draw up to 10.2 kW.
- Will it throttle? GPUs reduce their clocks when they run hot. A hall that cools well in spring but not at the August peak quietly slows the whole synchronised job, because the slowest GPU sets the pace.
- Will it stop? A power event that drops a single rack can stop a job using every node. The power path (feeds, UPS, generators) sets how often that happens.
- What does it cost? Power is passed through in GPU prices or billed directly; a less efficient hall costs more per useful hour.
- For the full story of power and cooling at frontier scale, see the site's AI Compute Stack series on power and cooling and water
- this part stays at Gopala's scale.
3. Sizing Maestro-1 from first principles
Maestro-1 in numbers
Real output of power_size.py (sustained draw and overheads are our assumptions).
Racks needed for 128 nodes, by rack power limit
Real output of power_size.py.
- Four numbers decide the footprint: power per node, how much of that it really draws while training, the extra for networking and storage, and the facility overhead (PUE: total facility power divided by IT power).
"""power_size.py โ how much power, cooling and space Maestro-1 needs.
Gopala rents rather than builds, but he still has to know these numbers:
they decide which data halls can take his cluster at all, and they are
a line on every invoice that passes through power.
"""
NODES = 128 # 1,024 GPUs, 8 per node
NODE_MAX_KW = 10.2 # DGX H100 maximum system power (NVIDIA user guide)
NODE_TYPICAL = 0.80 # our assumption: sustained draw ~80% of max in training
NET_STORAGE_KW = 0.12 # our assumption: switches, storage, management = +12% of IT
PUE = {"modern liquid-assisted hall": 1.25, "2025 global average": 1.54}
PRICE_KWH = 0.08 # our assumption, $ per kWh
RACK_LIMITS_KW = [15, 40, 130] # legacy, air-cooled AI-ready, liquid (GB200 NVL72 class)
def main():
it_max = NODES * NODE_MAX_KW * (1 + NET_STORAGE_KW)
it_avg = NODES * NODE_MAX_KW * NODE_TYPICAL * (1 + NET_STORAGE_KW)
print(f"IT load: {it_max/1000:.2f} MW design (max), {it_avg/1000:.2f} MW typical")
for name, pue in PUE.items():
fac = it_avg * pue
mwh = fac * 730 / 1000
print(f" {name:28s} PUE {pue:.2f}: facility {fac/1000:.2f} MW, "
f"{mwh:,.0f} MWh/month, ${mwh*1000*PRICE_KWH/1e3:,.0f}k/month")
print("racks needed:")
for lim in RACK_LIMITS_KW:
per = int(lim // NODE_MAX_KW)
if per == 0:
print(f" {lim:3d} kW racks: cannot hold one node"); continue
racks = -(-NODES // per)
print(f" {lim:3d} kW racks: {per} nodes/rack -> {racks} racks")
heat_btu = it_avg * 3412
print(f"heat to remove: {it_avg:,.0f} kW = {heat_btu/12000:,.0f} tons of cooling")
gpu_kw = NODES * 8 * 0.7
print(f"GPUs alone at 700 W TDP: {gpu_kw:.0f} kW "
f"({gpu_kw/(NODES*NODE_MAX_KW):.0%} of node max)")
if __name__ == "__main__":
main()What the numbers tell him:
- About 1.5 MW of IT capacity must be reserved even though the typical draw is closer to 1.2 MW: the facility has to be built for the peak, and a training job hits near-peak during dense phases.
- Rack density decides the room. In a legacy 15 kW hall, the cluster spreads over 128 racks, which means long cables between nodes and more switch layers. At 40 kW (a typical air-cooled AI-ready hall) it needs 43 racks; in a liquid-cooled hall, 11.
- Efficiency is real money. The gap between a modern hall and the global average is about $20k a month at our assumed $0.08/kWh: ~$240k a year, enough to pay an engineer.
- GPUs are only about half of a node's maximum power. CPUs, memory, NICs, NVSwitches, fans and power-conversion losses take the rest. Quoting "700 W ร 1,024" under-sizes by half.
4. Try your own cluster
Facility sizing calculator (our model)
5. Air, liquid, or both
Cooling options by rack density
Industry ranges, not vendor specifications.
Every watt that goes in comes out as heat. There are three common ways to remove it from an AI rack:
| Approach | Typical rack | How it works | For Maestro-1 |
|---|---|---|---|
| Air | up to ~30โ40 kW with hot/cold aisle containment | Fans push air through servers; room units (CRAH) cool the air with chilled water | Works for air-cooled H100 nodes at 3โ4 per rack |
| Rear-door heat exchangers | ~40โ70 kW | A water-cooled coil on the back of each rack takes the heat out of exhaust air | A common retrofit; denser racks in an old hall |
| Direct-to-chip liquid | 100+ kW | Cold plates on GPUs and CPUs, fed by a coolant distribution unit (CDU) | Required by GB200 NVL72-class racks; optional for H100 |
- H100 HGX nodes are sold in both air-cooled and liquid-cooled versions.
- Gopala's provider offers air-cooled nodes in contained 40 kW racks, which is fine.
- If his plan later moves to Blackwell-generation rack-scale systems, liquid cooling is not optional, and only some halls have it.
- He writes that into the renewal options of the contract.
- He asks for the hall's supply-air temperature history for last summer, and then watches GPU temperature and clock speed in his own telemetry (Part 7).
- A node that is 5โ10 ยฐC hotter than its neighbours under the same load is usually a rack airflow problem, not a GPU problem, and is cheap to fix if found early.
6. The power path, and where it fails
The power path to one GPU
Every link has its own redundancy design; the last two are the ones a renter can check.
- Electricity reaches a GPU through a chain: utility feed โ substation โ medium-voltage switchgear โ transformers โ UPS (batteries that bridge until generators start) โ power distribution units โ rack power strips โ the node's power supplies.
Each link has a redundancy design:
- N: no spare. One failure stops the load.
- N+1: one spare unit for a group.
- 2N: two fully independent paths (A and B feeds), each able to carry the whole load. Common in enterprise halls, and, as the cabling check below shows, not enough on its own for a 6-PSU H100 node.
- The node has its own redundancy too.
- NVIDIA's DGX H100 has six 3.3 kW power supplies in a 4+2 arrangement, and NVIDIA's design guide notes that at least four must be powered for the server to run.
- That is why the same guide asks for three independent power sources per rack, each feeding two of the six supplies, so any one source can fail and four remain.
- Gopala checks the arithmetic with a script below and asks to see the cabling, rack by rack.
- AI training halls sometimes choose less redundancy than enterprise halls (for example, no UPS on some cooling loads), trading a small risk of a stop for lower cost, because a training job can restart from a checkpoint while a bank cannot.
- That trade-off is acceptable for him only if his checkpoint strategy (Part 7) makes a stop cost minutes, not days.
7. Checking the power cabling with a script
Does a node survive losing one feed?
Real output of psu_feeds.py.
- Here the research changed Gopala's plan.
- His first checklist (and the first version of this article) asked for the usual enterprise A/B feeds.
- NVIDIA's SuperPOD design guide for H100 says something stricter: because four of the six power supplies must be energised, each rack needs three independent power sources, each connected to two of every node's six supplies, each from a separate upstream PDU, and each sized to carry half of the rack's expected peak.
The script checks why:
"""psu_feeds.py โ does a node survive losing one power feed?
A DGX H100 has six power supplies and needs at least four energised.
NVIDIA's SuperPOD design guide therefore asks for THREE independent
sources per rack, each feeding two of the six supplies. This checks
every possible wiring with two and with three feeds.
"""
from itertools import product
NEED = 4
def survives(wiring, lost):
return sum(1 for f in wiring if f != lost) >= NEED
def ok_all(w, feeds):
return all(survives(w, f) for f in feeds)
def main():
for feeds in ("AB", "ABC"):
ways = list(product(feeds, repeat=6))
good = [w for w in ways if ok_all(w, feeds)]
print(f"{len(feeds)} feeds: {len(good)} of {len(ways)} wirings survive any one feed failing")
cases = {"A/B, 3+3 (enterprise habit)": ("AAABBB", "AB"),
"A/B/C, 2+2+2 (NVIDIA design)": ("AABBCC", "ABC"),
"A/B/C, one cord on wrong strip": ("AAABCC", "ABC"),
"A/B/C, C strip never cabled": ("AAABBB", "ABC")}
for name, (w, feeds) in cases.items():
res = " ".join(f"lose {f}: {'runs' if survives(w, f) else 'STOPS'}" for f in feeds)
print(f" {name:32s} {res}")
print("lesson: with a 4-of-6 minimum, two feeds can never protect the node;")
print(" three feeds work only if every node is cabled 2+2+2 - check it rack by rack")
if __name__ == "__main__":
main()- With two feeds, no wiring protects the node: one feed always ends up with three or fewer supplies.
- With three, only the balanced 2+2+2 pattern (and its relabellings) survives the loss of any feed, and a single cord on the wrong strip quietly removes that protection.
- So the site visit includes walking a sample of racks and reading power strip labels, and the provider's power monitoring is asked to show per-outlet current, which reveals a mis-cabled supply as a strip carrying more than its share.
- A data hall built for 2N enterprise loads can host H100 nodes, but only if the rack power design follows the three-feed pattern.
- It is a cheap thing to check before delivery and an expensive thing to discover during the main run, when a routine UPS maintenance on one feed would stop every node in the hall at once.
8. How cold is cold enough?
Two industry frameworks set the vocabulary for cooling conversations:
- Air: ASHRAE's thermal guidelines define recommended and allowable ranges for server inlet air. Halls run inside the recommended envelope for reliability and use the wider allowable envelope only for short periods. Gopala asks for the hall's design inlet temperature and last summer's worst week, rather than for a single number.
- Liquid: ASHRAE classifies liquid-cooled environments by the maximum temperature of the facility water supplied to the equipment. The fifth edition (2021) renamed the classes by that upper limit: W17, W27, W32, W40 (new), W45 and W+ (โ ๏ธ via trade-press summaries). Warmer water means the facility can reject heat with dry coolers instead of chillers for more of the year, which lowers PUE and water use, but the equipment must be rated for it.
- For Maestro-1 on air-cooled H100s, the practical check is simpler: inside his own telemetry (Part 7), GPU temperatures under full load should be consistent across racks, and no GPU should report thermal slowdown.
- A rack row that runs several degrees hotter than the others is a facilities ticket, not a GPU ticket.
9. Training is a strange electrical load
- A synchronised training job is not a steady load.
- Thousands of GPUs compute together, then wait together for communication, then compute again, so power can swing quickly by a large fraction across the whole cluster.
- Meta reported that power consumption across its Llama 3 training cluster fluctuated by tens of megawatts in step with the job, which stretches the limits of the power grid.
- At 1.5 MW Gopala will not trouble a grid, but the swings still matter to the provider's UPS and generators.
Two practical consequences:
- Tell the provider when a large job starts and stops; some ask for it.
- Do not restart all 1,024 GPUs at the same second after a failure if the provider asks for a ramp. Part 7's restart logic can stagger it.
10. The site-visit checklist
A renter cannot design the building, but he can check it. Gopala's checklist, used on the scenario visit:
| Area | Question | Good answer |
|---|---|---|
| Power | Rack power limit, and is it contracted per rack? | โฅ 40 kW per rack for air-cooled H100, written into the order |
| Power | How many independent feeds per rack, and how are the six node PSUs split? | Three feeds from separate upstream PDUs, each node cabled 2+2+2 (the cabling check) |
| Power | UPS runtime and generator test schedule | Minutes of UPS, monthly generator tests announced in advance |
| Cooling | Supply temperature and the worst week last summer | Within ASHRAE's recommended range all year |
| Cooling | Containment, and blanking panels in empty slots | Hot or cold aisle contained; no gaps |
| Space | Are all 128 nodes in one hall, close to the spine switches? | Yes, with cable lengths that allow the planned fabric |
| People | Who swaps a node at 3 a.m., and where are spares? | Staff on site 24/7, spare nodes and GPUs in the building |
| Security | Who can enter the cage; how are failed drives handled? | Named access list, drives destroyed on site with a record |
| Water | How the hall uses water for cooling | A stated water-use figure (WUE) and source |
- On the visit, the hall meets every item except one: generator tests are monthly but not announced to tenants.
- Gopala gets a clause requiring 7 days'
- notice and agrees with the provider that tests will not be scheduled during the main run (maintenance windows).
- The hall itself is not a real site.
- Gopala gets a clause requiring 7 days'
11. Maintenance windows and change freezes
- Facilities have to be maintained: UPS batteries replaced, generators tested, switchgear inspected.
- Each event carries a small risk of a stop.
The agreement Gopala signs:
- A 30-day change freeze on his racks' power and cooling during the main run (months 9โ10.5), except for emergencies.
- 7 days' notice for any planned work that touches his power path.
- A shared calendar so the provider's maintenance and his checkpoints line up: take a checkpoint before planned work, not after.
12. If the next model needs liquid cooling
- Rack-scale systems such as NVIDIA's GB200 NVL72 put 72 GPUs in one liquid-cooled rack drawing on the order of 120 kW or more (โ ๏ธ exact figures vary by configuration), roughly ten times what the hall above was built for per rack.
- If Maestro-2 is trained on such systems, the facility question changes from "does the hall have enough space?"
- to "does the hall have facility water at the rack, coolant distribution units and a floor that can take the weight?".
- Only a subset of halls do, and they are booked early.
- That is why Part 4's contract includes an upgrade path, and why Gopala keeps a short list of providers with liquid-cooled capacity even though he does not need it yet.
- The site's cooling and water article covers the engineering.
- If Maestro-2 is trained on such systems, the facility question changes from "does the hall have enough space?"
13. Facility data a renter should ask for
| Signal | Where it comes from | What he uses it for |
|---|---|---|
| Per-outlet power on rack PDUs | Intelligent rack PDUs | Spotting mis-cabled supplies; real power per node |
| Rack inlet temperature | Sensors at the cold aisle | Separating airflow problems from GPU problems |
| Planned maintenance calendar | Facilities team | Taking a checkpoint before work on a feed |
| Incident reports | Facilities team | Matching a job failure to a power or cooling event |
| Monthly energy and PUE | Building management system | Cost and emissions reporting |
- Most providers can export some of this through an API or a portal.
- Getting it into the same dashboards as GPU metrics (Part 7) is what lets him answer the question "was that us or the building?" in minutes.
14. Five facility mistakes renters make
- Sizing to GPU TDP (700 W ร GPUs) instead of node maximum power, and running out of rack power.
- Accepting 2N A/B feeds for 6-PSU nodes without checking the cabling (the cabling check).
- Splitting the cluster across halls to fit low-density racks, then paying in cable lengths and extra switch layers.
- Never asking about maintenance windows until a generator test lands in the middle of a run.
- Treating hot nodes as bad GPUs and replacing hardware that was fine.
15. The environmental footprint
- Investors and design partners now ask.
- From the script: about 1,070โ1,320 MWh a month while the cluster runs hard, depending on the hall's efficiency.
- Emissions depend entirely on the local grid mix, which differs by more than ten times between regions, so Gopala asks the provider for the grid's carbon intensity and any renewable contracts that cover the hall, and reports the number in his model card later (Part 14) rather than making a claim he cannot back.
16. Delegation box
- The sizing script and the requirements
- The site visit and the checklist
- The change-freeze agreement
- Reading the provider's facility documents and summarising gaps
- Pulling temperature history into a chart
- Drafting the maintenance-window clause
- The provider's facilities team (they run the building)
- A facilities consultant for one day of the visit
- The provider's existing certifications (e.g. ISO 27001, SOC 2)
17. Budget and risk tracker
| Item | Status after Part 5 | Note |
|---|---|---|
| Spend to date | ~$45k (scenario) | Plus the visit and a consultant day |
| Facilities cost | Inside the GPU price | Power is passed through; efficiency matters at renewal |
| Time | 16 of ~78 weeks | Delivery begins month 3 |
| Power and cooling risk | Low for H100 air-cooled | Change freeze signed for the main run |
| Upgrade risk | Medium | Liquid cooling needed for future rack-scale systems |
| Throttling risk | Unknown | Watch GPU temperatures and clocks per node from day one |
18. The series
The Maestro: 18 parts in five phases
- Renting the building does not remove the physics: Maestro-1 needs about 1.5 MW of IT capacity, dense racks, a power path that will not stop the job, and a change freeze during the main run.
- Next, Part 6: the cluster arrives, and nothing is trusted until it is tested.
19. Reading the numbers
- Primary / standard
- DGX H100: 6 ร 3.3 kW PSUs, 4+2 redundancy, 10.2 kW maximum (NVIDIA user guide); four of six must be energised, three power sources per rack each feeding two PSUs, each path sized for half the rack's peak (NVIDIA SuperPOD H100 data-centre design guide). H100 700 W TDP. Llama 3 trained on 16,384 H100s; power swings across the cluster (Meta).
- Secondary โ ๏ธ
- ASHRAE liquid-cooling classes W17โW45 and W+ (5th edition naming, via trade press). Global average PUE ~1.54 in 2025 (Uptime Institute survey, as reported by secondary coverage); 1.52 capacity-weighted in Uptime's 2026 survey. Typical rack densities by cooling type are industry ranges.
- Our assumptions
- Sustained draw at 80% of maximum, +12% for network and storage, PUE 1.25 for a modern hall, $0.08/kWh, the site visit and agreements.