Compute Frontier: Why AI Keeps Building Bigger Clusters

The Efficiency Paradox

Cheaper per token does not mean cheaper frontier. Mixture-of-Experts (MoE) architectures attack one specific term in the compute equation — how many parameters actually participate in processing each token — and can cut that number by 20x or more. But the compute saved per token rarely stays saved. It gets spent on more training tokens (text, code, images, audio, video), longer context windows (more attention state per example), and reasoning-oriented post-training (RL, synthetic traces, inference-time search) — each of which increases total system compute even as compute-per-unit-of-capability falls. The simplified planning equation both the story and the lab below build on:

Real Training Runs

These are public, model-specific compute disclosures — not estimates of a company's total AI R&D spend. They show how differently a dense and a sparse (MoE) model of similar frontier ambition spend their compute budget.

GPU-hour comparisons are useful but not perfectly apples-to-apples: H100 and H800 differ in interconnect (H800 has reduced NVLink bandwidth for export-control reasons), software stacks differ, and "full training" boundaries vary by what each disclosure counts.

The comparison that matters: DeepSeek-V3 activates roughly 5.5% of its total parameters per token and trained in under a tenth of the GPU-hours Llama 3.1 405B needed, despite a comparable-or-larger total parameter count and a similar token count. That gap is the efficiency paradox made concrete — and it is exactly the gap that gets reinvested into the next thing on the list: more experiments, more modalities, more inference-time compute.

Why Concentrate Capacity

Owning GPUs is not the same as building a training computer. The harder engineering problem is connecting enough accelerators with adequate bandwidth, latency, memory locality, power, and cooling to keep them productive on one workload. A hyperscaler can own millions of accelerators spread across regions and still not be able to point more than a fraction of them at a single synchronous training job — topology, network bandwidth, and fault tolerance are the real constraint, not the fleet total.

The compute hierarchy runs from a single GPU (HBM + tensor cores), to an NVLink domain (tens of GPUs), to a rack or pod (hundreds to thousands), to a datacenter fabric (tens to hundreds of thousands), to a multi-site WAN-connected system (hundreds of thousands to 1M+). Each step up trades ease of scaling for a harder networking problem — and the entire point of a mega-cluster is to push that ceiling as high as possible before the network itself becomes the bottleneck.

The Hardware Timeline

Frontier compute stopped being just a GPU generation ago. The unit of design has climbed from chip → server → rack → pod → datacenter → multi-site fabric, and every generation improves raw arithmetic while memory capacity, bandwidth, and interconnect increasingly determine how much of that arithmetic is actually usable.

Peak numbers come from vendor specifications and use different precisions; they are not direct performance comparisons. The point is the direction of travel: more HBM capacity, faster links, larger coherent domains per generation.

Four bottlenecks determine how much of that peak arithmetic a real training job can actually use:

MoE Changes the Bottleneck

A sparse MoE model stores many experts but activates only a subset per token — a router network decides, per token, which experts to send it to. That can slash matrix-multiply work dramatically (DeepSeek-V3's ~5.5% activation ratio above is the clearest real example) while increasing pressure somewhere else entirely: memory placement (which experts live on which device) and interconnect traffic (the all-to-all communication needed to route tokens to experts that may live on a different accelerator). MoE does not remove the systems problem — it relocates it from arithmetic to routing and networking.

The Compute Lab, Reproduced in Python

The source project includes an interactive JavaScript simulator (rendered as sliders for total parameters, active-parameter ratio, training tokens, accelerator count, effective TFLOP/s, and cost per accelerator-hour) that converts a model design into wall-clock training time, GPU-hours, electricity, and rental-equivalent cost. Its stated methodology, quoted directly from the project's own README and page footers:

Below is that same methodology, reimplemented as original Python and independently executed against the DeepSeek-V3-like configuration from the table above, to confirm the arithmetic holds together end to end:

def training_flops(active_params, tokens):
    # C ~= 6 * active parameters * training tokens (the same 6N heuristic as CS336 Part 3)
    return 6 * active_params * tokens


def accelerator_hours(flops, effective_tflops):
    flops_per_second = effective_tflops * 1e12
    return flops / flops_per_second / 3600


def wall_clock_days(accel_hours, num_accelerators):
    return (accel_hours / num_accelerators) / 24


def bf16_checkpoint_bytes(total_params):
    return total_params * 2


def rental_cost(accel_hours, dollars_per_accel_hour):
    return accel_hours * dollars_per_accel_hour


def electricity_kwh(accel_hours, watts_per_accelerator=700):
    return accel_hours * watts_per_accelerator / 1000


# DeepSeek-V3-like configuration from the "Real Training Runs" table above
total_params = 671_000_000_000
active_params = 37_000_000_000
tokens = 14_800_000_000_000
num_accelerators = 2_048          # illustrative cluster size, not a disclosed figure
effective_tflops = 400              # sustained, not peak -- an assumption, not a disclosure
hourly_cost = 2.0

flops = training_flops(active_params, tokens)
hours = accelerator_hours(flops, effective_tflops)
days = wall_clock_days(hours, num_accelerators)
checkpoint = bf16_checkpoint_bytes(total_params)
cost = rental_cost(hours, hourly_cost)
energy = electricity_kwh(hours)

print(f"Training FLOPs: {flops:.3e}")
print(f"Accelerator-hours: {hours:,.0f}")
print(f"Wall-clock at {num_accelerators} accelerators: {days:,.1f} days")
print(f"BF16 checkpoint: {checkpoint/1e9:.1f} GB")
print(f"Rental-equivalent cost: ${cost:,.0f}")
print(f"Accelerator electricity: {energy:,.0f} kWh")

# Executed output for this exact configuration:
# Training FLOPs: 3.286e+24
# Accelerator-hours: 2,281,667
# Wall-clock at 2048 accelerators: 46.4 days
# BF16 checkpoint: 1342.0 GB
# Rental-equivalent cost: $4,563,333
# Accelerator electricity: 1,597,167 kWh

Two things worth checking against your intuition. First, the illustrative 2,048-accelerator, 400-sustained-TFLOP/s run above lands at roughly 46 days for a DeepSeek-V3-scale model — plausible against the ~2.788M actual H800-hours DeepSeek reported divided across whatever cluster size they actually used, though the source project is explicit that its own effective-TFLOP/s and cluster-size inputs are assumptions, not disclosures, so treat the exact day count as an order-of-magnitude planning number, not a reproduction of DeepSeek's real schedule. Second, this is a deliberately simple linear model: it does not capture communication overhead, topology, checkpointing, failures, or batch-size limits — the source project's own "lab warning" states this directly, and doubling accelerator count in the formula above does not model the sub-linear scaling that shows up in practice once a cluster gets large enough that the network, not the arithmetic, becomes the constraint.

Iteration Time as the Scarce Resource

A frontier research program is a loop, not a single run: hypothesize, train, evaluate, change the data or architecture, train again. Compressing a training run from months to weeks does not just save wall-clock time on one model — it multiplies the number of serious research cycles a team can run per year. If a training run consumes 30 million accelerator-hours, roughly 15,000 accelerators imply about 83 days at perfect (linear) scaling, while 150,000 accelerators imply about 8 days — real scaling is worse than linear at that scale, but the incentive to concentrate capacity is enormous regardless, because the scarce resource being purchased is often iteration time, not just raw FLOPs.

⚠️ Confidence Notes and Gaps

🔗 Full Reference List