Scaling Laws, GPUs and Distributed Training: How Frontier Models Get Built

The Big Picture

Language-model quality has improved predictably as three inputs grew together: parameters (N, the size of the model), data (D, the number of training tokens), and compute (C, the floating-point operations spent training). The observation that loss falls smoothly and predictably as these grow — a scaling law — is what turned "train a bigger model" from a gamble into an engineering plan. It is also why the rest of this article exists: turning more compute into a model means buying GPUs, and using thousands of GPUs on one model means splitting the work between them.

Empirical Scaling Laws

Kaplan et al. (OpenAI, 2020) trained many transformer language models of different sizes and found that test loss follows a power law in each of N, D and C when the other two aren't the bottleneck, over more than seven orders of magnitude. On a log-log plot a power law is a straight line: every 10× increase in compute buys the same fractional drop in loss. Other details — depth versus width, number of heads — mattered far less than total size within a wide range.

We ran the same kind of study in miniature: five byte-level GPTs from 12 thousand to 1.8 million non-embedding parameters, each trained on Python source code with validation loss measured at seven points during training. Each curve below is one model; the x-axis is training compute (6·N·D, explained in the next section).

Three things in the chart are the whole idea of scaling laws in miniature. At any fixed number of tokens, the bigger model has lower loss. At a fixed compute budget, the best model is neither the smallest nor the biggest. And the frontier — the best loss reachable at each budget — is close to a straight line on log-log axes, which is a power law: ours is L ∝ C^−0.102. The exponent depends on the data, tokenizer and architecture, so it is not comparable to published values; what carries over is the shape.

Counting Compute: C ≈ 6·N·D

Training compute is estimated with a rule of thumb that falls out of counting matrix multiplies (derived line by line in Resource Accounting). Each parameter takes part in one multiply-add per token on the forward pass — 2 FLOPs — and the backward pass costs about twice the forward, for gradients with respect to both activations and weights:

To turn FLOPs into time: an H100 does at most 989 × 10¹² dense BF16 FLOPs per second, and real training reaches roughly 40% of that (Llama 3 reported 38–43% model FLOPs utilization). Llama 3 70B's 6.3 × 10²⁴ FLOPs at 400 TFLOP/s is about 1.6 × 10¹⁰ GPU-seconds — roughly 4.4 million H100-hours, or about 11 days on 16,000 GPUs. That is an estimate from the formula, not a figure Meta reported for this model.

Compute-Optimal Training: Chinchilla

Given a fixed compute budget C = 6ND, you can train a big model on few tokens or a small model on many. Which is best? Hoffmann et al. (DeepMind, 2022) trained over 400 models and found that N and D should grow in equal proportion: doubling compute should mean roughly √2 more parameters and √2 more tokens. That works out to about 20 tokens per parameter. Their fitted loss function makes the trade-off explicit:

The paper's headline test: Chinchilla, 70B parameters on 1.4 trillion tokens, used the same compute as the 280B-parameter Gopher trained on 300 billion — 4× smaller, about 4.7× more data — and outperformed it across a large range of benchmarks. Many earlier large models, it turned out, had been undertrained.

Our miniature study shows the same effect: at each budget, the model that reaches the lowest loss changes as the budget grows.

The best model size rises with compute — from 12,288 to 589,824 parameters across 2½ orders of magnitude of budget — which is the qualitative Chinchilla result. The tokens-per-parameter column is not a reproduction of the 20:1 rule: our grid has only five model sizes, the learning rate isn't re-tuned or annealed for each budget as Chinchilla's were, and a byte-level model on 1.6 million tokens is far from the regime of the paper. Treat the trend as real and the ratios as noise.

Why GPUs: Thousands of Simple Cores

A transformer's compute is almost entirely matrix multiplication, and a matrix multiply is millions of independent multiply-adds. A CPU has a few dozen powerful cores built for sequential, branchy code with low latency. A GPU has thousands of simpler cores built for throughput: running the same instruction over huge amounts of data at once.

To multiply C = A × B on a GPU, the output matrix is cut into tiles. Each tile is assigned to a block of threads on one SM, which loads the matching strips of A and B from high-bandwidth memory (HBM) into fast on-chip shared memory, computes the tile's partial products on tensor cores, and accumulates. Thousands of tiles run in parallel. Loading each strip once into on-chip memory and reusing it many times is the key trick — it's what makes large matrix multiplies compute-bound rather than memory-bound, as the next sections show.

Inside an H100

NVIDIA's H100 (Hopper architecture) is the workhorse of 2023–2025 frontier training. The SXM version's headline numbers, from NVIDIA's specifications:

Precision is the biggest lever. Fewer bits per number means more multiplies per tensor-core instruction, less memory and less bandwidth:

Compute-Bound vs Memory-Bound

Whether a GPU is limited by arithmetic or by memory depends on the operation's arithmetic intensity: FLOPs performed per byte moved from memory. The H100's balance point is 989 × 10¹² FLOP/s ÷ 3.35 × 10¹² B/s ≈ 295 FLOPs per byte in BF16. Below that, the cores wait on memory; above it, memory waits on the cores. This is the roofline model.

We measured both on a CPU, which has the same roofline shape with smaller numbers:

This single idea explains a lot of practice. Big matrix multiplies (large batch × large hidden size) are compute-bound and use the GPU well. Elementwise operations, normalizations and softmax are memory-bound, which is why kernel fusion — doing several of them in one pass over memory, as FlashAttention does for attention — pays off. And decode at inference time is memory-bound because it multiplies the weights by a single token's vector. It's also why tiny models waste hardware: their matrices are too small to keep the cores busy, the same way our n = 32 multiply did.

Four Kinds of Parallelism

A 70B model doesn't fit on one GPU for training (see Training Memory), and even if it did, 6.3 × 10²⁴ FLOPs on one H100 would take a century and a half. Training is spread across many GPUs by splitting along four different axes:

The difference between tensor and pipeline parallelism is worth being precise about, because they're often blurred: tensor parallelism splits a layer across GPUs; pipeline parallelism splits the stack of layers. A fifth axis, context (sequence) parallelism, splits a long sequence across GPUs, and is how long-context training fits in memory. Real runs combine these — Llama 3's largest models used tensor, context, pipeline and fully sharded data parallelism at once across up to 16,000 H100s.

How Data-Parallel Training Works

Every step of data-parallel training has the same shape:

Because the loss is a mean over examples, the average of per-shard mean gradients equals the gradient of the full batch. Data parallelism is not an approximation: it computes the same step as one GPU with the whole batch, just faster. Our script checks this with four processes, each taking 4 of 16 sequences, using PyTorch's real all_reduce:

data parallel on 4 processes: max |grad difference| vs single big batch = 1.49e-08 over 84,288 gradient values

The difference is at the level of float32 rounding — the only difference is the order in which numbers were added. The all-reduce itself is usually a ring: each GPU sends and receives 2(n−1)/n of the gradient size, about twice the gradient size whatever the number of GPUs, which is why data parallelism scales to thousands of GPUs. NCCL implements it on NVIDIA hardware; our script uses the CPU backend, gloo.

Where the Memory Goes in Training

With mixed-precision training and the Adam optimizer, every parameter costs 16 bytes before any activations are stored — the accounting from the ZeRO paper:

Plain data parallelism keeps all 16 bytes on every GPU, which is why it alone can't train a 70B model. ZeRO (and PyTorch's FSDP, which implements the same idea) removes the redundancy by sharding across the n data-parallel GPUs: stage 1 shards the optimizer state, stage 2 also the gradients, stage 3 also the weights, which are gathered layer by layer just before they're needed. Try it:

Activations — each layer's intermediate outputs, saved for the backward pass — come on top and grow with batch size and sequence length. They're reduced by activation checkpointing (store only some layers' outputs and recompute the rest during backward, trading about a third more compute for memory), by tensor and sequence parallelism, and by smaller micro-batches.

The Real-World Stack

A frontier training cluster is thousands of servers, each with 8 GPUs connected by NVLink and NVSwitch; servers connect through InfiniBand or RoCE Ethernet networks. Parallelism is mapped onto this hierarchy: tensor parallelism inside a server where bandwidth is highest, pipeline stages across servers, data parallelism across the whole cluster.

At this scale failure is routine, not exceptional: the Llama 3 paper reports hundreds of unexpected interruptions during a 54-day period of pretraining, most attributed to hardware, which is why checkpointing and fast automatic restarts are part of the training system itself.

Run the Experiments

One script, three experiments: the roofline on your own CPU, the miniature scaling study with a compute-frontier fit, and a four-process data-parallel check with real torch.distributed all-reduce. It trains on the Python standard library's own source code, so it needs no downloads. About 8 minutes on a 4-thread CPU. Needs PyTorch 2.x.

"""Three experiments: the roofline on your own CPU, a miniature scaling-law study, and data-parallel all-reduce."""
import math, os, pathlib, sysconfig, time
import torch
import torch.distributed as dist
import torch.multiprocessing as mp
import torch.nn as nn
import torch.nn.functional as F

torch.set_num_threads(4)

# ======================= 1. Compute-bound vs memory-bound (roofline) =======================
def bench(fn, reps=10):
    fn(); t = time.perf_counter()
    for _ in range(reps): fn()
    return (time.perf_counter() - t) / reps

def roofline():
    print("matrix multiply (n x n) @ (n x n), float32:")
    for n in (32, 128, 512, 2048):
        a, b = torch.randn(n, n), torch.randn(n, n)
        t = bench(lambda: a @ b, 20 if n < 2048 else 5)
        flops, bytes_moved = 2 * n ** 3, 3 * n * n * 4
        print(f"  n={n:>4}: {flops / t / 1e9:7.1f} GFLOP/s   arithmetic intensity {flops / bytes_moved:6.1f} FLOP/byte")
    x, y = torch.randn(32_000_000), torch.randn(32_000_000)
    t = bench(lambda: x + y, 5)
    print(f"elementwise add, 32M floats: {3 * 4 * 32e6 / t / 1e9:.1f} GB/s, {32e6 / t / 1e9:.2f} GFLOP/s (1 FLOP per 12 bytes)")

# ======================= 2. A miniature scaling-law study =======================
root = pathlib.Path(sysconfig.get_paths()["stdlib"])
text = b"".join(p.read_bytes() for p in sorted(root.glob("*.py")))          # ~4.8 MB of real text
data = torch.tensor(list(text), dtype=torch.long)
split = int(0.95 * len(data))
train_data, val_data = data[:split], data[split:]
T, B = 64, 16

class Block(nn.Module):
    def __init__(self, d, h):
        super().__init__()
        self.h, self.ln1, self.ln2 = h, nn.LayerNorm(d), nn.LayerNorm(d)
        self.qkv, self.proj = nn.Linear(d, 3 * d), nn.Linear(d, d)
        self.up, self.down = nn.Linear(d, 4 * d), nn.Linear(4 * d, d)
    def forward(self, x):
        Bb, Tt, d = x.shape
        q, k, v = (t.view(Bb, Tt, self.h, d // self.h).transpose(1, 2) for t in self.qkv(self.ln1(x)).split(d, -1))
        x = x + self.proj(F.scaled_dot_product_attention(q, k, v, is_causal=True).transpose(1, 2).reshape(Bb, Tt, d))
        return x + self.down(F.gelu(self.up(self.ln2(x))))
class GPT(nn.Module):
    def __init__(self, d, L, h=4):
        super().__init__()
        self.tok, self.pos = nn.Embedding(256, d), nn.Embedding(T, d)
        self.blocks = nn.Sequential(*[Block(d, h) for _ in range(L)])
        self.ln, self.head = nn.LayerNorm(d), nn.Linear(d, 256, bias=False)
    def forward(self, x):
        return self.head(self.ln(self.blocks(self.tok(x) + self.pos(torch.arange(x.shape[1])))))

def batch(src, g):
    i = torch.randint(0, len(src) - T - 1, (B,), generator=g)
    return torch.stack([src[j:j + T] for j in i]), torch.stack([src[j + 1:j + T + 1] for j in i])

@torch.no_grad()
def val_loss(m):
    g = torch.Generator().manual_seed(123)
    return sum(F.cross_entropy(m(x).reshape(-1, 256), y.reshape(-1)).item()
               for x, y in (batch(val_data, g) for _ in range(25))) / 25

def scaling_study():
    print(f"\ncorpus: {len(data):,} byte tokens of Python standard-library source")
    configs = [(32, 1), (48, 2), (96, 2), (128, 3), (192, 4)]      # (width d, layers L)
    checkpoints = [50, 100, 200, 400, 800, 1200, 1600]
    runs = {}
    for d, L in configs:
        torch.manual_seed(0)
        model, N = GPT(d, L), 12 * L * d * d                       # N = non-embedding parameters
        opt = torch.optim.AdamW(model.parameters(), lr=2e-3, weight_decay=0.1)
        g, pts = torch.Generator().manual_seed(0), []
        for step in range(1, checkpoints[-1] + 1):
            for grp in opt.param_groups: grp["lr"] = 2e-3 * min(1, step / 50)
            x, y = batch(train_data, g)
            loss = F.cross_entropy(model(x).reshape(-1, 256), y.reshape(-1))
            opt.zero_grad(); loss.backward(); opt.step()
            if step in checkpoints:
                D = step * B * T
                pts.append((6 * N * D, D, val_loss(model.eval()))); model.train()
        runs[N] = pts
        print(f"  N={N:>9,}: val loss " + "  ".join(f"{l:.2f}" for _, _, l in pts) + f"   (after {checkpoints[-1] * B * T:,} tokens)")

    # compute frontier: best loss achievable at each compute budget, then a power-law fit L = a * C^-alpha
    all_pts = sorted((C, l) for pts in runs.values() for C, _, l in pts)
    frontier, best = [], float("inf")
    for C, l in all_pts:
        if l < best: best = l; frontier.append((C, l))
    xs, ys = [math.log(C) for C, _ in frontier], [math.log(l) for _, l in frontier]
    n = len(xs); mx, my = sum(xs) / n, sum(ys) / n
    slope = sum((a - mx) * (b - my) for a, b in zip(xs, ys)) / sum((a - mx) ** 2 for a in xs)
    print(f"compute frontier: {len(frontier)} points, fitted L ~ C^{slope:.3f}")

    # compute-optimal model size: at a fixed budget, which N reaches the lowest loss?
    def loss_at(pts, C):
        for (c0, _, l0), (c1, _, l1) in zip(pts, pts[1:]):
            if c0 <= C <= c1:
                w = (math.log(C) - math.log(c0)) / (math.log(c1) - math.log(c0))
                return l0 + w * (l1 - l0)
        return None
    print("compute-optimal model size at fixed budgets:")
    for C in (1e10, 3e10, 1e11, 3e11, 1e12, 3e12):
        cands = {N: loss_at(p, C) for N, p in runs.items() if loss_at(p, C) is not None}
        Nopt = min(cands, key=cands.get)
        row = "  ".join(f"{N:>7,}:{l:.2f}" for N, l in sorted(cands.items()))
        print(f"  C={C:.0e}: best N = {Nopt:>9,} (D = {C / 6 / Nopt:>10,.0f} tokens, {C / 6 / Nopt / Nopt:5.1f} tokens/param)   [{row}]")

# ======================= 3. Data parallelism: all-reduce == one big batch =======================
def worker(rank, world, shared_x, shared_y, init_state, out):
    os.environ.update(MASTER_ADDR="127.0.0.1", MASTER_PORT="29531")
    dist.init_process_group("gloo", rank=rank, world_size=world)
    torch.set_num_threads(1)
    model = GPT(48, 2); model.load_state_dict(init_state)
    shard = slice(rank * 4, rank * 4 + 4)                           # each rank gets 4 of the 16 sequences
    loss = F.cross_entropy(model(shared_x[shard]).reshape(-1, 256), shared_y[shard].reshape(-1))
    loss.backward()
    for p in model.parameters():                                    # all-reduce: sum, then average
        dist.all_reduce(p.grad, op=dist.ReduceOp.SUM); p.grad /= world
    if rank == 0:
        out.put(torch.cat([p.grad.flatten() for p in model.parameters()]))
    dist.destroy_process_group()

if __name__ == "__main__":
    roofline()
    scaling_study()
    torch.manual_seed(1)
    ref = GPT(48, 2)
    x, y = batch(train_data, torch.Generator().manual_seed(9))
    F.cross_entropy(ref(x).reshape(-1, 256), y.reshape(-1)).backward()      # one process, full batch of 16
    full = torch.cat([p.grad.flatten() for p in ref.parameters()])
    ctx = mp.get_context("spawn"); q = ctx.Queue()
    procs = [ctx.Process(target=worker, args=(r, 4, x, y, ref.state_dict(), q)) for r in range(4)]
    for p in procs: p.start()
    dp = q.get()
    for p in procs: p.join()
    print(f"\ndata parallel on 4 processes: max |grad difference| vs single big batch = {(dp - full).abs().max():.2e} "
          f"over {full.numel():,} gradient values")

Output (PyTorch on a 4-thread CPU; timings vary by machine, but the shapes of the results and the gradient check hold):

matrix multiply (n x n) @ (n x n), float32:
  n=  32:    14.2 GFLOP/s   arithmetic intensity    5.3 FLOP/byte
  n= 128:   193.4 GFLOP/s   arithmetic intensity   21.3 FLOP/byte
  n= 512:   491.3 GFLOP/s   arithmetic intensity   85.3 FLOP/byte
  n=2048:   367.9 GFLOP/s   arithmetic intensity  341.3 FLOP/byte
elementwise add, 32M floats: 12.8 GB/s, 1.07 GFLOP/s (1 FLOP per 12 bytes)

corpus: 4,776,939 byte tokens of Python standard-library source
  N=   12,288: val loss 3.43  3.01  2.70  2.50  2.36  2.24  2.14   (after 1,638,400 tokens)
  N=   55,296: val loss 3.14  2.86  2.55  2.38  2.20  2.01  1.89   (after 1,638,400 tokens)
  N=  221,184: val loss 2.92  2.61  2.40  2.17  1.85  1.69  1.59   (after 1,638,400 tokens)
  N=  589,824: val loss 2.82  2.50  2.34  2.01  1.68  1.56  1.48   (after 1,638,400 tokens)
  N=1,769,472: val loss 2.68  2.44  2.23  1.87  1.60  1.49  1.43   (after 1,638,400 tokens)
compute frontier: 15 points, fitted L ~ C^-0.102
compute-optimal model size at fixed budgets:
  C=1e+10: best N =    12,288 (D =    135,634 tokens,  11.0 tokens/param)   [ 12,288:2.88]
  C=3e+10: best N =    12,288 (D =    406,901 tokens,  33.1 tokens/param)   [ 12,288:2.50   55,296:2.91]
  C=1e+11: best N =    12,288 (D =  1,356,337 tokens, 110.4 tokens/param)   [ 12,288:2.21   55,296:2.45  221,184:2.75]
  C=3e+11: best N =    55,296 (D =    904,225 tokens,  16.4 tokens/param)   [ 55,296:2.16  221,184:2.37  589,824:2.59]
  C=1e+12: best N =   221,184 (D =    753,520 tokens,   3.4 tokens/param)   [221,184:1.89  589,824:2.18  1,769,472:2.47]
  C=3e+12: best N =   589,824 (D =    847,711 tokens,   1.4 tokens/param)   [589,824:1.67  1,769,472:2.06]

data parallel on 4 processes: max |grad difference| vs single big batch = 1.49e-08 over 84,288 gradient values

⚠️ Corrections and Notes

🔗 References