Systems: The Distributed Infrastructure Underneath Every Model in This Series

Why Every Article So Far Has Quietly Depended on This One

Every architecture in "How Each Network Architecture Actually Learned," every training technique in this series, and every open bet from "After Transformers" assumes something this article finally makes explicit: none of it runs on one machine. A frontier-scale model trains across hundreds or thousands of GPUs simultaneously, and making that work — keeping every GPU's copy of the model consistent, fitting a model far larger than any single GPU's memory, and recovering when hardware inevitably fails mid-run — is its own engineering discipline, with its own real, citable research literature.

This article picks up exactly where Article 1's compute section and Article 2's self-assessment item 5 left off, going from "know that these three parallelism strategies exist" to "understand mechanically what each one is actually doing."

The Simplest Strategy: Copy the Model, Split the Data

Data parallelism is the most intuitive of the three: put a full copy of the model on every GPU, split each training batch into chunks (one chunk per GPU), and have every GPU compute its own forward and backward pass (Article 4's Era 2 mechanics) on its chunk independently and simultaneously. The catch: after this, every GPU has computed a slightly different gradient (based on its own chunk of data), and all copies need to end up with the exact same, averaged gradient before taking an optimizer step — otherwise the model copies would drift apart and stop being the same model.

The mechanism that makes this efficient is ring-allreduce (popularized by Baidu's 2017 engineering work applying it to deep learning): arrange the GPUs in a logical ring, and have each one pass a piece of its gradient to its neighbor while receiving a piece from the other side, repeating until every GPU has the full, summed gradient. Its key property: the communication cost per GPU stays roughly constant no matter how many GPUs are in the ring — a critical scalability property, since a naive "send everything to one central GPU" approach would create a communication bottleneck that gets worse as you add more GPUs.

The Real Bottleneck Is Memory, Not Just Compute

Data parallelism's core assumption — a full copy of the model fits on every GPU — breaks down for genuinely large models: the model's weights, its gradients, and its optimizer state (Adam, Article 3, keeps extra per-parameter bookkeeping) together can far exceed a single GPU's memory, long before compute becomes the limiting factor. Rajbhandari et al.'s ZeRO paper (previously cited in Article 1) solves this by partitioning — rather than fully replicating — the optimizer state, gradients, and eventually the weights themselves across the data-parallel GPUs, with three increasingly aggressive stages:

This is a direct, practical illustration of a general systems principle: there's rarely a free lunch — ZeRO's memory savings come at the cost of additional communication to reconstruct the full weights or gradients when a GPU actually needs them for its local computation, an explicit memory-versus-communication tradeoff a systems-literate researcher needs to reason about directly.

Splitting a Single Layer's Math Across GPUs

Even with ZeRO, some individual layers in the largest models are themselves too large to compute on one GPU. Shoeybi et al.'s Megatron-LM (previously cited in Article 1) introduces tensor parallelism: split a single layer's weight matrix itself — column-wise or row-wise — across multiple GPUs, so each GPU computes only its slice of a matrix multiplication, then the partial results are combined (via another collective communication step, similar in spirit to Part 1's allreduce) to produce the layer's true output.

The key tradeoff: tensor parallelism requires communication within every single layer's forward and backward pass (not just once per batch, as in data parallelism's gradient sync), which demands very high-bandwidth, low-latency interconnects between GPUs — this is why tensor parallelism is typically used only within a single physical machine's tightly-connected GPUs, rather than across machines connected by slower networking.

Splitting the Network's Layers Across GPUs

Huang et al.'s GPipe (2019) introduced the other major way to split a too-large model: assign different consecutive layers to different GPUs (GPU 1 holds layers 1–10, GPU 2 holds layers 11–20, and so on), so an input flows through the GPUs in sequence, pipeline-style. The naive version of this has a serious efficiency problem — a "pipeline bubble," where GPU 2 sits idle waiting for GPU 1 to finish layers 1–10 on the first input before it has anything to do, and this idle time recurs at the start and end of every batch. GPipe's fix is to split each batch into smaller "micro-batches" and feed them through the pipeline in a staggered, overlapping sequence, similar in principle to how an actual factory assembly line keeps every station busy by having multiple items in progress simultaneously rather than processing one item fully before starting the next.

Real Frontier Training Runs Use All Three at Once

Narayanan et al.'s "Efficient Large-Scale Language Model Training on GPU Clusters" (2021) — often called the "PTD-P" paper — demonstrates that real large-scale training combines all three strategies simultaneously, each applied at the layer of the hardware hierarchy it suits best: tensor parallelism within a single machine's tightly-connected GPUs (Part 3's high-bandwidth requirement), pipeline parallelism across machines within a cluster, and data parallelism across separate groups of machines entirely (Part 1's more relaxed, less frequent communication requirement). This "3D parallelism" isn't a theoretical nicety — it's the actual production strategy behind most of the frontier-scale training runs referenced throughout this series.

Serving a Trained Model Is Its Own Systems Problem

Everything above concerns training; serving a trained model to real users at scale is a related but distinct systems challenge. Kwon et al.'s vLLM/PagedAttention paper (2023, previously referenced via vLLM in Article 1) identifies a specific memory-management inefficiency in naive LLM serving: each ongoing conversation's "attention cache" (intermediate values needed to continue generating text) was typically allocated as one large, contiguous memory block per request, wasting substantial memory to fragmentation and over-allocation — genuinely analogous to how an operating system manages memory in fixed-size pages rather than large contiguous blocks, which is exactly the analogy the paper's name invokes. This single systems insight is a major reason vLLM became the dominant serving engine referenced in "After Transformers."

Revisiting the OPT-175B Logbook, Mechanically

Article 1's Scenario B cited Meta's public OPT-175B training logbook as evidence that distributed training is dominated by infrastructure firefighting. With this article's vocabulary, that claim becomes mechanically specific: a single failed GPU in a 992-GPU 3D-parallel job (Part 5) can stall the entire pipeline or data-parallel group it belongs to, since tensor- and pipeline-parallel groups (Parts 3–4) are tightly synchronized by design — there's no way for the rest of the group to simply continue without the failed GPU's shard of the model. This is precisely why the logbook records ~35 manual restarts: each hardware failure or network communication error (NCCL failures, in the terminology of Part 1's collective communication) required stopping, diagnosing, and resuming from the most recent saved checkpoint, not a graceful degradation.

Key Papers to Read First

Courses to Complete

Real Scenario Walkthroughs

Self-Assessment Checklist

⚠️ What's Missing

Where This Series Goes Next

Article 11, the capstone, ties every foundational concept from all ten prior articles — the technical stack, self-assessment, math, deep learning concepts, generalization theory, RL, interpretability, alignment, world models, and this article's systems layer — back into one unified map, showing how they connect into a coherent picture of what a working AGI researcher actually needs to know.

🎥 Recommended Videos

🧭 Closing — Distributed Training Is a Systems Discipline, Not an Afterthought