Splitting the Giant: The Libraries That Spread One Model Across 16,000 GPUs
No frontier model fits on one chip. Between the framework (Part 2) and the kernels (Part 4) sits a layer of libraries whose only job is to split a model, its data and its optimizer across thousands of GPUs while keeping them busy: NVIDIA's Megatron-Core, Microsoft's DeepSpeed, PyTorch's FSDP2 and TorchTitan, Google's MaxText, and DeepSeek's in-house stack. This part explains what each one does, why pipelines leave GPUs idle, why mixture-of-experts models need their own tricks, and how to choose.
1. Case file
2. Timeline
Splitting models, 2018 โ 2026
Tap a milestone. Colour shows the idea: data and sharding, model splitting, pipelines, sequence, experts.
3. Five ways to split a model
Every library combines the same five ideas. Producing Intelligence Part 4 follows them through Meta's Llama 3 run; here is the short version, with what each one costs.
4. Strategy builder
Build a parallelism plan
Our own simplified model of a dense transformer trained in mixed precision with Adam: ~16 bytes per parameter of training state, plus an illustrative activation term. 80 GB GPUs, 8 per server. Not a substitute for a real memory profiler.
5. The libraries
Two families have emerged. Megatron-style libraries hand-engineer each transformer block for tensor and pipeline parallelism and squeeze out the last few percent on NVIDIA hardware. Sharding-first designs (FSDP2, TorchTitan, JAX with XLA) treat parallelism as a layout of arrays on a device mesh and compose the dimensions more freely. In 2026 they borrow from each other constantly: TorchTitan adds pipeline schedules and Float8, Megatron-Core adds FSDP, and both import ideas DeepSeek open-sourced.
6. Capability matrix
Library ร technique
From project documentation and papers. โ built in ยท โ partial or via integration ยท โ not a focus. Tap a cell.
7. The bubble problem
Pipelines have a structural flaw: the last stage cannot start until the first has finished, and the first sits idle while the last finishes the backward pass. That idle time is the bubble. With p stages and m micro-batches, GPipe's bubble fraction is (p โ 1) / (m + p โ 1). More micro-batches shrink it, but each one needs stored activations.
Pipeline schedule animator
4 stages. Forward = 1 time unit, backward = 2. Grey = idle. Our own simulation of the textbook schedules.
8. Mixture of experts makes it harder
Tokens on the move
Each token is routed to a few experts that may live on other GPUs: one all-to-all to send it, another to bring the result back. Tap Route.
Mixture-of-experts models such as DeepSeek-V3 activate only a fraction of their parameters per token, which makes them cheap to train for their size but heavy on communication: every MoE layer needs two all-to-all exchanges, often across InfiniBand between servers. The DeepSeek-V3 report describes custom cross-node all-to-all kernels that use both InfiniBand and NVLink while tying up few streaming multiprocessors, and DualPipe to hide that communication behind computation. DeepSeek later open-sourced these pieces (DeepEP for expert-parallel communication and DualPipe), and they have since been integrated into other stacks. If the router sends too many tokens to one expert, that GPU becomes the bottleneck: load balancing is a parallelism problem as much as a modelling one.
9. Long context
10. Who uses what
| Organisation | Parallelism stack | Source |
|---|---|---|
| Meta | In-house 4D (FSDP, TP, PP, CP) on PyTorch; TorchTitan is its open reference | Llama 3 paper; TorchTitan paper |
| Google DeepMind | JAX + XLA sharding, Pathways; MaxText as open reference | Google materials |
| DeepSeek | HAI-LLM (in-house) with DualPipe, DeepEP | V3 technical report |
| NVIDIA and many enterprises | Megatron-Core, NeMo | NVIDIA repository |
| Open-weight labs and startups | Megatron-Core, DeepSpeed, FSDP2/TorchTitan, Nanotron โ ๏ธ (varies by release) | Model cards and papers |
| OpenAI, Anthropic, xAI | Not disclosed | โ |
11. Reported performance
What the papers say
Each bar is a different kind of number from a different setup. Read the label, not the length.
MFU = model FLOPs utilisation, the share of the chips' peak compute spent on the model's maths. Llama 3: BF16 MFU 38โ43% on 8Kโ16K H100s (Meta). Megatron 2021: 52% of peak on 3,072 A100s for a 1T-parameter model (Narayanan et al.). TorchTitan: speedups over its own optimised baselines on H100s.
12. Choosing a library
Which library fits?
An editorial guide.
13. Reading the numbers
- Primary
- Llama 3: 16,384 H100s, 38โ43% MFU, 4D parallelism, 128K context with CP (Meta paper). DeepSeek-V3: 2,048 H800s, HAI-LLM, DualPipe, custom all-to-all kernels, 2.788M H800 GPU-hours (report). TorchTitan: +65.08% (8B, 128 GPUs), +12.59% (70B, 256 GPUs), +30% (405B, 512 GPUs) over optimised baselines; 405B throughput 76 โ 16 tokens/s/GPU from 32K to 262K context (paper). Megatron-Core announced Jan 2024 (NVIDIA repository). DeepSpeed accepted as a PyTorch Foundation hosted project, May 2025, after Microsoft contributed it to the Linux Foundation (PyTorch Foundation).
- Formulas
- GPipe and 1F1B bubble: (p โ 1)/(m + p โ 1). Interleaved schedules reduce it roughly by the number of chunks per GPU.
- Our models โ ๏ธ
- The strategy builder (16 bytes/parameter, a crude activation term, ideal sharding) and the pipeline animator (F = 1, B = 2, no communication time) are illustrations. Library usage by open-weight labs varies and is marked โ ๏ธ.
- Papers
- Megatron-LM (2019)
- Efficient large-scale training with Megatron-LM (2021)
- ZeRO (2019)
- GPipe (2018)
- Zero bubble pipeline parallelism
- DeepSpeed-Ulysses
- Ring attention
- PyTorch FSDP (2023)
- TorchTitan (ICLR 2025)
- The Llama 3 herd of models
- DeepSeek-V3 technical report
- Projects
- NVIDIA Megatron-LM / Megatron-Core
- PyTorch Foundation welcomes vLLM and DeepSpeed
- DeepEP ยท DualPipe
- MaxText
- Hugging Face: The Ultra-Scale Playbook
- Part 2: PyTorch vs JAX ยท Part 1: the map