Home โ€บ Blog โ€บ Parallelism libraries
Inside the ML Layer ยท Part 3 ยท Parallelism

Splitting the Giant: The Libraries That Spread One Model Across 16,000 GPUs

No frontier model fits on one chip. Between the framework (Part 2) and the kernels (Part 4) sits a layer of libraries whose only job is to split a model, its data and its optimizer across thousands of GPUs while keeping them busy: NVIDIA's Megatron-Core, Microsoft's DeepSpeed, PyTorch's FSDP2 and TorchTitan, Google's MaxText, and DeepSeek's in-house stack. This part explains what each one does, why pipelines leave GPUs idle, why mixture-of-experts models need their own tricks, and how to choose.

FL
FrontierAGI Team
โš ๏ธ How this was researched (as of 27 Oct 2026) Sources are the original papers (Megatron-LM, ZeRO, GPipe, PipeDream, zero-bubble pipelines, DeepSpeed-Ulysses, ring attention, PyTorch FSDP, TorchTitan), Meta's Llama 3 paper, DeepSeek's V3 technical report and open-sourced repositories, NVIDIA's Megatron-LM repository and the PyTorch Foundation. Utilisation figures come from different setups and are not comparable with each other. The strategy builder and pipeline animator are our own simplified models. See Reading the numbers.

1. Case file

๐Ÿงฉ
The parallelism layer, October 2026Layer 5 of 8 in the ML layer map
Main open librariesMegatron-Core ยท DeepSpeed ยท FSDP2/TorchTitan ยท MaxText
DeepSpeed governancePyTorch Foundation hosted project since May 2025
Megatron-CoreSplit out of Megatron-LM, Jan 2024
Llama 3 405B16,384 H100s ยท 4D parallelism ยท 38โ€“43% MFU
DeepSeek-V32,048 H800s ยท in-house HAI-LLM ยท 2.788M GPU-hours
TorchTitan (Llama 3.1 405B)+30% over baseline on 512 H100s

2. Timeline

Splitting models, 2018 โ†’ 2026

Tap a milestone. Colour shows the idea: data and sharding, model splitting, pipelines, sequence, experts.

Tap a milestone.

3. Five ways to split a model

Every library combines the same five ideas. Producing Intelligence Part 4 follows them through Meta's Llama 3 run; here is the short version, with what each one costs.

1. Data parallel (DP) + shardingEach group of GPUs sees different data. Plain DP copies the whole model to every GPU; ZeRO and FSDP shard optimizer state (stage 1), gradients (stage 2) and weights (stage 3) across them, gathering weights just in time.Cost: all-gather and reduce-scatter every layer; works across slower links.
2. Tensor parallel (TP)Each layer's big matrices are cut into slices on different GPUs, which compute partial results and combine them. Megatron-LM's 2019 design for transformer layers is still the standard.Cost: several all-reduces per layer; needs NVLink, so usually โ‰ค 8 GPUs (one server) or one NVL72 rack.
3. Pipeline parallel (PP)Consecutive layers live on different GPUs; micro-batches flow through like an assembly line.Cost: idle "bubbles" at the start and end of each step (section 7); little communication.
4. Context / sequence parallel (CP)A very long sequence is split along its length; attention needs keys and values from other chunks, passed in a ring or swapped all-to-all.Cost: extra attention communication; essential above ~32K tokens.
5. Expert parallel (EP)In mixture-of-experts models, different experts live on different GPUs and tokens are sent to whichever experts the router chooses.Cost: two all-to-alls per MoE layer, often across nodes.

4. Strategy builder

Build a parallelism plan

Our own simplified model of a dense transformer trained in mixed precision with Adam: ~16 bytes per parameter of training state, plus an illustrative activation term. 80 GB GPUs, 8 per server. Not a substitute for a real memory profiler.

Presets:

5. The libraries

Two families have emerged. Megatron-style libraries hand-engineer each transformer block for tensor and pipeline parallelism and squeeze out the last few percent on NVIDIA hardware. Sharding-first designs (FSDP2, TorchTitan, JAX with XLA) treat parallelism as a layout of arrays on a device mesh and compose the dimensions more freely. In 2026 they borrow from each other constantly: TorchTitan adds pipeline schedules and Float8, Megatron-Core adds FSDP, and both import ideas DeepSeek open-sourced.

6. Capability matrix

Library ร— technique

From project documentation and papers. โ— built in ยท โ— partial or via integration ยท โ€” not a focus. Tap a cell.

Tap a cell.

7. The bubble problem

Pipelines have a structural flaw: the last stage cannot start until the first has finished, and the first sits idle while the last finishes the backward pass. That idle time is the bubble. With p stages and m micro-batches, GPipe's bubble fraction is (p โˆ’ 1) / (m + p โˆ’ 1). More micro-batches shrink it, but each one needs stored activations.

Pipeline schedule animator

4 stages. Forward = 1 time unit, backward = 2. Grey = idle. Our own simulation of the textbook schedules.

ForwardBackwardIdle (bubble)
2018GPipeAll forwards, then all backwards. Simple; stores activations for all m micro-batches.
2019โ€“211F1B (PipeDream)Same bubble, but alternates forward and backward so each stage holds at most p micro-batches of activations.
2021Interleaved 1F1BMegatron gives each GPU several smaller chunks of layers (v), cutting the bubble roughly v-fold at the cost of more communication.
2023Zero bubbleSplits backward into input-gradient and weight-gradient work and schedules weight gradients into the gaps.
2024DualPipeDeepSeek feeds micro-batches from both ends and overlaps computation with communication; fewer bubbles than prior methods, per the V3 report.

8. Mixture of experts makes it harder

Tokens on the move

Each token is routed to a few experts that may live on other GPUs: one all-to-all to send it, another to bring the result back. Tap Route.

Mixture-of-experts models such as DeepSeek-V3 activate only a fraction of their parameters per token, which makes them cheap to train for their size but heavy on communication: every MoE layer needs two all-to-all exchanges, often across InfiniBand between servers. The DeepSeek-V3 report describes custom cross-node all-to-all kernels that use both InfiniBand and NVLink while tying up few streaming multiprocessors, and DualPipe to hide that communication behind computation. DeepSeek later open-sourced these pieces (DeepEP for expert-parallel communication and DualPipe), and they have since been integrated into other stacks. If the router sends too many tokens to one expert, that GPU becomes the bottleneck: load balancing is a parallelism problem as much as a modelling one.

9. Long context

Ring attention / context parallelThe sequence is cut into chunks; each GPU keeps its queries and passes keys and values around a ring so every chunk attends to every other. Meta used context parallelism to extend Llama 3 to 128K tokens; TorchTitan's tables show Llama 3.1 405B trained up to 262,144 tokens with CP 8 (at 16 tokens per second per GPU, against 76 at 32K).
DeepSpeed-UlyssesInstead of a ring, an all-to-all swaps the split from sequence to attention heads, so each GPU computes full attention for a subset of heads, then swaps back. Limited by the number of heads, but simple and fast on good networks.

10. Who uses what

OrganisationParallelism stackSource
MetaIn-house 4D (FSDP, TP, PP, CP) on PyTorch; TorchTitan is its open referenceLlama 3 paper; TorchTitan paper
Google DeepMindJAX + XLA sharding, Pathways; MaxText as open referenceGoogle materials
DeepSeekHAI-LLM (in-house) with DualPipe, DeepEPV3 technical report
NVIDIA and many enterprisesMegatron-Core, NeMoNVIDIA repository
Open-weight labs and startupsMegatron-Core, DeepSpeed, FSDP2/TorchTitan, Nanotron โš ๏ธ (varies by release)Model cards and papers
OpenAI, Anthropic, xAINot disclosedโ€”

11. Reported performance

What the papers say

Each bar is a different kind of number from a different setup. Read the label, not the length.

MFU = model FLOPs utilisation, the share of the chips' peak compute spent on the model's maths. Llama 3: BF16 MFU 38โ€“43% on 8Kโ€“16K H100s (Meta). Megatron 2021: 52% of peak on 3,072 A100s for a 1T-parameter model (Narayanan et al.). TorchTitan: speedups over its own optimised baselines on H100s.

12. Choosing a library

Which library fits?

An editorial guide.

VerdictParallelism libraries are where a lab's scale is won or lost. For dense models on NVIDIA GPUs, Megatron-Core remains the performance benchmark, while FSDP2 and TorchTitan offer a simpler, PyTorch-native path that has closed much of the gap. For mixture-of-experts models, DeepSeek's open-sourced communication tools changed what everyone else considers possible. On TPUs, JAX and XLA do most of this for you. The largest labs still build their own; what they publish suggests the ideas are now shared and the advantage lies in engineering and hardware co-design.

13. Reading the numbers

Primary
Llama 3: 16,384 H100s, 38โ€“43% MFU, 4D parallelism, 128K context with CP (Meta paper). DeepSeek-V3: 2,048 H800s, HAI-LLM, DualPipe, custom all-to-all kernels, 2.788M H800 GPU-hours (report). TorchTitan: +65.08% (8B, 128 GPUs), +12.59% (70B, 256 GPUs), +30% (405B, 512 GPUs) over optimised baselines; 405B throughput 76 โ†’ 16 tokens/s/GPU from 32K to 262K context (paper). Megatron-Core announced Jan 2024 (NVIDIA repository). DeepSpeed accepted as a PyTorch Foundation hosted project, May 2025, after Microsoft contributed it to the Linux Foundation (PyTorch Foundation).
Formulas
GPipe and 1F1B bubble: (p โˆ’ 1)/(m + p โˆ’ 1). Interleaved schedules reduce it roughly by the number of chunks per GPU.
Our models โš ๏ธ
The strategy builder (16 bytes/parameter, a crude activation term, ideal sharding) and the pipeline animator (F = 1, B = 2, no communication time) are illustrations. Library usage by open-weight labs varies and is marked โš ๏ธ.