Home โ€บ Blog โ€บ Splitting a Giant
Producing Intelligence ยท Part 4 ยท Parallelism

Splitting a Giant: How Llama 3 405B Was Divided Across 16,384 GPUs

Llama 3 405B does not fit on any single GPU, or even a single server. Training it means cutting the model, the data and even individual sequences into pieces, spreading them over thousands of H100s, and keeping all those pieces in step without drowning the network. Meta did it with "4D parallelism": tensor, context, pipeline and data parallelism layered on top of each other and matched to the shape of its network. This part explains each dimension, Meta's exact configurations, the memory arithmetic, the pipeline "bubble" problem, and why 38โ€“43% of peak is a good result.

FL
FrontierAGI Team
โš ๏ธ How this was researched The configurations come from Table 4 of Meta's paper "The Llama 3 Herd of Models" (section 3.3.2). The 8,192-GPU row and the 16,384-GPU efficiency figures are quoted directly in search results and in Meta's follow-up ISCA 2025 paper; the long-context row (CP 16, DP 8) comes from reproductions of the table in secondary summaries, and we checked that its numbers multiply out correctly. Meta has not published per-GPU memory breakdowns, so the memory calculator, parallelism animations and bubble simulator are simplified models. See Reading the numbers.

๐Ÿงณ The model does not fit

Part 3 fixed the plan: 405 billion parameters, 15.6 trillion tokens, up to 16,384 H100 GPUs. Now the problem is physical. In 16-bit precision the weights alone take about 810 GB. Training also needs gradients and optimiser state: with standard mixed-precision Adam, roughly 16 bytes per parameter, about 6.5 TB, before counting the activations saved during each forward pass. Each H100 has 80 GB of memory.

~6.5 TBweights + gradients + optimiser state for 405B (โ‰ˆ16 bytes/param)
80 GBmemory on one H100
4Dtensor ร— context ร— pipeline ร— data parallelism
43%best model FLOPs utilisation (8,192 GPUs)

So the model has to be split, and not in one way but four at once. For the basic concepts, see Scaling Laws, GPUs and Distributed Training; this article focuses on how Meta actually did it.

โœ‚๏ธ The four (and a half) ways to split

๐ŸŽฌ Parallelism visualiser

Pick a strategy to see what each of four GPUs holds and what has to cross the network.

๐Ÿ—บ๏ธ Llama 3's layout: TP 8 ร— CP ร— PP 16 ร— DP

Meta grouped GPUs in the order [TP, CP, PP, DP] and implemented data parallelism with FSDP, which shards parameters, gradients and optimiser state across data-parallel ranks. Table 4 of the paper lists the configuration for each stage of pre-training. In every row, multiplying the four dimensions gives the GPU count, and the global batch stays at about 16 million tokens.

๐Ÿ“‹ Configurations by training stage

Click a row to check the arithmetic.

Llama 3 paper Table 4: 8,192-GPU row quoted in search results; 16K-GPU throughput (400 and 380 TFLOPs/GPU) also reported in Meta's ISCA 2025 paper; long-context row (CP 16, DP 8) from secondary reproductions of the table.

Doubling from 8,192 to 16,384 GPUs doubled data parallelism (64 โ†’ 128), but to keep the global batch at 16M tokens each data-parallel group had to process half as many sequences, so pipelines had less work to overlap and MFU fell from 43% to 41%. Long-context training kept the GPU count but redistributed it: context parallelism of 16 splits each 131,072-token sequence across 16 GPU groups, leaving only 8 data-parallel replicas.

๐ŸŒ Matching the split to the network

The order is not arbitrary. Each dimension communicates with a different intensity, so Meta placed the chattiest one on the fastest links and the most tolerant one on the slowest (see the network map in Part 1).

๐Ÿ”— From NVLink to the building

Innermost to outermost, for the 8,192-GPU configuration.

๐Ÿงฎ Memory budget per GPU

๐ŸŽ›๏ธ Will it fit in 80 GB?

Simplified model. Defaults approximate Llama 3 405B's main stage; Meta's actual memory breakdown is not published.

Sharding
Activations

Weights 2 bytes/param, gradients 2, optimiser 12 (FP32 master copy + two Adam moments), each divided by TP ร— PP and, where sharded, by DP. Activations โ‰ˆ layers per stage ร— (sequence รท CP) ร— hidden ร— bytes-per-token-per-layer รท TP ร— PP micro-batches in flight (34 bytes with none recomputed, ~8 with selective recomputation, ~2 with full). Hidden size and layers scale with model size. Ignores buffers, fragmentation and embeddings.

๐Ÿซง Pipeline bubbles and schedules

Pipeline parallelism gives each group of GPUs a slice of layers (Llama 3: 126 layers across 16 stages). The catch is the bubble: at the start of each batch, later stages wait for the first micro-batch to reach them, and at the end, early stages sit idle. Splitting the batch into more micro-batches shrinks the bubble, and interleaved schedules give each GPU several smaller slices so the pipeline fills faster. Meta modified the interleaved schedule so the batch no longer had to be a multiple of the number of pipeline stages, accounted for the first stage's extra memory (embeddings and warm-up micro-batches), and co-designed layer counts with the parallelism layout to balance compute and memory across stages.

๐ŸŽ›๏ธ Bubble simulator

GPipe-style schedule: forward passes (blue) then backward passes (purple). Grey cells are idle time.

Bubble fraction โ‰ˆ (p โˆ’ 1) รท (v ร— m + p โˆ’ 1) for p stages, m micro-batches and v virtual stages (Narayanan et al., 2021). The chart draws the non-interleaved case; the interleaving slider only changes the computed fraction.

๐Ÿ“ Long context: context parallelism

Attention cost grows with the square of sequence length, and activations grow linearly, so a 131,072-token sequence cannot be handled by one GPU group. Context parallelism splits each sequence along its length across GPUs. Meta used an all-gather approach: each GPU computes queries for its own chunk but gathers the keys and values it needs from the others, and the design supports document masks, so separate documents packed into one sequence do not attend to each other. Related techniques such as ring attention pass key-value blocks around a ring instead. Even so, throughput fell to about 380 TFLOPs per GPU (โ‰ˆ38% MFU) at 131K tokens.

๐Ÿ“‰ What limits efficiency

๐Ÿ“Š Throughput by configuration

BF16 MFU and TFLOPs per GPU from the Llama 3 paper and Meta's ISCA 2025 paper; H100 dense BF16 peak โ‰ˆ 989 TFLOPs.

Against those forces, 38โ€“43% of theoretical peak on 16,384 GPUs is a strong result; the remaining gap is why labs keep investing in kernels, communication libraries and network design.

๐Ÿฆ™ Llama 3 case file: the split Order [TP, CP, PP, DP] ยท TP 8 inside each 8-GPU server (NVLink) ยท PP 16 across 126 layers ยท DP via FSDP ยท 8,192 GPUs: DP 64, 43% MFU ยท 16,384 GPUs: DP 128, 41% ยท long context (131K tokens): CP 16, DP 8, ~38% ยท global batch โ‰ˆ16M tokens throughout.

๐Ÿ•ฐ๏ธ Timeline

2018โ€“19 ยท Google's GPipe introduces micro-batched pipeline parallelism; Microsoft's PipeDream proposes 1F1B scheduling.
2019 ยท Nvidia's Megatron-LM introduces efficient tensor parallelism for transformers.
2019โ€“20 ยท Microsoft's ZeRO shards optimiser state, gradients and parameters across data-parallel ranks.
2021 ยท Megatron's interleaved pipeline schedule and "3D parallelism" for trillion-parameter training.
2022 ยท Sequence parallelism and selective activation recomputation (Korthikanti et al.); PyTorch FSDP.
2023 ยท Ring attention for very long contexts.
2024 ยท Llama 3 405B trained with 4D parallelism on up to 16,384 GPUs; DeepSeek-V3 combines expert parallelism with FP8 at scale.
2025 ยท Meta's ISCA paper details Llama 3's parallelism; workload-balanced 4D parallelism (WLB-LLM).

๐Ÿ”ญ What to watch

๐ŸŽฏ The bottom line Training Llama 3 405B meant splitting it four ways at once: tensor parallelism of 8 inside each server, pipeline parallelism of 16 across the model's 126 layers, context parallelism for long sequences, and FSDP data parallelism across everything else, ordered so the most talkative split used the fastest links. That layout kept 8,192 to 16,384 H100s working on one model at 38โ€“43% of peak. Every frontier lab solves the same puzzle; what differs is the model (dense or mixture-of-experts), the network, and how cleverly the pieces are scheduled.

๐Ÿ”ข Reading the numbers

๐Ÿ“š References and papers