From Checkpoint to Endpoint: How Llama 3 405B Is Served
Training produced a set of weights. Users get value only when those weights answer requests, quickly and cheaply, millions of times a day. Serving a 405-billion-parameter model raises its own engineering problems: in BF16 it does not fit in one server, every conversation drags a growing memory cache behind it, and each generated token has to read the whole model from memory. This part covers how Meta and the inference ecosystem solved those problems for Llama 3 405B: two-node pipelines, micro-batching, a careful FP8 recipe, and the engines that serve it.
1. From one giant job to millions of small ones
Parts 4 and 5 described one enormous, synchronised training job. Serving inverts almost every assumption.
| Training (Parts 4โ5) | Serving (this part) | |
|---|---|---|
| Unit of work | One job, months long | Millions of requests, seconds each |
| Goal | Maximise utilisation (MFU) | Meet latency targets at lowest cost per token |
| Scale | 16,384 GPUs together | Many independent replicas of 8โ16 GPUs |
| Memory holds | Weights, gradients, optimiser state, activations | Weights plus a KV cache per active conversation |
| Precision | BF16 with FP32 accumulation | BF16 or FP8, sometimes lower |
| Failure | One GPU fault stalls everyone; restart from checkpoint | Route traffic to another replica |
2. Prefill and decode
Every request has two phases. Prefill processes the whole prompt at once: thousands of tokens go through the model in parallel, so the GPUs are busy doing arithmetic (compute-bound). Decode then generates the answer one token at a time; each step does little arithmetic but must read all the weights and the cache from memory, so it is limited by memory bandwidth. Users feel prefill as the time to first token and decode as the speed of streaming text.
Anatomy of one request
Illustrative timings. Change the prompt and answer length.
3. Why BF16 needs two servers
At 2 bytes per parameter, 405B parameters need about 810 GB just for the weights. A standard server with eight 80 GB H100s has 640 GB of HBM. The model simply does not fit, before counting the KV cache. Meta's paper therefore serves the BF16 model on 16 GPUs across two machines: tensor parallelism inside each machine, where NVLink is fast, and pipeline parallelism between them, because the network between machines is much slower. It is the same principle as the training layout in Part 4: put the most communication-hungry split on the fastest links.
Does it fit?
Weights + KV cache vs GPU memory. Our arithmetic; real engines also need room for activations and runtime overhead (roughly 5โ10%).
4. Micro-batching across the pipeline
Pipelines waste time in training because of "bubbles": stages wait for each other during the forward and backward passes. Meta points out that inference has no backward pass, so the bubble problem largely disappears, and it uses micro-batching to keep both machines busy: while the second machine works on micro-batch 1, the first starts micro-batch 2. Meta tested two micro-batches on requests with 4,096 input tokens and 256 output tokens, for both prefill and decode, and reports higher throughput at the same local batch size, with some added latency.
Two-node pipeline, with and without micro-batching
Illustrative schedule. Node 1 holds the first half of the layers, node 2 the second.
5. The FP8 recipe
The other way to fit the model on one server is to make the numbers smaller. FP8 stores each value in one byte instead of two, halving the weight memory to about 405 GB, and H100 tensor cores do FP8 arithmetic at about twice the BF16 rate. The risk is accuracy: an 8-bit float has only a few bits of precision, so values must be rescaled into its narrow range. Meta's recipe, from the paper, is deliberately careful:
- Row-wise scaling. Each row of a weight or activation matrix gets its own scale factor, rather than one for the whole tensor, so one extreme value only affects its own row.
- Only the feed-forward layers. Most parameters and activations in the feed-forward (FFN) layers are quantised; self-attention parameters are not. The paper notes the FFN layers account for roughly half of inference compute time.
- First and last layers left alone. The first and last transformer layers stay in BF16, following earlier work on quantisation.
- Capped scale factors. Dynamic scale factors are capped at 1200. High-perplexity tokens such as dates can produce very large activations and extreme scales, which the cap guards against.
Meta checked quality by comparing reward-model scores of responses generated in BF16 and FP8 and concluded the FP8 approach has a negligible impact on the model's responses. vLLM ran the official FP8 405B model on a single node of 8รH100 or 8รA100 GPUs on launch day; Neural Magic published an FP8 checkpoint that runs on one 8รH100 node.
Per-tensor vs row-wise scaling
Illustrative 4ร6 matrix. Add an outlier and compare the rounding error. FP8 is simulated as 16 levels for clarity.
Which parts of Llama 3 405B are FP8?
126 transformer layers. Hover or tap a block.
6. The KV cache
During decode, the model reuses the keys and values it computed for every earlier token, instead of recomputing them. These are stored in the KV cache, and it grows with every token of every active conversation. For Llama 3 405B the size per token is:
2 (K and V) ร 126 layers ร 8 KV heads ร 128 dims ร 2 bytes = 516,096 bytes โ 0.5 MB
A single full 128K-token conversation therefore holds about 68 GB of cache in BF16, more than one H100's entire memory. Without grouped-query attention (8 shared KV heads instead of one per each of the 128 query heads) it would be 16 times larger. Other estimates of this number circulate, some much higher, depending on the assumptions; the formula above uses the published shape.
KV-cache calculator
Engines manage this memory carefully. PagedAttention (vLLM) stores the cache in fixed-size blocks like operating-system memory pages, so it does not need one contiguous slab per request. Chunked prefill splits long prompts into pieces that can be mixed with decode steps; vLLM enabled it automatically to support Llama 3.1's full 128K context at launch. Prefix caching reuses the cache for shared prompt beginnings such as system prompts.
7. Inference engines
Few users run Llama 3 405B with Meta's own code. They use inference engines that combine the techniques above with fast GPU kernels: vLLM (which partnered with Meta and Neural Magic for launch-day support), NVIDIA's TensorRT-LLM, SGLang, and others. Cloud providers offered the model through their APIs from release. The single most important scheduling idea is continuous batching.
Static vs continuous batching
Illustrative: six requests with different answer lengths on a 4-slot batch. Press play.
- Speculative decoding
- A small draft model proposes several tokens; the big model checks them in one pass, accepting the ones it agrees with.
- Disaggregated serving
- Run prefill and decode on separate GPU pools, since one is compute-bound and the other memory-bound.
- KV quantisation
- Store the cache in FP8 to halve its memory.
- Tensor parallelism
- Split each layer over the 8 GPUs of a node so each reads only an eighth of the weights per token.
8. Cost per token
A serving replica costs the same per hour whether it is busy or idle. Cost per token therefore comes down to how many tokens it produces per hour, which depends on batch size. Larger batches share each read of the weights across more users, raising throughput, but each user waits longer per token. Every provider picks a point on this curve.
Throughput vs latency
Illustrative model of one 8-GPU replica at an assumed $20/hour. Not measured prices.
For how these economics play out across the industry, see The Token Price Collapse and The Great Flip.
9. Distillation: the 405B as a teacher
Most traffic never touches the 405B model. The 8B and 70B siblings are far cheaper to serve. Meta changed the Llama 3.1 licence so developers could use the outputs of Llama models, including 405B, to improve other models, something the Llama 2 licence prohibited. Meta presented this as enabling synthetic data generation and distillation into smaller models. In practice the largest model's most valuable job is often to generate training data and judgments, not to answer users directly.
๐๏ธ Llama 3 case file: serving
- BF16
- 16 GPUs over 2 nodes; tensor parallel within, pipeline parallel across; micro-batching
- FP8
- Row-wise scales; FFN only; first and last layers in BF16; scale cap 1200; negligible reward-score impact
- Single node
- Official FP8 model on 8รH100 or 8รA100 (vLLM launch day)
- KV cache
- ~0.5 MB/token BF16 with 8 KV heads; ~68 GB for 128K tokens (our arithmetic)
- Context
- 128K, served with chunked prefill
- Licence
- Outputs may be used to improve other models (new in 3.1)
๐ What to watch
- Lower precision for serving (FP4 on Blackwell) and how much quality it costs.
- Disaggregated prefill/decode becoming the default architecture.
- Mixture-of-experts models, which change the memory and batching maths.
- Next in this series: operations, keeping the whole factory running.
๐ข Reading the numbers
- Two-node BF16 setup, TP/PP split, micro-batching experiment, the FP8 recipe (row-wise, FFN only, first/last layers, cap 1200), FFN share of compute and the negligible-impact finding come from the Llama 3 paper as quoted in search results.
- Single-node FP8 and chunked prefill come from vLLM's launch post and Neural Magic's model card. The licence change comes from Meta's announcement as quoted by Simon Willison and SD Times.
- Model shape comes from secondary model listings โ ๏ธ consistent with the public config.
- Memory and KV-cache figures are our arithmetic. Request timings, pipeline schedule, quantisation demo, batching animation and cost curve are illustrations.
- Meta's measured throughput and latency (Figure 24) are not reproduced because we could not retrieve the values.
๐ References
- Llama 3
- Llama Team, Meta: "The Llama 3 Herd of Models" (2024), section 6
- vLLM: Announcing Llama 3.1 Support in vLLM
- Neural Magic: Meta-Llama-3.1-405B-Instruct-FP8 model card
- Meta: Introducing Llama 3.1
- Simon Willison: Introducing Llama 3.1 (licence change)
- Prime Intellect: Llama 405B in BF16 across nodes
- Inference methods
- Kwon et al.: "Efficient Memory Management for LLM Serving with PagedAttention" (2023)
- Yu et al.: "Orca: A Distributed Serving System for Transformer-Based Generative Models" (OSDI 2022)
- Agrawal et al.: "Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve" (2024)
- Zhong et al.: "DistServe" (2024)
- Ainslie et al.: "GQA" (2023)
- Micikevicius et al.: "FP8 Formats for Deep Learning" (2022)
- Leviathan et al.: "Fast Inference from Transformers via Speculative Decoding" (2023)
- Related on this site
- Part 7: Is It Ready? ยท Part 4: Splitting a Giant ยท The Great Flip ยท The Memory Wall