Home โ€บ Blog โ€บ From Checkpoint to Endpoint
Producing Intelligence ยท Part 8 ยท Serving

From Checkpoint to Endpoint: How Llama 3 405B Is Served

Training produced a set of weights. Users get value only when those weights answer requests, quickly and cheaply, millions of times a day. Serving a 405-billion-parameter model raises its own engineering problems: in BF16 it does not fit in one server, every conversation drags a growing memory cache behind it, and each generated token has to read the whole model from memory. This part covers how Meta and the inference ecosystem solved those problems for Llama 3 405B: two-node pipelines, micro-batching, a careful FP8 recipe, and the engines that serve it.

FL
FrontierAGI Team
โš ๏ธ How this was researched Facts come from section 6 (inference) of Meta's paper "The Llama 3 Herd of Models" (arXiv 2407.21783) as quoted in search results, vLLM's Llama 3.1 launch post, Neural Magic's FP8 model card and Meta's launch announcement. The model's shape (126 layers, 128 query heads, 8 key-value heads, head size 128) comes from secondary model listings consistent with the public config. Memory and KV-cache figures are our own arithmetic, and the latency, batching and cost charts are illustrations, not Meta's measurements. See Reading the numbers.
810 GBBF16 weights; one 8ร—H100 server has 640 GB of HBM
16 GPUstwo nodes for BF16 inference in Meta's paper
8 GPUsone node for the official FP8 model (vLLM, launch day)
~0.5 MBKV cache per token in BF16 (our arithmetic)

1. From one giant job to millions of small ones

Parts 4 and 5 described one enormous, synchronised training job. Serving inverts almost every assumption.

Training (Parts 4โ€“5)Serving (this part)
Unit of workOne job, months longMillions of requests, seconds each
GoalMaximise utilisation (MFU)Meet latency targets at lowest cost per token
Scale16,384 GPUs togetherMany independent replicas of 8โ€“16 GPUs
Memory holdsWeights, gradients, optimiser state, activationsWeights plus a KV cache per active conversation
PrecisionBF16 with FP32 accumulationBF16 or FP8, sometimes lower
FailureOne GPU fault stalls everyone; restart from checkpointRoute traffic to another replica

2. Prefill and decode

Every request has two phases. Prefill processes the whole prompt at once: thousands of tokens go through the model in parallel, so the GPUs are busy doing arithmetic (compute-bound). Decode then generates the answer one token at a time; each step does little arithmetic but must read all the weights and the cache from memory, so it is limited by memory bandwidth. Users feel prefill as the time to first token and decode as the speed of streaming text.

Anatomy of one request

Illustrative timings. Change the prompt and answer length.

3. Why BF16 needs two servers

At 2 bytes per parameter, 405B parameters need about 810 GB just for the weights. A standard server with eight 80 GB H100s has 640 GB of HBM. The model simply does not fit, before counting the KV cache. Meta's paper therefore serves the BF16 model on 16 GPUs across two machines: tensor parallelism inside each machine, where NVLink is fast, and pipeline parallelism between them, because the network between machines is much slower. It is the same principle as the training layout in Part 4: put the most communication-hungry split on the fastest links.

Does it fit?

Weights + KV cache vs GPU memory. Our arithmetic; real engines also need room for activations and runtime overhead (roughly 5โ€“10%).

Sum of context across all active requests, in steps of 32K.

4. Micro-batching across the pipeline

Pipelines waste time in training because of "bubbles": stages wait for each other during the forward and backward passes. Meta points out that inference has no backward pass, so the bubble problem largely disappears, and it uses micro-batching to keep both machines busy: while the second machine works on micro-batch 1, the first starts micro-batch 2. Meta tested two micro-batches on requests with 4,096 input tokens and 256 output tokens, for both prefill and decode, and reports higher throughput at the same local batch size, with some added latency.

Two-node pipeline, with and without micro-batching

Illustrative schedule. Node 1 holds the first half of the layers, node 2 the second.

5. The FP8 recipe

The other way to fit the model on one server is to make the numbers smaller. FP8 stores each value in one byte instead of two, halving the weight memory to about 405 GB, and H100 tensor cores do FP8 arithmetic at about twice the BF16 rate. The risk is accuracy: an 8-bit float has only a few bits of precision, so values must be rescaled into its narrow range. Meta's recipe, from the paper, is deliberately careful:

Meta checked quality by comparing reward-model scores of responses generated in BF16 and FP8 and concluded the FP8 approach has a negligible impact on the model's responses. vLLM ran the official FP8 405B model on a single node of 8ร—H100 or 8ร—A100 GPUs on launch day; Neural Magic published an FP8 checkpoint that runs on one 8ร—H100 node.

Per-tensor vs row-wise scaling

Illustrative 4ร—6 matrix. Add an outlier and compare the rounding error. FP8 is simulated as 16 levels for clarity.

Which parts of Llama 3 405B are FP8?

126 transformer layers. Hover or tap a block.

Teal: FFN quantised to FP8. Grey: kept in BF16 (first and last layer entirely, and every self-attention block).

6. The KV cache

During decode, the model reuses the keys and values it computed for every earlier token, instead of recomputing them. These are stored in the KV cache, and it grows with every token of every active conversation. For Llama 3 405B the size per token is:

2 (K and V) ร— 126 layers ร— 8 KV heads ร— 128 dims ร— 2 bytes = 516,096 bytes โ‰ˆ 0.5 MB

A single full 128K-token conversation therefore holds about 68 GB of cache in BF16, more than one H100's entire memory. Without grouped-query attention (8 shared KV heads instead of one per each of the 128 query heads) it would be 16 times larger. Other estimates of this number circulate, some much higher, depending on the assumptions; the formula above uses the published shape.

KV-cache calculator

Engines manage this memory carefully. PagedAttention (vLLM) stores the cache in fixed-size blocks like operating-system memory pages, so it does not need one contiguous slab per request. Chunked prefill splits long prompts into pieces that can be mixed with decode steps; vLLM enabled it automatically to support Llama 3.1's full 128K context at launch. Prefix caching reuses the cache for shared prompt beginnings such as system prompts.

7. Inference engines

Few users run Llama 3 405B with Meta's own code. They use inference engines that combine the techniques above with fast GPU kernels: vLLM (which partnered with Meta and Neural Magic for launch-day support), NVIDIA's TensorRT-LLM, SGLang, and others. Cloud providers offered the model through their APIs from release. The single most important scheduling idea is continuous batching.

Static vs continuous batching

Illustrative: six requests with different answer lengths on a 4-slot batch. Press play.

๐Ÿ’ก More tricks in modern engines
Speculative decoding
A small draft model proposes several tokens; the big model checks them in one pass, accepting the ones it agrees with.
Disaggregated serving
Run prefill and decode on separate GPU pools, since one is compute-bound and the other memory-bound.
KV quantisation
Store the cache in FP8 to halve its memory.
Tensor parallelism
Split each layer over the 8 GPUs of a node so each reads only an eighth of the weights per token.

8. Cost per token

A serving replica costs the same per hour whether it is busy or idle. Cost per token therefore comes down to how many tokens it produces per hour, which depends on batch size. Larger batches share each read of the weights across more users, raising throughput, but each user waits longer per token. Every provider picks a point on this curve.

Throughput vs latency

Illustrative model of one 8-GPU replica at an assumed $20/hour. Not measured prices.

For how these economics play out across the industry, see The Token Price Collapse and The Great Flip.

9. Distillation: the 405B as a teacher

Most traffic never touches the 405B model. The 8B and 70B siblings are far cheaper to serve. Meta changed the Llama 3.1 licence so developers could use the outputs of Llama models, including 405B, to improve other models, something the Llama 2 licence prohibited. Meta presented this as enabling synthetic data generation and distillation into smaller models. In practice the largest model's most valuable job is often to generate training data and judgments, not to answer users directly.

๐Ÿ—‚๏ธ Llama 3 case file: serving

BF16
16 GPUs over 2 nodes; tensor parallel within, pipeline parallel across; micro-batching
FP8
Row-wise scales; FFN only; first and last layers in BF16; scale cap 1200; negligible reward-score impact
Single node
Official FP8 model on 8ร—H100 or 8ร—A100 (vLLM launch day)
KV cache
~0.5 MB/token BF16 with 8 KV heads; ~68 GB for 128K tokens (our arithmetic)
Context
128K, served with chunked prefill
Licence
Outputs may be used to improve other models (new in 3.1)
Verdict Serving Llama 3 405B is a memory problem first and a compute problem second. Meta's answers were conventional but carefully engineered: split the model the way the hardware dictates, keep the pipeline busy with micro-batches, and quantise only where it is safe. The most important result was that a conservative FP8 recipe let a frontier-class open model run on a single standard server, which is what made it practical for clouds and companies to host.

๐Ÿ‘€ What to watch

๐Ÿ”ข Reading the numbers

๐Ÿ“š References