Serving the Model: vLLM, SGLang, TensorRT-LLM and the Engines That Turn GPUs into Tokens
Training happens once; inference happens billions of times a day, and it is where AI factories earn their money. An inference engine takes a trained model and a stream of requests and turns GPUs into tokens as cheaply and quickly as possible. Its tricks have names (PagedAttention, continuous batching, prefix caching, speculative decoding, disaggregated prefill) and each one can double the tokens a GPU produces. This part explains them, with calculators, and compares the engines that implement them.
1. Case file
2. Timeline
Inference engines, 2019 โ 2026
Tap a milestone. Colour: research ideas, open engines, NVIDIA, lab disclosures.
3. Two phases: prefill and decode
Every request has two very different phases. Prefill reads the whole prompt in one parallel pass; it is a big matrix multiply and compute-bound, and its length sets the time to first token (TTFT). Decode then generates one token at a time, each needing a full read of the model's weights for a little arithmetic; it is memory-bound (Part 4's roofline), and sets the time per output token (TPOT). Almost every engine trick is about keeping decode busy.
One request, start to finish
Press Play. Illustrative timings for a 70B model: prefill of a 2,000-token prompt, then 12 output tokens.
4. The KV cache
To avoid recomputing attention over the whole conversation at every step, engines keep each token's attention keys and values in GPU memory: the KV cache. Its size is layers ร KV heads ร head size ร 2 (keys and values) ร bytes per number ร tokens. For long contexts and many users it outgrows the model itself, which is why architectures shrink it: grouped-query attention (GQA) shares keys and values across query heads, and DeepSeek's multi-head latent attention (MLA) stores a compressed latent instead.
KV-cache calculator
Our arithmetic from published model configurations. KV cache in BF16; weights at the stated precision; ignores activations and framework overhead.
5. PagedAttention
Early servers reserved a contiguous block of memory per request, sized for the longest possible answer. Most of it sat empty, and gaps between requests could not be reused. The vLLM paper (Kwon et al., SOSP 2023) found existing systems wasted much of their KV memory this way and borrowed an idea from operating systems: split the cache into small fixed-size blocks mapped through a table, allocate them on demand, and share them between requests that have the same prefix. The paper reported 2โ4ร higher throughput than prior systems at the same latency.
Contiguous vs paged
GPU memory for KV cache, 48 slots, three requests with different lengths. Illustration.
6. Continuous batching
Because decode is memory-bound, a GPU can generate tokens for many requests in nearly the time it takes for one: the weights are read once per step and reused across the batch. The question is how to form batches when answers have different lengths. Static batching waits for the whole batch to finish. Continuous (iteration-level) batching, introduced by Orca (OSDI 2022), lets requests join and leave at every step.
Static vs continuous
Four slots, eight requests of random lengths. Each column is one decode step. Our own simulation.
7. Prefix caching and RadixAttention
Many requests share a beginning: the same system prompt, the same few-shot examples, the earlier turns of a chat, the same document being questioned. If the engine keeps their KV cache, it can skip that part of the prefill. SGLang's RadixAttention (Zheng et al., 2023) stores cached prefixes in a radix tree and reuses the longest match automatically; vLLM V1 made prefix caching close to zero-overhead and on by default.
Prefix tree
Add requests and watch shared prefixes being reused. Token counts are illustrative.
8. Speculative decoding
Decode leaves arithmetic idle, so engines spend it on guessing. A small draft model (or extra heads on the main model, as in Medusa and EAGLE) proposes several tokens; the big model checks them all in one pass and keeps the longest correct run. The output is identical to normal decoding; only speed changes. Leviathan et al. (2023) give the expected tokens per step for draft length ฮณ and acceptance rate ฮฑ as (1 โ ฮฑฮณ+1) / (1 โ ฮฑ).
Speculative decoding calculator
Our application of the paper's formula. Draft cost is relative to one target-model step.
9. Quantisation for serving
Bytes per parameter, and what it buys
Weights for a 70B model at different precisions. Fewer bytes means fewer GPUs and faster memory-bound decode, at some accuracy risk.
Serving tolerates lower precision than training. FP8 weights and activations are now routine on Hopper and Blackwell; 4-bit weight methods (AWQ, GPTQ) are popular for open models on smaller GPUs; Blackwell adds hardware FP4. The KV cache can also be quantised to FP8, halving its size. Each step needs evaluation (Part 6): quantisation errors often appear on long reasoning chains before they show in short benchmarks.
10. Disaggregation and serving at scale
11. The engines
12. Feature matrix
Our reading of project documentation, October 2026. โ built in ยท โ partial ยท โ not a focus. Tap a cell.
13. Economics: throughput vs latency
Every engine faces the same trade-off. Bigger batches make each GPU-hour produce more tokens, which lowers cost, but every user waits longer between tokens. Providers sell different points on this curve: cheap batch APIs, standard chat, and premium "fast" tiers. The token-price article follows how this sets what you pay.
The serving curve
Our own simplified model: decode step time = 10 ms + 0.15 ms ร batch size, one replica of a 70B model on 8 GPUs costing $2 per GPU-hour. Not a benchmark.
14. Choosing an engine
Which engine fits?
An editorial guide.
15. Reading the numbers
- Primary
- vLLM/PagedAttention: 2โ4ร throughput (SOSP 2023). vLLM V1 alpha announced 27 Jan 2025 with "1.7ร speedup" (vLLM; benchmark conditions in the post). Dynamo: "up to 30ร" requests for DeepSeek-R1 on GB200 NVL72, more than 2ร Llama 70B on Hopper (NVIDIA). DeepSeek: 73.7K/14.8K tokens/s per H800 node, 608B/168B tokens per day, $87,072 daily cost at $2/GPU-hour, $562,027 theoretical revenue, 545% theoretical margin (DeepSeek disclosure via press). TGI maintenance mode (Hugging Face docs). Speculative formula: Leviathan et al., 2023.
- Secondary โ ๏ธ
- llm-d launch partners; TGI maintenance date (Dec 2025 per third parties).
- Our arithmetic
- KV per token: Llama 3 8B 0.13 MB, Llama 3 70B 0.33 MB (GQA, 8 KV heads), DeepSeek-V3 MLA ~0.07 MB (512 + 64 latent dims ร 61 layers), all BF16. Paging, batching, prefix-tree and serving-curve visuals are illustrations.
- Papers
- Orca: continuous batching (OSDI 2022)
- vLLM / PagedAttention (SOSP 2023)
- SGLang / RadixAttention
- DistServe ยท Splitwise
- Fast inference via speculative decoding (Leviathan et al.)
- EAGLE ยท Medusa
- Projects and disclosures
- vLLM V1 alpha
- NVIDIA: introducing Dynamo
- Hugging Face TGI documentation
- DeepSeek open-infra index (inference system overview)
- SCMP: DeepSeek's theoretical 545% margin
- PyTorch Foundation welcomes vLLM
- Part 6: evaluation ยท Part 4: kernels