Home โ€บ Blog โ€บ Inference engines
Inside the ML Layer ยท Part 7 ยท Inference

Serving the Model: vLLM, SGLang, TensorRT-LLM and the Engines That Turn GPUs into Tokens

Training happens once; inference happens billions of times a day, and it is where AI factories earn their money. An inference engine takes a trained model and a stream of requests and turns GPUs into tokens as cheaply and quickly as possible. Its tricks have names (PagedAttention, continuous batching, prefix caching, speculative decoding, disaggregated prefill) and each one can double the tokens a GPU produces. This part explains them, with calculators, and compares the engines that implement them.

FL
FrontierAGI Team
โš ๏ธ How this was researched (as of 27 Oct 2026) Sources are the Orca, PagedAttention (vLLM), SGLang, DistServe and speculative-decoding papers, the vLLM V1 announcement, NVIDIA's Dynamo announcement, Hugging Face's TGI documentation, DeepSeek's inference-system disclosure (Open Source Week, Febโ€“Mar 2025) and the PyTorch Foundation. Vendor speedups ("up to") are labelled as such. The calculators and timelines are our own simplified models. See Reading the numbers.

1. Case file

โšก
The inference layer, October 2026Layer 2 of 8 in the ML layer map
Leading open enginesvLLM ยท SGLang ยท TensorRT-LLM (+ Dynamo)
vLLM governancePyTorch Foundation hosted project since May 2025
PagedAttention (vLLM paper)2โ€“4ร— throughput vs prior systems
NVIDIA Dynamo (GTC, Mar 2025)"up to 30ร—" requests, DeepSeek-R1 on GB200 NVL72
DeepSeek V3/R1 serving (Feb 2025)~73.7K input ยท ~14.8K output tokens/s per H800 node
Hugging Face TGIMaintenance mode; docs recommend vLLM, SGLang

2. Timeline

Inference engines, 2019 โ†’ 2026

Tap a milestone. Colour: research ideas, open engines, NVIDIA, lab disclosures.

Tap a milestone.

3. Two phases: prefill and decode

Every request has two very different phases. Prefill reads the whole prompt in one parallel pass; it is a big matrix multiply and compute-bound, and its length sets the time to first token (TTFT). Decode then generates one token at a time, each needing a full read of the model's weights for a little arithmetic; it is memory-bound (Part 4's roofline), and sets the time per output token (TPOT). Almost every engine trick is about keeping decode busy.

One request, start to finish

Press Play. Illustrative timings for a 70B model: prefill of a 2,000-token prompt, then 12 output tokens.

Prefill processes all prompt tokens together; decode adds one token per step.

4. The KV cache

To avoid recomputing attention over the whole conversation at every step, engines keep each token's attention keys and values in GPU memory: the KV cache. Its size is layers ร— KV heads ร— head size ร— 2 (keys and values) ร— bytes per number ร— tokens. For long contexts and many users it outgrows the model itself, which is why architectures shrink it: grouped-query attention (GQA) shares keys and values across query heads, and DeepSeek's multi-head latent attention (MLA) stores a compressed latent instead.

KV-cache calculator

Our arithmetic from published model configurations. KV cache in BF16; weights at the stated precision; ignores activations and framework overhead.

5. PagedAttention

Early servers reserved a contiguous block of memory per request, sized for the longest possible answer. Most of it sat empty, and gaps between requests could not be reused. The vLLM paper (Kwon et al., SOSP 2023) found existing systems wasted much of their KV memory this way and borrowed an idea from operating systems: split the cache into small fixed-size blocks mapped through a table, allocate them on demand, and share them between requests that have the same prefix. The paper reported 2โ€“4ร— higher throughput than prior systems at the same latency.

Contiguous vs paged

GPU memory for KV cache, 48 slots, three requests with different lengths. Illustration.

Request ARequest BRequest CReserved, unusedFree

6. Continuous batching

Because decode is memory-bound, a GPU can generate tokens for many requests in nearly the time it takes for one: the weights are read once per step and reused across the batch. The question is how to form batches when answers have different lengths. Static batching waits for the whole batch to finish. Continuous (iteration-level) batching, introduced by Orca (OSDI 2022), lets requests join and leave at every step.

Static vs continuous

Four slots, eight requests of random lengths. Each column is one decode step. Our own simulation.

7. Prefix caching and RadixAttention

Many requests share a beginning: the same system prompt, the same few-shot examples, the earlier turns of a chat, the same document being questioned. If the engine keeps their KV cache, it can skip that part of the prefill. SGLang's RadixAttention (Zheng et al., 2023) stores cached prefixes in a radix tree and reuses the longest match automatically; vLLM V1 made prefix caching close to zero-overhead and on by default.

Prefix tree

Add requests and watch shared prefixes being reused. Token counts are illustrative.

8. Speculative decoding

Decode leaves arithmetic idle, so engines spend it on guessing. A small draft model (or extra heads on the main model, as in Medusa and EAGLE) proposes several tokens; the big model checks them all in one pass and keeps the longest correct run. The output is identical to normal decoding; only speed changes. Leviathan et al. (2023) give the expected tokens per step for draft length ฮณ and acceptance rate ฮฑ as (1 โˆ’ ฮฑฮณ+1) / (1 โˆ’ ฮฑ).

Speculative decoding calculator

Our application of the paper's formula. Draft cost is relative to one target-model step.

9. Quantisation for serving

Bytes per parameter, and what it buys

Weights for a 70B model at different precisions. Fewer bytes means fewer GPUs and faster memory-bound decode, at some accuracy risk.

Serving tolerates lower precision than training. FP8 weights and activations are now routine on Hopper and Blackwell; 4-bit weight methods (AWQ, GPTQ) are popular for open models on smaller GPUs; Blackwell adds hardware FP4. The KV cache can also be quantised to FP8, halving its size. Each step needs evaluation (Part 6): quantisation errors often appear on long reasoning chains before they show in short benchmarks.

10. Disaggregation and serving at scale

Disaggregated prefill and decodePrefill wants compute; decode wants memory bandwidth and big batches. Running them on separate GPU pools and shipping the KV cache between them (DistServe, Splitwise, 2024) lets each pool use its best parallelism. NVIDIA's Dynamo (GTC, March 2025) orchestrates this across engines, with KV-aware routing; NVIDIA reports "up to 30ร—" more requests served for DeepSeek-R1 on GB200 NVL72 and more than 2ร— for Llama 70B on Hopper. llm-d, a Kubernetes-native project launched by Red Hat with industry partners in 2025 โš ๏ธ, does similar on top of vLLM.
Expert parallelism at scale: DeepSeekDeepSeek disclosed its V3/R1 serving system in Febโ€“Mar 2025: prefill and decode on separate nodes, large expert parallelism across many GPUs, computation overlapped with communication. It reported ~73.7K input and ~14.8K output tokens per second per H800 node, 608B input and 168B output tokens in one day, and a theoretical cost-profit margin of 545% if all tokens were billed at R1 prices (with GPUs costed at an assumed $2/hour). It noted real revenue is much lower.

11. The engines

12. Feature matrix

Our reading of project documentation, October 2026. โ— built in ยท โ— partial ยท โ€” not a focus. Tap a cell.

Tap a cell.

13. Economics: throughput vs latency

Every engine faces the same trade-off. Bigger batches make each GPU-hour produce more tokens, which lowers cost, but every user waits longer between tokens. Providers sell different points on this curve: cheap batch APIs, standard chat, and premium "fast" tiers. The token-price article follows how this sets what you pay.

The serving curve

Our own simplified model: decode step time = 10 ms + 0.15 ms ร— batch size, one replica of a 70B model on 8 GPUs costing $2 per GPU-hour. Not a benchmark.

14. Choosing an engine

Which engine fits?

An editorial guide.

VerdictThe open inference stack has consolidated around vLLM and SGLang, with NVIDIA's TensorRT-LLM and Dynamo as the performance-first option on its own hardware and llama.cpp for local use. The big gains now come from system design: disaggregating prefill and decode, routing requests to where their cache already lives, and spreading mixture-of-experts models across many GPUs, as DeepSeek showed. Because inference runs continuously, a 2ร— engine improvement is worth more to an AI factory than almost any other software change.

15. Reading the numbers

Primary
vLLM/PagedAttention: 2โ€“4ร— throughput (SOSP 2023). vLLM V1 alpha announced 27 Jan 2025 with "1.7ร— speedup" (vLLM; benchmark conditions in the post). Dynamo: "up to 30ร—" requests for DeepSeek-R1 on GB200 NVL72, more than 2ร— Llama 70B on Hopper (NVIDIA). DeepSeek: 73.7K/14.8K tokens/s per H800 node, 608B/168B tokens per day, $87,072 daily cost at $2/GPU-hour, $562,027 theoretical revenue, 545% theoretical margin (DeepSeek disclosure via press). TGI maintenance mode (Hugging Face docs). Speculative formula: Leviathan et al., 2023.
Secondary โš ๏ธ
llm-d launch partners; TGI maintenance date (Dec 2025 per third parties).
Our arithmetic
KV per token: Llama 3 8B 0.13 MB, Llama 3 70B 0.33 MB (GQA, 8 KV heads), DeepSeek-V3 MLA ~0.07 MB (512 + 64 latent dims ร— 61 layers), all BF16. Paging, batching, prefix-tree and serving-curve visuals are illustrations.