Frontier Model Production Engineering: Training, Serving and Operating Models at Scale
A research checkpoint is a file. A production model is a service that trains across tens of thousands of GPUs without losing weeks to failures, and then answers millions of users fast and cheaply. This visual deep dive walks the whole path โ parallelism, memory, MoE, reliability, serving, prefill and decode, KV cache, speculative decoding, kernels, quantization, metrics and deployment โ with interactive diagrams that compute the real numbers.
01 The Production Challenge
Training produces a checkpoint: hundreds of billions of numbers plus research code written to run experiments, not to serve customers. Production engineering turns that into a service with guarantees โ fast first tokens, steady streaming, high GPU utilization, survival of hardware failures, and a cost per million tokens a business can live with. It happens on both sides of the checkpoint: engineering the training run so it finishes, and engineering the serving system so it scales.
02 The Distributed Training Stack
No single GPU can hold a frontier model, its gradients and its optimizer state, let alone process trillions of tokens in reasonable time. Training splits the work along several axes at once โ "3D", "4D" or "5D" parallelism. Pick a strategy to see what each GPU holds:
| Strategy | What is split | Communication | Origin / example |
|---|---|---|---|
| Data parallel | The batch; each GPU has a full model replica | All-reduce gradients every step | PyTorch DDP |
| Tensor parallel | Each weight matrix, column- or row-wise | All-reduce inside every layer โ needs fast NVLink, so kept within a server | Megatron-LM (Shoeybi et al., 2019) |
| Pipeline parallel | Layers into stages; micro-batches flow through | Point-to-point activations between stages | GPipe (Huang et al., 2019), 1F1B schedules |
| Expert parallel | MoE experts across GPUs | All-to-all token dispatch and combine | GShard, DeepSeek-V3 |
| Context / sequence parallel | The sequence (tokens) itself | Exchange keys/values (ring attention) or all-gather | Megatron sequence parallelism, Llama 3 context parallelism |
| Gradient accumulation | A large batch into micro-batches over time | None until the final update | Standard in every trainer |
Pipeline parallelism has a catch: while the first micro-batch travels through the stages, later stages sit idle, and they idle again as the pipeline drains. More micro-batches shrink that "bubble":
03 Memory Engineering During Training
With mixed-precision Adam, each parameter costs about 16 bytes before any activations: 2 for the BF16 weight, 2 for its gradient, and 12 of optimizer state (an FP32 master copy plus two FP32 Adam moments). That is why a 70B model needs over a terabyte just for model state. ZeRO (the idea behind DeepSpeed ZeRO and PyTorch FSDP) shards these pieces across data-parallel GPUs instead of replicating them:
04 MoE Production Engineering
Mixture-of-Experts models (see Part 1) are cheap per token but hard to run: experts live on different GPUs, so every MoE layer sends each token to its experts' GPUs and back โ an all-to-all exchange twice per layer. And tokens don't spread evenly: if the router over-picks one expert, its GPU becomes the straggler everyone waits for. Systems cap each expert at a capacity:
- Top-k / top-1
- Load-balancing loss or bias
- Capacity limits
- Distribute experts across GPUs and nodes
- Balance compute and memory
- Replicate hot experts
- Efficient token dispatch (e.g. DeepEP)
- Overlap with compute
- Limit nodes per token
- Expert caching
- Large expert-parallel serving
- Fewer cross-node hops
05 Training Reliability at Scale
At frontier scale, hardware failure is not an exception โ it is the weather. Meta's Llama 3 paper reports the numbers from a 54-day snapshot of pre-training on 16,384 H100 GPUs:
Because failures are certain, the question is how often to checkpoint. Checkpoint too often and you spend all your time writing; too rarely and each failure throws away hours. Young's classic approximation gives the sweet spot:
06 Inference Serving Architecture
A request's journey from user to streamed tokens passes through several layers. Press play:
The scheduler is where the biggest gains are. Static batching runs a batch until its longest request finishes, leaving finished slots empty. Continuous batching (iteration-level scheduling, from Orca, OSDI 2022) admits a new request the moment another finishes. And paged KV cache (PagedAttention, from vLLM) stops reserving maximum-length memory per request. This simulator measures all four combinations on the same 400 requests:
"""Static vs continuous batching, contiguous vs paged KV cache: a tiny LLM-serving simulator.
Time advances in decode steps. Each step, every running request produces one token. A GPU has room for a
fixed number of KV-cache slots (tokens). Static batching waits for a whole batch to finish before admitting
new requests; continuous batching (Orca / vLLM style) admits a waiting request the moment one finishes.
Contiguous allocation reserves max_len slots per request up front; paged allocation takes 16-token blocks
as the sequence grows. Standard library only.
"""
import random
KV_SLOTS, MAX_LEN, BLOCK = 16_384, 2_048, 16
def workload(n, seed=7):
rng = random.Random(seed)
# prompt 50-1000 tokens, output 20-1000 tokens: long-tailed like real chat traffic
return [(rng.randint(50, 1000), min(int(rng.expovariate(1 / 250)) + 20, 1000)) for _ in range(n)]
def simulate(reqs, continuous, paged):
waiting, running, t, done, lat = list(enumerate(reqs)), [], 0, 0, []
used_hist = []
def need(p, gen): # KV slots a request holds right now
return -(-(p + gen) // BLOCK) * BLOCK if paged else MAX_LEN
while waiting or running:
if continuous or not running: # admission
while waiting:
i, (p, o) = waiting[0]
if sum(need(r[1], r[3]) for r in running) + need(p, 1) > KV_SLOTS: break
waiting.pop(0); running.append([i, p, o, 0])
t += 1
for r in running: r[3] += 1
used_hist.append(sum(r[1] + r[3] for r in running) / KV_SLOTS)
for r in [r for r in running if r[3] >= r[2]]:
running.remove(r); done += 1; lat.append(t)
toks = sum(o for _, o in reqs)
return t, toks / t, sum(lat) / len(lat), sum(used_hist) / len(used_hist)
reqs = workload(400)
print(f"400 requests, {sum(o for _, o in reqs):,} output tokens, GPU KV budget {KV_SLOTS:,} token slots\n")
print(" scheduler steps tokens/step mean completion step KV slots holding real tokens")
base = None
for name, c, pg in [("static batch + contiguous KV", False, False), ("continuous + contiguous KV", True, False),
("static batch + paged KV", False, True), ("continuous + paged KV", True, True)]:
t, tps, ml, util = simulate(reqs, c, pg)
base = base or tps
print(f" {name:<29} {t:>6,} {tps:>11.1f} {ml:>20,.0f} {util:>27.0%} ({tps / base:.1f}x)")Output (deterministic โ fixed random seed):
400 requests, 103,109 output tokens, GPU KV budget 16,384 token slots scheduler steps tokens/step mean completion step KV slots holding real tokens static batch + contiguous KV 34,107 3.0 17,227 13% (1.0x) continuous + contiguous KV 13,564 7.6 6,606 34% (2.5x) static batch + paged KV 11,591 8.9 5,551 39% (2.9x) continuous + paged KV 5,345 19.3 2,314 85% (6.4x)
The simulator counts decode steps, not seconds: in a real GPU a larger batch makes each step slightly slower, so real gains are smaller than 6.4ร โ the vLLM paper reports 2โ4ร throughput over prior systems at the same latency. The direction and the reason are the same: 13% of KV slots held real tokens with contiguous reservation and static batches, 85% with paging and continuous batching.
07 Prefill vs Decode: Two Different Workloads
Generating a response has two phases. Prefill reads the whole prompt in one parallel pass and builds the KV cache; it is compute-bound, and it determines time to first token. Decode generates one token at a time, and each step must read every weight from memory to do very little arithmetic per request; it is memory-bandwidth-bound and determines the time between tokens. Batching more requests per decode step is the main way to use a GPU's compute:
Mixing them on one GPU causes interference: a long prefill stalls every decoding request sharing its batch. Chunked prefill (Sarathi-Serve) splits long prompts into chunks that piggyback on decode steps; disaggregation separates them entirely.
08 Disaggregated Prefill and Decode
Disaggregated serving runs prefill and decode on separate GPU pools. A prefill worker processes the prompt, then transfers the KV cache over the network (RDMA or NVLink) to a decode worker, which streams the tokens. Each pool can use its own batch size, parallelism and even hardware, so TTFT and inter-token latency can be tuned independently.
09 KV-Cache Engineering
The KV cache โ keys and values of every earlier token โ is often the largest consumer of serving memory, and managing it decides how many requests fit at once. Before paging, systems reserved a contiguous region per request at maximum length; the vLLM paper measured that 60โ80% of KV memory in existing systems was wasted by fragmentation and over-reservation.
10 Speculative Decoding
Decode is memory-bound: checking five tokens costs the big model almost the same as generating one, because the weights are read once either way. Speculative decoding exploits this. A small, fast draft proposes k tokens; the large target model verifies all of them in a single forward pass, keeps the longest accepted prefix, and adds one token of its own. With the right acceptance rule the output distribution is exactly the target model's (Leviathan et al., 2023; Chen et al., 2023).
Production variants avoid a separate draft model: Medusa adds extra decoding heads to the target; EAGLE drafts from the target's own hidden features (EAGLE-3 reports 3.0โ6.5ร speedups); Apple's ReDrafter uses a recurrent draft head; n-gram / prompt lookup copies spans from the context; and DeepSeek-V3's multi-token-prediction module can serve as a draft. vLLM, SGLang and TensorRT-LLM all support several of these. Gains shrink at large batch sizes, where decode becomes less memory-bound.
11 Kernel and Execution Optimization
A GPU computes far faster than it can move data to and from its main memory (HBM). Many operations in a transformer โ layer norms, activations, softmax โ do little arithmetic per byte, so their cost is the memory round trip. Kernel fusion runs several of them in one kernel, keeping intermediate results in fast on-chip memory:
12 Quantization and Precision Engineering
Fewer bits per weight means less memory, less bandwidth per decode step, and faster math on hardware that supports the format. The trade-off is accuracy, managed with calibration, per-channel or per-block scaling, and quantization-aware training.
| Precision | Typical use | Benefits | Trade-offs |
|---|---|---|---|
| FP32 | Master weights, optimizer state | Highest precision | 4 bytes per value; slow |
| BF16 | Standard training and inference | FP32's range, stable training | Twice FP8's memory |
| FP16 | Training (with loss scaling) and inference | Good precision | Narrow range โ overflow risk |
| FP8 | Large-scale training (DeepSeek-V3) and inference | Half of BF16's memory, ~2ร tensor throughput on H100 | Needs fine-grained scaling; hardware support (Hopper and later) |
| INT8 | Inference (weights and/or activations) | Lower memory, higher throughput | Possible quality loss; outliers need handling (SmoothQuant) |
| INT4 / FP4 | Aggressive inference (GPTQ, AWQ; NVFP4 on Blackwell) | Very low memory, high decode throughput | Larger quality risk; more careful calibration |
13 Production Metrics
Users feel two latencies: how long until the first token appears (TTFT), and how fast the rest streams (inter-token latency). Operators watch throughput, utilization and cost. Their relationship:
14 Reliability and Deployment
New model versions roll out like any risky software change โ gradually, with metrics gates and a fast way back:
15 The End-to-End Production Path
Next: The Frontier Model Market Landscape โ who builds, trains, serves and sells these models, and what they cost.