Always On: The Inference Engineers Who Serve Models to Billions
Training happens a few times a year. Inference happens every second, for every user, forever, and it is where most AI compute is heading. Turning a trained model into a service that answers millions of people within strict latency targets, at a price that makes business sense, is the job of the inference platform team. How Llama 3 405B is served (FP8, KV cache, continuous batching) is in Producing Intelligence Part 8. This part is about running inference as a product: routing, disaggregated serving, caching, rollouts, capacity and the engineers on call for all of it.
1. Where we are in the stack
The AI factory stack, by role
2. The roles: inference performance and platform engineers
Inference teams usually split in two. Sarvam AI's posting for a Performance Engineer, Inference describes a team divided between GPU kernels and the serving stack, with on-call ownership of an inference service-level objective (SLO) and experience with vLLM, SGLang and TensorRT-LLM as strong pluses. Postings for Model Serving Engineers ask for 6+ years in distributed systems or ML platforms and responsibilities such as continuous batching, request multiplexing and multi-tenant routing that balances latency, throughput and cost.
Two roles, two skill profiles
Our reading of postings (1โ5). The day timelines are composite illustrations.
3. Inference as a product
An inference platform is judged by a handful of numbers: time to first token (how fast an answer starts), time per output token (how fast it streams), throughput (tokens per second per GPU), cost per million tokens and availability. Traffic comes in different shapes: interactive chat, long-running agents, offline batch jobs, and internal workloads such as reinforcement-learning rollouts that generate training data. Tightening the latency target means smaller batches, which means more GPUs.
SLO calculator
Illustrative model of one model's fleet. Batch size is chosen to meet the per-token latency target.
4. What lives at this layer
The inference platform stack
5. Disaggregated serving
The biggest architectural change in inference since continuous batching is disaggregation: running the two phases of a request on separate pools of GPUs. Prefill (reading the prompt) is compute-bound; decode (generating tokens) is memory-bandwidth-bound (see Part 6). On the same GPU they interfere: a long prompt's prefill stalls everyone else's streaming. Split them, and each pool can be sized, batched and even built from different hardware for its own job. The cost is moving each request's KV cache from the prefill pool to the decode pool quickly.
NVIDIA's open-source Dynamo framework is built around this. It works with TensorRT-LLM, vLLM and SGLang engines, routes each request to the worker most likely to already hold its KV cache (tracked in a global radix tree) while balancing load, and moves KV caches between prefill and decode workers with its NIXL transfer library. llm-d, a Kubernetes-native project, runs prefill, decode and routing as separately scaled services โ ๏ธ secondary.
Combined vs disaggregated
Illustrative model: how prompt and answer lengths change the case for splitting the phases.
6. Case study: Meta
Meta serves Llama-based models to Meta AI across its apps, its smart glasses and internal workloads such as RLHF. In an October 2025 engineering post on scaling LLM inference, Meta described moving toward disaggregated prefill and decode tiers, which could let compute-heavy hardware handle prefill and memory-bandwidth-heavy hardware handle decode, and N-dimensional parallelism across nodes (context, pipeline, expert and tensor parallel, with separate data parallel replicas). For mixture-of-experts models it flagged all-to-all communication as 10โ30% of latency. A PyTorch blog post with the vLLM community reported that prefill/decode disaggregation is enabled in Meta's internal inference stack serving large-scale Meta traffic, with better time-to-first-token and time-per-token than its previous stack, and that optimisations are being upstreamed to vLLM.
7. Routing and caching
Many requests share their beginning: the same system prompt, the same document, the growing history of an agent. If a request lands on a server that already has that prefix's KV cache, its prefill is mostly free. Cache-aware routing trades a little load balance for a lot of reuse; KV caches are increasingly tiered from GPU memory to host memory and SSD so more prefixes stay warm.
Round-robin vs cache-aware routing
Illustrative: 8 servers, requests from 6 popular prefixes (system prompts, documents, agent sessions).
8. Rollouts and capacity
Shipping a new model version is a deployment like any other: a canary on a small share of traffic, automatic comparison of quality and latency, then a gradual ramp, with instant rollback. Capacity is the harder problem. Interactive traffic follows the sun, with peaks and troughs, but GPUs cost the same at 4am as at noon. Platforms fill the troughs with work that can wait: batch jobs, evaluations, and reinforcement-learning rollouts that generate training data.
Filling the troughs
Illustrative 24-hour traffic for a fleet sized for its peak.
9. When it breaks
Inference detective
Pick a symptom.
10. Handoffs and careers
Who the inference team works with
- Typical backgrounds
- Backend and distributed-systems engineers from high-traffic services, GPU performance engineers, ML engineers who deployed their own models.
- Core skills
- vLLM/SGLang/TensorRT-LLM internals, batching and KV-cache management, load balancing and routing, autoscaling, latency analysis, capacity and cost modelling, on-call discipline.
- Growth path
- Inference engineer โ owner of a model's serving fleet or the routing layer โ head of inference platform.
- Further reading
- From Checkpoint to Endpoint ยท The Great Flip ยท How LLM Inference Works
๐๏ธ Case file: Meta
- Architecture
- Moving to disaggregated prefill/decode tiers; N-D parallelism across nodes.
- Results
- Disaggregation live for large-scale Meta traffic, better TTFT and per-token latency; upstreamed to vLLM.
- Bottleneck
- All-to-all communication 10โ30% of MoE inference latency.
- Capacity
- MTIA 450/500 for GenAI inference (from early 2027); reported ~$21B CoreWeave deal.
- Not published
- Request volumes and per-model fleet sizes.
๐ Next in the series
Part 11: Observability, reliability and on-call โ the SREs and production engineers who watch the whole factory and answer the pager.
๐ข Reading the numbers
- Meta's architecture and 10โ30% all-to-all figure come from its October 2025 engineering post; disaggregation in production from the PyTorch blog.
- MTIA roadmap and CoreWeave deal come from 2026 reporting; the deal size is as reported.
- Dynamo facts come from NVIDIA's documentation; llm-d details are โ ๏ธ secondary.
- Role descriptions come from Sarvam AI and model-serving postings.
- SLO calculator, disaggregation simulator, routing demo and traffic curve are illustrations.
๐ References
- Meta
- Meta Engineering: Scaling LLM inference โ innovations (October 2025)
- PyTorch blog: Disaggregated inference at scale (Meta and vLLM)
- VKTR: Meta expands in-house AI chip roadmap (MTIA)
- Trending Topics: Meta and CoreWeave $21B deal for inference
- Serving systems
- NVIDIA Dynamo architecture ยท Disaggregated serving
- Roles
- Sarvam AI: Performance Engineer, Inference (posting)
- Related on this site
- From Checkpoint to Endpoint ยท Part 9: Research tooling ยท The Token Price Collapse