Home โ€บ Blog โ€บ The Inference Platform
Who Runs the AI Factory ยท Part 10 ยท Inference platform

Always On: The Inference Engineers Who Serve Models to Billions

Training happens a few times a year. Inference happens every second, for every user, forever, and it is where most AI compute is heading. Turning a trained model into a service that answers millions of people within strict latency targets, at a price that makes business sense, is the job of the inference platform team. How Llama 3 405B is served (FP8, KV cache, continuous batching) is in Producing Intelligence Part 8. This part is about running inference as a product: routing, disaggregated serving, caching, rollouts, capacity and the engineers on call for all of it.

FL
FrontierAGI Team
โš ๏ธ How this was researched Sources are Meta's October 2025 engineering post on scaling LLM inference, the PyTorch blog on disaggregated inference at Meta, NVIDIA Dynamo documentation, reporting on Meta's MTIA roadmap and CoreWeave deal, and real job postings (no OpenAI, Anthropic or xAI inference postings were found). llm-d details are โš ๏ธ secondary. Meta does not publish request volumes, so we give none. The SLO calculator, disaggregation simulator, routing demo and traffic curve are illustrations. See Reading the numbers.

1. Where we are in the stack

The AI factory stack, by role

2. The roles: inference performance and platform engineers

Inference teams usually split in two. Sarvam AI's posting for a Performance Engineer, Inference describes a team divided between GPU kernels and the serving stack, with on-call ownership of an inference service-level objective (SLO) and experience with vLLM, SGLang and TensorRT-LLM as strong pluses. Postings for Model Serving Engineers ask for 6+ years in distributed systems or ML platforms and responsibilities such as continuous batching, request multiplexing and multi-tenant routing that balances latency, throughput and cost.

Two roles, two skill profiles

Our reading of postings (1โ€“5). The day timelines are composite illustrations.

3. Inference as a product

An inference platform is judged by a handful of numbers: time to first token (how fast an answer starts), time per output token (how fast it streams), throughput (tokens per second per GPU), cost per million tokens and availability. Traffic comes in different shapes: interactive chat, long-running agents, offline batch jobs, and internal workloads such as reinforcement-learning rollouts that generate training data. Tightening the latency target means smaller batches, which means more GPUs.

SLO calculator

Illustrative model of one model's fleet. Batch size is chosen to meet the per-token latency target.

4. What lives at this layer

The inference platform stack

5. Disaggregated serving

The biggest architectural change in inference since continuous batching is disaggregation: running the two phases of a request on separate pools of GPUs. Prefill (reading the prompt) is compute-bound; decode (generating tokens) is memory-bandwidth-bound (see Part 6). On the same GPU they interfere: a long prompt's prefill stalls everyone else's streaming. Split them, and each pool can be sized, batched and even built from different hardware for its own job. The cost is moving each request's KV cache from the prefill pool to the decode pool quickly.

NVIDIA's open-source Dynamo framework is built around this. It works with TensorRT-LLM, vLLM and SGLang engines, routes each request to the worker most likely to already hold its KV cache (tracked in a global radix tree) while balancing load, and moves KV caches between prefill and decode workers with its NIXL transfer library. llm-d, a Kubernetes-native project, runs prefill, decode and routing as separately scaled services โš ๏ธ secondary.

Combined vs disaggregated

Illustrative model: how prompt and answer lengths change the case for splitting the phases.

6. Case study: Meta

Meta serves Llama-based models to Meta AI across its apps, its smart glasses and internal workloads such as RLHF. In an October 2025 engineering post on scaling LLM inference, Meta described moving toward disaggregated prefill and decode tiers, which could let compute-heavy hardware handle prefill and memory-bandwidth-heavy hardware handle decode, and N-dimensional parallelism across nodes (context, pipeline, expert and tensor parallel, with separate data parallel replicas). For mixture-of-experts models it flagged all-to-all communication as 10โ€“30% of latency. A PyTorch blog post with the vLLM community reported that prefill/decode disaggregation is enabled in Meta's internal inference stack serving large-scale Meta traffic, with better time-to-first-token and time-per-token than its previous stack, and that optimisations are being upstreamed to vLLM.

2024 Llama 3.1 405B served in FP8 on single 8-GPU nodes (Producing Intelligence Part 8).
Oct 2025 Engineering post: disaggregated tiers, N-D parallelism, all-to-all costs.
2025โ€“26 Prefill/decode disaggregation live in Meta's internal stack; contributions to vLLM.
2026 Reported ~$21 billion CoreWeave deal through 2032 focused on running models (inference); MTIA 450 and 500 chips targeted at generative-AI inference, with mass deployment planned from early 2027.

7. Routing and caching

Many requests share their beginning: the same system prompt, the same document, the growing history of an agent. If a request lands on a server that already has that prefix's KV cache, its prefill is mostly free. Cache-aware routing trades a little load balance for a lot of reuse; KV caches are increasingly tiered from GPU memory to host memory and SSD so more prefixes stay warm.

Round-robin vs cache-aware routing

Illustrative: 8 servers, requests from 6 popular prefixes (system prompts, documents, agent sessions).

8. Rollouts and capacity

Shipping a new model version is a deployment like any other: a canary on a small share of traffic, automatic comparison of quality and latency, then a gradual ramp, with instant rollback. Capacity is the harder problem. Interactive traffic follows the sun, with peaks and troughs, but GPUs cost the same at 4am as at noon. Platforms fill the troughs with work that can wait: batch jobs, evaluations, and reinforcement-learning rollouts that generate training data.

Filling the troughs

Illustrative 24-hour traffic for a fleet sized for its peak.

9. When it breaks

Inference detective

Pick a symptom.

10. Handoffs and careers

Who the inference team works with

๐Ÿงญ Getting into the role
Typical backgrounds
Backend and distributed-systems engineers from high-traffic services, GPU performance engineers, ML engineers who deployed their own models.
Core skills
vLLM/SGLang/TensorRT-LLM internals, batching and KV-cache management, load balancing and routing, autoscaling, latency analysis, capacity and cost modelling, on-call discipline.
Growth path
Inference engineer โ†’ owner of a model's serving fleet or the routing layer โ†’ head of inference platform.
Further reading
From Checkpoint to Endpoint ยท The Great Flip ยท How LLM Inference Works

๐Ÿ—‚๏ธ Case file: Meta

Architecture
Moving to disaggregated prefill/decode tiers; N-D parallelism across nodes.
Results
Disaggregation live for large-scale Meta traffic, better TTFT and per-token latency; upstreamed to vLLM.
Bottleneck
All-to-all communication 10โ€“30% of MoE inference latency.
Capacity
MTIA 450/500 for GenAI inference (from early 2027); reported ~$21B CoreWeave deal.
Not published
Request volumes and per-model fleet sizes.
Verdict Inference has become the main business of the AI factory, and its platform is converging on a clear design: disaggregated prefill and decode, cache-aware routing, tiered KV caches and fleets that fill idle hours with batch and RL work. The engineers who run it combine high-traffic service operations with GPU-level performance knowledge, and every millisecond and percentage point they save shows up directly in the price of a token.

๐Ÿ‘€ Next in the series

Part 11: Observability, reliability and on-call โ€” the SREs and production engineers who watch the whole factory and answer the pager.

๐Ÿ”ข Reading the numbers

๐Ÿ“š References