Serving at Scale
Phase 4 is shipping. Training happens once; serving happens every second a client uses the product, and its cost decides whether the business works. This part puts Maestro-1 on the 64 serving GPUs reserved in Part 4: how memory, not compute, limits how many people can use it at once, why 8-bit weights change everything for this model, how batching and caching multiply throughput, the serving stack, routing and autoscaling, and what one million output tokens actually costs.
serving_capacity.py and batching_sim.py were run (real output below); the first uses our assumptions for achieved bandwidth and context length, the second a simulated workload. serve.sh was not executed for this article. Prices and usage are the scenario. Disclosure: Claude, made by Anthropic, helps write this site.
1. Mission log: months 4โ15
- The post-trained open model serves design partners on the 64-GPU serving cluster (Kubernetes, Part 7).
- Writes
serving_capacity.py; finds BF16 leaves almost no room for users; moves to FP8 weights and KV cache.
- Maestro-1 rolled out behind Maestro-0 with a canary (Part 17); Maestro-0 kept warm for rollback.
2. Why serving decides the business
- Every client action is a bill.
- A coding agent sends long contexts and generates many tokens per task (Part 16).
- The serving cluster runs all day, every day, unlike the training run.
- The plan in Part 3 assumed $108k a month for 64 serving GPUs.
- At $2.35 per GPU-hour that is fixed; what varies is how much useful work those GPUs do.
- Serving efficiency therefore turns directly into price, margin and how many clients the company can take.
Two numbers that set the economics
Real output of serving_capacity.py (our assumptions).
3. Memory, not compute, is the limit
- Generating a token means reading the model and every conversation's memory.
- Each decode step reads the active weights once for the whole batch, and the key-value (KV) cache of every live conversation.
- An H100 can move about 3.35 TB/s from its HBM memory (NVIDIA); decoding is usually bound by that bandwidth, not by arithmetic.
- The KV cache grows with every token of context.
- Maestro-1's grouped-query attention keeps it at about 160 KB per token (Part 9).
- A coding conversation with 16,000 live tokens needs about 2.6 GB.
- And all 65.2B weights must fit too.
- In BF16 that is about 130 GB, almost all of two H100s.
"""serving_capacity.py โ how many users can 64 H100s serve, and at what cost?
Memory per replica = weights + KV cache for active conversations.
Decode speed is limited by reading weights and KV cache from HBM each
step. Architecture from Part 9; bandwidth and efficiency are our
assumptions (H100 SXM HBM3 ~3.35 TB/s is NVIDIA's figure).
"""
TOTAL, ACTIVE = 65.2e9, 11.7e9
KV_PER_TOKEN = 2 * 40 * 8 * 128 * 2 # bytes, BF16 KV, GQA (Part 9) = 160 KB
HBM, BW, EFF = 80e9, 3.35e12, 0.5 # per GPU; EFF = achieved share of bandwidth (assumption)
GPUS, PRICE = 64, 2.35
CTX = 16_000 # average live context per conversation (assumption)
def replica(bytes_per_w, gpus):
weights = TOTAL * bytes_per_w
free = gpus * HBM * 0.9 - weights
seqs = int(free // (KV_PER_TOKEN * CTX))
# per decode step: read active expert weights (all tokens in batch share them) + every sequence's KV
step_bytes = ACTIVE * bytes_per_w + seqs * KV_PER_TOKEN * CTX
step_s = step_bytes / (gpus * BW * EFF)
return weights, seqs, seqs / step_s
def main():
print(f"KV cache per token: {KV_PER_TOKEN/1024:.0f} KB; per {CTX:,}-token conversation: {KV_PER_TOKEN*CTX/1e9:.2f} GB")
for name, b, g in [("BF16 weights, 2 GPUs/replica", 2, 2), ("FP8 weights, 1 GPU/replica", 1, 1), ("FP8 weights, 2 GPUs/replica", 1, 2)]:
w, seqs, tps = replica(b, g)
if seqs <= 0:
print(f"{name:30s} weights {w/1e9:.0f} GB -> does not fit"); continue
reps = GPUS // g
total_tps = tps * reps
cost = GPUS * PRICE / (total_tps * 3600) * 1e6
print(f"{name:30s} {reps:2d} replicas x {seqs:3d} live chats = {reps*seqs:4d}; "
f"{total_tps:8,.0f} tokens/s; ${cost:.2f} per million output tokens")
if __name__ == "__main__":
main()Conversations served at once on 64 GPUs
Real output of serving_capacity.py (our assumptions: 16K live tokens, 50% of peak bandwidth).
- The finding.
- In BF16, the weights fill two GPUs so completely that each replica holds only five conversations: the cluster would serve 160 users at once at $2.85 per million tokens.
- With 8-bit (FP8) weights and KV cache on two GPUs, each replica holds about 30 conversations: 960 at once, at about $1.17 per million tokens, roughly 2.4 times cheaper.
- An MoE model's large total size is its serving weakness; quantisation is how start-ups live with it.
4. Eight-bit weights, carefully
- H100s run FP8 arithmetic natively.
- Weights and KV cache in FP8 halve memory and bandwidth compared with BF16.
- Quality must be measured, not assumed.
- The full evaluation harness (Part 12) runs on the FP8 model; any drop beyond the confidence interval blocks the switch.
- Sensitive parts (the router, some attention layers) can stay in higher precision if needed.
- More aggressive options exist.
- 4-bit weights would fit a replica on one GPU, at greater quality risk; he keeps it as a future experiment, not a launch plan.
5. Batching: serving many people at once
- GPUs are efficient only with many requests in flight.
- Static batching waits for a whole batch to finish before taking new requests, so short answers wait for long ones.
- Continuous batching, introduced by Orca (Yu et al., 2022), admits a new request as soon as any slot frees.
"""batching_sim.py โ static vs continuous batching on one replica.
Requests arrive at random with random output lengths. Static batching
waits for the whole batch to finish before admitting new requests;
continuous batching (Orca, 2022; used by vLLM and others) admits a new
request as soon as any slot frees. One 'step' = one token for every
active request. SIMULATED workload, fixed seed.
"""
import random
def run(mode, slots=32, n=600, rate=0.9, seed=5):
rng = random.Random(seed)
arrivals, t = [], 0.0
for _ in range(n):
t += rng.expovariate(rate); arrivals.append((t, rng.choice([40, 80, 200, 600, 1500])))
queue, active, done, step, i = [], [], [], 0, 0
while len(done) < n:
while i < n and arrivals[i][0] <= step:
queue.append(arrivals[i]); i += 1
if mode == "continuous" or not active:
while queue and len(active) < slots:
a, L = queue.pop(0); active.append([a, L, L])
for r in active: r[2] -= 1
for r in [r for r in active if r[2] <= 0]:
active.remove(r); done.append(step + 1 - r[0])
step += 1
done.sort()
return step, done[len(done) // 2], done[int(len(done) * 0.95)]
def main():
for mode in ("static", "continuous"):
steps, p50, p95 = run(mode)
print(f"{mode:10s} finished 600 requests in {steps:6d} steps; latency p50 {p50:6.0f}, p95 {p95:6.0f} steps")
if __name__ == "__main__":
main()Static vs continuous batching (simulated)
Real output of batching_sim.py; units are decode steps.
- Reading it.
- The same hardware finished the workload about 2.9 times faster with continuous batching, and the median request waited about a third as long.
- This is why every modern inference engine uses it.
6. PagedAttention and prefix caching
- KV cache memory was often wasted.
- Reserving the maximum length for each conversation leaves most of it empty.
- vLLM's PagedAttention (Kwon et al., 2023) stores the cache in small pages like an operating system's virtual memory, nearly eliminating waste; the paper reported 2โ4ร higher throughput than earlier systems at similar latency.
- Prefix caching reuses shared beginnings.
- Agent conversations repeat the same system prompt and the same repository files every step.
- Keeping their KV cache and reusing it skips recomputation; Part 16 shows it cuts the cost of a long task by about five times.
- Routing must cooperate.
- A conversation goes back to the replica that holds its cached prefix, or the cache is useless.
7. Try a serving setup
Serving capacity calculator (our model)
8. Speculative decoding and other speed-ups
- Speculative decoding.
- A small draft model proposes several tokens; the large model checks them in one pass and keeps the ones it agrees with (Leviathan et al., 2023).
- Output is unchanged; speed improves when the draft is often right, which is common for code.
- Others he keeps on the list.
- Expert-parallel serving across replicas for very large batches.
- Separating prompt processing (prefill) from generation (decode) onto different GPUs, which some large services do to keep latency steady.
- Order of work.
- FP8 and prefix caching first (biggest wins, lowest risk), speculative decoding second, the rest only when measurements show the need.
9. The serving stack
From a client request to a token
The same Prometheus and alerting as training (Part 7), different dashboards.
- Open-source engines do the heavy lifting.
- vLLM and SGLang are widely used open inference engines; NVIDIA's TensorRT-LLM is another option. The site's ML Layer article on inference engines compares them.
#!/bin/bash # serve.sh โ one Maestro-1 replica on 2 GPUs with an open inference engine # (vLLM-style flags from its documentation; verify against your version). vllm serve /models/maestro-1-fp8 \ --tensor-parallel-size 2 \ --quantization fp8 --kv-cache-dtype fp8 \ --max-model-len 65536 \ --enable-prefix-caching \ --gpu-memory-utilization 0.90 \ --max-num-seqs 64 \ --port 8000 # Kubernetes runs 32 of these behind a router that sends a conversation back to # the replica holding its cached prefix (Part 16's cost model depends on it).
10. Autoscaling and the reserved block
- Demand follows working hours; the reserved GPUs do not.
- 64 GPUs are paid for all month (Part 4). At night, idle replicas run evaluation sweeps and loop attempts (Part 11) instead of sitting empty.
- Peaks overflow to on-demand capacity.
- When queues grow beyond a latency target, extra replicas start on the burst pool (Part 4); they take minutes to load weights, so scaling reacts to trends, not single spikes.
- A latency target per product.
- Interactive chat: first token in about a second. Background agent tasks: throughput matters more than latency, so they fill the gaps.
11. Latency: what users actually feel
- Two numbers matter for interactive use.
- Time to first token: how long before anything appears, dominated by processing the prompt (prefill).
- Time per output token: how fast the answer streams, dominated by decode bandwidth.
- Long contexts make prefill expensive.
- An agent step may send tens of thousands of tokens; without prefix caching, every step re-processes them.
- With caching, only the new part of the context is processed.
- Targets by product.
- IDE assistant: first token around one second, steady streaming.
- PR bot and CI fixer: background tasks where minutes are fine, so they are batched for throughput and scheduled into quiet hours.
What sets latency
Each part has its own fix.
12. Where the replicas run
- Clients care where their code is processed.
- Some contracts require processing in a named region (Part 14).
- The serving cluster sits with a different provider from the training block (Part 4), in a region chosen for the first partners.
- A second region later.
- Adding a region means a second copy of the weights, a second sandbox fleet and routing by client; it is planned when a client in another region signs, not before.
13. Serving the same model the harness tested
- Small serving changes can change answers.
- Quantisation, a new engine version, different sampling settings or a changed system prompt.
- The rule.
- The exact serving configuration (engine version, precision, sampling, prompts) is part of the model version, evaluated by the harness (Part 12) and shipped through the canary (Part 17).
- A nightly job replays a fixed set of tasks against production and compares results with the harness run.
14. The serving budget
| Item | Monthly (scenario) | Driver |
|---|---|---|
| 64 reserved GPUs | $108k (Part 3) | Fixed by contract |
| Burst GPUs | $5โ15k | Peaks above the reserved capacity |
| Storage and networking | $6k | Weights, logs, KV cache offload |
| Sandboxes (CPU) | $9k | Agent tasks running tests |
- The reserved cost is fixed; the cost per task is not.
- Every improvement in this part lowers cost per solved task without changing the bill, which is why Part 16 can price competitively.
15. Load testing before clients do
- Every new model version and engine version is load-tested on a copy of the serving setup.
- Replay a day of recorded traffic (prompts with client data removed) at one, two and four times its real rate.
- Record throughput, first-token latency, 95th-percentile latency and errors at each level.
- The test answers three questions.
- At what load does latency break the target?
- How many replicas does the expected peak need, with headroom?
- Does anything fail differently from the previous version (memory errors, timeouts)?
- The result sets the autoscaling thresholds.
- Scaling starts well before the breaking point, because new replicas take minutes to load 65 GB of FP8 weights.
16. Graceful degradation
- When capacity runs out, degrade instead of failing.
- Background tasks (PR bot, CI fixer) are paused first; interactive requests keep priority.
- Very long contexts are summarised or truncated with a notice rather than rejected.
- If Maestro-1 replicas are unhealthy, traffic can fall back to the warm Maestro-0 deployment for simple requests.
- Clients are told.
- A status page and per-request headers show when a degraded mode is active.
17. Seven serving mistakes
- Sizing by compute when memory is the limit.
- Shipping BF16 weights for a large MoE model and wondering why it is expensive.
- Quantising without re-running the evaluation harness.
- Routing without regard to cached prefixes.
- Paying for idle reserved GPUs at night.
- Optimising average latency and ignoring the 95th percentile.
- Building a custom inference engine instead of using an open one.
18. Delegation box
- Precision and capacity decisions
- Latency targets per product
- Switch-over and rollback calls
- Load tests and capacity reports
- Summaries of latency incidents
- Tuning engine settings in a test environment
- Open inference engines
- Reserved serving GPUs and a burst pool
- Kubernetes and the GPU Operator
19. Budget and risk tracker
| Item | Status after Part 15 | Note |
|---|---|---|
| Spend to date | ~$8.6M (scenario) | Serving since month 4 plus Phase 2โ3 |
| Serving | Maestro-1 FP8, 32 replicas (scenario) | ~960 concurrent conversations (our model) |
| Time | Month ~15 | Partners and pilots on Maestro-1 |
| Cost risk | Lower | FP8 + prefix caching |
| Quality risk from FP8 | Watch | Harness re-run each version |
| Peak-load risk | Watch | Burst pool, background work fills gaps |
20. The series
The Maestro: 18 parts in five phases
- Serving a 65B-parameter MoE model is a memory problem: BF16 weights left room for five conversations per replica, FP8 for thirty.
- Continuous batching and prefix caching multiply what the same GPUs can do.
- Next, Part 16: turning a fast model into products clients pay for.
21. Reading the numbers
- Primary / standard
- H100 SXM HBM3 ~3.35 TB/s and FP8 support (NVIDIA). PagedAttention / vLLM, 2โ4ร throughput vs prior systems (Kwon et al., 2023). Continuous batching (Orca, Yu et al., 2022). Speculative decoding (Leviathan et al., 2023).
- Secondary โ ๏ธ
- None in this part.
- Our assumptions
- Achieved bandwidth, live context per conversation, replica layouts, prices per token, simulated workload and all deployment outcomes.