Home โ€บ Blog โ€บ The Maestro ยท Part 15
The Maestro ยท Part 15 ยท Phase 4 ยท ๐Ÿ”„ Operator hat

Serving at Scale

Phase 4 is shipping. Training happens once; serving happens every second a client uses the product, and its cost decides whether the business works. This part puts Maestro-1 on the 64 serving GPUs reserved in Part 4: how memory, not compute, limits how many people can use it at once, why 8-bit weights change everything for this model, how batching and caching multiply throughput, the serving stack, routing and autoscaling, and what one million output tokens actually costs.

FL
FrontierAGI Team
๐Ÿ”„ Operator๐Ÿง  ML engineer๐Ÿ’ผ Founder๐Ÿ’ป Coder
โš ๏ธ How to read this (as of 30 Oct 2026) PagedAttention/vLLM, Orca, speculative decoding and H100 specifications are cited from their papers and NVIDIA. serving_capacity.py and batching_sim.py were run (real output below); the first uses our assumptions for achieved bandwidth and context length, the second a simulated workload. serve.sh was not executed for this article. Prices and usage are the scenario. Disclosure: Claude, made by Anthropic, helps write this site.

1. Mission log: months 4โ€“15

Months 4โ€“12 ยท Maestro-0 in service
  • The post-trained open model serves design partners on the 64-GPU serving cluster (Kubernetes, Part 7).
Months 13โ€“14 ยท Preparing Maestro-1
  • Writes serving_capacity.py; finds BF16 leaves almost no room for users; moves to FP8 weights and KV cache.
Month 14โ€“15 ยท Switch-over
  • Maestro-1 rolled out behind Maestro-0 with a canary (Part 17); Maestro-0 kept warm for rollback.

2. Why serving decides the business

Two numbers that set the economics

$108kper month for 64 reserved serving GPUs (Part 3)
$1.17per million output tokens at the chosen setup (our model)
960conversations served at once (our model)
~35,600output tokens per second across the cluster (our model)

Real output of serving_capacity.py (our assumptions).

3. Memory, not compute, is the limit

serving_capacity.py ยท Python 3, no dependencies
"""serving_capacity.py โ€” how many users can 64 H100s serve, and at what cost?

Memory per replica = weights + KV cache for active conversations.
Decode speed is limited by reading weights and KV cache from HBM each
step. Architecture from Part 9; bandwidth and efficiency are our
assumptions (H100 SXM HBM3 ~3.35 TB/s is NVIDIA's figure).
"""
TOTAL, ACTIVE = 65.2e9, 11.7e9
KV_PER_TOKEN = 2 * 40 * 8 * 128 * 2          # bytes, BF16 KV, GQA (Part 9) = 160 KB
HBM, BW, EFF = 80e9, 3.35e12, 0.5             # per GPU; EFF = achieved share of bandwidth (assumption)
GPUS, PRICE = 64, 2.35
CTX = 16_000                                   # average live context per conversation (assumption)

def replica(bytes_per_w, gpus):
    weights = TOTAL * bytes_per_w
    free = gpus * HBM * 0.9 - weights
    seqs = int(free // (KV_PER_TOKEN * CTX))
    # per decode step: read active expert weights (all tokens in batch share them) + every sequence's KV
    step_bytes = ACTIVE * bytes_per_w + seqs * KV_PER_TOKEN * CTX
    step_s = step_bytes / (gpus * BW * EFF)
    return weights, seqs, seqs / step_s

def main():
    print(f"KV cache per token: {KV_PER_TOKEN/1024:.0f} KB; per {CTX:,}-token conversation: {KV_PER_TOKEN*CTX/1e9:.2f} GB")
    for name, b, g in [("BF16 weights, 2 GPUs/replica", 2, 2), ("FP8 weights, 1 GPU/replica", 1, 1), ("FP8 weights, 2 GPUs/replica", 1, 2)]:
        w, seqs, tps = replica(b, g)
        if seqs <= 0:
            print(f"{name:30s} weights {w/1e9:.0f} GB -> does not fit"); continue
        reps = GPUS // g
        total_tps = tps * reps
        cost = GPUS * PRICE / (total_tps * 3600) * 1e6
        print(f"{name:30s} {reps:2d} replicas x {seqs:3d} live chats = {reps*seqs:4d}; "
              f"{total_tps:8,.0f} tokens/s; ${cost:.2f} per million output tokens")

if __name__ == "__main__":
    main()
$ python3 serving_capacity.py KV cache per token: 160 KB; per 16,000-token conversation: 2.62 GB BF16 weights, 2 GPUs/replica 32 replicas x 5 live chats = 160; 14,682 tokens/s; $2.85 per million output tokens FP8 weights, 1 GPU/replica 64 replicas x 2 live chats = 128; 12,654 tokens/s; $3.30 per million output tokens FP8 weights, 2 GPUs/replica 32 replicas x 30 live chats = 960; 35,598 tokens/s; $1.17 per million output tokens

Conversations served at once on 64 GPUs

BF16 weights, 2 GPUs per replica160
FP8 weights, 1 GPU per replica128
FP8 weights, 2 GPUs per replica (chosen)960

Real output of serving_capacity.py (our assumptions: 16K live tokens, 50% of peak bandwidth).

4. Eight-bit weights, carefully

5. Batching: serving many people at once

batching_sim.py ยท Python 3, no dependencies; simulated workload
"""batching_sim.py โ€” static vs continuous batching on one replica.

Requests arrive at random with random output lengths. Static batching
waits for the whole batch to finish before admitting new requests;
continuous batching (Orca, 2022; used by vLLM and others) admits a new
request as soon as any slot frees. One 'step' = one token for every
active request. SIMULATED workload, fixed seed.
"""
import random

def run(mode, slots=32, n=600, rate=0.9, seed=5):
    rng = random.Random(seed)
    arrivals, t = [], 0.0
    for _ in range(n):
        t += rng.expovariate(rate); arrivals.append((t, rng.choice([40, 80, 200, 600, 1500])))
    queue, active, done, step, i = [], [], [], 0, 0
    while len(done) < n:
        while i < n and arrivals[i][0] <= step:
            queue.append(arrivals[i]); i += 1
        if mode == "continuous" or not active:
            while queue and len(active) < slots:
                a, L = queue.pop(0); active.append([a, L, L])
        for r in active: r[2] -= 1
        for r in [r for r in active if r[2] <= 0]:
            active.remove(r); done.append(step + 1 - r[0])
        step += 1
    done.sort()
    return step, done[len(done) // 2], done[int(len(done) * 0.95)]

def main():
    for mode in ("static", "continuous"):
        steps, p50, p95 = run(mode)
        print(f"{mode:10s} finished 600 requests in {steps:6d} steps; latency p50 {p50:6.0f}, p95 {p95:6.0f} steps")

if __name__ == "__main__":
    main()
$ python3 batching_sim.py # simulated requests, 32 slots static finished 600 requests in 28702 steps; latency p50 13432, p95 26571 steps continuous finished 600 requests in 9802 steps; latency p50 4082, p95 7788 steps

Static vs continuous batching (simulated)

Static: steps to finish 600 requests28,702
Continuous: steps to finish9,802
Static: median latency13,432
Continuous: median latency4,082

Real output of batching_sim.py; units are decode steps.

6. PagedAttention and prefix caching

7. Try a serving setup

Serving capacity calculator (our model)

8. Speculative decoding and other speed-ups

9. The serving stack

From a client request to a token

๐ŸŒAPI gatewayAuth, rate limits, usage metering
๐ŸงญRouterSends a conversation to the replica with its cache
โš™๏ธInference engineContinuous batching, paged KV, FP8
๐Ÿ–ฅ๏ธ2 ร— H100One replica of Maestro-1
๐Ÿ“ŠTelemetryLatency, errors, tokens, cost per request

The same Prometheus and alerting as training (Part 7), different dashboards.

serve.sh ยท vLLM-style launch sketch ยท not executed for this article
#!/bin/bash
# serve.sh โ€” one Maestro-1 replica on 2 GPUs with an open inference engine
# (vLLM-style flags from its documentation; verify against your version).
vllm serve /models/maestro-1-fp8 \
  --tensor-parallel-size 2 \
  --quantization fp8 --kv-cache-dtype fp8 \
  --max-model-len 65536 \
  --enable-prefix-caching \
  --gpu-memory-utilization 0.90 \
  --max-num-seqs 64 \
  --port 8000
# Kubernetes runs 32 of these behind a router that sends a conversation back to
# the replica holding its cached prefix (Part 16's cost model depends on it).

10. Autoscaling and the reserved block

11. Latency: what users actually feel

What sets latency

๐Ÿงพ PrefillProcess the prompt; cut by prefix caching
๐Ÿ” DecodeOne token at a time; bound by memory bandwidth
โณ QueueingWaiting for a batch slot; cut by continuous batching
๐ŸŒ Network and toolsTool calls and sandbox time in agent steps

Each part has its own fix.

12. Where the replicas run

13. Serving the same model the harness tested

14. The serving budget

ItemMonthly (scenario)Driver
64 reserved GPUs$108k (Part 3)Fixed by contract
Burst GPUs$5โ€“15kPeaks above the reserved capacity
Storage and networking$6kWeights, logs, KV cache offload
Sandboxes (CPU)$9kAgent tasks running tests

15. Load testing before clients do

16. Graceful degradation

17. Seven serving mistakes

18. Delegation box

๐Ÿ™‹ Gopala himself
  • Precision and capacity decisions
  • Latency targets per product
  • Switch-over and rollback calls
๐Ÿค– AI agents
  • Load tests and capacity reports
  • Summaries of latency incidents
  • Tuning engine settings in a test environment
๐Ÿค Bought, rented or borrowed
  • Open inference engines
  • Reserved serving GPUs and a burst pool
  • Kubernetes and the GPU Operator

19. Budget and risk tracker

ItemStatus after Part 15Note
Spend to date~$8.6M (scenario)Serving since month 4 plus Phase 2โ€“3
ServingMaestro-1 FP8, 32 replicas (scenario)~960 concurrent conversations (our model)
TimeMonth ~15Partners and pilots on Maestro-1
Cost riskLowerFP8 + prefix caching
Quality risk from FP8WatchHarness re-run each version
Peak-load riskWatchBurst pool, background work fills gaps

20. The series

The Maestro: 18 parts in five phases

Verdict
  • Serving a 65B-parameter MoE model is a memory problem: BF16 weights left room for five conversations per replica, FP8 for thirty.
    • Continuous batching and prefix caching multiply what the same GPUs can do.
    • Next, Part 16: turning a fast model into products clients pay for.

21. Reading the numbers

Primary / standard
H100 SXM HBM3 ~3.35 TB/s and FP8 support (NVIDIA). PagedAttention / vLLM, 2โ€“4ร— throughput vs prior systems (Kwon et al., 2023). Continuous batching (Orca, Yu et al., 2022). Speculative decoding (Leviathan et al., 2023).
Secondary โš ๏ธ
None in this part.
Our assumptions
Achieved bandwidth, live context per conversation, replica layouts, prices per token, simulated workload and all deployment outcomes.