Home › Blog › Production Agent Architecture
Deep Dive · With runnable platform experiments 🏛️

Production Agent Architecture: End-to-End System Design

Every earlier piece in this series covered one part of an agent: the model, its context, retrieval, tools, planning, memory, multiple agents, MCP. This one puts them together into a platform that serves real users reliably — the control plane that decides, the execution plane that acts, and the infrastructure that makes it durable, secure, observable and affordable. It closes with five experiments on the platform mechanics that decide whether an agent survives production: durable execution, model routing, tracing, scaling and circuit breakers.

How to read this: this is the capstone of the How LLMs Are Built series (piece fifteen) and links to the earlier articles for each component rather than repeating them. "Our run" numbers come from the script in Run the Experiments, which simulates models, tools and failures so that the platform logic — queues, leases, checkpoints, routing, tracing, rate limits, breakers — can be run and measured. The costs are in relative units, not real prices.

From Idea to Production

A production agent platform combines models, planning, memory, tools and infrastructure to reliably solve real-world tasks. It handles complex multi-step tasks, integrates with many external systems, remembers and learns over time, supports multiple specialized agents, runs reliably, securely and at scale, provides observability and control, includes human oversight where needed, and serves real users and real business workflows. The gap between a demo and that list is almost entirely infrastructure.

Key Design Principles

PrincipleMeaning
ModularEach component can be improved or replaced independently
ReliableFault tolerant, with retries and recovery
SecureControlled access to data and actions
ScalableSupports many concurrent users and agents
ObservableFull visibility into behaviour and costs
Cost-efficientModel and tool routing, caching
Human-in-the-loopApproval for sensitive actions
FlexibleSupports different models, tools and use cases

High-Level Architecture

Reading the platform top to bottom:

1Users and applications — web chat, mobile apps, API and SDK integrations, internal enterprise tools.
2API gateway — authentication (SSO, API keys), rate limiting, tenant isolation, request routing, logging and monitoring.
3Agent runtime (the control plane) — planner/orchestrator, model router, context builder, policy and safety, task manager (queueing, scheduling, retries, recovery).
4Shared services — memory service (working, episodic, semantic, artifact), RAG service (ingestion, embeddings, retrieval and reranking), state and checkpoint store, cache layer (prompt, retrieval, tool and application caches).
5Execution layer — MCP and tool gateway, subagents (research, coding, analysis, review), sandboxed execution (code, shell, browser automation in isolated containers with resource limits), human approval (requests, feedback, audit trail).
6External systems, reached via MCP and APIs — databases, file systems, SaaS applications, internal APIs, web search, enterprise systems (CRM, ERP).

Each layer has its own article in this series: the gateway and serving, the context builder, RAG, tools, MCP, memory and checkpoints, subagents.

The Agent Execution Loop

For a request such as "Investigate this customer escalation and draft a response": receive the request via API or UI → load context (memory, RAG, tool schemas, state) → plan or update the plan → choose the next action (a tool, a subagent, or a request for approval) → execute it → observe the result → update state and memory → replan if needed → loop until the success criteria are met → return the final result: the answer, actions taken, artifacts, updated memory. The loop itself is described in Agentic Systems; what production adds is that every step runs inside the infrastructure below.

Control Plane vs Execution Plane

Borrowed from networking and cloud infrastructure, the split separates deciding from doing, for reliability, security and scale:

Control plane — the brains, orchestration, governance
Planning and task decomposition · model routing · context and memory management · tool selection and permissions · task scheduling and queueing · state and checkpoint management · policy, safety and validation · human approval workflows · observability and monitoring · cost and usage control.
Execution plane — actions, tools, external systems
LLM inference (multiple models) · tool execution via MCP · sandboxed code execution · browser and web automation · database and file access · external APIs and SaaS · subagent workers · document processing · long-running background jobs · parallel task execution.

The control plane sends plans, policies and tool calls down; the execution plane sends results, observations and errors back up. The payoff: execution workers can be scaled, sandboxed and restarted freely because they hold no decisions, and policy is enforced in one place instead of inside every tool.

Core Components

ComponentResponsibilities
API gatewayAuth, rate limits, routing, multi-tenant support, monitoring and logging
Agent runtimeManages the full execution loop, coordinates components, handles errors and recovery
Planner / orchestratorDecomposes goals, tracks dependencies, decides next actions
Model routerSelects the best model per task; balances cost, latency and quality; supports multiple providers
Context builderAssembles instructions, memory, RAG results and tool schemas; manages context size
Memory serviceWorking, episodic and semantic memory; stores artifacts and summaries; retrieves relevant memories
RAG serviceIngestion, embeddings, retrieval and reranking, grounded context
State / checkpoint storeTask state and progress, resumable workflows, long-running task persistence
Tool / MCP gatewayConnects to MCP servers, tool discovery and routing, authentication
Sandboxed executionSecure code, shell and browser execution; isolated environments; resource and time limits
Human approval serviceApproval for sensitive actions; shows plan, impact and details; audit trail
Job queue / schedulerAsync and long-running tasks, retries and backoff, distributed worker management
ObservabilityTraces, logs, metrics; token usage and cost; tool calls and performance
CachingPrompt, retrieval and tool caches; configurable time-to-live and invalidation
Policy and safetyAccess control and permissions; input/output validation; data-loss and prompt-injection protection

Worked Example: A Customer Escalation

  1. The user submits a request: "Review the latest customer escalation and prepare a response."
  2. The agent retrieves context: customer history from the CRM via MCP, past tickets from the database, relevant documents via RAG.
  3. Specialized agents work in parallel: a research agent finds similar issues, an analysis agent examines logs, a coding agent checks code — all through MCP tools.
  4. The planner consolidates their findings, checks for contradictions and creates a plan.
  5. The agent drafts a response using the retrieved information and analysis.
  6. Human approval: the draft and key findings go to a support manager.
  7. Send: after approval, the response goes out via email or the CRM (MCP).
  8. Update memory and close: save the outcome, key learnings and artifacts; mark the task complete.

Our run traces a simulated version of this request through every component (section 3 of the output):

3) trace of one request (ms, relative model cost)
   request: escalation             313.2 ms ####################################### 
      gateway: auth + rate limit        4.1 ms  
      memory service                   10.1 ms # 
      RAG retrieval                    30.1 ms ### 
      context builder                  40.2 ms ##### 
      planner (large model)            80.1 ms ########## cost 12
      MCP: crm.get_ticket              40.2 ms ##### 
      subagents (parallel)             91.0 ms ########### 
      drafting (small model)           50.2 ms ###### cost 2
      policy + output checks            6.1 ms  
      human approval (async)            1.1 ms  

A trace like this answers the questions production raises: where the time went (the parallel subagents and the planner), where the money went (the large-model planning call cost six times the small-model draft), and which span to fix first.

Reliability and Failure Recovery

Production reliability practices: automatic retries with exponential backoff; fallback to alternate models or tools; resuming from checkpoints after failures; graceful degradation when a tool is unavailable; timeout handling and cancellation; and classifying errors so each kind gets the right strategy. Two mechanisms matter most for agents, because agents run long and act on the world:

Durable execution
Jobs live on a queue; stateless workers take a job under a lease; the job's state is checkpointed after every step; if a worker dies, the lease expires, the job returns to the queue, and the next worker resumes from the last checkpoint. Workflow engines built for this, such as Temporal, apply the same idea.
Idempotent side effects
A step can run twice — a worker may crash after sending an email but before recording that it did. Attaching an idempotency key to each side effect (as payment APIs such as Stripe do) lets the receiving system ignore the repeat, so the effect happens exactly once.

Our run put 40 jobs on a queue served by six workers that crash 4% of the time on every step:

1) 40 jobs, 6 stateless workers that crash 4% of the time per step
   duplicate sends prevented: 1, jobs completed: 40, resumed from checkpoint: 8, worker crashes: 11; emails actually sent: 40

11 crashes, and still every job completed — 8 of them by resuming from a checkpoint rather than starting over. Once a job was retried after its email had already gone out, and the idempotency key stopped the duplicate, so exactly 40 emails were sent. For degraded dependencies, a circuit breaker stops calling a failing service after repeated errors and sends traffic to a fallback until a cool-down passes:

5) 20 requests through a circuit breaker during a 9-request outage: primary 8, fallback (primary failed) 4, fallback (breaker open) 8
   calls sent to the failing API: 4

During a nine-request outage the breaker let only four requests hit the failing API — three to trip it and one probe after the cool-down — and served the rest from cache.

Cost Efficiency and Model Routing

Not every step needs the biggest model. A model router sends each step to the cheapest model likely to succeed, trading cost against quality; research systems such as RouteLLM learn routers from preference data. Our run simulated 5,000 agent steps with a small model (cost 1, weaker on hard steps) and a large one (cost 12):

all small model cost 1.00 88.4% success all large model cost 12.00 96.2% success route by difficulty cost 2.14 92.1% success route: medium+hard to large cost 5.38 95.3% success
Our run: relative cost per step (bars) and step success rate for four routing policies. Routing only medium and hard steps to the large model kept 99% of its quality at 45% of its cost.

The numbers depend entirely on the assumed accuracies and prices, and a real router must estimate difficulty — misrouting a hard step to a small model is the main risk. The shape is general, though: most agent steps are easy, so routing is usually the largest cost lever after caching.

Scalability and Performance

Practices: stateless agent runtimes (state lives in the stores, so any worker can serve any request); async job queues and distributed workers; parallel task execution; caching at multiple levels; model routing on cost, latency and quality; auto-scaling on demand. One constraint caps all of it — shared limits downstream. Our run measured throughput as workers were added, against a model API that allows 40 calls per second:

4) model calls per second vs workers (each call 50 ms; API rate limit 40/s)
    1 workers:  20.0 calls/s
    2 workers:  40.0 calls/s
    4 workers:  41.0 calls/s
    8 workers:  42.3 calls/s
   16 workers:  45.0 calls/s

Throughput doubles from one worker to two, then flattens at the API's limit (the small excess is calls already in flight when the measurement ends). Beyond that, extra workers only queue. Scaling an agent platform means finding the real bottleneck — usually model rate limits, tool APIs or a database — and raising, sharding or caching around it.

Observability and Monitoring

Distributed tracing end to end, as in the trace above; logs of all model calls, tool calls and decisions; metrics for latency, cost and success rate; dashboards and alerts; the ability to inspect a task's history and intermediate steps; and to debug failed runs and replay them when needed. OpenTelemetry's semantic conventions for generative AI give a vendor-neutral way to record model and tool spans, so traces from different components line up.

Security and Governance

Authentication (SSO, API keys, OAuth); fine-grained tool permissions; tenant and data isolation; audit logging of all actions; input and output validation and content filtering; secrets management and key rotation. For agents specifically, keep consequential actions behind the human approval service, run generated code only in the sandbox, and treat everything returned by tools as untrusted — the details are in Tool Calling and MCP.

Production Best Practices

Start with clear, well-defined use cases. Use modular, replaceable components. Implement strong monitoring from day one. Enforce permissions and human approval. Test with realistic data and failure scenarios — crash workers, fail tools, hit rate limits, as our experiments do. Continuously evaluate and improve over time — see Evaluating Agents.

Production agents in one paragraph
A production agent platform separates a control plane that plans, routes, builds context, enforces policy and schedules work from an execution plane that runs models, tools, sandboxes and subagents against external systems. What makes it production-grade is infrastructure: jobs on durable queues with checkpoints and idempotent side effects, so crashes cost nothing; circuit breakers and fallbacks for failing dependencies; model routing and caching for cost; stateless workers for scale, up to the real downstream limit; end-to-end tracing for visibility; and permissions, sandboxes and human approval for safety. The model supplies judgment — the platform supplies dependability.

Run the Experiments

One script, standard library only, runs in about twenty seconds (most of it the scaling measurement).

"""Production agent platform mechanics, measured.

1. Durable execution: 40 agent jobs on a queue, stateless workers that crash mid-job, leases that expire and
   re-queue work, checkpoints to resume from, and idempotency keys so side effects happen exactly once.
2. Model routing: send easy steps to a small model and hard ones to a large model; cost vs quality.
3. Distributed tracing: one request's span tree with latency and cost per component.
4. Scaling: throughput as workers are added, against a shared model-API rate limit.
5. Circuit breaker: stop calling a failing tool and fall back.
Models, tools and failures are simulated; the platform logic is real. Standard library only.
"""
import collections, queue, random, threading, time

# ---------------- 1. durable execution: queue + leases + checkpoints + idempotency ----------------
rng = random.Random(3)
STEPS = ["load context", "plan", "call CRM", "draft reply", "send email", "update memory"]
checkpoints, sent, lock = {}, {}, threading.Lock()                 # shared durable stores
jobs = queue.Queue(); stats = collections.Counter()
for j in range(40): jobs.put(f"job-{j:02d}")
def send_email(job, idem_key):                                     # external side effect with an idempotency key
    with lock:
        if idem_key in sent: stats["duplicate sends prevented"] += 1; return
        sent[idem_key] = job
def worker(wid):
    while True:
        try: job = jobs.get(timeout=0.2)
        except queue.Empty: return
        start = checkpoints.get(job, 0)                            # resume where the last worker stopped
        if start: stats["resumed from checkpoint"] += 1
        for i in range(start, len(STEPS)):
            if STEPS[i] == "send email": send_email(job, f"{job}:send")
            if rng.random() < 0.04:                                # the worker crashes mid-step
                stats["worker crashes"] += 1
                jobs.put(job)                                      # lease expires -> job goes back on the queue
                break
            time.sleep(0.002)
            with lock: checkpoints[job] = i + 1                    # checkpoint after each completed step
        else:
            stats["jobs completed"] += 1
threads = [threading.Thread(target=worker, args=(w,)) for w in range(6)]
for t in threads: t.start()
for t in threads: t.join()
print("1) 40 jobs, 6 stateless workers that crash 4% of the time per step")
print("   " + ", ".join(f"{k}: {v}" for k, v in sorted(stats.items())) + f"; emails actually sent: {len(sent)}")

# ---------------- 2. model routing ----------------
rng = random.Random(4)
MODELS = {"small": (1, {"easy": 0.97, "medium": 0.85, "hard": 0.55}),       # (relative cost, success by difficulty)
          "large": (12, {"easy": 0.98, "medium": 0.95, "hard": 0.88})}
steps = [rng.choice(["easy"] * 6 + ["medium"] * 3 + ["hard"]) for _ in range(5000)]
def run(policy):
    cost = ok = 0
    for d in steps:
        m = policy(d); c, acc = MODELS[m]; cost += c; ok += rng.random() < acc[d]
    return cost / len(steps), ok / len(steps)
print("\n2) routing 5,000 agent steps (60% easy, 30% medium, 10% hard); relative cost per step")
for name, pol in [("all small model", lambda d: "small"), ("all large model", lambda d: "large"),
                  ("route by difficulty", lambda d: "large" if d == "hard" else "small"),
                  ("route: medium+hard to large", lambda d: "small" if d == "easy" else "large")]:
    c, q = run(pol); print(f"   {name:<28} cost {c:>5.2f}   step success {q:.1%}")

# ---------------- 3. distributed tracing ----------------
spans = []
class Span:
    def __init__(self, name, parent=None, cost=0.0): self.name, self.parent, self.cost = name, parent, cost
    def __enter__(self): self.t0 = time.perf_counter(); return self
    def __exit__(self, *a): self.ms = (time.perf_counter() - self.t0) * 1000; spans.append(self)
def work(s): time.sleep(s)
with Span("request: escalation") as root:
    with Span("gateway: auth + rate limit", root): work(0.004)
    with Span("context builder", root):
        with Span("memory service", root): work(0.010)
        with Span("RAG retrieval", root): work(0.030)
    with Span("planner (large model)", root, cost=12): work(0.080)
    with Span("MCP: crm.get_ticket", root): work(0.040)
    with Span("subagents (parallel)", root) as sub:
        ts = [threading.Thread(target=work, args=(d,)) for d in (0.06, 0.09)]
        [t.start() for t in ts]; [t.join() for t in ts]
    with Span("drafting (small model)", root, cost=2): work(0.050)
    with Span("policy + output checks", root): work(0.006)
    with Span("human approval (async)", root): work(0.001)
print("\n3) trace of one request (ms, relative model cost)")
for s in [root] + [s for s in spans if s is not root]:
    ind = "" if s is root else "   "
    bar = "#" * int(s.ms / 8)
    print(f"   {ind}{s.name:<30} {s.ms:>6.1f} ms {bar} {'cost ' + str(s.cost) if s.cost else ''}")

# ---------------- 4. horizontal scaling against a rate limit ----------------
class RateLimiter:                                                 # token bucket: the model API allows 40 calls/s
    def __init__(self, rate, burst=2): self.rate, self.burst, self.tokens, self.last, self.lock = rate, burst, burst, time.perf_counter(), threading.Lock()
    def take(self):
        while True:
            with self.lock:
                now = time.perf_counter(); self.tokens = min(self.burst, self.tokens + (now - self.last) * self.rate); self.last = now
                if self.tokens >= 1: self.tokens -= 1; return
            time.sleep(0.002)
def throughput(n_workers, seconds=3.0):
    rl, done, stop = RateLimiter(40), [0], time.perf_counter() + seconds
    def w():
        while time.perf_counter() < stop:
            rl.take(); time.sleep(0.05); done[0] += 1              # each model call takes 50 ms
    ts = [threading.Thread(target=w) for _ in range(n_workers)]; [t.start() for t in ts]; [t.join() for t in ts]
    return done[0] / seconds
print("\n4) model calls per second vs workers (each call 50 ms; API rate limit 40/s)")
for n in (1, 2, 4, 8, 16):
    print(f"   {n:>2} workers: {throughput(n):>5.1f} calls/s")

# ---------------- 5. circuit breaker ----------------
class Breaker:
    def __init__(self, threshold=3, cooldown=5): self.fails, self.threshold, self.cooldown, self.open_until = 0, threshold, cooldown, -1
    def call(self, t, primary, fallback):
        if t < self.open_until: return fallback(), "fallback (breaker open)"
        try: r = primary(t); self.fails = 0; return r, "primary"
        except Exception:
            self.fails += 1
            if self.fails >= self.threshold: self.open_until = t + self.cooldown
            return fallback(), "fallback (primary failed)"
def search_api(t):
    if 3 <= t < 12: raise ConnectionError("503")                   # an outage from t=3 to t=11
    return "fresh results"
b, log = Breaker(), collections.Counter()
for t in range(20): _, how = b.call(t, search_api, lambda: "cached results"); log[how] += 1
print(f"\n5) 20 requests through a circuit breaker during a 9-request outage: " + ", ".join(f"{k} {v}" for k, v in log.items()))
print("   calls sent to the failing API: {}".format(log["fallback (primary failed)"]))

Output (thread scheduling makes crash counts and timings vary slightly between runs):

1) 40 jobs, 6 stateless workers that crash 4% of the time per step
   duplicate sends prevented: 1, jobs completed: 40, resumed from checkpoint: 8, worker crashes: 11; emails actually sent: 40

2) routing 5,000 agent steps (60% easy, 30% medium, 10% hard); relative cost per step
   all small model              cost  1.00   step success 88.4%
   all large model              cost 12.00   step success 96.2%
   route by difficulty          cost  2.14   step success 92.1%
   route: medium+hard to large  cost  5.38   step success 95.3%

3) trace of one request (ms, relative model cost)
   request: escalation             313.2 ms ####################################### 
      gateway: auth + rate limit        4.1 ms  
      memory service                   10.1 ms # 
      RAG retrieval                    30.1 ms ### 
      context builder                  40.2 ms ##### 
      planner (large model)            80.1 ms ########## cost 12
      MCP: crm.get_ticket              40.2 ms ##### 
      subagents (parallel)             91.0 ms ########### 
      drafting (small model)           50.2 ms ###### cost 2
      policy + output checks            6.1 ms  
      human approval (async)            1.1 ms  

4) model calls per second vs workers (each call 50 ms; API rate limit 40/s)
    1 workers:  20.0 calls/s
    2 workers:  40.0 calls/s
    4 workers:  41.0 calls/s
    8 workers:  42.3 calls/s
   16 workers:  45.0 calls/s

5) 20 requests through a circuit breaker during a 9-request outage: primary 8, fallback (primary failed) 4, fallback (breaker open) 8
   calls sent to the failing API: 4

⚠️ Notes

The source infographic is an accurate reference architecture. Three things it implies but doesn't show:
  • "Resume from checkpoints" is not enough on its own for agents that act: a step can run twice across a crash, so side effects need idempotency keys to happen exactly once.
  • "Auto-scaling on demand" stops at downstream limits. Model-API rate limits, tool APIs and databases cap throughput however many workers you add.
  • The model router has to estimate difficulty. Its savings come with the risk of sending hard steps to a weak model, so routing needs evaluation like any other component.
Sources: Anthropic's agent-building guidance; RouteLLM (learned model routing); OpenTelemetry semantic conventions for generative AI; the circuit-breaker pattern (Nygard, Release It!; Fowler); Stripe's idempotent requests; Temporal's durable execution. The experiments' models, tools and prices are simulated; all "our run" output is real.

🔗 References