Harness Production Engineering: Running Agents Reliably at Scale
A capable model plus a clever prompt makes a demo. A production agent service needs durable state that survives crashes, asynchronous jobs that run for hours, tools that execute safely, sandboxes, approvals, tracing, retries that don't double-charge anyone, and an architecture that separates control from execution. This visual deep dive covers each layer โ with interactive diagrams, a runnable durability simulation, and the 2026 platform features that implement them.
01 The Production Challenge
A frontier model brings capability โ reasoning, learned tool use, multi-step planning, multimodal understanding, knowledge. It becomes useful in the real world only inside a harness that makes it work reliably: orchestration and state, tools and environments, memory and context management, permissions and approvals, observability, failure recovery and scalable infrastructure.
02 Durable Agent Runtime
An agent run is a long program whose steps are slow, expensive and sometimes irreversible โ model calls, tool calls, emails, payments. If the worker dies halfway, you want to resume, not restart, and you must not repeat side effects. That is what durable execution provides: persisted run state and step history, idempotent steps, automatic retries with timeouts, pause/resume and cancellation, and deterministic replay where possible (as in durable-workflow engines such as Temporal).
The same idea at scale โ 2,000 simulated 40-step runs with a 1.5% crash chance per step. In the checkpoint-only runtime, a crash can land after a side effect happened but before the checkpoint was saved, so the retried step repeats it:
"""Why agent runs need durable execution: crashes, checkpoints and idempotency.
A simulated 40-step agent run (model calls and tool calls, ~20 s each). Five steps have side effects:
send an email, charge a card, create a ticket, write a record, post a message. The worker process
crashes with probability 1.5% per step (deploys, OOM, lost node, provider timeout).
Runtimes compared, 2,000 runs each:
restart - no persisted state: a crash restarts the whole run from step 1
checkpoint - state saved after every step: a crash resumes from the last checkpoint,
but the step in flight is retried blindly (its side effect may already have happened)
checkpoint+idem - same, plus an idempotency key per side-effecting step, so a retried step's
side effect is applied at most once
Standard library only.
"""
import random
STEPS, STEP_S, CRASH_P = 40, 20, 0.015
SIDE_EFFECTS = {6, 14, 22, 30, 38}
def run(mode, rng):
step, elapsed, crashes, effects = 0, 0, 0, {}
applied_keys = set()
while step < STEPS:
elapsed += STEP_S
if step in SIDE_EFFECTS:
key = f"step-{step}"
if mode != "checkpoint+idem" or key not in applied_keys:
effects[step] = effects.get(step, 0) + 1 # the external action happens
applied_keys.add(key)
if rng.random() < CRASH_P: # crash after doing the work, before saving
crashes += 1
if mode == "restart":
step = 0 # everything is lost
continue # checkpointed modes redo only this step
step += 1
dupes = sum(n - 1 for n in effects.values())
return elapsed / 60, crashes, dupes
print(f"{STEPS}-step run, {STEP_S} s per step, {CRASH_P:.1%} crash chance per step; 2,000 runs per runtime\n")
print(f" {'runtime':<18}{'mean time':>11}{'p95 time':>10}{'crashes/run':>13}{'duplicate side effects':>24}{'runs with a duplicate':>23}")
for mode in ["restart", "checkpoint", "checkpoint+idem"]:
rng = random.Random(3)
res = [run(mode, rng) for _ in range(2000)]
times = sorted(t for t, _, _ in res)
print(f" {mode:<18}{sum(times) / len(times):>8.1f} min{times[int(0.95 * len(times))]:>7.1f} min"
f"{sum(c for _, c, _ in res) / len(res):>13.2f}{sum(d for *_, d in res):>24}{sum(1 for *_, d in res if d) / len(res):>22.1%}")
print(f"\nWithout crashes the run takes {STEPS * STEP_S / 60:.1f} min.")Output:
40-step run, 20 s per step, 1.5% crash chance per step; 2,000 runs per runtime runtime mean time p95 time crashes/run duplicate side effects runs with a duplicate restart 18.7 min 36.7 min 0.88 3469 43.4% checkpoint 13.5 min 14.0 min 0.65 188 8.9% checkpoint+idem 13.5 min 14.0 min 0.65 0 0.0% Without crashes the run takes 13.3 min.
Checkpointing removes almost all the wasted time โ the long tail disappears โ but only idempotency keys stop duplicate emails and charges. Both are needed.
03 Long-Running and Asynchronous Execution
Tasks that run for minutes or hours can't hold an HTTP request open. The pattern: the client starts a task, gets an ID back immediately, and a job queue hands the work to a pool of workers; the client polls a status API or receives progress events, and can cancel. Scheduled (cron) tasks, worker autoscaling and time limits complete the picture.
background: true returns a response ID immediately; the model keeps working server-side and the client polls (or streams) until the response completes, avoiding timeouts on long reasoning and agent tasks. Background responses are stored for about ten minutes to allow polling.04 Context Lifecycle Engineering
In production, context is managed as a lifecycle, not a prompt: new information arrives, a selector keeps what's relevant, the history is compacted to preserve key facts, and an updated working context goes into the next step โ repeated until the task ends.
05 State and Memory Persistence
Separate what the model is thinking about now (working context) from durable task state. Each kind of state gets a store suited to it:
06 Tool Execution Architecture
Between "the model wants to call a tool" and "the tool ran" sit several gates. A tool registry holds schemas and metadata; a router selects the implementation; each call is validated, permission-checked, executed in a sandbox with timeouts, and returned as a structured result โ with retries, circuit breakers and detailed logs around it. Pick a scenario to see where each call stops:
07 MCP in Production (2026)
The Model Context Protocol connects agents to tools, data and systems through MCP servers (see MCP explained). Its 2026-07-28 specification is its biggest production-oriented revision so far:
Mcp-Session-Id header are gone; every request carries its protocol version and client capabilities, so any server instance can answer it โ ordinary HTTP load balancing works. Servers that need state mint explicit handles passed as tool arguments.tools/list, resources/list and similar results carry ttlMs and cacheScope (public or private) hints, modelled on HTTP caching.08 Sandboxes and Controlled Environments
Agents that run code or operate computers need isolated environments with real capabilities and hard limits: a file system, shell and code runtime, browser or computer use, controlled package installation, network policies (allow/deny), resource limits (CPU, memory, time), snapshots and mounts, and clean, reproducible starting states.
09 Guardrails and Human Approval
Guardrails check inputs, outputs and tool calls against policy; sensitive actions pause for human review. Pick an action the agent wants to take:
- Harmful content
- Policy checks
- Injection detection
- Response filtering
- Safety and PII checks
- Allowed tools
- Parameter checks and limits
- Least privilege
- Data controls
- Audit trail
10 Orchestration and Handoffs
An orchestrator or planner decomposes the task, routes pieces to specialist agents and coordinates the result. Two patterns dominate: handoffs (control passes to another agent together with context and state) and agents as tools (the orchestrator calls a specialist and keeps control). Either way, loops must be bounded and one agent must own the final answer.
11 Observability and Tracing
You can't fix what you can't see. A trace records one run as a tree of spans โ every model call (tokens, latency), tool call (inputs, outputs), handoff, guardrail, approval, retry and state change. Hover the spans:
12 Reliability Engineering
Agents call flaky networks, rate-limited models and third-party APIs thousands of times. Standard distributed-systems techniques apply โ and matter more, because the agent may otherwise "reason" its way into repeating a failed action.
13 Deployment Architecture
Separate the control plane (orchestration, configuration, monitoring, policy) from the execution plane (scalable workers and tool infrastructure), and make everything multi-tenant. Hover a component:
14 Production Metrics and Engineering Checklist
15 End-to-End Flow: From User Goal to Reliable Result
Next: The Harness Market Landscape โ who builds the frameworks, sandboxes, workflows and observability around frontier models.