Home โ€บ Blog โ€บ Harness Production Engineering
Harness Engineering (Agentic Scaffoldings) ยท Part 2 ยท Snapshot: September 2026 ๐Ÿ—๏ธ

Harness Production Engineering: Running Agents Reliably at Scale

A capable model plus a clever prompt makes a demo. A production agent service needs durable state that survives crashes, asynchronous jobs that run for hours, tools that execute safely, sandboxes, approvals, tracing, retries that don't double-charge anyone, and an architecture that separates control from execution. This visual deep dive covers each layer โ€” with interactive diagrams, a runnable durability simulation, and the 2026 platform features that implement them.

How to read this: this is the production lens of the series; Part 1 covers the research frontier. Interactive figures compute from the formulas shown or are labelled illustrative. Platform features (OpenAI's Agents SDK and background mode, Anthropic's long-running harness guidance, the MCP 2026-07-28 specification) come from public documentation linked in References, checked in September 2026; the patterns apply to any stack.

01 The Production Challenge

A frontier model brings capability โ€” reasoning, learned tool use, multi-step planning, multimodal understanding, knowledge. It becomes useful in the real world only inside a harness that makes it work reliably: orchestration and state, tools and environments, memory and context management, permissions and approvals, observability, failure recovery and scalable infrastructure.

Model capability what the model can do
๐Ÿง  Reasoning
๐Ÿ”ง Tool use (learned)
๐Ÿ—บ๏ธ Multi-step planning
๐Ÿ–ผ๏ธ Multimodal understanding
๐Ÿ“š Knowledge
Production harness what makes it work reliably
๐ŸŽ›๏ธ Orchestration & state
๐Ÿงฐ Tools & environments
๐Ÿ—‚๏ธ Memory & context management
๐Ÿ” Permissions & approvals
๐Ÿ”ญ Observability & tracing
๐Ÿฉน Failure recovery
๐Ÿ“ˆ Scalable infrastructure
Agent service
Runs for hours
Survives failures
Acts safely
Debuggable
Meets SLAs

02 Durable Agent Runtime

An agent run is a long program whose steps are slow, expensive and sometimes irreversible โ€” model calls, tool calls, emails, payments. If the worker dies halfway, you want to resume, not restart, and you must not repeat side effects. That is what durable execution provides: persisted run state and step history, idempotent steps, automatic retries with timeouts, pause/resume and cancellation, and deterministic replay where possible (as in durable-workflow engines such as Temporal).

โ–ถ Crash in the middle A 12-step run; step 7 sends an email (โœ‰). Press run: the worker crashes right after the email goes out, before the step is checkpointed.

The same idea at scale โ€” 2,000 simulated 40-step runs with a 1.5% crash chance per step. In the checkpoint-only runtime, a crash can land after a side effect happened but before the checkpoint was saved, so the retried step repeats it:

"""Why agent runs need durable execution: crashes, checkpoints and idempotency.

A simulated 40-step agent run (model calls and tool calls, ~20 s each). Five steps have side effects:
send an email, charge a card, create a ticket, write a record, post a message. The worker process
crashes with probability 1.5% per step (deploys, OOM, lost node, provider timeout).

Runtimes compared, 2,000 runs each:
  restart            - no persisted state: a crash restarts the whole run from step 1
  checkpoint         - state saved after every step: a crash resumes from the last checkpoint,
                       but the step in flight is retried blindly (its side effect may already have happened)
  checkpoint+idem    - same, plus an idempotency key per side-effecting step, so a retried step's
                       side effect is applied at most once
Standard library only.
"""
import random

STEPS, STEP_S, CRASH_P = 40, 20, 0.015
SIDE_EFFECTS = {6, 14, 22, 30, 38}

def run(mode, rng):
    step, elapsed, crashes, effects = 0, 0, 0, {}
    applied_keys = set()
    while step < STEPS:
        elapsed += STEP_S
        if step in SIDE_EFFECTS:
            key = f"step-{step}"
            if mode != "checkpoint+idem" or key not in applied_keys:
                effects[step] = effects.get(step, 0) + 1      # the external action happens
                applied_keys.add(key)
        if rng.random() < CRASH_P:                            # crash after doing the work, before saving
            crashes += 1
            if mode == "restart":
                step = 0                                      # everything is lost
            continue                                          # checkpointed modes redo only this step
        step += 1
    dupes = sum(n - 1 for n in effects.values())
    return elapsed / 60, crashes, dupes

print(f"{STEPS}-step run, {STEP_S} s per step, {CRASH_P:.1%} crash chance per step; 2,000 runs per runtime\n")
print(f"   {'runtime':<18}{'mean time':>11}{'p95 time':>10}{'crashes/run':>13}{'duplicate side effects':>24}{'runs with a duplicate':>23}")
for mode in ["restart", "checkpoint", "checkpoint+idem"]:
    rng = random.Random(3)
    res = [run(mode, rng) for _ in range(2000)]
    times = sorted(t for t, _, _ in res)
    print(f"   {mode:<18}{sum(times) / len(times):>8.1f} min{times[int(0.95 * len(times))]:>7.1f} min"
          f"{sum(c for _, c, _ in res) / len(res):>13.2f}{sum(d for *_, d in res):>24}{sum(1 for *_, d in res if d) / len(res):>22.1%}")
print(f"\nWithout crashes the run takes {STEPS * STEP_S / 60:.1f} min.")

Output:

40-step run, 20 s per step, 1.5% crash chance per step; 2,000 runs per runtime

   runtime             mean time  p95 time  crashes/run  duplicate side effects  runs with a duplicate
   restart               18.7 min   36.7 min         0.88                    3469                 43.4%
   checkpoint            13.5 min   14.0 min         0.65                     188                  8.9%
   checkpoint+idem       13.5 min   14.0 min         0.65                       0                  0.0%

Without crashes the run takes 13.3 min.

Checkpointing removes almost all the wasted time โ€” the long tail disappears โ€” but only idempotency keys stop duplicate emails and charges. Both are needed.

03 Long-Running and Asynchronous Execution

Tasks that run for minutes or hours can't hold an HTTP request open. The pattern: the client starts a task, gets an ID back immediately, and a job queue hands the work to a pool of workers; the client polls a status API or receives progress events, and can cancel. Scheduled (cron) tasks, worker autoscaling and time limits complete the picture.

โ–ถ Job queue and worker pool Tasks arrive in a burst; workers drain the queue. Computed.
Public example โ€” OpenAI background mode: in the Responses API, setting background: true returns a response ID immediately; the model keeps working server-side and the client polls (or streams) until the response completes, avoiding timeouts on long reasoning and agent tasks. Background responses are stored for about ten minutes to allow polling.

04 Context Lifecycle Engineering

In production, context is managed as a lifecycle, not a prompt: new information arrives, a selector keeps what's relevant, the history is compacted to preserve key facts, and an updated working context goes into the next step โ€” repeated until the task ends.

โ–ถ The context lifecycle Each step of the loop, animated.
Public example โ€” Anthropic's long-running agent harness (November 2025): an initializer agent sets up the environment on the first run โ€” a feature list, a git repository, an init script and a progress file โ€” and each later coding agent session makes incremental progress on one feature, commits, and updates the progress notes, so a fresh context window can pick up exactly where the last one stopped. Anthropic pairs this with compaction and structured note-taking.

05 State and Memory Persistence

Separate what the model is thinking about now (working context) from durable task state. Each kind of state gets a store suited to it:

โ–ถ The memory and state layer Click a store.
โœ“Resume from any step ยท long-term task continuity.
โœ“Separation of context and durable state.
โœ“Provenance and auditing ยท shared state across subagents and sessions.

06 Tool Execution Architecture

Between "the model wants to call a tool" and "the tool ran" sit several gates. A tool registry holds schemas and metadata; a router selects the implementation; each call is validated, permission-checked, executed in a sandbox with timeouts, and returned as a structured result โ€” with retries, circuit breakers and detailed logs around it. Pick a scenario to see where each call stops:

โ–ถ A tool call's journey Six gates between the model and the outside world.

07 MCP in Production (2026)

The Model Context Protocol connects agents to tools, data and systems through MCP servers (see MCP explained). Its 2026-07-28 specification is its biggest production-oriented revision so far:

โ–ถ Why "stateless" matters for scaling Requests from three clients across three server instances.
๐Ÿ”“ Stateless protocol coreThe initialize handshake and the Mcp-Session-Id header are gone; every request carries its protocol version and client capabilities, so any server instance can answer it โ€” ordinary HTTP load balancing works. Servers that need state mint explicit handles passed as tool arguments.
๐Ÿ—ƒ๏ธ Cacheable list resultstools/list, resources/list and similar results carry ttlMs and cacheScope (public or private) hints, modelled on HTTP caching.
๐Ÿ” Authorization hardeningCloser to production OAuth 2.0 / OpenID Connect: issuer validation (RFC 9207), issuer-bound client credentials, and Client ID Metadata Documents replacing Dynamic Client Registration.
๐Ÿงฉ Extensions frameworkA governed way to add capabilities: Tasks, MCP Apps, Enterprise Managed Authorization and OAuth Client Credentials are extensions.
โณ TasksFor long-running work a server can return a durable task handle; the client polls or subscribes and can cancel. Opt-in on both sides.
โ–ถ What a cache hint saves Agents often re-list tools on every step. Computed.

08 Sandboxes and Controlled Environments

Agents that run code or operate computers need isolated environments with real capabilities and hard limits: a file system, shell and code runtime, browser or computer use, controlled package installation, network policies (allow/deny), resource limits (CPU, memory, time), snapshots and mounts, and clean, reproducible starting states.

โ–ถ Control plane vs sandbox The harness never runs model-generated code itself. Press play.
Public example โ€” OpenAI Agents SDK sandbox agents (April 2026): the SDK separates the harness (control plane: agent loop, model calls, tool routing, handoffs, approvals, tracing, run state) from compute (the sandbox: files, shell, packages, mounted storage such as S3 or GCS, exposed ports). Workspace manifests describe the starting files, repositories and mounts, and built-in snapshotting and rehydration let a run resume in a fresh container if the original fails or expires.

09 Guardrails and Human Approval

Guardrails check inputs, outputs and tool calls against policy; sensitive actions pause for human review. Pick an action the agent wants to take:

โ–ถ The approval flow Layered guardrails route each action by risk.
Input guardrails
  • Harmful content
  • Policy checks
  • Injection detection
Output guardrails
  • Response filtering
  • Safety and PII checks
Tool guardrails
  • Allowed tools
  • Parameter checks and limits
Policy enforcement
  • Least privilege
  • Data controls
  • Audit trail
Public example โ€” OpenAI Agents SDK: input and output guardrails run alongside the agent and can halt a run ("tripwires"), and human-in-the-loop support lets a run pause on a tool call that needs approval, persist its state, and resume after a person approves or rejects it.

10 Orchestration and Handoffs

An orchestrator or planner decomposes the task, routes pieces to specialist agents and coordinates the result. Two patterns dominate: handoffs (control passes to another agent together with context and state) and agents as tools (the orchestrator calls a specialist and keeps control). Either way, loops must be bounded and one agent must own the final answer.

โ–ถ Orchestrator and specialists "Analyse last quarter's churn and draft a retention plan." Press play.
โœ“Planner vs worker roles ยท specialist subagents ยท model routing per task or step.
โœ“Bounded loops with termination criteria ยท explicit ownership of the final response.

11 Observability and Tracing

You can't fix what you can't see. A trace records one run as a tree of spans โ€” every model call (tokens, latency), tool call (inputs, outputs), handoff, guardrail, approval, retry and state change. Hover the spans:

โ–ถ A run trace Illustrative spans on a real-time axis.
Hover or tap a span.
Public example โ€” OpenAI Agents SDK tracing: tracing is on by default and records LLM generations, tool calls, handoffs, guardrails and custom spans for each run, viewable in a traces dashboard and exportable to other observability tools. OpenTelemetry's GenAI semantic conventions are the emerging vendor-neutral format.

12 Reliability Engineering

Agents call flaky networks, rate-limited models and third-party APIs thousands of times. Standard distributed-systems techniques apply โ€” and matter more, because the agent may otherwise "reason" its way into repeating a failed action.

โ–ถ Exponential backoff with jitter 100 agents hit a rate limit at the same moment. When do they retry? Computed.
โ–ถ Circuit breaker Stop hammering a failing tool; probe it, then recover. Press play.
โฑ๏ธ Exponential backoff & retriesWith jitter and retry budgets.
๐Ÿ”‘ Idempotency keysAvoid duplicate side effects (section 02).
๐Ÿ“ฎ Dead-letter queuesPark failed tasks for inspection instead of retrying forever.
๐Ÿฉบ Health checks & circuit breakersDetect and isolate failing tools.
๐Ÿšฆ Rate-limit handlingRespect model and tool quotas; queue instead of failing.
๐Ÿ”€ Failover & graceful degradationFall back to another model or a reduced feature set.
๐Ÿงฉ Partial completionReturn useful results even if some steps failed โ€” and say which.
โ†ฉ๏ธ Rollback / compensationUndo side effects when a multi-step action fails midway (the saga pattern).

13 Deployment Architecture

Separate the control plane (orchestration, configuration, monitoring, policy) from the execution plane (scalable workers and tool infrastructure), and make everything multi-tenant. Hover a component:

โ–ถ A reference architecture Users โ†’ gateway โ†’ control plane โ†’ execution plane โ†’ streaming result.
Hover or tap a component.

14 Production Metrics and Engineering Checklist

Task success rateend to end, verified โ€” the metric that matters
Completion without human rescueautonomy actually achieved
Retry ratemodel and tools; rising = something is degrading
Tool success rateper tool; feeds circuit breakers
Human intervention rateapprovals and escalations per task
p50 / p95 latencyend to end, and queue time to start
Tokens and cost per taskwatch for runaway loops
State-recovery successresumed runs that completed
Policy violations / blocked actionsguardrail and sandbox events
Availability and error ratethe service-level view
โ–ถ Engineering checklist Tick what your agent service already has.

15 End-to-End Flow: From User Goal to Reliable Result

1User goalnatural language or API request
2API gatewayauth, routing
3Durable runtimecreate or resume the run
4Load state & contextmemory, artifacts
5Plandecompose, select tools
6Choose model & toolroute, validate, authorize
7Executein sandbox / via MCP
8Observetrace, monitor
9Persist statecheckpoint, artifacts
10Verify / approvehuman-in-the-loop if needed
11Continue / retryhandle errors
12Final resultstream or return
13Trace & metricslogs, dashboards
The production principle: optimize the whole agent system โ€” state, context, tools, permissions, scheduling, reliability and observability โ€” not just prompts or model calls.
Harness production engineering in one paragraph
A production agent is a distributed system whose most unpredictable component is the model. Treat each run as a durable workflow: checkpoint every step, make every side effect idempotent, and let runs pause, resume and survive crashes. Push long work into queues and background tasks, manage context as a lifecycle with compaction and durable notes, and keep state in stores suited to it. Put every tool call through validation, permissions, a sandbox and a circuit breaker; route sensitive actions through guardrails and human approval; bound every loop and give one agent ownership of the answer. Trace everything, retry with jittered backoff, and separate the control plane from the execution plane. In 2026 the platforms โ€” durable runtimes, sandbox agents, background modes and a stateless MCP โ€” increasingly provide these pieces; the engineering is in wiring them together and measuring end-to-end task success.

Next: The Harness Market Landscape โ€” who builds the frameworks, sandboxes, workflows and observability around frontier models.

โš ๏ธ Notes

The source infographic's public examples check out against the documentation: OpenAI's background mode, Agents SDK sandbox agents (April 2026), guardrails with human review and tracing; Anthropic's compaction and long-running harness guidance; and the MCP 2026-07-28 specification's stateless core, cacheable list results, authorization hardening and extensions. One refinement: it lists long-running Tasks among core MCP innovations โ€” in the 2026-07-28 specification Tasks are an opt-in extension (alongside MCP Apps and enterprise authorization) rather than part of the protocol core. The durability simulation uses stated crash and timing assumptions; the trace, queue and approval diagrams are illustrative; backoff and cache figures are computed.

๐Ÿ”— References