Home โ€บ Blog โ€บ Harness Case Studies
Harness Engineering (Agentic Scaffoldings) ยท Part 4 ยท Snapshot: September 2026 ๐Ÿ”

Harness Case Studies: OpenAI Codex, Claude Code Auto Mode and Gemini Deep Research

Three real agent products, three harness designs. Codex runs many coding agents in parallel in isolated worktrees and cloud sandboxes. Claude Code's auto mode replaces most permission prompts with a sandbox and a two-stage safety classifier โ€” and publishes how often that classifier misses. Gemini Deep Research turns one question into a long, iterative, cited investigation. This visual deep dive takes each apart from public documentation, with interactive diagrams and a runnable simulation of approval fatigue using the published numbers.

How to read this: this is the case-study lens of the series (research, production, market). Everything here comes from the companies' public documentation and engineering posts, checked in September 2026 and linked in References. Internal architectures are not public; where a diagram simplifies or infers, it says so.

01 Case Study 1: OpenAI Codex โ€” a Coding-Agent Harness

Theme: end-to-end software engineering with multiple agents, worktrees and background tasks.

Codex is OpenAI's AI coding agent for end-to-end engineering work โ€” features, refactors, migrations, tests, docs. It runs in the CLI, IDE extensions and cloud, and since February 2026 in a desktop Codex app built to manage many agents over long periods (Windows followed in March 2026). The same month OpenAI shipped GPT-5.3-Codex and GPT-5.3-Codex-Spark as its underlying models.

โ–ถ Parallel agents in isolated worktrees Three tasks from one repository, running at once. Press play.

Each agent gets its own git worktree โ€” a separate working copy of the same repository โ€” so parallel agents never edit the same files. Inside each, the agent runs a loop:

โ–ถ Inside one agent The typical loop, from task to reviewable result.

What keeps it safe is a two-dimensional control: a sandbox mode (what the agent can touch) and an approval policy (when it must ask). Try combinations:

โ–ถ Codex sandbox ร— approval policy Simplified from the Codex security documentation.
Cloud tasks use a two-phase runtime: a setup phase can reach the network to install dependencies, then the agent phase runs offline by default unless internet access is enabled for that environment; secrets configured for the environment are available only during setup and removed before the agent starts. Repository instructions live in an AGENTS.md file; background automations run scheduled jobs โ€” issue triage, alert monitoring, CI/CD tasks โ€” in their own worktrees.
Parallel agents
  • Worktrees and cloud environments
Background jobs
  • Issue triage, alerts, CI/CD tasks
Skills & team instructions
  • AGENTS.md, repo conventions
Checkpoints & artifacts
  • Diffs, logs, test results
Observability
  • Progress, status, run history

02 Case Study 2: Claude Code Auto Mode โ€” Safer Autonomous Tool Use

Theme: reduce approval fatigue while preserving safety through layered safeguards.

Anthropic found that Claude Code users approved 93% of permission prompts โ€” and that the more prompts people see, the less attention each gets. Two harness features attack this. Sandboxing lets Claude work freely inside OS-level file-system and network boundaries (Seatbelt on macOS, bubblewrap on Linux), which cut permission prompts by 84% in internal use. Auto mode goes further: model-based classifiers approve or block actions instead of a human, with two layers of defence โ€” one for what Claude reads, one for what Claude does. Pick an action:

โ–ถ The auto-mode pipeline How each proposed action is handled.

The transcript classifier (running on Claude Sonnet 4.6) works in two stages: a fast stage 1 told to "err on the side of blocking" that outputs a single yes/no, and a stage 2 that reasons carefully about anything stage 1 flagged. Crucially it sees only the user's messages and the executable action โ€” assistant prose and tool outputs are stripped, so neither the agent nor injected content can argue with it. Anthropic published how well it works:

โ–ถ Published classifier error rates From Anthropic's engineering post, evaluated on real traffic and synthetic attacks.
False positive = a safe action wrongly blocked (measured on 10,000 real tool calls). False negative = a genuinely overeager or dangerous action allowed. Anthropic calls the 17% miss rate on real overeager actions "the honest number" โ€” auto mode is a trade-off against no guardrails, not a replacement for sandboxing.
More published details: reads, search, code navigation and plan-mode transitions are allow-listed; file edits inside the project directory skip the classifier (they're reviewable and reversible). Classifiers run when work is delegated to a subagent and again when results come back. After 3 consecutive or 20 total blocked actions, the session stops and escalates to a human.

03 Approval Fatigue, Simulated

How do manual prompts, a sandbox, auto mode and their combination compare over a week of agent work? This simulation uses Anthropic's published rates plus stated assumptions about how often actions are dangerous and how human attention decays:

"""Approval fatigue: manual prompts vs a sandbox vs a classifier (auto mode), on the same agent session.

Published inputs (Anthropic engineering posts):
  - users approve 93% of permission prompts
  - sandboxing cut permission prompts by 84% in internal use
  - auto-mode classifier, full two-stage pipeline: 0.4% false positives, 17% false negatives on
    real overeager actions
Assumptions (ours, adjustable): 2,000 actions per week of sessions; 1% of actions are genuinely
dangerous/overeager; a human reviewer's chance of catching a dangerous prompt starts at 90% and
decays as prompts pile up in a session (fatigue); a sandbox boundary contains 80% of dangerous actions
(the rest - e.g. deleting files inside the project - happen within it). Standard library only.
"""
import random

ACTIONS, DANGER_P, CONTAIN = 2_000, 0.01, 0.8
AUTO_FPR, AUTO_FNR, SANDBOX_CUT = 0.004, 0.17, 0.84

def human_catch(prompts_so_far):
    return max(0.35, 0.90 - 0.004 * prompts_so_far)      # fatigue: attention decays with volume

def simulate(mode, rng):
    prompts = executed_danger = blocked_safe = caught = 0
    session_prompts = 0
    for i in range(ACTIONS):
        if i % 100 == 0: session_prompts = 0              # a new session every 100 actions
        danger = rng.random() < DANGER_P
        if mode == "manual prompts":
            ask = rng.random() < 0.6                      # ~60% of actions need a prompt
        elif mode in ("sandbox", "sandbox + auto"):
            ask = rng.random() < 0.6 * (1 - SANDBOX_CUT)  # most actions run freely inside the boundary
            if mode == "sandbox + auto": ask = False      # the classifier handles what the boundary doesn't
            if danger and rng.random() < CONTAIN: caught += 1; continue   # the boundary contains it
            if mode == "sandbox + auto":
                if danger:
                    if rng.random() < AUTO_FNR: executed_danger += 1
                    else: caught += 1
                elif rng.random() < AUTO_FPR: blocked_safe += 1
                continue
        else:                                             # auto mode: a classifier replaces the human
            if danger:
                if rng.random() < AUTO_FNR: executed_danger += 1
                else: caught += 1
            elif rng.random() < AUTO_FPR: blocked_safe += 1
            continue
        if not ask:
            if danger: executed_danger += 1
            continue
        prompts += 1; session_prompts += 1
        if danger:
            if rng.random() < human_catch(session_prompts): caught += 1
            else: executed_danger += 1
    return prompts, executed_danger, caught, blocked_safe

print(f"{ACTIONS:,} agent actions, {DANGER_P:.0%} dangerous; 200 simulated weeks per mode\n")
print(f"   {'mode':<16}{'prompts shown':>15}{'dangerous executed':>21}{'dangerous stopped':>20}{'safe actions blocked':>23}")
for mode in ["manual prompts", "sandbox", "auto mode", "sandbox + auto"]:
    rng = random.Random(4)
    r = [simulate(mode, rng) for _ in range(200)]
    avg = [sum(x[k] for x in r) / len(r) for k in range(4)]
    print(f"   {mode:<16}{avg[0]:>15,.0f}{avg[1]:>21.1f}{avg[2]:>20.1f}{avg[3]:>23.1f}")
print("\nManual prompts miss dangerous actions that never asked, and fatigue erodes the rest; layers compound.")

Output:

2,000 agent actions, 1% dangerous; 200 simulated weeks per mode

   mode              prompts shown   dangerous executed   dangerous stopped   safe actions blocked
   manual prompts            1,200                 11.1                 9.2                    0.0
   sandbox                     191                  3.9                16.4                    0.0
   auto mode                     0                  3.3                16.8                    8.1
   sandbox + auto                0                  0.8                18.9                    7.7

Manual prompts miss dangerous actions that never asked, and fatigue erodes the rest; layers compound.

Showing people 1,200 prompts a week does not make them safe: most dangerous actions never asked, and fatigue erodes the rest. A sandbox removes most prompts and contains most damage; the classifier catches much of what remains at the cost of a few wrongly blocked actions; stacked, fewer than one of the week's 20 dangerous actions gets through (0.8 on average). The exact numbers depend on the assumptions โ€” the ordering is the lesson, and it is why Anthropic ships the layers together.

04 Case Study 3: Google Gemini Deep Research โ€” a Long-Horizon Research Harness

Theme: autonomous, iterative research with citations, tools and asynchronous execution.

Gemini Deep Research is an autonomous research agent: it plans, searches, reads, notices gaps in what it knows, searches again, and writes a cited report. In April 2026 Google launched Deep Research and Deep Research Max on Gemini 3.1 Pro, available through the Gemini app and โ€” for developers โ€” exclusively through the Interactions API. Max spends more test-time compute for long, accuracy-critical investigations across hundreds of sources.

โ–ถ The iterative research loop "How are European utilities financing grid storage?" Press play; watch the knowledge gaps shrink.
โ–ถ Interactive vs asynchronous research Toggle the two modes.
๐ŸŒ Web search & retrievalGoogle Search as the default source.
๐Ÿ“ File uploads & file storesPDFs, CSVs, images, audio and video; File Search over private data โ€” web access can be turned off entirely.
๐Ÿ”Œ Remote MCP serversCustom and proprietary data sources through MCP.
๐Ÿ”— URL context & code executionRead specific pages; compute over data.
๐Ÿ—บ๏ธ Collaborative plan reviewThe user can edit the research plan before it runs.
๐Ÿ“Š Structured, cited outputsReports with citations; charts and infographics inline (added April 2026).

05 Cross-Case Comparison

โ–ถ Compare by dimension Click a dimension to highlight how each product handles it.

06 What These Cases Teach Us

1Durable state and checkpoints matter. Long-running tasks need persistent state, artifacts and the ability to resume.
2Tool execution is mediated by a runtime. Agents don't act directly โ€” tools run in controlled environments (worktrees, sandboxes, APIs).
3Context selection and compression are core. Repos, files, web content and enterprise data must be selected, structured and fit in context.
4Safety is layered. Input and output checks, permissions, sandboxing, approvals and classifiers each reduce risk โ€” none is enough alone.
5Verification is domain-specific. Tests and builds for code, policy checks for tool use, citations and source checking for research.
6Parallelism and subagents increase throughput โ€” but need isolation, result integration and state management.
7Background execution turns agents into services. Scheduled and asynchronous runs enable real-world, long-horizon work.
8Strong harnesses compensate for model weaknesses with retries, observability, bounded loops and external validation.

07 Model vs Harness: Same Model, Different Outcomes

The same frontier model โ€” its reasoning, tool use, multimodal understanding and long-context processing โ€” performs very differently depending on the harness around it. Toggle harness components on and off to see which failure each one prevents (illustrative):

โ–ถ Build the harness A 40-step coding task with one flaky test, one misleading web page and one context overflow.

08 A Reusable End-to-End Harness Pattern

Distilled from the three cases โ€” the same twelve steps appear, in different forms, in each:

1User goaltask, question or issue
2Load contextrepos, files, web, enterprise data
3Planbreak down tasks
4Choose tool or subagenttool, MCP, delegate
5Permission & safety checkclassifier, approval, policy, sandbox
6Executein a controlled environment
7Observeresults, logs, artifacts
8Verifytests, citations, policy checks
9Update statecheckpoints, memory
10Replan / retryif needed
11Final artifactcode, report, action taken
12Trace & evaluatelogs, metrics, feedback

09 What Remains Proprietary or Unknown

These products are unusually well documented โ€” Anthropic even published its classifier's error rates โ€” but important details stay private:

1Exact internal prompts and system messages
2Model-side reasoning traces
3Detailed routing, planning and scheduling policies
4Proprietary evaluation data and benchmarks
5Model selection and ensemble logic
6Infrastructure, storage and scaling architecture
7Exact safety thresholds and classifier criteria
8Internal tools and datasets
The cases in one paragraph
Codex, Claude Code and Gemini Deep Research wrap frontier models in very different harnesses for very different jobs โ€” parallel coding in isolated worktrees, safe autonomy through sandboxes and classifiers, and long iterative research with citations โ€” yet they share one architecture: load and select context, plan, route every action through a permission layer into a controlled environment, verify with domain-specific checks, persist state, and run in the background. The most instructive detail is Anthropic's candour: a two-stage classifier that wrongly blocks 0.4% of safe actions still misses 17% of genuinely overeager ones, which is exactly why production harnesses stack defences โ€” sandbox, classifier, bounded denials, human escalation โ€” instead of trusting any single one.

Next: Product case study #1 โ€” the OpenAI Agents API: the Codex harness as a managed production runtime.

โš ๏ธ Notes

The source infographic's descriptions match the public documentation; refinements: (1) its "Model vs Harness" panel uses dated examples ("GPT-5 / Claude 4 / Gemini 2.5") โ€” in September 2026 these products run on newer models (for example GPT-5.3-Codex, Sonnet 4.6 for the auto-mode classifier, Gemini 3.1 Pro for Deep Research); (2) the Claude Code panel omits the published error rates โ€” full pipeline 0.4% false positives and 17% false negatives on real overeager actions โ€” which are essential to judging the safeguard; (3) it describes Deep Research Max as launched "2026" โ€” specifically April 2026, on Gemini 3.1 Pro, via the Interactions API; (4) the Codex permission model is two settings (sandbox mode and approval policy), which the infographic merges into "isolated worktrees / cloud envs". Worktree, loop, research and harness-builder diagrams are simplified illustrations; the approval simulation combines published rates with stated assumptions.

๐Ÿ”— References