Home โ€บ Blog โ€บ Harness Engineering
Deep Dive ยท With a same-model ablation ๐Ÿงฐ

LLM Harness Engineering: Engineering the System Around the Model

The same model can be a chatbot that invents answers or an agent that reliably resolves tickets. The difference is the harness โ€” the context, tools, environment, controls and feedback loops built around it. This piece defines harness engineering, maps its components and flow, and measures the claim directly: one unchanged model, five harnesses, from bare chat to production-ready.

How to read this: this article pulls together ideas the How LLMs Are Built series covered component by component, under the name the industry now uses for them. Published results are cited. "Our run" numbers come from the script in Run the Ablation, whose model is simulated with fixed habits so that only the harness varies โ€” it demonstrates the effect of each layer, not the performance of a real model. For an interactive tour of every layer, see The Complete Stack.

What Is Harness Engineering?

A harness is the system around the model that makes it useful. It provides context, tools, memory and an environment; guides multi-step behaviour and planning; adds verification, safety and guardrails; handles errors, retries and recovery; integrates with real-world systems โ€” and so turns a foundation model into a reliable product. The term gained currency when OpenAI described "harness engineering" in February 2026: an internal team built and shipped a product of roughly a million lines of code written by Codex agents, with the engineers' job being to design the environment โ€” the harness โ€” that made that reliable.

Model vs Harness

The model (unchanged)

Pretrained weights ยท general capabilities ยท the same model in every example below ยท not aware of your specific tools, data or policies.

The harness (everything else)

Instructions and context ยท memory and state management ยท tools and external systems ยท planning and orchestration ยท verification and guardrails ยท error handling and recovery ยท observability and evaluation.

The best-known published demonstration is SWE-agent (Yang et al., 2024). With the same underlying model, a purpose-built agentโ€“computer interface โ€” commands for viewing, searching and editing files designed for a model rather than a human โ€” solved 12.5% of SWE-bench issues, against 3.8% for the previous best approach, which used retrieval. The weights didn't change; the harness did.

Same Model, Five Harnesses

We measured the claim with an ablation. One simulated model with fixed habits โ€” it invents answers when it has no data, malforms tool arguments 15% of the time, sometimes claims it acted when it didn't, and follows instructions injected into data 30% of the time โ€” ran 400 support tasks ("analyze the customer's data and send a response"), 80 of them containing an injected instruction. Each harness adds one layer:

0% 0 bare chat 0% 0 + context & RAG 72% 13 + tools 99% 20 + verification & retry 98% 0 + guardrails & approval goal metunsafe actions (count)
Our run: goals met (green) and unsafe actions (red) as each harness layer is added around the identical model.
HarnessGoal metGrounded answerAction takenUnsafe actionsModel calls / task
bare chat0.0%0.0%0.0%01.00
+ context & RAG0.0%88.8%0.0%01.00
+ tools71.8%91.2%79.0%131.92
+ verification & retry99.0%99.2%99.8%202.24
+ guardrails & approval98.0%98.5%98.0%02.23
1Bare chat achieves nothing: without data every answer is invented, and without tools nothing gets done.
2Context and RAG make answers grounded, but the system still can't act.
3Tools let it act โ€” and immediately introduce new failures: malformed calls, claimed-but-not-done actions, and unsafe actions from injected instructions.
4Verification and retry fix the reliability problems โ€” but not the safety one. Unsafe actions stay, because checking whether the work was done doesn't check whether it should have been done.
5Guardrails and approval bring unsafe actions to zero, at a small cost: a few more tasks stop for a human instead of completing.

Each layer fixes some failures and exposes others, and reliability and safety need different layers. That is harness engineering in one table.

Key Goals of a Harness

GoalMeaning
CapabilityGive access to data and tools
ReliabilityHandle errors and uncertainty
ControlEnforce policies and permissions
VerifiabilityCheck accuracy and grounding
AdaptabilitySupport different domains and use cases
MaintainabilityMonitor, evaluate and improve over time

High-Level Architecture

The harness sits between the user or application, external systems and the model. Inside it: context, planning, memory, tools, state, verification, policies and observability. It connects the foundation model to external systems (APIs, databases, files, SaaS) and to an environment for doing things safely (sandboxes, code execution, browsers). The model is one component; the harness decides everything it sees and everything it's allowed to do.

The End-to-End Harness Flow

  1. User goal โ€” "Analyze customer data and draft a response."
  2. Context builder โ€” system instructions, conversation history, retrieved knowledge, relevant memory, tool definitions, the current message.
  3. Planner / orchestrator โ€” decompose the task, decide the next action, track progress, choose tools or subagents, handle dependencies.
  4. Tool and environment execution โ€” select and validate the tool, check permissions, execute via MCP or APIs, use a sandbox when needed, get a structured result.
  5. Observe and update โ€” process the result, update memory and task state, check for errors, decide the next step.
  6. Verify and guardrail โ€” check grounding, validate output, enforce safety policies, request human approval if needed, detect and handle failures.
  7. Final result โ€” complete the goal, return a verified answer, update memory, log traces and metrics, feed the improvement loop. Otherwise replan, retry or continue until the goal is complete.

Core Components of a Harness

ComponentAddressesDeep dive
Context engineeringAssemble the right information: instructions, history, memory, RAG, toolsContext Engineering
Memory and stateWorking, episodic, semantic and artifact memory; checkpoints; long-term storageMemory & Planning
Planning and orchestrationTask decomposition, tool selection, multi-step controlAgentic Systems
Tools and integrationsAPIs, databases, files, code execution, browsers via MCP or connectorsTool Calling, MCP
Environment and sandboxingIsolated execution for code, shell, browsers and risky actionsProduction Agents
Verification and evaluationCheck correctness, grounding and constraint complianceEvaluating Agents
Permissions and safetyAuthentication, authorization, tool restrictions, human approvalsTool security
Error handling and recoveryRetries, replanning, fallbacks, exception handlingReliability
Observability and loggingTraces, tool calls, latency, tokens, cost, success ratesObservability
Continuous improvementUse failures and feedback to improve prompts, tools, policies or fine-tuned modelsImprovement loop

Handling Failures

Failures occur โ€” tool errors, invalid output, missing data, timeouts. The harness detects and classifies them (what type, what the error message says, whether retry is possible), picks a recovery strategy (retry with backoff, fix arguments and retry, use an alternative tool, replan with more context, ask the user for clarification, escalate to a human), and continues execution from the last checkpoint toward the goal. In our ablation, the retry layer turned malformed tool calls from failed tasks into recovered ones, and checking the claimed action against what actually happened caught the model saying "sent" when it hadn't.

Safety, Permissions and Control

Keep the model within the right boundaries: authentication and identity (SSO, service accounts); authorization with tool-level access control and least privilege; policy enforcement on actions and outputs; sandboxing for code, shell and browsers; human approval for sensitive actions; sensitive-data handling to prevent leakage; audit logging of actions, tool calls and decisions; and content safety filtering. The ablation's lesson applies here: these controls address a different failure class from reliability features, and a harness needs both.

Observe, Evaluate, Improve

Observability shows what the harness is doing: traces (end-to-end execution timelines), model calls (prompts, responses, tokens, latency), tool calls (inputs, outputs, success or failure), metrics (cost, latency, success and error rates), dashboards and alerts. Evaluation measures the harness itself: task success, trajectory quality, tool correctness, groundedness, safety compliance, reliability, efficiency and human intervention rate. The continuous improvement loop โ€” collect data, analyze failures, update prompts, tools, policies or models, test with regression suites and benchmarks, redeploy โ€” is how a harness gets better, and it's usually cheaper and faster than changing the model.

Worked Example: A Customer Support Agent

  1. Receive ticket โ€” a customer reports an issue with their account.
  2. Retrieve data โ€” RAG and tools fetch the customer profile, past tickets and account details.
  3. Analyze โ€” identify the root cause using the retrieved data and reasoning.
  4. Draft response โ€” a personalized, accurate response using company guidelines.
  5. Verify โ€” check tone, grounding and policy compliance.
  6. Approve โ€” request human approval if needed.
  7. Take action โ€” send the response via email or CRM and update the ticket.
  8. Finish โ€” confirm success, update memory and log the trace.

Every step except "analyze" and "draft" is harness, not model โ€” which is why teams that ship reliable agents spend most of their effort there.

Harness engineering in one paragraph
The harness is everything around the model: what it sees (context, memory, retrieval), what it can do (tools, environments, sandboxes), how it proceeds (planning, state, recovery), what it's allowed to do (permissions, policies, approval), and how you know it worked (verification, observability, evaluation, improvement). With the model held fixed, the harness decides whether it invents answers or completes tasks, and whether it's merely reliable or also safe โ€” and because it can be changed, tested and shipped far faster than the model, it's where most of the engineering of an AI product happens.

Run the Ablation

One script, standard library only, runs instantly.

"""Same model, five harnesses: an ablation study of harness engineering.

One simulated model with fixed habits - it invents answers when it has no data, sometimes malforms tool
arguments, sometimes claims it did something it didn't, and sometimes follows instructions injected into
the data it reads - runs 400 customer-support tasks inside five harnesses, each adding one layer.
The model never changes; only the system around it does. Standard library only.
"""
import random

class Model:
    """The unchanged model: fixed probabilities for each habit."""
    def __init__(self, rng): self.rng = rng
    def answer(self, facts):
        if not facts: return "invented"                                   # no data -> plausible hallucination
        return "correct" if self.rng.random() < 0.90 else "wrong detail"
    def tool_args(self): return "valid" if self.rng.random() < 0.85 else "malformed"
    def claims_done_without_acting(self): return self.rng.random() < 0.08
    def follows_injection(self): return self.rng.random() < 0.30

LEVELS = ["bare chat", "+ context & RAG", "+ tools", "+ verification & retry", "+ guardrails & approval"]

def run_task(level, injected, rng):
    m, calls = Model(rng), 0
    facts = level >= 1                                                    # context builder retrieves ticket + policy
    ans = m.answer(facts); calls += 1
    if level >= 3 and ans == "wrong detail":                             # verify the draft against the sources
        ans = m.answer(facts); calls += 1
    sent_to, leaked = None, False
    if level >= 2:                                                        # tools: the harness can act
        if injected and m.follows_injection():
            if level >= 4: pass                                           # guardrail: flagged as untrusted, blocked
            else: leaked = True
        if m.claims_done_without_acting() and level < 3:
            pass                                                          # says "sent" but never called the tool
        else:
            for attempt in range(3 if level >= 3 else 1):                 # retries only with verification layer
                calls += 1
                if m.tool_args() == "valid": sent_to = "customer"; break
            if level >= 4 and ans != "correct": sent_to = None            # approval step rejects a bad draft
    goal = ans == "correct" and sent_to == "customer"
    return {"goal": goal, "grounded": ans == "correct", "acted": sent_to is not None,
            "unsafe": leaked, "calls": calls}

rng = random.Random(11)
tasks = [rng.random() < 0.2 for _ in range(400)]                          # 20% of tickets contain an injected instruction
print(f"400 support tasks ('analyze the customer's data and send a response'), {sum(tasks)} with injected instructions\n")
print("   harness                     goal met   grounded answer   action taken   unsafe actions   model calls/task")
rows = []
for lvl, name in enumerate(LEVELS):
    r = [run_task(lvl, inj, rng) for inj in tasks]
    s = {k: sum(x[k] for x in r) for k in ("goal", "grounded", "acted", "unsafe", "calls")}
    rows.append(s)
    print(f"   {name:<26} {s['goal'] / 4:>7.1f}%   {s['grounded'] / 4:>14.1f}%   {s['acted'] / 4:>11.1f}%   "
          f"{s['unsafe']:>14}   {s['calls'] / 400:>15.2f}")
print("\nthe model object and its probabilities were identical in every row; only the harness changed")

Output (deterministic โ€” fixed random seed):

400 support tasks ('analyze the customer's data and send a response'), 80 with injected instructions

   harness                     goal met   grounded answer   action taken   unsafe actions   model calls/task
   bare chat                      0.0%              0.0%           0.0%                0              1.00
   + context & RAG                0.0%             88.8%           0.0%                0              1.00
   + tools                       71.8%             91.2%          79.0%               13              1.92
   + verification & retry        99.0%             99.2%          99.8%               20              2.24
   + guardrails & approval       98.0%             98.5%          98.0%                0              2.23

the model object and its probabilities were identical in every row; only the harness changed

โš ๏ธ Notes

The source infographic is accurate. Two additions from the ablation and the sources:
  • The "weak vs strong harness" panel suggests one jump. In practice each layer fixes some failures and introduces or reveals others โ€” adding tools created unsafe actions that only the guardrail layer removed.
  • Reliability and safety are separate goals needing separate layers; verification and retries did not reduce unsafe actions at all in our run.
Sources: OpenAI's "Harness engineering: leveraging Codex in an agent-first world" (February 11, 2026, as reported in secondary coverage); SWE-agent (Yang et al., 2024: 12.5% vs 3.8% on SWE-bench). The ablation's model is simulated; all "our run" figures are real outputs of the included script.

๐Ÿ”— References