LLM Harness Engineering: Engineering the System Around the Model
The same model can be a chatbot that invents answers or an agent that reliably resolves tickets. The difference is the harness โ the context, tools, environment, controls and feedback loops built around it. This piece defines harness engineering, maps its components and flow, and measures the claim directly: one unchanged model, five harnesses, from bare chat to production-ready.
What Is Harness Engineering?
A harness is the system around the model that makes it useful. It provides context, tools, memory and an environment; guides multi-step behaviour and planning; adds verification, safety and guardrails; handles errors, retries and recovery; integrates with real-world systems โ and so turns a foundation model into a reliable product. The term gained currency when OpenAI described "harness engineering" in February 2026: an internal team built and shipped a product of roughly a million lines of code written by Codex agents, with the engineers' job being to design the environment โ the harness โ that made that reliable.
Model vs Harness
The model (unchanged)
Pretrained weights ยท general capabilities ยท the same model in every example below ยท not aware of your specific tools, data or policies.The harness (everything else)
Instructions and context ยท memory and state management ยท tools and external systems ยท planning and orchestration ยท verification and guardrails ยท error handling and recovery ยท observability and evaluation.The best-known published demonstration is SWE-agent (Yang et al., 2024). With the same underlying model, a purpose-built agentโcomputer interface โ commands for viewing, searching and editing files designed for a model rather than a human โ solved 12.5% of SWE-bench issues, against 3.8% for the previous best approach, which used retrieval. The weights didn't change; the harness did.
Same Model, Five Harnesses
We measured the claim with an ablation. One simulated model with fixed habits โ it invents answers when it has no data, malforms tool arguments 15% of the time, sometimes claims it acted when it didn't, and follows instructions injected into data 30% of the time โ ran 400 support tasks ("analyze the customer's data and send a response"), 80 of them containing an injected instruction. Each harness adds one layer:
| Harness | Goal met | Grounded answer | Action taken | Unsafe actions | Model calls / task |
|---|---|---|---|---|---|
| bare chat | 0.0% | 0.0% | 0.0% | 0 | 1.00 |
| + context & RAG | 0.0% | 88.8% | 0.0% | 0 | 1.00 |
| + tools | 71.8% | 91.2% | 79.0% | 13 | 1.92 |
| + verification & retry | 99.0% | 99.2% | 99.8% | 20 | 2.24 |
| + guardrails & approval | 98.0% | 98.5% | 98.0% | 0 | 2.23 |
Each layer fixes some failures and exposes others, and reliability and safety need different layers. That is harness engineering in one table.
Key Goals of a Harness
| Goal | Meaning |
|---|---|
| Capability | Give access to data and tools |
| Reliability | Handle errors and uncertainty |
| Control | Enforce policies and permissions |
| Verifiability | Check accuracy and grounding |
| Adaptability | Support different domains and use cases |
| Maintainability | Monitor, evaluate and improve over time |
High-Level Architecture
The harness sits between the user or application, external systems and the model. Inside it: context, planning, memory, tools, state, verification, policies and observability. It connects the foundation model to external systems (APIs, databases, files, SaaS) and to an environment for doing things safely (sandboxes, code execution, browsers). The model is one component; the harness decides everything it sees and everything it's allowed to do.
The End-to-End Harness Flow
- User goal โ "Analyze customer data and draft a response."
- Context builder โ system instructions, conversation history, retrieved knowledge, relevant memory, tool definitions, the current message.
- Planner / orchestrator โ decompose the task, decide the next action, track progress, choose tools or subagents, handle dependencies.
- Tool and environment execution โ select and validate the tool, check permissions, execute via MCP or APIs, use a sandbox when needed, get a structured result.
- Observe and update โ process the result, update memory and task state, check for errors, decide the next step.
- Verify and guardrail โ check grounding, validate output, enforce safety policies, request human approval if needed, detect and handle failures.
- Final result โ complete the goal, return a verified answer, update memory, log traces and metrics, feed the improvement loop. Otherwise replan, retry or continue until the goal is complete.
Core Components of a Harness
| Component | Addresses | Deep dive |
|---|---|---|
| Context engineering | Assemble the right information: instructions, history, memory, RAG, tools | Context Engineering |
| Memory and state | Working, episodic, semantic and artifact memory; checkpoints; long-term storage | Memory & Planning |
| Planning and orchestration | Task decomposition, tool selection, multi-step control | Agentic Systems |
| Tools and integrations | APIs, databases, files, code execution, browsers via MCP or connectors | Tool Calling, MCP |
| Environment and sandboxing | Isolated execution for code, shell, browsers and risky actions | Production Agents |
| Verification and evaluation | Check correctness, grounding and constraint compliance | Evaluating Agents |
| Permissions and safety | Authentication, authorization, tool restrictions, human approvals | Tool security |
| Error handling and recovery | Retries, replanning, fallbacks, exception handling | Reliability |
| Observability and logging | Traces, tool calls, latency, tokens, cost, success rates | Observability |
| Continuous improvement | Use failures and feedback to improve prompts, tools, policies or fine-tuned models | Improvement loop |
Handling Failures
Failures occur โ tool errors, invalid output, missing data, timeouts. The harness detects and classifies them (what type, what the error message says, whether retry is possible), picks a recovery strategy (retry with backoff, fix arguments and retry, use an alternative tool, replan with more context, ask the user for clarification, escalate to a human), and continues execution from the last checkpoint toward the goal. In our ablation, the retry layer turned malformed tool calls from failed tasks into recovered ones, and checking the claimed action against what actually happened caught the model saying "sent" when it hadn't.
Safety, Permissions and Control
Keep the model within the right boundaries: authentication and identity (SSO, service accounts); authorization with tool-level access control and least privilege; policy enforcement on actions and outputs; sandboxing for code, shell and browsers; human approval for sensitive actions; sensitive-data handling to prevent leakage; audit logging of actions, tool calls and decisions; and content safety filtering. The ablation's lesson applies here: these controls address a different failure class from reliability features, and a harness needs both.
Observe, Evaluate, Improve
Observability shows what the harness is doing: traces (end-to-end execution timelines), model calls (prompts, responses, tokens, latency), tool calls (inputs, outputs, success or failure), metrics (cost, latency, success and error rates), dashboards and alerts. Evaluation measures the harness itself: task success, trajectory quality, tool correctness, groundedness, safety compliance, reliability, efficiency and human intervention rate. The continuous improvement loop โ collect data, analyze failures, update prompts, tools, policies or models, test with regression suites and benchmarks, redeploy โ is how a harness gets better, and it's usually cheaper and faster than changing the model.
Worked Example: A Customer Support Agent
- Receive ticket โ a customer reports an issue with their account.
- Retrieve data โ RAG and tools fetch the customer profile, past tickets and account details.
- Analyze โ identify the root cause using the retrieved data and reasoning.
- Draft response โ a personalized, accurate response using company guidelines.
- Verify โ check tone, grounding and policy compliance.
- Approve โ request human approval if needed.
- Take action โ send the response via email or CRM and update the ticket.
- Finish โ confirm success, update memory and log the trace.
Every step except "analyze" and "draft" is harness, not model โ which is why teams that ship reliable agents spend most of their effort there.
Run the Ablation
One script, standard library only, runs instantly.
"""Same model, five harnesses: an ablation study of harness engineering.
One simulated model with fixed habits - it invents answers when it has no data, sometimes malforms tool
arguments, sometimes claims it did something it didn't, and sometimes follows instructions injected into
the data it reads - runs 400 customer-support tasks inside five harnesses, each adding one layer.
The model never changes; only the system around it does. Standard library only.
"""
import random
class Model:
"""The unchanged model: fixed probabilities for each habit."""
def __init__(self, rng): self.rng = rng
def answer(self, facts):
if not facts: return "invented" # no data -> plausible hallucination
return "correct" if self.rng.random() < 0.90 else "wrong detail"
def tool_args(self): return "valid" if self.rng.random() < 0.85 else "malformed"
def claims_done_without_acting(self): return self.rng.random() < 0.08
def follows_injection(self): return self.rng.random() < 0.30
LEVELS = ["bare chat", "+ context & RAG", "+ tools", "+ verification & retry", "+ guardrails & approval"]
def run_task(level, injected, rng):
m, calls = Model(rng), 0
facts = level >= 1 # context builder retrieves ticket + policy
ans = m.answer(facts); calls += 1
if level >= 3 and ans == "wrong detail": # verify the draft against the sources
ans = m.answer(facts); calls += 1
sent_to, leaked = None, False
if level >= 2: # tools: the harness can act
if injected and m.follows_injection():
if level >= 4: pass # guardrail: flagged as untrusted, blocked
else: leaked = True
if m.claims_done_without_acting() and level < 3:
pass # says "sent" but never called the tool
else:
for attempt in range(3 if level >= 3 else 1): # retries only with verification layer
calls += 1
if m.tool_args() == "valid": sent_to = "customer"; break
if level >= 4 and ans != "correct": sent_to = None # approval step rejects a bad draft
goal = ans == "correct" and sent_to == "customer"
return {"goal": goal, "grounded": ans == "correct", "acted": sent_to is not None,
"unsafe": leaked, "calls": calls}
rng = random.Random(11)
tasks = [rng.random() < 0.2 for _ in range(400)] # 20% of tickets contain an injected instruction
print(f"400 support tasks ('analyze the customer's data and send a response'), {sum(tasks)} with injected instructions\n")
print(" harness goal met grounded answer action taken unsafe actions model calls/task")
rows = []
for lvl, name in enumerate(LEVELS):
r = [run_task(lvl, inj, rng) for inj in tasks]
s = {k: sum(x[k] for x in r) for k in ("goal", "grounded", "acted", "unsafe", "calls")}
rows.append(s)
print(f" {name:<26} {s['goal'] / 4:>7.1f}% {s['grounded'] / 4:>14.1f}% {s['acted'] / 4:>11.1f}% "
f"{s['unsafe']:>14} {s['calls'] / 400:>15.2f}")
print("\nthe model object and its probabilities were identical in every row; only the harness changed")Output (deterministic โ fixed random seed):
400 support tasks ('analyze the customer's data and send a response'), 80 with injected instructions
harness goal met grounded answer action taken unsafe actions model calls/task
bare chat 0.0% 0.0% 0.0% 0 1.00
+ context & RAG 0.0% 88.8% 0.0% 0 1.00
+ tools 71.8% 91.2% 79.0% 13 1.92
+ verification & retry 99.0% 99.2% 99.8% 20 2.24
+ guardrails & approval 98.0% 98.5% 98.0% 0 2.23
the model object and its probabilities were identical in every row; only the harness changedโ ๏ธ Notes
- The "weak vs strong harness" panel suggests one jump. In practice each layer fixes some failures and introduces or reveals others โ adding tools created unsafe actions that only the guardrail layer removed.
- Reliability and safety are separate goals needing separate layers; verification and retries did not reduce unsafe actions at all in our run.