Home โ€บ Blog โ€บ Harness & Scaffolding Research
Harness Engineering (Agentic Scaffoldings) ยท Part 1 ยท Snapshot: September 2026 ๐Ÿงฐ

Harness & Scaffolding Research and Innovation: What's Changing Around the Models

The same model can be a chatbot or an agent that works for hours โ€” the difference is the harness around it. This visual deep dive covers the research frontier of that harness: context engineering, compaction, structured and long-horizon memory, planning and replanning, subagents, dynamic tool loading, sandboxes, verifiers inside the loop, learning from environments and adaptive harnesses โ€” with the 2026 benchmarks that show where agents still fail.

How to read this: every diagram is interactive. Figures labelled "computed" follow the formula shown; figures labelled "illustrative" show the shape of an idea, not a measurement. Benchmark findings and product features come from the papers and documentation linked in References, checked in September 2026. For the fundamentals, see LLM Harness Engineering and Context Engineering & Memory.

01 The Big Picture: From Prompting to Model-Native Harnesses

In 2023 most LLM use was a single turn: prompt in, answer out, no state. Today's agents run inside a harness โ€” the software that decides what the model sees, which tools it can call, what it remembers, how its work is checked and how long it keeps going. Toggle between the two:

โ–ถ Traditional prompting vs a modern harness Toggle to compare; the dashed lines show data flowing.

02 The Context Engineering Frontier

Context is a scarce resource: every token competes for the model's attention, and models use long contexts unevenly โ€” Liu et al. found accuracy drops when the relevant fact sits in the middle of a long input ("lost in the middle"). Anthropic's guidance frames the goal as finding "the smallest possible set of high-signal tokens" for each step. The techniques:

โฑ๏ธ Just-in-time retrievalKeep lightweight references (file paths, queries, links) and load content on demand, instead of pre-loading everything.
๐ŸŽฏ Selective loadingLoad only relevant documents, code or memories for this step.
๐Ÿ’พ Prompt cachingReuse the stable prefix (system prompt, tools, documents) across calls โ€” cheaper and faster.
๐Ÿ—œ๏ธ Token-efficient representationsStructured, compressed tool outputs; return IDs and summaries, not raw dumps.
๐Ÿ“Š Explicit token budgetsAllocate the window across instructions, retrieved content, history, tools and output.
โ–ถ Dynamic context assembly A context selector fills the window each step. Drag the budgets; watch what fits.
Computed from illustrative per-item sizes you can see in the legend: instructions 3,000 tokens, each tool schema 500, each document chunk 1,500, each history turn 800, output reserve 8,000. Real sizes vary widely; the point is that tools and history grow quietly until they crowd out everything else.

03 Compaction and Context Compression

Long tasks produce histories many times larger than any window. Compaction summarizes a trajectory nearing the limit and restarts with the summary โ€” Anthropic describes it, with structured note-taking and multi-agent designs, as its main tool for long-horizon work. The question is what to keep:

TypeExamplesApproach
Raw historyFull chat, tool logs, intermediate thoughtsSummarize; drop low value
Task stateCurrent goal, progress, decisionsKeep structured, lossless
EvidenceKey documents, outputs, resultsKeep selectively, with citations
Unresolved workOpen questions, next stepsKeep structured
NoiseRepetitive, low-value steps; old tool outputsCompress or discard (clearing old tool results is the safest first cut)
โ–ถ Watch a trajectory get compacted Each block is a step; gold blocks hold something the agent must remember. Press play.

The same experiment as a runnable simulation over 50 random 300-step trajectories:

"""Four ways to keep a long agent trajectory inside a fixed context window.

A simulated 300-step coding task. Every step adds a tool result to the history (most are noise: logs,
file listings, retries). A few steps produce things the agent must still know at the end: decisions,
constraints, facts, open questions. At the end the agent is asked about all of them.

Strategies, with a 32,000-token window:
  none        - keep everything; the task fails once history exceeds the window
  truncate    - drop the oldest steps when full (sliding window)
  summarize   - when full, compress old history into a summary; each summary pass keeps an item with p=0.8
  structured  - when full, move key items into a structured state block (decisions, constraints, facts,
                open questions) - each item is extracted with p=0.95, then kept losslessly; compress the rest
Standard library only.
"""
import random

WINDOW, STEPS, SUMMARY_KEEP, EXTRACT = 32_000, 300, 0.8, 0.95
KINDS = ["decision", "constraint", "fact", "open question"]

def trajectory(seed):
    rng = random.Random(seed)
    steps = []
    for i in range(STEPS):
        tokens = rng.randint(150, 900)                    # tool outputs vary in size
        key = rng.choice(KINDS) if rng.random() < 0.07 else None
        steps.append((i, tokens, key))
    return steps

def run(strategy, steps, rng):
    history, state, summary_items, summary_tokens, peak, failed, passes = [], [], [], 0, 0, False, 0
    for step in steps:
        history.append(step)
        used = sum(t for _, t, _ in history) + summary_tokens + 60 * len(state)
        if used > WINDOW:
            if strategy == "none":
                failed = True; break
            old, history = history[: len(history) // 2], history[len(history) // 2:]   # compact the older half
            passes += 1
            if strategy == "truncate":
                pass                                               # key items in dropped steps are lost
            elif strategy == "summarize":
                summary_items = [k for k in summary_items if rng.random() < SUMMARY_KEEP]
                summary_items += [s for s in old if s[2] and rng.random() < SUMMARY_KEEP]
                summary_tokens = 1_500
            elif strategy == "structured":
                state += [s for s in old if s[2] and rng.random() < EXTRACT]   # extracted once (p=0.95), then kept losslessly
                summary_tokens = 800
        used = sum(t for _, t, _ in history) + summary_tokens + 60 * len(state)
        peak = max(peak, used)
    keys = [s for s in steps if s[2]]
    known = {s[0] for s in history if s[2]} | {s[0] for s in summary_items} | {s[0] for s in state}
    recalled = 0 if failed else sum(1 for s in keys if s[0] in known)
    return failed, recalled, len(keys), peak, passes

print(f"{STEPS}-step trajectory, {WINDOW:,}-token window; results over 50 random trajectories\n")
print(f"   {'strategy':<12}{'finished':>10}{'key items recalled':>22}{'peak context':>15}{'compactions':>13}")
for strategy in ["none", "truncate", "summarize", "structured"]:
    rng = random.Random(1)
    res = [run(strategy, trajectory(seed), rng) for seed in range(50)]
    done = sum(not f for f, *_ in res)
    rec = sum(r for _, r, _, _, _ in res) / max(1, sum(k for _, _, k, _, _ in res))
    peak = sum(p for *_, p, _ in res) / len(res)
    comp = sum(c for *_, c in res) / len(res)
    print(f"   {strategy:<12}{done:>7}/50{rec:>21.0%}{peak:>12,.0f} tok{comp:>12.1f}")
total = sum(t for _, t, _ in trajectory(0))
print(f"\nThe raw history of one trajectory is ~{total:,} tokens, {total / WINDOW:.1f}x the window.")

Output:

300-step trajectory, 32,000-token window; results over 50 random trajectories

   strategy      finished    key items recalled   peak context  compactions
   none              0/50                   0%      31,681 tok         0.0
   truncate         50/50                  17%      31,940 tok         8.1
   summarize        50/50                  47%      31,947 tok         8.8
   structured       50/50                  95%      31,947 tok         8.8

The raw history of one trajectory is ~153,539 tokens, 4.8x the window.

Every strategy that compacts finishes the task; they differ in what the agent still knows. Truncation silently forgets early decisions. Repeated free-text summaries lose a little more on each pass โ€” information decays geometrically. Writing key items to a structured state block once, then keeping it losslessly, preserves nearly everything. The retention probabilities are assumptions; the ranking is the lesson.

04 Structured Memory for Agents

Agents increasingly separate memory into types, each with its own write and retrieve policy โ€” an idea running from MemGPT's operating-system-style memory tiers (2023) and Generative Agents' memory stream to today's products, such as Anthropic's memory tool and file-based project memories. Click a memory type:

โ–ถ The memory flow Events โ†’ memory manager โ†’ typed stores โ†’ retrieval for the current task.

05 Long-Horizon Memory Research (2026)

Three 2026 benchmarks probe memory in agents that act, not just chat โ€” and each finds serious gaps.

  • Microsoft Research, May 2026
  • Memory formed during agent execution across tool-use dialogue (ฯ„ยฒ-bench), deep research, coding (SWE-Gym) and computer use (WebArena-Infinity)
  • Reports memory-isolated scores, separating memory from reasoning and tool skill
  • Finding: in its pilot, no memory strategy raised resolve rates overall; gains depended on the domain (e.g. summary memory +17.5 points on telecom, structured memory +14.0 on airline)
  • July 2026
  • Prospective memory: acting on an intention at a future cue ("remind me when X") while doing other work, across a simulated seven-day week
  • Finding: the best configuration (a GPT-5.4 agent) reached only 65.1% F1, and no single strategy won across models
  • Tongyi Lab, August 2026
  • 1,705 tasks, 212 tools, 13 domains in a stateful mobile sandbox where data, permissions and device state change after every action
  • Scores tool use, memory usage, skill usage and sub-agent collaboration
  • Finding: basic tool use is strongest (13-model mean 76.6%); overall scores sit around 57%

Prospective memory is the least familiar of these ideas, so here it is as a timeline:

โ–ถ Delayed intentions On Monday the user asks: "When the invoice arrives, forward it to accounting" and "Remind me on Thursday to book flights." Illustrative.

06 Planning and Replanning

Agents break goals into steps, act, observe, and revise. The research frontier is less about making a plan than about changing it well: noticing a broken tool, a failed step or an impossible task, and replanning โ€” or refusing.

โ–ถ The planning loop, with a failure Goal: "Book the cheapest refundable flight to Berlin and add it to my calendar."
๐ŸŒณ Goal decompositionHierarchical plans: goals โ†’ subgoals โ†’ actions.
๐Ÿ” Feedback-conditioned planningUpdate the plan from each result (ReAct-style interleaving of reasoning and acting).
๐Ÿšซ Infeasibility detectionRecognize impossible or underspecified tasks and ask or refuse instead of inventing a solution.
๐Ÿ› ๏ธ Robustness to tool noiseRetry, switch to an alternative tool, or route around a broken one.
๐Ÿ›‘ Stop conditionsKnow when to finish, escalate to a human or give up.
2026 evidence โ€” Agent Planning Benchmark (APB, June 2026): 4,209 multimodal cases across 22 domains and five settings โ€” holistic planning, feedback-conditioned step-wise planning, and robustness to extraneous tools, broken tools and unsolvable tasks. Across 12 models it finds systematic weaknesses in long-horizon planning, tool-noise robustness, calibrated refusal and inference-time refinement; using its diagnostics to refine plans improved execution on ToolSandbox and ฯ„ยฒ-bench tasks.

07 Subagents and Delegation

A lead agent can hand focused sub-tasks to subagents, each with a clean context window, then combine their condensed results. The lead's context stays small; the total work is spread out โ€” and total token use rises.

โ–ถ Context windows under delegation Each subagent explores in its own window and returns a short result. Illustrative sizes, computed totals.
Documented example โ€” Anthropic's multi-agent research system (June 2025): a lead agent (Claude Opus 4) directing parallel subagents (Claude Sonnet 4) outperformed a single Opus 4 agent by 90.2% on Anthropic's internal research evaluation. Anthropic found token usage alone explained 80% of the performance variance, and multi-agent runs used about 15ร— more tokens than chats โ€” worth it for broad, parallelizable research, not for tightly coupled tasks such as most coding.

08 Tool Discovery and Dynamic Capability Loading

Loading every tool definition up front wastes context and confuses tool choice once there are dozens of tools. The frontier is to discover and load capabilities at runtime: search a tool registry, fetch MCP servers' tools on demand, load skills (folders of instructions and scripts) only when relevant, and let the model compose tools in code.

โ–ถ Load everything vs search on demand Tokens spent on tool definitions before the model does any work. Sizes are adjustable assumptions.
Upfront = tools ร— schema size. On demand = a search tool (~500 tokens, assumed) + the tools actually loaded. Anthropic reports that its Tool Search Tool (with defer_loading) cut tool-definition tokens by about 85% in internal tests while improving tool-selection accuracy, e.g. Opus 4.5 from 79.5% to 88.1% on an MCP evaluation.
๐Ÿงฉ Built-in toolsCode execution, search, editing.
๐Ÿ”Ž Tool searchFind relevant tools by description when needed.
๐Ÿ”Œ Remote MCP serversExternal capabilities over the Model Context Protocol (see MCP).
๐Ÿ“š SkillsAgent Skills (Anthropic, October 2025): folders of instructions, scripts and resources loaded progressively โ€” only a short description sits in context until needed.
๐Ÿงฎ Programmatic compositionThe model writes code that chains tools, so intermediate results never enter the context (programmatic tool calling; CodeAct).

09 Sandboxed Environments and Computer Use

Agents that run code, browse and operate computers need an environment where their actions are real but contained. A sandbox gives them a file system, shell, browser or GUI and APIs โ€” behind explicit permissions for network, time, resources and file access. Toggle the permissions and watch what gets through:

โ–ถ Sandbox boundary The agent attempts six actions. Click a permission to allow or deny it.
Why it matters: computer-use benchmarks such as OSWorld and WebArena put agents in real operating systems and websites, and SWE-agent showed that the agentโ€“computer interface โ€” how commands and file views are presented โ€” changes success rates substantially. Sandboxes are also the main defence against prompt injection turning into damage: an injected instruction can only do what the sandbox permits.

10 Verification and Evaluators Inside the Loop

Rather than trusting a single output, harnesses check work before returning it: run the tests, validate the schema, check claims against sources, ask a critic model. If a check fails, feedback goes back to the model for another attempt. How much does that help? It depends on how good the verifier is:

โ–ถ Retry until the verifier passes Computed exactly for independent attempts.
Each attempt is good with probability p. A good output passes with probability (1 โˆ’ false-reject rate); a bad one slips through with probability (1 โˆ’ catch rate). The loop returns the first output that passes, or the last attempt. Final correct = ฮฃ over attempts of P(reach attempt) ร— p ร— (1 โˆ’ f), plus the last attempt's unchecked result. Real attempts are correlated, so real gains are smaller.
๐Ÿง Verifier / critic agentsA second model checks reasoning and claims.
๐Ÿงช Environment checksRun code, execute tests, inspect the resulting state.
๐Ÿ“ Schema validationStructured outputs guaranteed to parse.
๐Ÿ“Ž Source groundingAnswers must cite retrieved evidence.
โœ… Evidence-based completion"Done" requires proof โ€” test output, a changed state โ€” not the model's say-so.

11 Agent Learning from Environments

Harnesses are becoming places where agents learn, not just act: reinforcement learning on tool outcomes, training in executable environments with verifiable rewards, reflection on failures, curricula of harder tasks, and skill libraries that grow with experience.

โ–ถ A growing skill library Voyager-style: solve a task, verify it, save the working code as a reusable skill. Illustrative.
๐ŸŽฏ RL with tool outcomesReward success and failure of real tool use; now a standard stage of frontier post-training.
๐Ÿ—๏ธ Executable environmentsSimulated tasks and real tools with verifiable rewards (SWE-Gym, ฯ„ยฒ-bench-style environments; MobilePA-Bench is designed to double as an RL foundation).
๐Ÿชž Self-improvement from experienceReflect on failures and update strategy (Reflexion, 2023).
๐Ÿ“ˆ Curriculum learningIncrease task complexity as the agent improves (Voyager's automatic curriculum).
โš–๏ธ Value models & process supervisionReward good intermediate steps, not just final answers.

12 Adaptive Harnesses

A fixed harness over-spends on easy tasks and under-protects risky ones. Adaptive harnesses choose their configuration per task. Set the task and see what an adaptive policy would pick (an illustrative rule set, not a product):

โ–ถ Configure by task Four factors in, a harness configuration out.

13 Frontier Benchmark Evidence (2026)

Recent benchmarks agree: the weak points are long horizons, state and tool use โ€” exactly the harness's job.

BenchmarkWhat it testsScaleKey finding
Agent Planning Benchmark (Jun 2026)Holistic and step-wise planning; extraneous and broken tools; unsolvable tasks4,209 cases, 22 domains, 12 modelsSystematic weaknesses in long-horizon planning, tool-noise robustness and calibrated refusal
MemGym (May 2026)Memory formed during execution: tool-use dialogue, research, coding, computer use5 tracks, 4 regimesMemory strategy gains are domain-dependent; aggressive compression alone doesn't help
PM-Bench (Jul 2026)Prospective memory: delayed intentions and monitoring over a simulated week8 models ร— 8 agent configurationsBest: 65.1% F1; no strategy dominates
MobilePA-Bench (Aug 2026)Stateful mobile planning: tools, memory, skills, sub-agents, permissions1,705 tasks, 212 tools, 13 modelsBasic tool use ~77% mean; overall ~57% โ€” advanced dimensions lag

14 Open Research Problems

1Faithful long-term memory (accurate, non-hallucinated)
2Preventing memory contamination (irrelevant or wrong memories)
3Robust plan repair under failures and environment shifts
4Scalable, generalizable verification
5Security: prompt and tool injection, data exfiltration, safe tool use
6Uncertainty calibration: when to ask, refuse or escalate
7Credit assignment in long, sparse-reward trajectories
8Reliable termination: no loops, genuine completion
9Coordinating subagents: division of labour, conflict resolution
10Evaluating real-world task completion beyond exact-answer metrics

15 Research Frontier Map: Evolution of the Harness

1Promptingsingle turn
2Context engineeringretrieval, compression
3Tool-using agentfunction calling
4Stateful agentmemory, basic planning
5Long-horizon agentplanning, replanning
6Subagent & skill ecosystemdelegation, dynamic tools
7Verified persistent agentlearning, self-improvement
โœ“Less reliance on giant prompts; more context compaction and structured memory.
โœ“Dynamic tool discovery (MCP, tool search, skills); sandboxed execution and computer use.
โœ“Subagents and multi-agent systems for broad tasks.
โœ“Long-running, stateful tasks with benchmarked planning and memory; verifier-driven completion criteria.
The harness frontier in one paragraph
Agent capability in 2026 comes as much from the harness as from the model. The research frontier treats context as a budget to engineer, compacts long trajectories into structured state rather than lossy summaries, splits memory into typed stores with explicit write and retrieve policies, plans with the expectation of failure, delegates to subagents with clean contexts, loads tools and skills only when needed, runs actions inside permissioned sandboxes, and checks work with verifiers before claiming success โ€” increasingly adapting all of this to each task. The 2026 benchmarks show how far there is to go: agents still forget delayed intentions, lose state, mishandle broken tools and fail to refuse impossible tasks. Those failures are now the research agenda.

Next: Harness Production Engineering โ€” durable runtimes, sandboxes, approvals, tracing and reliability for agents in production.

โš ๏ธ Notes

The four 2026 benchmarks named in the source infographic are real and correctly attributed in spirit; refinements made here: (1) MemGym's headline result is not simply that "agents forget earlier context" โ€” its pilot found that no memory strategy improved resolve rates overall and that gains were domain-dependent, and its contribution is memory-isolated scoring of memory formed during execution; (2) MobilePA-Bench is broader than state loss: it scores tool use, memory, skills and sub-agent collaboration in a stateful mobile sandbox, with overall scores around 57%; (3) the token budget table (instructions 20%, retrieved content 40%, and so on) is an illustrative allocation, not a published standard; (4) in the Sandbox panel "browser / computer use" and "GUI" describe the same capability class. Interactive figures marked illustrative show ideas, not measurements; the compaction simulation's retention rates are stated assumptions.

๐Ÿ”— References

Agent Planning Benchmark: A Diagnostic Framework for Planning Capabilities in LLM Agents (arXiv:2606.04874) MemGym: a Long-Horizon Memory Environment for LLM Agents (arXiv:2605.20833) Microsoft Research โ€” MemGym publication page PM-Bench: Evaluating Prospective Memory in LLM Agents (arXiv:2607.12385) MobilePA-Bench: Benchmarking Mobile Planner Agents on Complex Real-World Tasks (arXiv:2608.23035) Tongyi-MAI โ€” MobilePA-Bench repository Anthropic โ€” Effective context engineering for AI agents Anthropic โ€” How we built our multi-agent research system Anthropic โ€” Advanced tool use (Tool Search Tool, programmatic tool calling) Claude Platform Docs โ€” Tool search tool Anthropic โ€” Equipping agents for the real world with Agent Skills Anthropic โ€” Building effective agents Liu et al. โ€” Lost in the Middle: How Language Models Use Long Contexts (arXiv:2307.03172) Packer et al. โ€” MemGPT: Towards LLMs as Operating Systems (arXiv:2310.08560) Park et al. โ€” Generative Agents (arXiv:2304.03442) Yao et al. โ€” ReAct: Synergizing Reasoning and Acting (arXiv:2210.03629) Shinn et al. โ€” Reflexion (arXiv:2303.11366) Wang et al. โ€” Voyager: An Open-Ended Embodied Agent (arXiv:2305.16291) Wang et al. โ€” Executable Code Actions Elicit Better LLM Agents (CodeAct) (arXiv:2402.01030) Yang et al. โ€” SWE-agent: Agent-Computer Interfaces (arXiv:2405.15793) Xie et al. โ€” OSWorld (arXiv:2404.07972) Zhou et al. โ€” WebArena (arXiv:2307.13854) Model Context Protocol โ€” specification and documentation