Harness & Scaffolding Research and Innovation: What's Changing Around the Models
The same model can be a chatbot or an agent that works for hours โ the difference is the harness around it. This visual deep dive covers the research frontier of that harness: context engineering, compaction, structured and long-horizon memory, planning and replanning, subagents, dynamic tool loading, sandboxes, verifiers inside the loop, learning from environments and adaptive harnesses โ with the 2026 benchmarks that show where agents still fail.
01 The Big Picture: From Prompting to Model-Native Harnesses
In 2023 most LLM use was a single turn: prompt in, answer out, no state. Today's agents run inside a harness โ the software that decides what the model sees, which tools it can call, what it remembers, how its work is checked and how long it keeps going. Toggle between the two:
02 The Context Engineering Frontier
Context is a scarce resource: every token competes for the model's attention, and models use long contexts unevenly โ Liu et al. found accuracy drops when the relevant fact sits in the middle of a long input ("lost in the middle"). Anthropic's guidance frames the goal as finding "the smallest possible set of high-signal tokens" for each step. The techniques:
03 Compaction and Context Compression
Long tasks produce histories many times larger than any window. Compaction summarizes a trajectory nearing the limit and restarts with the summary โ Anthropic describes it, with structured note-taking and multi-agent designs, as its main tool for long-horizon work. The question is what to keep:
| Type | Examples | Approach |
|---|---|---|
| Raw history | Full chat, tool logs, intermediate thoughts | Summarize; drop low value |
| Task state | Current goal, progress, decisions | Keep structured, lossless |
| Evidence | Key documents, outputs, results | Keep selectively, with citations |
| Unresolved work | Open questions, next steps | Keep structured |
| Noise | Repetitive, low-value steps; old tool outputs | Compress or discard (clearing old tool results is the safest first cut) |
The same experiment as a runnable simulation over 50 random 300-step trajectories:
"""Four ways to keep a long agent trajectory inside a fixed context window.
A simulated 300-step coding task. Every step adds a tool result to the history (most are noise: logs,
file listings, retries). A few steps produce things the agent must still know at the end: decisions,
constraints, facts, open questions. At the end the agent is asked about all of them.
Strategies, with a 32,000-token window:
none - keep everything; the task fails once history exceeds the window
truncate - drop the oldest steps when full (sliding window)
summarize - when full, compress old history into a summary; each summary pass keeps an item with p=0.8
structured - when full, move key items into a structured state block (decisions, constraints, facts,
open questions) - each item is extracted with p=0.95, then kept losslessly; compress the rest
Standard library only.
"""
import random
WINDOW, STEPS, SUMMARY_KEEP, EXTRACT = 32_000, 300, 0.8, 0.95
KINDS = ["decision", "constraint", "fact", "open question"]
def trajectory(seed):
rng = random.Random(seed)
steps = []
for i in range(STEPS):
tokens = rng.randint(150, 900) # tool outputs vary in size
key = rng.choice(KINDS) if rng.random() < 0.07 else None
steps.append((i, tokens, key))
return steps
def run(strategy, steps, rng):
history, state, summary_items, summary_tokens, peak, failed, passes = [], [], [], 0, 0, False, 0
for step in steps:
history.append(step)
used = sum(t for _, t, _ in history) + summary_tokens + 60 * len(state)
if used > WINDOW:
if strategy == "none":
failed = True; break
old, history = history[: len(history) // 2], history[len(history) // 2:] # compact the older half
passes += 1
if strategy == "truncate":
pass # key items in dropped steps are lost
elif strategy == "summarize":
summary_items = [k for k in summary_items if rng.random() < SUMMARY_KEEP]
summary_items += [s for s in old if s[2] and rng.random() < SUMMARY_KEEP]
summary_tokens = 1_500
elif strategy == "structured":
state += [s for s in old if s[2] and rng.random() < EXTRACT] # extracted once (p=0.95), then kept losslessly
summary_tokens = 800
used = sum(t for _, t, _ in history) + summary_tokens + 60 * len(state)
peak = max(peak, used)
keys = [s for s in steps if s[2]]
known = {s[0] for s in history if s[2]} | {s[0] for s in summary_items} | {s[0] for s in state}
recalled = 0 if failed else sum(1 for s in keys if s[0] in known)
return failed, recalled, len(keys), peak, passes
print(f"{STEPS}-step trajectory, {WINDOW:,}-token window; results over 50 random trajectories\n")
print(f" {'strategy':<12}{'finished':>10}{'key items recalled':>22}{'peak context':>15}{'compactions':>13}")
for strategy in ["none", "truncate", "summarize", "structured"]:
rng = random.Random(1)
res = [run(strategy, trajectory(seed), rng) for seed in range(50)]
done = sum(not f for f, *_ in res)
rec = sum(r for _, r, _, _, _ in res) / max(1, sum(k for _, _, k, _, _ in res))
peak = sum(p for *_, p, _ in res) / len(res)
comp = sum(c for *_, c in res) / len(res)
print(f" {strategy:<12}{done:>7}/50{rec:>21.0%}{peak:>12,.0f} tok{comp:>12.1f}")
total = sum(t for _, t, _ in trajectory(0))
print(f"\nThe raw history of one trajectory is ~{total:,} tokens, {total / WINDOW:.1f}x the window.")Output:
300-step trajectory, 32,000-token window; results over 50 random trajectories strategy finished key items recalled peak context compactions none 0/50 0% 31,681 tok 0.0 truncate 50/50 17% 31,940 tok 8.1 summarize 50/50 47% 31,947 tok 8.8 structured 50/50 95% 31,947 tok 8.8 The raw history of one trajectory is ~153,539 tokens, 4.8x the window.
Every strategy that compacts finishes the task; they differ in what the agent still knows. Truncation silently forgets early decisions. Repeated free-text summaries lose a little more on each pass โ information decays geometrically. Writing key items to a structured state block once, then keeping it losslessly, preserves nearly everything. The retention probabilities are assumptions; the ranking is the lesson.
04 Structured Memory for Agents
Agents increasingly separate memory into types, each with its own write and retrieve policy โ an idea running from MemGPT's operating-system-style memory tiers (2023) and Generative Agents' memory stream to today's products, such as Anthropic's memory tool and file-based project memories. Click a memory type:
05 Long-Horizon Memory Research (2026)
Three 2026 benchmarks probe memory in agents that act, not just chat โ and each finds serious gaps.
- Microsoft Research, May 2026
- Memory formed during agent execution across tool-use dialogue (ฯยฒ-bench), deep research, coding (SWE-Gym) and computer use (WebArena-Infinity)
- Reports memory-isolated scores, separating memory from reasoning and tool skill
- Finding: in its pilot, no memory strategy raised resolve rates overall; gains depended on the domain (e.g. summary memory +17.5 points on telecom, structured memory +14.0 on airline)
- July 2026
- Prospective memory: acting on an intention at a future cue ("remind me when X") while doing other work, across a simulated seven-day week
- Finding: the best configuration (a GPT-5.4 agent) reached only 65.1% F1, and no single strategy won across models
- Tongyi Lab, August 2026
- 1,705 tasks, 212 tools, 13 domains in a stateful mobile sandbox where data, permissions and device state change after every action
- Scores tool use, memory usage, skill usage and sub-agent collaboration
- Finding: basic tool use is strongest (13-model mean 76.6%); overall scores sit around 57%
Prospective memory is the least familiar of these ideas, so here it is as a timeline:
06 Planning and Replanning
Agents break goals into steps, act, observe, and revise. The research frontier is less about making a plan than about changing it well: noticing a broken tool, a failed step or an impossible task, and replanning โ or refusing.
07 Subagents and Delegation
A lead agent can hand focused sub-tasks to subagents, each with a clean context window, then combine their condensed results. The lead's context stays small; the total work is spread out โ and total token use rises.
08 Tool Discovery and Dynamic Capability Loading
Loading every tool definition up front wastes context and confuses tool choice once there are dozens of tools. The frontier is to discover and load capabilities at runtime: search a tool registry, fetch MCP servers' tools on demand, load skills (folders of instructions and scripts) only when relevant, and let the model compose tools in code.
09 Sandboxed Environments and Computer Use
Agents that run code, browse and operate computers need an environment where their actions are real but contained. A sandbox gives them a file system, shell, browser or GUI and APIs โ behind explicit permissions for network, time, resources and file access. Toggle the permissions and watch what gets through:
10 Verification and Evaluators Inside the Loop
Rather than trusting a single output, harnesses check work before returning it: run the tests, validate the schema, check claims against sources, ask a critic model. If a check fails, feedback goes back to the model for another attempt. How much does that help? It depends on how good the verifier is:
11 Agent Learning from Environments
Harnesses are becoming places where agents learn, not just act: reinforcement learning on tool outcomes, training in executable environments with verifiable rewards, reflection on failures, curricula of harder tasks, and skill libraries that grow with experience.
12 Adaptive Harnesses
A fixed harness over-spends on easy tasks and under-protects risky ones. Adaptive harnesses choose their configuration per task. Set the task and see what an adaptive policy would pick (an illustrative rule set, not a product):
13 Frontier Benchmark Evidence (2026)
Recent benchmarks agree: the weak points are long horizons, state and tool use โ exactly the harness's job.
| Benchmark | What it tests | Scale | Key finding |
|---|---|---|---|
| Agent Planning Benchmark (Jun 2026) | Holistic and step-wise planning; extraneous and broken tools; unsolvable tasks | 4,209 cases, 22 domains, 12 models | Systematic weaknesses in long-horizon planning, tool-noise robustness and calibrated refusal |
| MemGym (May 2026) | Memory formed during execution: tool-use dialogue, research, coding, computer use | 5 tracks, 4 regimes | Memory strategy gains are domain-dependent; aggressive compression alone doesn't help |
| PM-Bench (Jul 2026) | Prospective memory: delayed intentions and monitoring over a simulated week | 8 models ร 8 agent configurations | Best: 65.1% F1; no strategy dominates |
| MobilePA-Bench (Aug 2026) | Stateful mobile planning: tools, memory, skills, sub-agents, permissions | 1,705 tasks, 212 tools, 13 models | Basic tool use ~77% mean; overall ~57% โ advanced dimensions lag |
14 Open Research Problems
15 Research Frontier Map: Evolution of the Harness
Next: Harness Production Engineering โ durable runtimes, sandboxes, approvals, tracing and reliability for agents in production.