The Top of the Stack: Agent Runtimes, Tool Protocols and the Software That Lets Models Act
A model on its own only produces text. An agent runtime wraps it in a loop: decide, call a tool, read the result, decide again, until the task is done. Around that loop sit protocols that connect models to tools (MCP) and to other agents (A2A), frameworks that structure the work, sandboxes that keep code execution safe, and tracing that shows what happened. This final part of the series explains that top layer, and how agents feed back into every layer below.
1. Case file
2. Timeline
From ReAct to foundations, 2022 โ 2026
Tap a milestone. Colour: research, frameworks, lab SDKs and APIs, protocols and foundations.
3. The agent loop
Almost every agent, however elaborate, is the loop the ReAct paper described in 2022: the model reasons, chooses an action, the runtime executes it and returns an observation, and the model continues. The runtime's job is everything the model cannot do itself: run tools, keep state, stop runaway loops, retry failures, ask a human when needed, and keep the context window from overflowing by summarising or dropping old material ("compaction").
Watch an agent fix a bug
A simplified coding task. Step through; the gauge shows the context window filling, then being compacted. Illustration.
4. Tool calling, from model to engine
Tool use starts in training: models are post-trained (Part 5) to emit structured calls, a tool name plus JSON arguments matching a schema. The serving engine (Part 7) can enforce the schema with constrained decoding, masking tokens that would break the JSON, which SGLang and vLLM support. Models can now request several calls at once ("parallel tool calls"), and the runtime runs them concurrently. Because every tool definition sits in the prompt, large tool sets cost tokens on every call; runtimes increasingly load tools on demand.
5. MCP and A2A
Why a protocol: N ร M versus N + M
Without a shared protocol, every AI app needs a custom integration for every tool. With one, each side implements the protocol once.
An MCP exchange
Simplified JSON-RPC messages between a client and a server. Step through.
6. Runtime patterns
Four ways to structure an agent
Tap a pattern.
7. The frameworks
Two camps have formed. The lab SDKs (OpenAI's, Anthropic's, Google's) are thin and opinionated, built around their own models and hosted tools, though most accept other models. The independent frameworks (LangGraph, CrewAI, LlamaIndex, Microsoft's) are model-agnostic and add orchestration, state and integrations. All have converged on the same primitives: tools, handoffs or sub-agents, guardrails, memory, MCP support and tracing.
8. Feature matrix
Our reading of documentation, October 2026. โ built in ยท โ partial or via integration ยท โ not a focus. Tap a cell.
9. Sandboxes and isolation
Agents that write and run code, browse or control computers must be isolated from the systems they run on. Options trade isolation strength against start-up time and overhead. AWS built Firecracker (paper at NSDI 2020) for Lambda; its specification targets about 125 ms from start to the guest's init process with under 5 MiB of memory overhead, and it underlies several agent-sandbox services.
Isolation vs start-up
A qualitative illustration. Positions are our judgement, not measurements. Tap a point.
10. Agents at AI-factory scale
Agents change the economics of everything below them. One task can mean dozens of model calls, each re-sending a growing context, so prefix caching (Part 7) becomes essential: most of each call is identical to the last. Long-running agents hold KV cache for minutes, shifting serving toward memory. And every agent run is potential training data: agentic RL (Part 5) trains models on exactly these loops, judged by agent evaluations (Part 6).
What does one agent task cost?
Our own model: context grows linearly each step; cached input is billed at a discount. Prices are inputs, not quotes.
11. Observability and safety
- Tracing: each agent run becomes a tree of spans (agent invocation, model calls, tool executions). OpenTelemetry's GenAI semantic conventions define span types such as
invoke_agentandexecute_tool; they are still in development, and vendor tools (LangSmith, Langfuse, Arize, the lab SDKs' built-in tracing) fill the gap. - Guardrails: input and output checks that run alongside the agent and can stop it; built into the OpenAI Agents SDK and others.
- Permissions: which tools an agent may call without asking, with human approval for destructive or external actions.
- Evaluation: agent benchmarks and production replays (Part 6) catch regressions when models, prompts or tools change.
12. Choosing a runtime
Which runtime fits?
An editorial guide.
13. The whole stack
Inside the ML Layer: the series map
Eight parts, from the top of the stack to the GPU. Tap to open a part.
An agent answering a question touches every layer in this series: its runtime calls a model served by an inference engine using kernels on GPUs, a model that was post-trained with RL, checked by evaluation harnesses, trained with parallelism libraries on a framework. Part 1's trace followed that path from the bottom; this series has now opened each box. The companion series, Inside the Data Layer, follows the data that feeds them.
14. Reading the numbers
- Primary
- Agentic AI Foundation formed 9 Dec 2025 with MCP, goose and AGENTS.md; 10,000+ MCP servers; platinum members include AWS, Anthropic, Block, Bloomberg, Cloudflare, Google, Microsoft, OpenAI (Linux Foundation). A2A Linux Foundation project announced 23 Jun 2025 (Linux Foundation). Claude Code SDK renamed Claude Agent SDK, 29 Sep 2025 (Anthropic). Firecracker ~125 ms to guest init, <5 MiB overhead (Firecracker specification).
- Secondary โ ๏ธ
- Microsoft Agent Framework (Oct 2025, unifying AutoGen and Semantic Kernel) per third-party coverage. OpenTelemetry GenAI conventions status per third-party write-ups.
- Our illustrations
- Agent loop and context gauge, integration counts, MCP messages (simplified), pattern diagrams, sandbox positions and the cost calculator.
- Papers
- ReAct (2022) ยท Toolformer (2023)
- Firecracker (NSDI 2020)
- Protocols and foundations
- Model Context Protocol specification
- Linux Foundation: Agentic AI Foundation
- Linux Foundation: Agent2Agent project
- Frameworks
- OpenAI Agents SDK
- Anthropic: building agents with the Claude Agent SDK
- Google Agent Development Kit
- LangGraph
- Microsoft Agent Framework overview (third party)
- OpenTelemetry GenAI conventions (write-up)
- Part 7: inference engines ยท Part 1: the map