Seeing Inside Agent Runs: Observability, Tracing and Cost
When a web service fails, it usually throws an error, and dashboards turn red. When an agent fails, it often returns a confident, wrong answer with status 200, after quietly spending forty model calls going round in circles. Ordinary monitoring sees none of that. Agent observability records every model call, tool call and decision as a trace, so people can see what the agent did, why, how long it took and what it cost. This part explains traces and spans for agents, the emerging OpenTelemetry standard, where an agent's tokens actually go, how runaway loops are stopped, and the unit economics every agent product has to get right.
1. Case file
create_agent, invoke_agent, tool execution, model inference2. Silent failures
None of these show up as HTTP errors or CPU alarms. They show up only in the content and shape of the run, so that is what agent observability records. (For monitoring the GPUs and servers underneath, see AI factory observability.)
3. An agent trace
One coding task, as a waterfall
Each bar is a span; time runs left to right. Tap a row for details. Our illustrative trace, assumed prices.
4. The OpenTelemetry standard
OpenTelemetry is the open standard most companies already use for traces and metrics. Its generative-AI working group defines semantic conventions: agreed names for what to record, so that any framework's traces can be read by any backend. They cover model calls (provider, model, input and output tokens, finish reason), agent operations (create_agent, invoke_agent, split into remote and local calls), tool execution and MCP. In 2026 the GenAI, provider-specific and MCP conventions moved into their own repository; the project still marks them as in development. Even unstable, they matter: an agent framework that emits standard spans can be monitored by whichever tool a company already uses, instead of locking it into one vendor's dashboard. Prompts and outputs can be recorded too, but are often switched off or redacted by default because they may contain personal data.
5. What to watch
| Signal | Why it matters | Typical alert |
|---|---|---|
| Task success rate | The real quality number, measured by automatic checks, user feedback or sampled review (Evaluation Layer Part 7) | Drop after a model or prompt change |
| Steps and tokens per task | Efficiency; a rising median often means the agent is struggling | p95 steps above budget |
| Tool error rate | Broken integrations, permission problems, rate limits | One tool above its usual error level |
| Cache hit rate | Cost and latency (Part 6) | Sudden fall after a prompt edit |
| Time to first action, total duration | User experience; long tails | Duration p99 |
| Blocked actions | Policy hits at the gateway and sandbox (Part 4): mistakes or attacks | Spike in egress or permission denials |
| Cost per successful task | The business number: total spend divided by tasks that actually worked | Above target margin |
6. Runaway loops and budgets
Because an agent decides for itself when it is done, every production system needs hard limits: a maximum number of steps, tokens or dollars per task, a wall-clock timeout, and loop detection (the same tool with the same arguments several times in a row). When a limit is hit, the agent should stop cleanly, save its state and report, not just vanish. Budgets also apply per user and per organisation, enforced at the gateway, so one buggy integration cannot spend a month's budget overnight. Traces of stopped runs are the most valuable debugging data a team has, and, once labelled, the raw material for new evaluations and training environments (Part 8).
7. Unit economics
Cost per successful task
Our model with assumed prices. Failed attempts still cost money, so success rate divides everything.
8. The series
Inside the Agent Layer: eight parts
9. Reading the numbers
- Primary
- OpenTelemetry GenAI semantic conventions (OpenTelemetry project): span types and development status.
- Secondary โ ๏ธ
- 2026 repository move and v1.41 agent span details (vendor blogs); a claim of stabilisation in June 2026 (unconfirmed).
- Our illustrations
- Example trace, prices and cost breakdown; unit-economics calculator.