Home โ€บ Blog โ€บ Observability and cost
Inside the Agent Layer ยท Part 7 ยท Observability

Seeing Inside Agent Runs: Observability, Tracing and Cost

When a web service fails, it usually throws an error, and dashboards turn red. When an agent fails, it often returns a confident, wrong answer with status 200, after quietly spending forty model calls going round in circles. Ordinary monitoring sees none of that. Agent observability records every model call, tool call and decision as a trace, so people can see what the agent did, why, how long it took and what it cost. This part explains traces and spans for agents, the emerging OpenTelemetry standard, where an agent's tokens actually go, how runaway loops are stopped, and the unit economics every agent product has to get right.

FL
FrontierAGI Team
โš ๏ธ How to read this (as of 28 Oct 2026) The OpenTelemetry GenAI conventions are, per the project, still in development status; one secondary source claims they were stabilised in mid-2026, which we could not confirm (โš ๏ธ). The example trace, cost breakdown and unit-economics calculator are our illustrations with assumed prices, not any provider's rates.

1. Case file

๐Ÿ“ˆ
Agent observability, October 2026Part 7 of the agent layer
Unit of observationA trace per task; a span per model call, tool call or sub-agent
OpenTelemetry GenAI conventionsModel, agent and MCP spans; moved to a dedicated repository in 2026; development status
Agent span typescreate_agent, invoke_agent, tool execution, model inference
ToolsAgent tracing platforms (open source and hosted) plus the big observability vendors
The two questionsDid it do the right thing? What did it cost per successful task?

2. Silent failures

Wrong but confidentThe agent says "fixed the bug" and the tests it chose to run pass; the bug is still there.
LoopsThe agent retries the same failing command, or re-reads the same file, until it hits a limit.
Wrong toolIt searches the web for something that was in the database, or calls a write tool when it meant to read.
Quiet cost blow-upsA prompt change doubles the average context, and the monthly bill doubles with no error anywhere.

None of these show up as HTTP errors or CPU alarms. They show up only in the content and shape of the run, so that is what agent observability records. (For monitoring the GPUs and servers underneath, see AI factory observability.)

3. An agent trace

One coding task, as a waterfall

Each bar is a span; time runs left to right. Tap a row for details. Our illustrative trace, assumed prices.

4. The OpenTelemetry standard

OpenTelemetry is the open standard most companies already use for traces and metrics. Its generative-AI working group defines semantic conventions: agreed names for what to record, so that any framework's traces can be read by any backend. They cover model calls (provider, model, input and output tokens, finish reason), agent operations (create_agent, invoke_agent, split into remote and local calls), tool execution and MCP. In 2026 the GenAI, provider-specific and MCP conventions moved into their own repository; the project still marks them as in development. Even unstable, they matter: an agent framework that emits standard spans can be monitored by whichever tool a company already uses, instead of locking it into one vendor's dashboard. Prompts and outputs can be recorded too, but are often switched off or redacted by default because they may contain personal data.

5. What to watch

SignalWhy it mattersTypical alert
Task success rateThe real quality number, measured by automatic checks, user feedback or sampled review (Evaluation Layer Part 7)Drop after a model or prompt change
Steps and tokens per taskEfficiency; a rising median often means the agent is strugglingp95 steps above budget
Tool error rateBroken integrations, permission problems, rate limitsOne tool above its usual error level
Cache hit rateCost and latency (Part 6)Sudden fall after a prompt edit
Time to first action, total durationUser experience; long tailsDuration p99
Blocked actionsPolicy hits at the gateway and sandbox (Part 4): mistakes or attacksSpike in egress or permission denials
Cost per successful taskThe business number: total spend divided by tasks that actually workedAbove target margin

6. Runaway loops and budgets

Because an agent decides for itself when it is done, every production system needs hard limits: a maximum number of steps, tokens or dollars per task, a wall-clock timeout, and loop detection (the same tool with the same arguments several times in a row). When a limit is hit, the agent should stop cleanly, save its state and report, not just vanish. Budgets also apply per user and per organisation, enforced at the gateway, so one buggy integration cannot spend a month's budget overnight. Traces of stopped runs are the most valuable debugging data a team has, and, once labelled, the raw material for new evaluations and training environments (Part 8).

7. Unit economics

Cost per successful task

Our model with assumed prices. Failed attempts still cost money, so success rate divides everything.

8. The series

Inside the Agent Layer: eight parts

VerdictAgents fail quietly, so the agent layer records everything: a trace per task, a span per model and tool call, tagged with tokens, cost, the user and the agent. Standard OpenTelemetry conventions are arriving, slowly. The number that decides whether an agent product survives is cost per successful task, which means quality and cost have to be watched together. And the traces collected here feed the last part of the stack: Part 8, RL environments, where agents are trained to do better.

9. Reading the numbers

Primary
OpenTelemetry GenAI semantic conventions (OpenTelemetry project): span types and development status.
Secondary โš ๏ธ
2026 repository move and v1.41 agent span details (vendor blogs); a claim of stabilisation in June 2026 (unconfirmed).
Our illustrations
Example trace, prices and cost breakdown; unit-economics calculator.