# AI Agent Observability: What to Trace and What It Catches

> AI agent observability means tracing tool calls, reasoning steps, and handoffs, not just tokens and latency. Here's what to instrument and why.

Source: https://last9.io/blog/ai-agent-observability-best-practices/

An AI agent can return a 200 status, finish in normal time, and still have called the wrong tool, hallucinated an argument, or handed off bad context to the next step. Observing a single LLM call catches slow responses and token spend. It does not catch that.

Agent observability means tracing the actual chain of decisions, tool calls, and handoffs an agent made, not just whether the final response arrived and how much it cost.

If the question is about a single model call, tokens, latency, cost, [Last9's LLM observability guide](https://last9.io/blog/llm-observability/) covers that layer directly. This piece is about the layer on top of it: what changes once that call is one step inside an autonomous, multi-step agent loop.

## What makes agent observability different from LLM observability?

A single LLM call is a request and a response. An agent is a loop: the model decides what to do, calls a tool, reads the result, decides again, and repeats until it judges the task done. That loop is not predictable the way a normal function call is.

Unlike code with a fixed control flow, where you can trace an input through the same logic every time to find a bug, the same agent input can trigger a different sequence of tool calls, pull back different retrieved documents, and produce a different final response on separate runs. There is no single path to inspect, only the path a given run took.

That non-determinism is why status codes and latency graphs stop being reliable signals. An agent that worked correctly in a demo can start picking the wrong tool in production days later, and an AI agent monitoring setup built around uptime and response time will keep reporting a healthy 200 status the entire time.

The request succeeded. The agent still did the wrong thing. Catching that requires the reasoning and tool-call chain itself, because the outcome alone looks fine.

## What should you trace in an agent run?

Four categories of spans cover most of what goes wrong in a production agent run:

**Tool-call spans** capture the tool name, the arguments passed to it, the return value, retry count, and duration. This is what surfaces a hallucinated argument or a silent retry loop that never shows up as an error.

**Reasoning spans** record the model's plan, which action it picked, what it observed back, and what it decided next. This is what reveals plan drift, the agent abandoning its original approach mid-task, or a wrong branch taken at a decision point.

**State and handoff spans** capture the context before and after each step, and the payload passed at a sub-agent handoff. This matters most in multi-agent systems, where a second agent operates on the first agent's output and has no way to know if that output was already wrong.

**Memory and retrieval spans** track what was queried, what came back, and how relevant or fresh it was. This is what catches a stale memory read or a retrieval that pulled the wrong entity into context.

Metrics worth tracking on top of these spans: tool selection accuracy, task completion rate, and how often a run required a retry or a human handoff. Latency and token cost still matter, but they answer "was it fast and cheap." Tool selection accuracy and task completion answer "was it right."

## How does OpenTelemetry represent an agent trace?

OpenTelemetry's [GenAI semantic conventions](https://github.com/open-telemetry/semantic-conventions-genai/blob/main/docs/gen-ai/gen-ai-agent-spans.md), maintained in their own dedicated repository as of 2026, define a specific span structure for this.

A top-level `invoke_agent` span represents the full agent run. Under it, `chat` spans represent each individual model call, and `execute_tool` spans represent each tool invocation the model requests. A typical trace looks like `invoke_agent` → `chat` → `execute_tool` → `chat` → `execute_tool` → `chat`, repeating until the agent finishes.

If semantic conventions are new to you, [this overview of OpenTelemetry semantic conventions](https://last9.io/blog/opentelemetry-semantic-conventions/) explains how they work across signals.

The `invoke_agent` span itself carries agent-specific attributes the spec defines directly:

- `gen_ai.agent.id` and `gen_ai.agent.name` identify which agent ran.
- `gen_ai.agent.description` gives, in the spec's own words, "the free-form description of the invoked GenAI agent."
- `gen_ai.conversation.id` ties the run to a specific conversation thread, so multi-turn context can be correlated across it.

These attributes are what separates an agent trace from a plain chat trace: identity and conversation continuity, on top of the model name and token count a chat span already has.

Where a model is calling tools over the [Model Context Protocol](https://modelcontextprotocol.io/), MCP adds its own span layer on top of `execute_tool`, carrying `mcp.method.name`, `mcp.session.id`, and a JSON-RPC request ID.

W3C Trace Context propagation links the MCP server's spans as children of the client's spans. That keeps a tool call traceable end to end even when the tool itself runs as a separate service.

As of this writing, the agent-span model is marked development status in the specification itself, meaning the overall shape is stabilizing but individual attribute names can still change. Check the current spec before you build dashboards or alerts on these exact field names.

## What are the real failure modes to watch for?

Beyond the wrong-tool and hallucinated-argument cases already covered, a few patterns show up repeatedly in production agent traces:

Looping is an agent that keeps retrying a tool call or revisiting the same reasoning step without making progress toward the task, often invisible in aggregate metrics since each individual call still completes normally.

Context loss across turns is an agent that correctly identifies what needs to happen early in a multi-turn interaction, then fails to apply that correctly once the conversation moves on, because the relevant detail from turn one didn't carry forward into the reasoning at turn four.

Incomplete handoffs are specific to multi-agent systems: agent A finishes its part and passes a result to agent B, but the handoff payload is missing something agent B needed, and agent B has no signal that anything is wrong, it just proceeds on incomplete information.

None of these produce an error, so a check on whether the run completed without an exception misses all three. Only the reasoning and handoff spans show them.

## Is this different for multi-agent systems?

Yes, in one way: a single agent's mistakes are contained to its own reasoning chain, but a multi-agent system's mistakes propagate.

If agent A retrieves stale data and hands it to agent B, agent B's reasoning is now built on bad input it has no way to detect, and whatever agent B produces looks locally correct in its own trace even though the whole chain is wrong.

A conversation or workflow ID that persists across the handoff is what lets you walk back from a wrong final answer to the agent-to-agent transition where the bad data entered the chain, rather than re-checking every agent in the chain one at a time.

## How do you add observability to an AI agent?

The span model above turns into a short rollout:

1. **Pick one agent.** Start with the agent that handles the most consequential task in production, not every agent at once.
2. **Wrap each run in an `invoke_agent` span.** Set `gen_ai.agent.name`, `gen_ai.agent.id`, and `gen_ai.conversation.id` so runs can be grouped by agent and by conversation.
3. **Record model and tool calls as child spans.** Use `chat` spans for model calls and `execute_tool` spans for tool calls, with the tool name, arguments, result, and retry count on each.
4. **Capture handoffs and retrievals.** At every sub-agent transition, record the payload that was passed. On every memory or retrieval read, record what came back.
5. **Track correctness next to latency and cost.** Add tool selection accuracy, task completion rate, and retry or human-handoff rate to the same dashboard as latency and token spend.

If your agents run on LangChain or LangGraph, [this guide to LangChain observability](https://last9.io/blog/langchain-observability/) covers the framework-specific setup.

## How does Last9 fit into this?

Last9's [`last9-genai` SDK](https://last9.io/docs/integrations/python-genai-sdk/) builds on the same OTel GenAI conventions covered above rather than replacing them. Each agent's spans carry `gen_ai.agent.name` and `gen_ai.agent.id` per the standard.

The SDK adds conversation threading, so a multi-agent handoff can be followed by conversation ID rather than reconstructed manually across separate traces. It also rolls cost attribution up per workflow rather than per individual call. The [launch post for `last9-genai`](https://last9.io/blog/last9-genai-llm-observability-sdk/) walks through the conversation model in more detail.

One current gap: tool call arguments and results are captured as span events today, not yet promoted onto the parent span the way the tool-call spans above describe. If your primary debugging need is inspecting exactly what argument a tool call received, that data is tracked, but not yet at the top level.

In the product, [Agents Monitoring](https://last9.io/docs/agents-monitoring/) reads any GenAI-convention telemetry and shows conversations, token usage, latency, cost, and success rate, with links from a slow or failing call to its full trace. For coding agents specifically, see [coding agent observability](https://last9.io/coding-agent-observability/).

## Where to start with agent tracing

A 200 status and a fast response time tell you the agent finished. They don't tell you it finished correctly.

Tracing tool calls, reasoning steps, and handoffs directly, using the span structure OpenTelemetry already defines for this, is what catches an agent that picked the wrong tool, hallucinated an argument, or passed bad context downstream while every individual call looked healthy.

Instrument one agent with the four span types above, review its failed runs, and expand from there once you know what a real failure looks like in your own traces.

## FAQ

### What is AI agent observability?

AI agent observability is the practice of tracing an autonomous agent's full decision chain, including its reasoning steps, tool calls, and any handoffs to other agents, rather than only monitoring the latency, cost, and success rate of the underlying model calls. It answers whether the agent did the right thing, where standard monitoring answers only whether the request succeeded.

### How is agent observability different from LLM observability?

LLM observability tracks a single model call: tokens used, latency, and response quality for that one request. Agent observability tracks the full multi-step loop an autonomous agent runs, including which tools it called, what arguments it passed, how its plan evolved across steps, and what it handed off to other agents, since an agent can complete successfully at the API level while still having made a wrong decision partway through.

### How do you add observability to an AI agent?

Start with one production agent. Wrap each run in an OpenTelemetry `invoke_agent` span with the agent name, agent ID, and conversation ID. Record each model call as a child `chat` span and each tool call as a child `execute_tool` span with its arguments and result. Capture the payload at every handoff and the results of every retrieval, then track tool selection accuracy and task completion rate alongside latency and cost.

### What does OpenTelemetry's invoke_agent span track?

OpenTelemetry's `invoke_agent` span represents a complete agent execution and carries agent-specific attributes including `gen_ai.agent.id`, `gen_ai.agent.name`, `gen_ai.agent.description`, and `gen_ai.conversation.id`. Child `chat` spans represent individual model calls within that run, and child `execute_tool` spans represent each tool the agent invokes, giving a full parent-child trace of one agent run from start to finish.

### Can an AI agent fail without throwing an error?

Yes, and this is the main reason agent-specific tracing matters. An agent that selects the wrong tool, hallucinates a tool argument, loses context across conversation turns, or passes an incomplete handoff to another agent typically completes its run normally at the infrastructure level, a 200 status, normal latency, no exception. The failure only shows up in the reasoning and tool-call content itself, which standard uptime or latency monitoring never inspects.

### Do I need different tracing for multi-agent systems versus a single agent?

The same span types apply, but multi-agent systems add a specific new risk: one agent's error becomes another agent's bad input. State and handoff spans that capture what was passed at each agent-to-agent transition, tied together with a persistent conversation or workflow ID, are what let you trace a wrong final answer back to the specific handoff where bad data entered the chain, rather than needing to inspect every agent in the pipeline individually.

### What are some examples of agent observability platforms?

Datadog, Splunk, and Google Cloud offer agent observability inside their broader monitoring products. Langfuse, LangSmith from LangChain, and Arize focus on LLM and agent tracing and evaluation.

Last9 is the stronger fit when you want agent traces in the same place as the rest of your telemetry. Its Agents Monitoring view reads standard OpenTelemetry GenAI telemetry with no Last9-specific SDK required, and a slow or failing agent call links straight to the full request trace, including the endpoint that triggered it, retries, and downstream work. That matters when the root cause is a slow database or a failing API rather than the model.

Whichever platform you pick, check that it accepts standard OTel GenAI spans, since that decides whether you can change backends later without re-instrumenting your agents.