AI-Powered Self-Hosted Blog

Designing Observability for AI Agents — Tracing Tool Calls and Visualizing Failures with OpenTelemetry

Published

Traditional applications usually fail at identifiable boundaries. An HTTP request returns 500, a database query times out, or a worker crashes and emits an exception.
AI agents fail differently.
An agent can return HTTP 200, complete every tool call, and still produce the wrong result. It may search the wrong database, call the same tool three times, misunderstand a tool response, or spend 40 seconds exploring a path that should have taken two. These failures are often silent, expensive, and difficult to reproduce.
Observability for agents is therefore not simply “logging plus an LLM column.” It requires a deliberate model of agent behavior: what constitutes a unit of work, which operations deserve spans, how semantic failures are detected, and how sensitive data is protected. OpenTelemetry provides a practical foundation because it offers vendor-neutral traces, metrics, logs, context propagation, and integrations with existing visualization systems.

Why Agents Fail Softly

A conventional service often has a binary outcome: success or failure. An agent has several layers of outcome:

  1. Query the United States sales database.
  2. Receive a valid result.
  3. Calculate a percentage correctly from the wrong dataset.
  4. Return a confident answer with HTTP 200.
    From an infrastructure perspective, every component worked. From the user’s perspective, the system failed.
    This distinction should shape the observability design from the beginning. Technical failures and semantic failures need separate signals, dashboards, alerts, and evaluation methods.

Model the Agent as a Trace

The most useful unit of observability is usually one complete agent run. A trace should show the entire decision path, not just the final model request.
A typical trace might look like this:

agent.run
├── llm.call
├── tool.call: search_orders
├── llm.call
├── tool.call: calculate_refund_rate
├── llm.call
└── final.answer

The root agent.run span should include stable identifiers such as:

agent.name = "refund-assistant"
agent.version = "2026.09"
model.name = "gpt-5"
tool.name = "search_orders"
tool.call.index = 2
tool.retry_count = 1
llm.input_tokens = 1840
llm.output_tokens = 312
agent.total_tool_calls = 4

The exact attribute names should be standardized within the organization. Consistency matters more than inventing a perfect schema, because it enables comparisons across agents and versions.

Detecting Loops and Suspicious Behavior

Agent traces become valuable when they expose patterns that ordinary logs hide. Several simple rules can detect many operational and semantic risks.
A repeated call to the same tool three times with substantially identical arguments may indicate a loop. More than ten tool calls in one run may indicate uncontrolled exploration. A tool error followed by endless retries can quickly consume both budget and latency. A final answer without a retrieval, calculation, or citation span may indicate unsupported reasoning.
Useful derived metrics include:

Visualizing Failures as Paths

A flat log list makes agent behavior difficult to understand. A trace timeline shows where time was spent and which operation led to the final answer.
Suppose a customer asks for an invoice. A trace may reveal:

agent.run: 42.8s
├── llm.call: 1.8s
├── tool.call search_customers: 0.4s
├── llm.call: 2.1s
├── tool.call search_invoices: 0.7s
├── llm.call: 1.9s
├── tool.call search_invoices: 0.6s
├── llm.call: 2.0s
└── tool.call browser_search: 32.4s

The user sees only a delayed answer. The trace shows that the agent abandoned the internal invoice system and spent most of its time browsing the public web.
Agent workflows are not always simple trees. Loops, retries, parallel tool calls, and delegated sub-agents can form a directed graph. OpenTelemetry spans still provide a useful structure, but additional attributes may be necessary:

Why OpenTelemetry Is a Strong Foundation

A custom logging format can record agent events, but it usually leaves teams with disconnected timestamps, inconsistent identifiers, and no standard propagation across services. An LLM-specific monitoring product may provide convenient dashboards, but it can be difficult to combine those signals with database, queue, and HTTP traces.
OpenTelemetry connects the agent to the rest of the system. A trace can begin at an API gateway, continue through an orchestration service, enter an agent loop, cross a tool implemented by another service, and end in a response handler. The same trace ID can be searched in a standard backend such as Jaeger, Grafana Tempo, or another compatible platform.
It also supports a gradual rollout. Start with traces for a small percentage of production requests, then add metrics and logs. Use tail-based sampling to retain suspicious traces, errors, long-running requests, and high-cost runs while sampling ordinary successful traffic more aggressively.

Protecting Prompts, Tool Data, and Secrets

The most useful trace content is often the most sensitive. Prompts can contain customer information. Tool responses may include financial records. Authorization headers, API keys, and full database rows must never be copied into telemetry by default.
A safer design records metadata and redacted summaries:

tool.name = "search_orders"
tool.status = "success"
tool.result_count = 12
tool.result_schema = "OrderSummary"
tool.output_hash = "..."

If raw prompts or outputs are needed for debugging, store them separately with strict access controls, encryption, short retention, and explicit sampling. Apply redaction before data reaches the telemetry exporter, not after it arrives in a shared dashboard.
Token counts and payload sizes are often sufficient for cost analysis. Full content should be treated as an exception requiring a clear operational reason.

Measuring Correctness, Not Just Reliability

Latency, error rate, and availability remain important, but they do not measure whether an answer is correct. Semantic quality requires additional instrumentation.
An evaluation harness can replay a fixed set of representative tasks against each agent version. For each run, it can compare:

Start With a Small, Useful Schema

A reliable implementation does not need to instrument everything on day one. Begin with one root span per agent run and child spans for model and tool calls. Add four categories of derived signals:

  1. Technical errors and timeouts.
  2. Loop, retry, and budget violations.
  3. Evidence and semantic-quality indicators.
  4. Sensitive-data and retention metadata.
    Then build dashboards that answer operational questions quickly: