AI Agent Observability: Tracing Tool Calls, Token Spend, and Prompt Audit Trails

AI agent observability means instrumenting each agent run as a distributed trace: one call tree that records every model invocation, every tool call, and the token count at each step. Without it, when something goes wrong you have only the final output and no view into what the agent actually did.

When your AI agent costs too much, hangs for 45 seconds, or returns the wrong answer, the first question your team asks is: what did it actually do? For most teams right now, the honest answer is: we do not know. The agent’s behavior lives in a black box, and the only artifacts on hand are the final input and the final output.

This post covers how to instrument production AI agent workloads with the OpenTelemetry GenAI semantic conventions so you can reconstruct any agent run after the fact, attribute token spend to each step, and build a prompt audit trail with proper privacy controls. (A separate guide covers hardening the observability MCP servers your agents connect to - Sentry, Grafana, PagerDuty. That one is about securing external tool connections. This one is about instrumenting the agent itself.)

The Problem: When Your Agent Misbehaves, Can You Reconstruct What It Saw?

Standard application tracing answers “what did this request do.” An agent needs the same answer, but an agent’s “request” is not a single function call - it is a tree. A top-level invocation fans out into one or more model calls. Each model call may request tool calls. Each tool call may trigger additional model calls. The agent cycles until it decides it is done.

Why a Single Input/Output Log Is Not Enough

If you only capture the initial prompt and the final response, you discard everything in between. That middle is exactly where misbehavior lives. The agent called the wrong tool. The agent ran a retry loop nobody expected. One model call consumed 80% of the token budget. A tool returned an error the agent silently ignored and then hallucinated past.

None of that is visible from the final output. You need the full tree: every model call with its token counts and finish reason, every tool call with its name and type, and the timing at each node. That is what the OpenTelemetry GenAI semantic conventions provide.

The Standard: OpenTelemetry GenAI Semantic Conventions

The OpenTelemetry project maintains standardized span names, attribute keys, and metric instruments for LLM and agent telemetry under the Semantic Conventions for Generative AI.

Where They Live Now and What “Development” Actually Means

Repository: As of mid-2026, the conventions have moved out of the main open-telemetry/semantic-conventions repository into a dedicated open-telemetry/semantic-conventions-genai repository. The old pages at opentelemetry.io/docs/specs/semconv/gen-ai/ now show a relocation banner: “GenAI semantic conventions have moved to the OpenTelemetry GenAI semantic conventions repository. This page has moved and is no longer maintained in this repository.” Cite and link to the new repo.

Stability: Every document in the set is marked Status: Development - OpenTelemetry’s label for experimental conventions. Every gen_ai.* attribute carries a Development badge. Attribute names can still change before a Stable release. The conventions are in production use at a number of organizations and are ready to adopt, but pin the convention version in your instrumentation library configuration. A post that presents these attribute names as permanent is going to age badly, and the stability page in the spec is clear about why.

The rename you need to know about: older tutorials and some instrumentation libraries still use gen_ai.system to identify the AI provider. The current attribute is gen_ai.provider.name. If you are auditing existing traces and seeing the old key, that is the explanation.

Modeling an Agent Run as a Span Tree

The core artifact is the span tree. The OpenTelemetry SIG GenAI Observability blog models an agent run as three primary span types, nested by trace context:

• invoke_agent spans at the root, representing the top-level agent operation.
• chat spans for each individual model call.
• execute_tool spans for each tool invocation.

For richer frameworks, the agent spans document also defines create_agent, invoke_workflow, and plan spans.

Span Names, Required Attributes, and Nesting

Span names are prescribed by the convention. An inference span SHOULD be named {gen_ai.operation.name} {gen_ai.request.model} - for example, chat claude-opus-4-8 or chat gpt-4. A tool span SHOULD be named execute_tool {gen_ai.tool.name}. An agent-creation span SHOULD be named create_agent {gen_ai.agent.name}.

Two attributes are Required on every GenAI span:

• gen_ai.operation.name: the operation type. Well-known values include chat, generate_content, text_completion, embeddings, create_agent, invoke_agent, execute_tool, and memory operations (create_memory, delete_memory). When a predefined value applies, it MUST be used.
• gen_ai.provider.name: the provider identifier. Values include openai, anthropic, aws.bedrock, gcp.gen_ai, and gcp.vertex_ai. This is the discriminator that selects provider-specific telemetry behavior. For Claude-based workloads, the Anthropic provider overlay requires this set to "anthropic" and states it SHOULD be provided at span creation time.

Trace context propagation is what makes the nesting work. Each child span inherits the trace ID and references its parent span ID. When you view the trace in any OTel-compatible backend, you see a waterfall: invoke_agent at the root, the first chat call beneath it, the execute_tool span under that, and the second chat call following the tool’s return.

graph TD
    A["invoke_agent my-research-agent<br/>gen_ai.agent.name: my-research-agent<br/>gen_ai.conversation.id: sess-abc123<br/>gen_ai.agent.version: 2.1.0<br/>total duration: 12.4s"]
    A --> B["chat claude-opus-4-8<br/>gen_ai.usage.input_tokens: 1024<br/>gen_ai.usage.output_tokens: 312<br/>gen_ai.response.finish_reasons: tool_calls<br/>duration: 3.1s"]
    B --> C["execute_tool web_search<br/>gen_ai.tool.name: web_search<br/>gen_ai.tool.type: function<br/>gen_ai.tool.call.id: call_001<br/>duration: 2.8s"]
    C --> D["chat claude-opus-4-8<br/>gen_ai.usage.input_tokens: 2048<br/>gen_ai.usage.output_tokens: 891<br/>gen_ai.usage.reasoning.output_tokens: 340<br/>gen_ai.response.finish_reasons: stop<br/>duration: 6.5s"]

This span tree is the artifact that answers every post-incident question: what did the agent call, in what order, how many tokens did each step use, and exactly where did the 12.4 seconds go.

Token Spend by Step

Because each chat span sits in the tree under the root invoke_agent, per-step token attribution comes directly from the trace structure.

Per-span token attributes (Recommended):

• gen_ai.usage.input_tokens: input tokens consumed on this call.
• gen_ai.usage.output_tokens: output tokens produced on this call.
• gen_ai.usage.reasoning.output_tokens: reasoning tokens for extended-thinking or chain-of-thought models. This is the attribute most cost analyses miss: reasoning tokens are billed but are not present in the visible completion text, so a count that only looks at output_tokens undercounts the actual cost.

The billable-token rule: the spec states directly that “When systems report both used tokens and billable tokens, instrumentation MUST report billable tokens.” The metric is cost-aligned, not just usage-aligned.

The aggregate metric: gen_ai.client.token.usage is a Histogram instrument with unit {token}, dimensioned by gen_ai.token.type (input or output) and by model. This gives you cross-run aggregates for dashboards and alerts, alongside per-trace attribution for incident investigation.

With this in place, you can answer: the second model call in last night’s agent run consumed 891 output tokens and 340 reasoning tokens. That one step accounted for 62% of the total run cost.

Timing: Where the 45 Seconds Went

The metrics document defines a complete set of timing instruments at every level of the agent call tree.

Client-side call timing:

Metric	What it measures
gen_ai.client.operation.duration	Full latency for one model call
gen_ai.client.operation.time_to_first_chunk	Time to first token, for streaming responses
gen_ai.client.operation.time_per_output_chunk	Per-chunk streaming latency

Agent and orchestration timing:

Metric	What it measures
gen_ai.invoke_agent.duration	Full agent run, root to finish
gen_ai.execute_tool.duration	Time for each tool invocation
gen_ai.workflow.duration	Time for workflow-level orchestration

Server-side timing (self-hosted inference only):

Metric	What it measures
gen_ai.server.request.duration	Server latency per request
gen_ai.server.time_to_first_token	Server-side TTFT
gen_ai.server.time_per_output_token	Server-side generation rate

Because each span also carries its own start and end timestamp, any latency spike traces directly back to a node in the tree: the model taking longer than expected, an external tool with high tail latency, or a retry loop that should not have been retrying.

How Do You Trace Tool Calls in an AI Agent?

The execute_tool span carries a full attribute set for reconstructing the tool-call sequence:

• gen_ai.tool.name (Required): the tool’s name, as declared in the agent’s tool list.
• gen_ai.tool.call.id (Recommended): the identifier the model assigned in its tool-call request. Use this to correlate the model’s request with the tool’s response.
• gen_ai.tool.description (Recommended): the description the model was given for this tool. Having this in the span tells you what the model understood the tool was for.
• gen_ai.tool.type (Recommended): one of function, extension, or datastore.
• gen_ai.tool.call.arguments (Opt-In): the arguments the model passed to the tool.
• gen_ai.tool.call.result (Opt-In): what the tool returned.

Arguments and results are content attributes, covered below. The metadata attributes (tool.name, tool.call.id, tool.type) are captured by default and give you the tool-call sequence without exposing potentially sensitive inputs and outputs.

How Do You Build an Agent Prompt Audit Trail?

Most tutorials on LLM observability skip the privacy dimension entirely, showing you how to capture full prompts with no mention of who else can read them. The OpenTelemetry GenAI conventions get this right from the start.

What is captured by default - metadata only:

By default, instrumentation captures enough to answer operational questions without touching message content.

What is opt-in - content:

Content attributes require an explicit decision to enable.

graph LR
    subgraph ON ["Metadata: On by Default"]
        M1["gen_ai.request.model"]
        M2["gen_ai.usage.input_tokens"]
        M3["gen_ai.usage.output_tokens"]
        M4["gen_ai.response.finish_reasons"]
        M5["span duration"]
        M6["gen_ai.tool.name"]
        M7["gen_ai.tool.call.id"]
        M8["gen_ai.tool.type"]
    end
    subgraph OPT ["Content: Opt-In"]
        C1["gen_ai.system_instructions"]
        C2["gen_ai.input.messages"]
        C3["gen_ai.output.messages"]
        C4["gen_ai.tool.call.arguments"]
        C5["gen_ai.tool.call.result"]
    end
    FLAG["OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT=true"] --> C1
    FLAG --> C2
    FLAG --> C3
    FLAG --> C4
    FLAG --> C5

The metadata side answers operational questions by default. The content side answers audit questions when deliberately enabled. Enable content capture with a plan for PII, and use the external-storage offload for large payloads.

Content capture “SHOULD be controlled by an explicit user opt-in, for example OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT.” The SIG GenAI Observability blog states it plainly: “By default, no prompt content or tool arguments are captured with GenAI telemetry, as these can contain sensitive data.”

For large payloads, the spec defines an “upload content to external storage” hook so message bodies do not bloat the span. A system prompt that runs to several thousand tokens belongs in external storage, not embedded in every trace entry.

The practical recommendation for teams building a prompt audit trail: enable content capture in a separate, access-controlled trace pipeline, run the captured content through a PII scanner before it reaches your backend, and use the external-storage hook for payloads above a size threshold you define. The standard already separates always-safe metadata from opt-in content - use that separation. (For the broader agent security threat model that puts observability gaps in context, see Threat Modeling AI Agents with STRIDE and MCP.)

Correlating Sessions and Agent Versions

The invoke_agent span is one run. A multi-turn conversation is many runs, potentially from different users. Two attributes handle the correlation:

gen_ai.conversation.id (Conditionally Required when available): “The unique identifier for a conversation (session, thread), used to store and correlate messages within this conversation.” Set this to a stable session identifier and every run in the conversation shares it. From your trace backend, you can then query the full session history as a set of related agent runs.

Agent identity attributes on the invoke_agent span:

• gen_ai.agent.name: the agent’s declared name.
• gen_ai.agent.id: a stable, unique identifier for this agent.
• gen_ai.agent.description: what the agent does.
• gen_ai.agent.version: the current version of the agent’s configuration.

gen_ai.agent.version is the one to wire up for rollout comparisons. When you update the system prompt or change the tool list, increment this value. Comparing token spend, latency distributions, and finish-reason breakdowns between versions becomes a standard trace query rather than a custom log parse - because you have a structured field to filter on.

How to Actually Emit This: The Instrumentation and Pipeline

You do not write these spans by hand. Two production-ready instrumentation libraries emit the GenAI conventions automatically, both exporting over OTLP:

• OpenLLMetry (Traceloop, Apache-2.0): a set of OpenTelemetry instrumentations for LLM and agent frameworks. Traceloop co-leads the OTel GenAI semantic-convention working group, so its output tracks the evolving spec.
• OpenInference (Arize, open-source): complementary OTel conventions and instrumentors for LLM and agent apps, native to the open-source Arize Phoenix collector and UI, but usable with any OTel-compatible backend.

Both export over OTLP - the same wire protocol your existing services use. The downstream pipeline is the same OpenTelemetry Collector you already run:

graph LR
    A["Agent Application<br/>auto-instrumented via<br/>OpenLLMetry or OpenInference"]
    A -->|"OTLP<br/>gRPC or HTTP"| B["OpenTelemetry Collector<br/>optional: PII filter processor"]
    B --> C["Arize Phoenix<br/>open-source"]
    B --> D["Langfuse<br/>/api/public/otel"]
    B --> E["General APM<br/>Grafana, Datadog, etc."]

Because all backends read the same OTel GenAI span attributes, the span tree, token counts, and timing data are identical regardless of which backend you query. Instrument once, read anywhere.

The Collector is the right place to insert a PII-filtering processor if you have enabled content capture. The agent emits the full content; the Collector scrubs it before forwarding to the backend.

For Anthropic workloads specifically: the anthropic.md provider overlay in semantic-conventions-genai requires gen_ai.provider.name set to "anthropic", provided at span creation time. The span otherwise inherits the full set of base inference attributes. Sibling overlays for OpenAI, AWS Bedrock, Azure AI Inference, and MCP follow the same pattern with their respective provider name values.

A Minimal Instrumentation Checklist

Before adding vendor-specific extensions, validate these foundations:

1. Install an OTel GenAI instrumentation library. OpenLLMetry or OpenInference, depending on your agent framework and preferred backend. Both handle span emission automatically.

2. Configure the OTLP exporter. Point it at your OpenTelemetry Collector. Use gRPC or HTTP/protobuf.

3. Pin the convention version. The GenAI conventions are at Development status. Pin the version in your instrumentation library config and treat upgrades as requiring an attribute audit.

4. Set gen_ai.provider.name on every agent span. Confirm you are using the current attribute, not the retired gen_ai.system.

5. Verify the span tree structure. Check that invoke_agent appears at the root, chat spans nest under it, and execute_tool spans appear as children of the model call that requested the tool - not floating at the root.

6. Confirm token attributes on every chat span. gen_ai.usage.input_tokens and gen_ai.usage.output_tokens should be present. For extended-thinking models, verify gen_ai.usage.reasoning.output_tokens as well.

7. Decide on content capture. If you need a prompt audit trail, enable OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT, put a PII-filtering processor in the Collector pipeline, and configure external-storage offload for large payloads.

8. Set gen_ai.conversation.id. Required for any multi-turn agent where you need to link runs into sessions.

9. Set agent identity attributes. gen_ai.agent.name and gen_ai.agent.version at minimum, so prompt-change rollouts are queryable from trace data.

10. Verify billable-token reporting. For models that distinguish used tokens from billable tokens, confirm your instrumentation is reporting the billable count, as the spec requires.

Frequently Asked Questions

Are the OpenTelemetry GenAI semantic conventions stable yet?

No. They are at OpenTelemetry’s “Development” stability tier, and every gen_ai.* attribute carries a Development badge - meaning attribute names can still change before a Stable release. They are already in use in production environments and are ready to adopt, but pin the convention version in your instrumentation library and plan for churn as the spec evolves. The conventions also moved out of the main OpenTelemetry semconv repo into a dedicated semantic-conventions-genai repository; the old opentelemetry.io/docs/specs/semconv/gen-ai/ pages now show a relocation banner.

Does OpenTelemetry capture my agent’s prompts by default?

No. By default, instrumentation captures metadata only: model names, token counts, durations, and finish reasons. The content-bearing attributes (gen_ai.input.messages, gen_ai.output.messages, gen_ai.system_instructions, gen_ai.tool.call.arguments, gen_ai.tool.call.result) are Opt-In and disabled by default because they routinely contain sensitive data. Enable them explicitly - for example with OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT=true - and for large payloads use the external-storage offload hook so message bodies do not appear inline in every trace entry.

How do I track token spend for each step of an agent run?

Each chat span carries gen_ai.usage.input_tokens and gen_ai.usage.output_tokens. Because those spans nest under the root invoke_agent span, you get per-step attribution directly from the trace tree - no custom logging needed. The gen_ai.client.token.usage histogram metric provides aggregate counts across runs, dimensioned by input/output type and model. For extended-thinking models, gen_ai.usage.reasoning.output_tokens captures reasoning tokens separately: these are billed but absent from the visible completion text, so ignoring them produces an understated cost figure. The spec also requires instrumentation to report billable tokens (not just used tokens) when both counts are available.

How do I trace the tool calls my agent makes?

Emit one execute_tool span per tool invocation, named execute_tool {gen_ai.tool.name}, as a child of the chat span that requested it. The Required attribute is gen_ai.tool.name. Recommended attributes are gen_ai.tool.call.id (correlates the request to the response), gen_ai.tool.description (what the model understood the tool to do), and gen_ai.tool.type with values function, extension, or datastore. The tool’s arguments and result (gen_ai.tool.call.arguments, gen_ai.tool.call.result) are Opt-In content, off by default.

Do I have to use a specific observability vendor for AI agent tracing?

No. Because the GenAI semantic conventions are a shared OpenTelemetry standard, you instrument once - with OpenLLMetry, OpenInference, or another OTel-native library - export over OTLP through the OpenTelemetry Collector, and read the identical span attributes in any OTel-compatible backend: Arize Phoenix, Langfuse (which exposes an OTLP endpoint at /api/public/otel), Grafana, Honeycomb, Datadog, or others. The attribute names are portable. You are not locked into a single vendor’s dashboard.