Agentic AI Observability with OpenTelemetry: Seeing Inside the Black Box

Has anyone played around with OpenLLMetry - the open source SDK that builts on top of OpenTelemetry?

By Amal V S, Backend & AI Systems Architect · Wed Sep 02 2026

Has anyone played around with OpenLLMetry - the open source SDK that builts on top of OpenTelemetry?

AI agents are no longer demos. They're calling tools, chaining LLM calls, retrying failed steps, and making autonomous decisions in production systems — often with real money and real users on the other end. And yet, when one of these agents misbehaves, most teams are still debugging with print() statements and vibes.

This post is a practical, no-hype look at how OpenTelemetry (OTel) is being used to bring real observability to agentic AI systems: what's standardized today, what's still shifting under your feet, and how to instrument an agent so you can actually explain why it did what it did.

1. Why agentic systems break traditional observability

Classic APM answers "where did it fail and how long did it take." Agentic systems introduce three problems traditional monitoring wasn't built for:

  • Non-determinism. The same input can produce a different tool call, a different plan, or a different number of steps on every run. Static dashboards and fixed alert thresholds don't hold up.

  • Invisible reasoning. The "decision" happens inside a model's forward pass. You don't get a stack trace when an agent picks the wrong tool — you get a plausible-looking wrong answer.

  • Deep, dynamic call chains. A single user request can fan out into retrieval calls, multiple LLM invocations, tool/function calls, sub-agent delegation, and memory reads/writes — a graph, not a line.

The failure mode in agentic systems is rarely a crash. It's a semantic failure: the agent didn't throw an exception, it just called the wrong function, looped on a bad plan, or hallucinated a parameter. You can't catch that with uptime checks — you need to reconstruct the actual execution path, step by step, after the fact.

This is exactly the gap distributed tracing was designed to close for microservices twenty years ago. The same tracing primitives — traces, spans, and structured attributes — turn out to map onto agent execution surprisingly well.

2. An open, standards-based approach: OWASP AOS + OpenTelemetry

Rather than every vendor inventing its own proprietary trace format for "AI observability," the industry is converging on two open efforts working together:

  • OWASP Agent Observability Standard (AOS) — a project with the explicit mission of turning agents "from black boxes into trustworthy systems through standardized observability," built on three pillars: instrumentation (via OpenTelemetry), traceability (end-to-end execution flow), and inspectability (insight into what actually happened inside a run).

  • OpenTelemetry (OTel) — the CNCF observability framework already running in most production stacks, providing the transport and data model: logs, metrics, and traces, all vendor-neutral and exportable to any compatible backend.

The pairing matters because it means agent telemetry doesn't need a separate pipeline. It rides on the same collectors, the same backends, and the same operational muscle memory your team already has for regular services.

3. What OpenTelemetry actually standardizes for GenAI (and what it doesn't — yet)

This is the part worth being precise about, because a lot of blog content overstates how "done" this is.

OpenTelemetry's GenAI semantic conventions define a shared gen_ai.* vocabulary for model calls and agent steps — attribute names like gen_ai.provider.name, gen_ai.operation.name, request/response model identifiers, and input/output token counts. As of mid-2026, these conventions:

  • Live in their own dedicated repository, separated from the core semantic conventions so they can iterate independently of OTel's stable release cycle.

  • Are marked Development status — OpenTelemetry's maturity ladder puts this below Alpha, meaning the spec explicitly says it should not be relied on for production without expecting change, and may shift without notice.

  • Have no tagged release of their own yet, even though the core OTel semantic conventions have shipped stable versions.

  • Still show real attribute drift across SDKs — one instrumentation library emits gen_ai.request.model, another emits llm.model_name for the same thing. If you build a dashboard against one and your team switches libraries, spans silently stop matching your filters.

What is reasonably solid:

  • Model/inference spans — operations like inference, embeddings, retrieval, and tool execution, tagged with provider, model, and token usage attributes. Token-usage metrics in particular are worth adopting early: they're your cheapest cost-attribution tool, and a single mis-scoped agent loop burning tokens is the kind of thing someone will ask you about in a postmortem.

  • Agent and framework spans — operations named create_agent, invoke_agent, invoke_workflow, plan, and execute_tool. These give you a reconstructible path through an agent's execution — which step called which tool, in what order — without exposing the model's actual internal reasoning tokens.

  • MCP (Model Context Protocol) conventions — a parallel, also-Development set of conventions for client/server spans, so tool servers accessed via MCP show up in the same trace graph as everything else.

The honest takeaway: instrument now, but wrap the attribute names behind your own thin abstraction (a small span helper or a Collector-side processor that normalizes renamed attributes) so that when the spec stabilizes — and the names shift — you have one place to update, not every call site in your codebase.

4. The building blocks: logs, metrics, traces

OpenTelemetry gives you three correlated signal types, and agentic systems genuinely benefit from all three:

  • Traces — the end-to-end path of a single agent run: the user prompt, every LLM call, every tool invocation, every retrieval, as parent/child spans. This is what lets you answer "what actually happened on this run."

  • Metrics — numeric, aggregable KPIs: token counts, latency per operation, tool-call error rates, cost per session. This is what lets you answer "is this getting worse over time, and by how much."

  • Logs — timestamped structured events, including the actual prompt/completion content. Content belongs here, not in span attributes: attributes are indexed and size-limited, and dumping raw prompt text into them is a good way to leak PII into your backend and blow past ingestion limits. Span events (structured, droppable at the Collector level without touching app code) are the better home for this.

5. Instrumenting an agent: manual vs. auto-instrumentation

Auto-instrumentation (zero code)

For popular SDKs (OpenAI, Anthropic, LangChain, LlamaIndex), community instrumentation packages patch the client automatically:

pip install opentelemetry-distro opentelemetry-instrumentation
pip install opentelemetry-instrumentation-openai
FROM python:3.13-alpine

ENV OTEL_EXPORTER_OTLP_ENDPOINT=otel-collector:4317

RUN pip install --no-cache-dir \
    fastapi \
    opentelemetry-distro \
    opentelemetry-instrumentation \
    opentelemetry-instrumentation-openai

COPY main.py .

CMD ["opentelemetry-instrument", "uvicorn", "main:app", "--host", "0.0.0.0", "--port", "8000"]

Every API call automatically produces a span carrying gen_ai.* attributes and token counts — no changes to your business logic.

Manual instrumentation (code-based)

When you need custom span names, agent-specific attributes, or you're wrapping a framework without auto-instrumentation support, instrument explicitly:

from openai import OpenAI
from opentelemetry import trace
from opentelemetry.instrumentation.openai_v2 import OpenAIInstrumentor

OpenAIInstrumentor().instrument()
tracer = trace.get_tracer("agent.research")

client = OpenAI()

def run_agent(task: str) -> str:
    with tracer.start_as_current_span("agent.run") as span:
        span.set_attributes({
            "gen_ai.system": "openai",
            "agent.name": "research-agent",
            "agent.task": task[:200],
        })

        # Retrieval creates its own child span
        context_docs = retrieve_context(task)

        messages = [
            {"role": "system", "content": "You are a research assistant."},
            {"role": "user", "content": task},
        ]

        tool_calls_count = 0
        while True:
            response = client.chat.completions.create(
                model="gpt-4o",
                messages=messages,
                tools=TOOLS,
            )
            choice = response.choices[0]
            span.set_attribute("agent.tool_calls_count", tool_calls_count)

            if choice.finish_reason != "tool_calls":
                return choice.message.content
            tool_calls_count += 1
            # ...execute tool, append result to messages, loop

Frameworks with native OTel support make this even simpler. PydanticAI, for example, exposes instrumentation as a constructor flag:

from pydantic_ai import Agent

agent = Agent(
    "google-gla:gemini-2.0-flash",
    system_prompt="Be concise, reply with one sentence.",
    instrument=True,
)

result = agent.run_sync('Where does "hello world" come from?')

6. The Collector: your one export pipeline

Instead of wiring every service directly to your observability backend, route everything through the OpenTelemetry Collector — a single, backend-agnostic pipeline for receiving, processing, and exporting telemetry:

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318

processors:
  batch:
  resourcedetection:
    detectors: ["env", "system"]

exporters:
  otlp/traces:
    endpoint: tempo:4317
  otlphttp/metrics:
    endpoint: http://mimir:9009/otlp
  otlphttp/logs:
    endpoint: http://loki:3100/otlp

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [batch, resourcedetection]
      exporters: [otlp/traces]
    metrics:
      receivers: [otlp]
      processors: [batch, resourcedetection]
      exporters: [otlphttp/metrics]
    logs:
      receivers: [otlp]
      processors: [batch, resourcedetection]
      exporters: [otlphttp/logs]

This is also the right layer to add a processor that normalizes attribute drift (rewriting llm.model_name → gen_ai.request.model, for example) so your dashboards don't silently lose data when a library upgrade changes attribute names underneath you.

7. A fully open-source backend: Grafana LGTM

Because everything above speaks OTLP, you can point it at any compatible backend. A common fully open-source, self-hostable stack is Grafana's LGTM:

  • Loki — logs (including your prompt/completion span events)

  • Grafana — dashboards and trace exploration

  • Tempo — traces (your agent execution graphs)

  • Mimir — metrics (token usage, latency, cost over time)

With this in place, even a deliberately trivial "hello world" agent reveals something useful: how resource-hungry a single LLM round trip actually is, and how much structure a "simple" prompt-response interaction hides once you look at the trace.

8. What this buys you — and what it doesn't

What you get:

  • A reconstructible execution path for every agent run — which tool was called, in what order, with what latency and cost.

  • Quantitative trend data (token spend, latency, error rate) alongside qualitative trace inspection.

  • Vendor neutrality — swap LLM providers or observability backends without rewriting instrumentation, because you instrumented to a standard, not a product.

What you don't get, yet:

  • A model's actual internal reasoning. Agent spans record that a decision point occurred and what it produced — not the hidden computation that produced it.

  • Attribute stability. Nothing in the gen_ai.* namespace is Stable as of this writing; expect renames as the spec matures, and design your instrumentation layer so those renames are a one-line fix, not a grep-and-replace across your codebase.

  • Automatic coverage. Agent and framework spans only appear if the framework or instrumentation you're using actually emits them — plugging in a bare LLM client without agent-level spans still leaves the "why did it call that tool" question unanswered.

9. Takeaways

Agentic AI observability isn't a new discipline requiring new tools — it's distributed tracing applied to a new kind of non-deterministic call graph, using an emerging but already-useful shared vocabulary. The pragmatic path in 2026 is:

  1. Instrument now using OTel's GenAI conventions, because token and latency visibility pays for itself immediately.

  2. Isolate the vendor-specific and spec-specific bits behind a thin wrapper or Collector-side processor, since the conventions are still marked Development and will shift.

  3. Keep content out of attributes — put prompts/completions in span events or logs, not indexed span attributes, for both cost and privacy reasons.

  4. Route through a Collector to an open backend (Tempo/Loki/Mimir/Grafana, or any OTLP-compatible system) so you're never locked into a single AI-observability vendor.

The black box doesn't have to stay a black box — it just needs the same tracing discipline we already trust for every other distributed system, applied one span at a time to the agent loop.