The AI Observability Stack: Traces, Evaluations, and Failure Reports

A user reports a bad answer. You pull the logs. You get a wall of prompt text, token counts, and a 200 OK.

Nothing in that wall tells you which retrieval step returned the wrong document, which tool call passed a malformed argument, or which model decision sent the run down the wrong branch. The request succeeded. The output was wrong. Your monitoring says everything is fine.

That gap is where AI observability lives. Not "more logging" — a trace-to-evaluation pipeline whose only real output is a cheaper next repair. If your stack cannot shorten the path from symptom to reproducible cause to a shipped change, you have instrumentation, not observability.

This article maps the layers of that pipeline and the seams where it usually leaks. It is not a vendor comparison. The architecture outlasts the tooling, even though the tooling is what changes every quarter.

Why Logs Stop Working When the System Starts Reasoning

Traditional observability rests on three pillars: metrics, logs, and traces. They were built for deterministic systems, where the same input yields the same output and a log line is a sufficient explanation of what happened. Uptime, latency, throughput, and provider error rates remain necessary in AI systems. They are not replaced. They are just no longer sufficient.

LLM and agent applications are probabilistic, context-dependent, and multi-step. A single run may involve a retrieval query, a reranker, three model calls, two tool invocations, and a final synthesis step. The output is a function of all of them, plus the prompt revision, plus the model version, plus whatever the retrieval layer happened to return that day.

When something breaks, the question is no longer "is the service up?" It is "this specific run produced this specific output for this specific reason." Answering that requires execution context: what the system saw, what it decided, what it passed forward. Error codes do not carry that. Prompt text alone does not carry it either.

So the unit of analysis shifts. From the service to the run. From the aggregate to the causal chain.

That shift is what makes AI observability a distinct discipline rather than a rebrand of application monitoring. And it is why teams that bolt an LLM dashboard onto existing infrastructure often end up with more data and no better answers.

The thesis for everything below: observability earns its cost only when it shortens the distance between a symptom and a fix. Every layer either contributes to that or becomes expensive storage.

The Four Layers of an AI Observability Stack

Before drilling into any single layer, hold the whole map. Four layers, each with a distinct job.

Layer 1 — Instrumentation. What the application emits. Spans for model calls, retrieval steps, tool invocations, and agent decisions, plus token counts, latency, cost, and error types. A span is one timed, correlated unit of work — the smallest thing you would want to see on its own. This layer is the raw material.

Layer 2 — Trace storage and correlation. How spans are joined into a single run, how runs are grouped into sessions, and how a trace ID becomes the handle you paste into a bug report. Correlation is the difference between a pile of events and a story.

Layer 3 — Evaluation. The judgments attached to traces. Automated, human, or implicit. This is where data becomes a verdict.

Layer 4 — Reporting and action. Dashboards, alerts, regression gates, and the failure report a human actually reads. A regression gate is a test that blocks or flags a change when a known failure reappears. This is where verdicts become decisions.

The layers are easy to buy and hard to connect. That is the seam problem, and it is the reason so many stacks underperform their component list.

A trace without an evaluation is a receipt. It proves something happened. It does not tell you whether it was good.

An evaluation without a trace is an opinion. It gives you a score with no path back to the mechanism that produced it.

The value lives in the join. Everything else is plumbing.

One structural note that matters for portability: OpenTelemetry-based conventions are becoming a common substrate for the instrumentation layer, and vendors including Microsoft's Foundry have built tracing on those standards rather than on proprietary span formats. Semantic conventions are the shared field names and relationships that let different tools read the same span the same way. That matters less for what you can measure today and more for what it costs to change platforms later. If your spans are emitted in a proprietary shape, switching vendors means re-instrumenting the application. Standards-based emission is the hedge.

Treat that as directional, not settled. The standards are still moving, and vendor support varies by framework and by how much of the agent lifecycle each platform actually covers.

What a Trace Actually Has to Capture

A useful trace record answers a specific set of questions. If any of them is missing, you will eventually hit a failure you cannot localize.

• Which model and version handled this step?
• Which prompt or template revision was used?
• Which retrieval steps ran, and what did they return?
• Which tools were called, with what arguments, and what came back?
• What was the final output?
• What did the run cost in tokens and latency?

That list is the difference between a debuggable record and a log stream.

Span granularity is a design decision, not a default. Too coarse, and you cannot localize the failure — you know the run was slow, not which step. Too fine, and the trace becomes unreadable and expensive to store. The right granularity is the level at which you would actually make a change. If you would never swap out a sub-step independently, it probably does not need its own span.

Multi-step and multi-agent runs raise the bar. When tasks are decomposed and distributed across agents, you need semantic conventions for agent roles, task hierarchy, and handoffs. Without them, cross-agent coordination is invisible, task outcomes cannot be evaluated, and retry patterns cannot be analyzed. Microsoft has described this gap directly: traditional telemetry conventions were optimized for single-agent reasoning paths and lack the semantic depth to capture collaborative workflows. The conventions are still being standardized, which means teams building agentic systems today are partly designing their own semantics.

Then there is governance metadata, which belongs on the trace from day one. Three flags in particular:

• Was tracing enabled for this run?
• Was sensitive input scrubbed before logging?
• Was an evaluation configured?

These three flags are what let a security or compliance function discover AI systems nobody registered. A trace record, aggregated across an organization, can become a useful inventory signal — but only when coverage is consistent and the metadata is populated the same way across systems. Partial coverage produces a partial inventory, and a partial inventory is worse than an honest gap because it looks complete.

Which brings us to the failure mode worth naming explicitly: traces that capture prompts and outputs verbatim become a second, unmanaged copy of your data. A privacy and retention problem disguised as a debugging feature.

The practical rule: decide the redaction boundary before you turn on full capture. Retrofitting it means deleting history you already relied on, and the deletion is usually the easy part. The hard part is the audit trail you no longer have.

From Traces to Evaluations: Closing the Loop

Observability produces data. Evaluation produces decisions. The gap between them is where most teams stall: they have traces and no verdicts.

The first move is to define success before choosing a method. Accuracy, groundedness, task completion, safety, and user satisfaction are different targets. They require different evaluators, and a single score that blends them will be useless for diagnosis.

Three sources of judgment, with real tradeoffs:

Automated evaluators. Cheap, fast, and themselves probabilistic. They need calibration against human judgment, and the calibration has to be measured on the cases you care about — not the easy ones.

Explicit human feedback. Trustworthy, slow, sparse. Thumbs up and down, star ratings, written comments. Consistent negative feedback is a strong signal. The problem is volume: you will never label enough traffic to cover your long tail.

Implicit signals. Rephrased questions, retries, repeated queries, abandoned sessions. Free, abundant, and ambiguous. A user rephrasing a question is evidence that something went wrong. It is not a labeled failure. Treat it as one and you will distort your metrics — you will count confusion, impatience, and genuine errors in the same bucket.

Implicit signals are the most underused and most misread source in the stack. Use them to find candidates for review, not to compute a quality score.

The next decision is offline versus online evaluation. Offline evaluation runs against a curated dataset and gives you reproducibility and regression detection. Online evaluation runs against live traffic and gives you reality. You need both, because they answer different questions. Offline tells you whether a change broke something you already knew about. Online tells you whether the system is failing in ways you have not seen yet.

Now the decision boundary that matters most for teams starting out: begin with a small, hand-labeled set of real failing traces rather than a large synthetic benchmark. The failing traces are the ones your users already found for you. As a starting heuristic, twenty real failures beat two thousand generated cases for finding the bug you actually have. That is a preference, not a law — if your failure surface is genuinely broad and your real-traffic volume is thin, generated cases can earn their place.

One calibration risk to keep visible. An automated evaluator that agrees with humans on your easy cases may disagree on exactly the hard cases you care about. Aggregate agreement numbers hide this. Measure agreement on the hard slice specifically, or you will ship an evaluator that is confident precisely where it is wrong.

Failure Reports: Turning Incidents into Repairs

A failure report is not a dashboard screenshot. It is a short, reproducible record:

• The trace ID
• The input
• The observed output
• The expected behavior
• The suspected layer
• The change that would prevent recurrence

The test of a good report is simple: can a teammate who was not on call reproduce the failure from it without asking a question? If not, the report is incomplete, and the next person pays the cost you avoided.

Cluster failures before fixing them. Ten traces that share a retrieval miss are one bug, not ten. Clustering is what converts a stream of complaints into a prioritized queue, and it is the step most teams skip because each individual complaint feels urgent.

Cost and latency belong in the report alongside quality. An agent that calls a model five times for marginal quality improvement is a failure of design, not just of budget. Real-time monitoring catches the pathological version of this — a bug causing an API loop — but the subtler version, where the extra calls are intentional and unjustified, only shows up when you look at cost per run next to quality per run.

Then close the loop with regression gates. Promote the traces behind fixed failures into a standing evaluation set. Run that set on every model or prompt change. This is the mechanism that stops the same bug from silently returning, and it is the point where observability stops being a cost center and starts being a ratchet.

The anti-pattern to name: teams that instrument heavily but never write reports end up with expensive storage and unchanged quality. They have built a very thorough record of problems they never fixed.

Where This Stack Gets Expensive, Fragile, or Misleading

The layers are straightforward to describe and hard to run. Five failure modes separate a working stack from a demo.

Storage and sampling. Full-fidelity capture of prompts, outputs, and retrieval context grows fast. Sampling saves money but can hide the rare failure you most need to see — and rare failures are usually the expensive ones. The tension has no clean resolution; it has a policy. Decide what you sample, what you keep in full, and how long each tier lives.

Vendor lock-in at the instrumentation layer. Covered above, but it belongs on this list because the cost is deferred and therefore easy to ignore. Proprietary span formats are cheap today and expensive at migration.

Evaluator drift. As models and prompts change, an evaluator calibrated last quarter may quietly stop measuring what you think it measures. The score stays stable. The meaning does not. Re-calibrate on a schedule, and treat a suspiciously stable score after a major model change as a warning sign rather than a win.

The monitorability ceiling. This one is an open question, not a settled fact, and it deserves to be stated as such. As models accomplish more with fewer visible reasoning tokens, the trace may capture the output while losing the reasoning that produced it. More capable systems are also becoming more aware of their own chains of thought. Observability of behavior is not the same as observability of intent, and the gap between them may widen. OpenAI's chief scientist has framed monitorability as a live concern: progress in intelligence does not guarantee progress in alignment. For teams building on frontier models, this is a ceiling you do not control. The operational implication for the rest of us is narrower than the debate: instrument what is externally visible — inputs, outputs, tool actions, retrieval results, policy checks — and do not assume hidden reasoning will be available to you.

Governance overlap. The same trace data that helps you debug is the data a security function needs to find unregistered AI systems. Observability decisions are compliance decisions, whether or not anyone framed them that way. The three metadata flags from the instrumentation layer are the concrete link.

And one honest gap: there is no settled convention yet for evaluating multi-agent task outcomes. Teams building agentic systems are partly designing their own semantics, which means their evaluation results are partly incomparable to anyone else's. That is not a reason to wait. It is a reason to document your semantics explicitly.

A Practical Path: What to Instrument First

Start with one workflow that has real users and real failures. Instrument it end to end before generalizing to the rest of the application. A single well-instrumented workflow teaches you more about your stack's seams than a broad, shallow rollout.

The sequence I would follow:

1. Emit correlated spans with a stable trace ID. Nothing else works until runs are joinable.
2. Attach cost and latency to those spans. These are cheap to add and immediately useful.
3. Add one evaluator targeting the failure you already know about. Choose the cheapest evaluator that directly observes the failure mechanism: a deterministic assertion or schema check where the failure is mechanical, a retrieval check where grounding is the issue, and a calibrated model-based or human evaluator only where the judgment genuinely requires one. Before you let any evaluator gate a change, validate it against a small labeled slice of hard cases.
4. Write the first failure report. Force the artifact into existence before the tooling feels ready.

Keep the first evaluation set small and adversarial. Twenty real failing traces beat two thousand generated ones.

Define the redaction boundary and the retention window before capture scales, not after. This is the step teams most reliably defer, and it is the one with the worst retrofit cost.

Set one alert that matters — a quality threshold or a cost spike — rather than a dashboard nobody opens. A dashboard is a place data goes to be ignored. An alert is a decision waiting to happen.

The skills that pay off here are narrower than the tool landscape suggests. Distributed tracing and span design. Evaluation design and human-agreement measurement. And the ability to read a trace as a causal story rather than a log dump. Teams that build those three capabilities stop treating observability as a purchase and start treating it as a feedback loop.

The Metric That Matters Most

Judge your observability stack by one number: how long it takes a new engineer to go from a user complaint to a reproducible cause and a shipped fix.

That is the north-star outcome, not the only measurement. You still need the supporting numbers that tell you whether a repair was safe and affordable: does the failure recur, does the evaluator still agree with humans on hard cases, what did the fix cost in latency and tokens, and did anything cross a safety or privacy line. Repair time tells you whether the loop works. The supporting measures tell you whether it is working on the right things.

If repair time is falling, the stack is working, regardless of which vendor you chose or how many spans you emit. If it is not falling, you have instrumentation — a very complete record of problems you have not fixed.

So the leverage question is not "what should we instrument?" It is: which single failure, if it became impossible to repeat, would remove the most support load or the most customer risk?

Find that one. Instrument it end to end. Write the report. Promote the trace into a regression gate. Then do it again with the next one.

The stack is not the point. The shrinking repair time is.