Observability · LLM observability & tracing
Make agent failures diagnosable instead of mysterious
One trace ID that explains the whole run.
You probably have this problem if
- A customer reports a bad answer from Tuesday and you cannot reconstruct what happened
- Debugging means grepping application logs and guessing
- You cannot say what percentage of runs fail, because failures have no categories
- Nobody knows which step in a multi-step agent is the slow one
- You have prompt/response logs, but nothing links them into a single run
What you end up with
Any bad run can be pulled up, replayed, and explained in minutes.
- Usually part of
- Agent Production Readiness Audit
- $4,000 – $6,000
- Phases
- 5 steps, detailed below with effort per step
- Domains I have done this in
- HR analyticsTelecom data platformsWorkflow automation
Why this is the thing to fix
Agents fail in ways conventional services do not: the code ran correctly and the output was still wrong. Stack traces and HTTP status codes tell you nothing, because nothing threw.
What you need instead is the decision path — every prompt, every tool call with its arguments, every retry, every token count — tied together under one trace ID. Without it, mean time to diagnosis is measured in days and a meaningful share of failures are simply never explained.
The second-order benefit matters more than the debugging: once failures are traced, they can be categorised, and once categorised they can be counted. Then you fix by frequency times severity instead of by whoever complained loudest.
The minimum viable agent trace
Every span an agent emits should carry these fields. This is the spec I implement — copy it. Most teams are missing rows 4, 8, 9 and 11, which are precisely the ones that make cost and correctness debuggable.
| Field | Why it earns its place |
|---|---|
| trace_id | One ID spanning the entire run, propagated across every service. Without this you have logs, not traces. |
| step_index | Position in the run. Makes loops visible as a pattern rather than a mystery. |
| step_type | model_call · tool_call · retrieval · guardrail. Lets you aggregate latency and cost by kind of work. |
| prompt_version | The exact prompt artefact used. This is what makes a reported failure reproducible after a deploy. |
| model_id | Pinned version, not a floating alias. Catches silent fallbacks to a more expensive model. |
| tokens_in / tokens_out | Per call, not per run. Cost attribution is impossible without this granularity. |
| cache_read_tokens | Distinguishes a working cache from an expensive one. A low hit rate on the 1-hour tier costs money. |
| cost_usd | Computed at write time. Costs derived later from aggregate token counts are always wrong. |
| tool_name / tool_args | The arguments, not just the name. Hallucinated arguments are invisible otherwise. |
| tool_args_valid | Whether validation passed. Rejection rate per tool is one of the best quality signals available and almost nobody tracks it. |
| stop_reason | end_turn · max_tokens · tool_use · error. A length-based stop treated as success is a silent truncation bug. |
| retry_count | Retries are billed and add latency. Untracked, that time and money appears to vanish into the model call. |
| latency_ms | Per step. A run-level number cannot tell you which step to fix. |
| outcome | success · refused · failed · escalated. The field that turns traces into countable failure rates. |
| failure_category | Tagged against a named taxonomy. Free-text notes cannot be aggregated. |
Vendor choice matters far less than field coverage. Langfuse, LangSmith, Braintrust, Phoenix or plain OpenTelemetry all work — a tool with these fields beats a better tool without them.
What to alert on — and what not to
Most agent alerting is either absent or so noisy it gets muted. Alert on trends and rates, not thresholds and individual events.
| Alert on | Not on | Why |
|---|---|---|
| Cost per conversation drifting up | Monthly budget exceeded | A unit-cost alert fires while it is still a bug. A budget alert fires after the money is gone. |
| p95 latency per operation | Average latency | Averages hide exactly the runs where users abandon. |
| Tool rejection rate rising | Individual rejected calls | The rate is the signal; single events are noise. |
| Rate of length-capped responses | Any single truncation | A rising rate means your limits no longer fit your traffic — a leading indicator. |
| Failure rate by category | Total error count | An undifferentiated count cannot be prioritised. |
| Repeated identical tool calls in a run | High step counts alone | Repetition is the clearest signal of a stuck agent; a long run may be legitimate. |
Every alert needs a defined action. An alert nobody knows how to respond to gets muted within two weeks, and then you have no alerting and a false sense of having some.
How I actually do it
With effort per step, so you can judge whether to hire me or hand this to someone on your team.
- 10.5 day
Map the run
Walk one real request end to end and identify every point where a decision is made or a boundary crossed. That list becomes the span set.
- 22 days
Instrument spans
Implement the field spec above, propagating one trace ID across services. Existing tool choice is respected — I do not replace a working stack to prove a point.
- 30.5 day
Define the failure taxonomy
Name your failure modes so they can be tagged and counted. I start from my published taxonomy and specialise it to your agent.
- 41 day
Dashboards and alerts
Cost per conversation, p95 by operation, failure rate by category, tool rejection rate. Each alert gets a documented response.
- 50.5 day
Capture path into evals
One click from a bad trace to a new eval case, so diagnosis feeds prevention instead of stopping at explanation.
What you keep
- Step-level tracing across the whole run with one propagated trace ID
- The full field spec implemented, including cost and validity fields
- A failure taxonomy specialised to your agent, with tagging in place
- Dashboards for cost per conversation, p95 by operation, and failure rate by category
- Alerts with documented responses, tuned to rates rather than thresholds
Failure modes this prevents
From the failure taxonomy — each links to the causes and fixes in full.
medium · common
The same question gives different answers each time
Some variation is inherent to sampling.
high · common
Output is cut off mid-way and nothing reports an error
The model hits its output limit and stops.
medium · common
Average latency looks fine but some users wait forty seconds
Multi-step agents have long tails by construction: each step adds latency, and retries multiply it.
Questions
Which LLM observability tool should we use?
Field coverage matters far more than vendor. Langfuse, LangSmith, Braintrust, Arize Phoenix and plain OpenTelemetry can all carry the spec above; a tool with complete fields beats a better tool with gaps.
If you have no strong preference: self-hosted Langfuse if data residency matters, a managed option if you would rather not run it. Decide on residency and pricing model, not features.
Is OpenTelemetry enough for LLM tracing?
Yes, technically — the semantics for LLM spans are conventions on top of ordinary tracing, and if you already run OTel it is the lowest-friction path.
What you give up is the purpose-built UI: side-by-side prompt diffing, replay, and eval integration. If your team lives in traces already, OTel is fine. If nobody will open the tracing UI unprompted, a dedicated tool gets used more.
How much latency does tracing add?
Single-digit milliseconds per span if you batch and export asynchronously, which is negligible against model calls measured in seconds.
The real cost is storage and retention, not latency. Keep full traces for a short window and sampled traces for longer — but keep all failed runs, because those are the ones you will need.
We already log prompts and responses. Is that observability?
No, and this is the most common misconception. Unlinked prompt/response pairs cannot answer "what did this run do and why" — they have no trace ID, no step ordering, no tool arguments, no retry count, no stop reason.
Logs tell you what was said. A trace tells you what was decided. Debugging an agent needs the second.
Want this done, or just want a second opinion on it?
The method above is genuinely what I do — if your team can run it themselves, run it. If you would rather it were done in a week by someone who has done it before, that is what the agent production readiness audit is for.