Decision support
Five agent metrics worth reporting upward
And four that look impressive while telling you nothing.
The question
What should I actually be tracking and reporting about our agent?
The short answer
Five: cost per conversation, eval pass rate, failure rate by named category, p95 latency per operation, and containment or escalation rate. Those five answer every question an executive asks — is it working, is it getting better, what does it cost, and is it getting worse — and each one is a leading indicator rather than a post-mortem. Stop reporting total requests, total spend and model benchmark scores; none of them tell you whether the thing works.
Written for
- Engineering leaders reporting to a CEO or board
- Product leaders accountable for an AI feature
- Anyone asked "how is the AI working?" and lacking a real answer
Why the answer is that
The reason "how is the AI working?" is hard to answer is that most teams track volume and spend, which describe usage rather than quality. You can serve a million requests badly.
The five below share a property: each is a rate or a per-unit number, so it stays meaningful as you grow and it moves before an incident rather than after. Totals only tell you what already happened.
One practical note on reporting: pair each metric with its trend, not its absolute value. An eval pass rate of 82% means nothing on its own. Eighty-two percent and falling for three weeks is the most important sentence in your update.
The five to report
Definition, what a healthy value looks like, and the executive question it answers.
| Metric | How to define it | Answers |
|---|---|---|
| Cost per conversation | Total spend divided by completed conversations, attributed by feature and customer | "What does this cost us, and does it scale?" Also the earliest warning of a regression — it moves before anything else does. |
| Eval pass rate | Percentage of your eval suite passing, reported per category rather than as one number | "Is it getting better or worse?" The only metric that answers this credibly. |
| Failure rate by category | Failures tagged against named categories, as a rate of total runs | "What is actually going wrong, and what should we fix first?" Frequency times severity gives you the priority order. |
| p95 latency per operation | 95th percentile, per operation type — never an average | "Is it fast enough?" The average hides exactly the runs where users abandon. |
| Containment or escalation rate | Share of interactions resolved without a human, or handed off | "Is this actually saving us anything?" The closest thing to a business outcome metric. |
Five is deliberate. A dashboard with thirty metrics is a dashboard nobody reads, and the ones that get dropped are always the leading indicators.
Stop reporting these
Common on AI dashboards, and each one can improve while your product gets worse.
| Vanity metric | Why it misleads | Report instead |
|---|---|---|
| Total requests or messages | Measures usage, not value. Rises when the agent is unhelpful and people retry. | Containment rate |
| Total monthly spend | Grows with success and with waste identically. Cannot distinguish them. | Cost per conversation |
| Average latency | Hides the tail where users actually abandon. | p95 per operation |
| Model benchmark scores | Measures the model on someone else's tasks, not your system on yours. | Eval pass rate on your own cases |
| Thumbs up / down rate | Very few users rate anything, and those who do are not representative. | Failure rate by category, from traces |
| Uptime | An agent can be 100% available and consistently wrong. | Eval pass rate |
The shared flaw: every one of these can improve while the product degrades. That is the definition of a vanity metric, and it is why "the numbers look fine" and "customers are unhappy" so often coexist.
Get the answer for your own system in four minutes
The free scorecard produces a score out of 100, a breakdown across reliability, cost and observability, and your gaps ranked by production risk with a fix and effort estimate for each. It prints to a PDF you can take into a planning meeting — which is usually more persuasive than an argument.
Run the scorecardIf you are the engineer who found this page: Written to be forwarded. If you are the engineer who found this, send the relevant one up — it is framed for the person who has to approve the work.
If you decide to do the work
The engagements this decision leads to, each with the full method and effort estimates — whether your team runs it or I do.
Questions
What metrics should I report to my board about our AI?
Cost per conversation, eval pass rate, failure rate by named category, p95 latency per operation, and containment rate. Together they answer whether it works, whether it is improving, what it costs, and whether it is degrading.
Report each with its trend rather than its absolute value. "82% and falling for three weeks" is the sentence that matters; "82%" on its own is not information.
Why is total LLM spend the wrong thing to track?
Because it rises with success and with waste in exactly the same way, so it cannot tell you which is happening. Spend doubling because usage doubled is good news; spend doubling because conversation history is unbounded is a bug.
Cost per conversation separates them. It is also the earliest warning signal you have — it moves days before anything else does.
How do I measure whether an AI agent is actually good?
Eval pass rate on cases drawn from your own real traffic, reported per category. Nothing else answers the question credibly — model benchmarks measure someone else's tasks, and user ratings come from an unrepresentative few.
If your team cannot produce this number, that is the finding. It means no one can currently tell whether last month's changes helped.
How often should these be reviewed?
Weekly at team level, monthly at leadership level, and alerted on continuously — the alerts should fire on trends rather than thresholds, because a threshold alert fires when it is already too late.
The review cadence matters less than someone owning it. A dashboard with no named owner stops being looked at within a month.
Want a second opinion on the decision?
Describe your situation and I will tell you what I would do — including when that is "nothing" or "handle it internally". No charge for that, and it is genuinely how a lot of these conversations end.