Cost & latency · LLM cost reduction
Cut agent spend without quietly cutting quality
Usually 40–70% recoverable, and the levers are ranked.
You probably have this problem if
- Spend multiplied with no corresponding increase in users
- You cannot say what a single conversation costs
- The provider invoice is the first place you learn about a change
- One frontier model serves every task, including trivial ones
- There is no ceiling that stops a runaway loop over a weekend
What you end up with
A materially lower bill, with evidence that quality held.
- Usually part of
- Agent Production Readiness Audit
- $4,000 – $6,000
- Phases
- 5 steps, detailed below with effort per step
- Domains I have done this in
- HR analyticsRecruitmentLead generation
Why this is the thing to fix
Almost every expensive agent is expensive for structural reasons, not because the model's price is unreasonable. Conversation history resent in full on every turn. Tool definitions billed on every call, used or not. Retries nobody counts. A frontier model doing classification.
That means the savings are usually large and boring. Across the engagements I have run, 40–70% is a normal recovery, and almost all of it comes from the first three levers below.
The important discipline is the pairing: every cost change goes through an eval suite. Cost reduction without quality measurement is not optimisation, it is a gamble you have not priced. This is the main reason I push for evals first when both are missing.
The levers, in the order I attempt them
Ranked by return per day of engineering. Percentages are typical recovery from engagements and public pricing arithmetic, not guarantees — run your own numbers in the calculator.
| Lever | Typical recovery | Effort | The catch |
|---|---|---|---|
| Route by task difficulty | 40–70% | 3–5 days | Needs evals per tier, or you are trading quality blind |
| Cache stable prompt prefixes | 20–40% of input cost | 1–2 days | Cache writes cost 1.25–2× base; a low hit rate loses money |
| Bound conversation history | 20–50% on long chats | 2–4 days | Naive truncation drops your system prompt — re-assert constraints |
| Trim tool definitions | 8–25% of input cost | 1–2 days | Vague descriptions cause hallucinated arguments; be shorter, not vaguer |
| Fix the retry cause | 5–15% | 2–4 days | Usually unactionable tool errors; fixes latency at the same time |
| Batch non-urgent work | 50% on eligible traffic | 2–3 days | Only for work that tolerates delay |
| Cap tokens per request and per run | Prevents tail events | 1 day | Not a saving — an insurance policy against a five-figure weekend |
Do them in this order. Routing first because it is the largest and its measurement infrastructure (per-tier evals) is what makes the rest safe.
Cost attribution tags to add before you optimise
You cannot reduce what you cannot attribute. Log these on every call, then chart cost per conversation as a first-class product metric. This is a day of work and it is the prerequisite for everything above.
- 1feature — which product surface triggered this call
- 2tenant / customer — so a single heavy account is visible
- 3task_type — classification, extraction, generation, routing (this is what tells you what to route)
- 4model_id — pinned version, to catch silent fallbacks to a pricier model
- 5tokens_in / tokens_out / cache_read_tokens — separately, per call
- 6cost_usd — computed at write time, never derived later from aggregates
- 7retry_count — so retry spend is a line item rather than invisible
- 8step_index — so per-turn cost growth over a conversation is visible as a curve
Skip this and every optimisation becomes an argument about which change caused the invoice to move.
How I actually do it
With effort per step, so you can judge whether to hire me or hand this to someone on your team.
- 11 day
Attribute the spend
Add the tags above and chart cost per conversation by feature and tenant. Usually reveals one or two surprises within a day.
- 20.5 day
Profile a real run
Decompose one representative conversation into prompt, history, retrieved context, tool definitions and output. The proportions tell you which lever applies.
- 31–3 days if none exists
Establish the quality baseline
An eval suite, or the subset of one, so every subsequent change can be checked. Without this the rest is guesswork.
- 43–8 days depending on scope
Apply levers in order, measuring each
One change at a time with an eval run between, so a quality regression is attributable to the change that caused it.
- 51 day
Install the ceilings
Per-request token caps, per-user daily limits, and a global breaker that degrades to a cheaper model rather than spending without bound.
What you keep
- Cost attribution by feature, tenant and task type on a dashboard
- A decomposed cost profile of your real traffic
- Routing implemented with per-tier quality verified by evals
- Caching, history bounds and tool-definition trimming where they pay
- Hard ceilings: per-request, per-user and a tested global circuit breaker
- A written record of what changed, what it saved, and what quality did
Failure modes this prevents
From the failure taxonomy — each links to the causes and fixes in full.
high · common
Spend jumped several times over with no corresponding traffic increase
Month-over-month spend multiplies while usage looks flat.
critical · very common
The agent loops, repeating the same step until something kills it
The agent calls a tool, dislikes the result, and calls it again with near-identical arguments — indefinitely.
medium · common
Average latency looks fine but some users wait forty seconds
Multi-step agents have long tails by construction: each step adds latency, and retries multiply it.
Questions
How much can we realistically cut our LLM costs?
40–70% is a normal recovery on an agent that has not been optimised, and most of it comes from routing tasks to cheaper models plus caching stable prompt prefixes.
The honest caveat: that range assumes nobody has done this work yet. If you already route and cache, remaining wins are smaller and come from history bounds and tool-definition trimming.
Will using a cheaper model hurt quality?
For classification, extraction, routing and formatting, usually not measurably — those tasks do not need frontier reasoning. For genuinely hard reasoning, yes.
The point is that this is a measurable question, not a matter of opinion. Route the task, run your evals on that tier, and read the number. Teams that skip the measurement either overpay everywhere out of caution or degrade quality without noticing.
Does prompt caching always save money?
No. A cache write costs more than ordinary input — 1.25× base for a 5-minute cache and 2× for an hour — while a read costs about 0.1×. It only pays once content is genuinely re-read: after one read for the short tier, two for the long one.
Caching a prefix that is never reused makes it more expensive. Check your actual hit rate before assuming the cache is helping.
What is the fastest cost win we can do ourselves this week?
Add cost attribution tags and chart cost per conversation. It is about a day of work, it saves nothing directly, and it is still the right first move — because it usually surfaces one feature or tenant consuming a disproportionate share.
After that, check your tool-definition token count. Teams are routinely surprised by a factor of two or three, and trimming is a day's work.
Want this done, or just want a second opinion on it?
The method above is genuinely what I do — if your team can run it themselves, run it. If you would rather it were done in a week by someone who has done it before, that is what the agent production readiness audit is for.