The message that kicked it off was four words: "Is this bill right?"
Our finance lead had dropped a screenshot of the usage dashboard into Slack. The total at the bottom was just under $14,000 for the month. When we'd shipped the first AI feature six months earlier, it had been closer to $1,800.
Nobody had approved a jump like that because there was never a jump to approve. It had crept up instead — more users, another feature, slightly longer prompts, an extra model call somewhere — until the line item was growing faster than the feature was earning.
I said I'd take a look, figuring I'd find some obvious waste and knock maybe 10–15% off.
A few weeks later the bill was around $4,000. The product behaved the same, our quality numbers hadn't moved, and users noticed nothing.
Finance did.
What I didn't expect was that most of the savings had nothing to do with the expensive-looking model calls. They came from boring things we'd stopped paying attention to.
And I nearly broke the product on the way there.
I was looking in the wrong place
My first instinct was the flagship model. Those calls were slow and expensive per request, so they were easy to blame.
But the provider dashboard only showed total tokens and spend. It couldn't tell me why we were spending it.
So before touching anything, I added LLM observability.
I wired our call paths into Langfuse and tagged traces by feature, model, input and output tokens, latency, and estimated cost.
The important part wasn't any one metric. It was tracing the whole workflow.
One user action might trigger retrieval, classification, the main generation, and occasionally a retry. Per call, several of those looked trivial. Per user workflow, the expensive paths jumped out immediately.
It was maybe a day of plumbing, and nearly everything useful afterward came from reading the numbers.
The biggest chunk of spend wasn't the handful of flagship generations. It was input tokens on our cheapest, highest-volume calls.
We were shipping roughly the same bloated context thousands of times a day, and because each request cost a fraction of a cent, nobody had cared.
A fraction of a cent times enough traffic stops being a fraction of a cent.
That was the first lesson: don't optimize LLM spend from the invoice. Instrument the path that creates the invoice.
The system prompt had become a junk drawer
The first big win came from the prompt itself, which had quietly grown to around 1,800 tokens.
It hadn't started that way. It had accumulated.
A response comes back in the wrong tone, so someone adds "Always respond in…". A format breaks, so someone adds "Never return…". Someone finds an edge case and adds another rule without noticing that something near the top already says almost the same thing.
Six months of that across a few people, and the system prompt had become a history of every small problem we'd ever had.
As far as I could tell, nobody had read the whole thing start to finish in months.
So I did.
A good third was redundant, described behavior the current model handled fine on its own, or guarded against product behavior that no longer existed.
I rewrote it instead of editing around the edges.
About 1,800 tokens became roughly 600.
The six worked examples we'd added to enforce formatting got the same treatment. The model had improved since we'd written them, and two examples held the behavior that six used to.
The other four were baggage by then.
Prompt cleanup alone accounted for roughly 30% of the original LLM bill.
To stop the same bloat coming back six months later, we gave each call path a rough token budget — system instructions, retrieved context, history, and response.
Not because every request has to hit an exact number, but because a context window shouldn't be treated as free real estate just because the model accepts it.
Now if someone wants to add 500 permanent tokens to a high-volume prompt, that cost is visible before it ships.
The optimization that backfired
Here's where I got cocky.
The prompt win was easy, so I went looking for the next obvious thing.
On our RAG paths we were passing too many retrieved chunks into context — often pages of text when the answer lived in two paragraphs.
The fix looked trivial: retrieve fewer chunks.
I dropped top_k from five to two, watched the token count fall, saw the bill tick down, and moved on feeling clever.
A few days later, a customer asked a question and got a confident, well-formatted, incomplete answer.
The missing piece had been in a chunk the retriever no longer returned.
The model hadn't failed. We'd starved it of the evidence.
That was the nasty part: nothing crashed. There was no exception and no alert. The system simply produced a plausible wrong answer and carried on.
That one stung.
I'd been treating the problem as token reduction when the real problem was cost without dropping quality.
You can cut an LLM bill by 100% by turning the feature off. That's not optimization.
So I backed the retrieval change out and stopped guessing.
Before making any more large changes, I built an eval set from a few hundred real, production-shaped requests.
I made sure it wasn't just the happy path. It included ambiguous inputs, support-ticket edge cases, retrieval failures we'd seen before, formatting problems, and the awkward tail where cheaper models were most likely to fail.
For deterministic tasks we could score against known-good outputs. For open-ended generations, we used quality criteria instead.
Nothing fancy.
What mattered was having a repeatable scorecard after every change.
Then I went back to retrieval and did it properly.
The goal was never less context.
It was less irrelevant context.
We tuned retrieval against the eval set to find how much we could trim before answer quality moved.
That was more work than changing top_k, but it meant we were removing noise instead of removing evidence.
Using the expensive model for boring work
The next large saving was model routing.
We had a very sophisticated reason for sending everything to the flagship model: it's what we'd wired up first.
When I pulled a few hundred real requests, the mismatch was obvious.
Classification, short extraction, reformatting, simple transformations, questions where retrieval had already done most of the work — none of those needed the strongest model.
Only a genuinely hard tail did.
So I added routing.
Simple rules handled the obvious cases. A cheap model classified the fuzzy middle. Easy work went to a smaller model, and the flagship kept the harder requests.
That was roughly another 25%.
But model routing creates its own failure mode.
A cheap model answering an easy question is great. A cheap model thinking a hard question is easy is how you degrade a product quietly.
After the retrieval incident, I didn't want another silent failure.
So for tasks we could validate, I stopped treating routing as a one-shot decision.
The smaller model attempted the task first. If the output failed validation or hit one of our uncertainty conditions, the request escalated to the stronger model.
Cheap when possible, expensive when necessary.
That was safer than pretending we could build a perfect "easy versus hard" classifier.
The router itself also costs money and adds latency, so we measured that rather than assuming it away.
For us it was small compared with the savings. In another workload, it might not be.
Real traffic repeats itself constantly
The last meaningful chunk came from caching.
We used three levels, safest first.
Provider-side prompt caching was the easy one.
A large part of every request started with the same static instructions, so once that prefix could be reused, we stopped paying full input price to process the identical block repeatedly.
Not free. Just much cheaper repeated input.
Then came exact-match caching.
A surprising amount of traffic was genuinely identical — same common question, same input, same valid answer.
Those requests don't need another model call if we've already computed an answer that is still valid.
Semantic caching was the dangerous one.
The idea is seductive: embed the incoming request, find a similar past one, reuse its answer. It works.
It can also return a beautifully written wrong answer — two questions can sit almost on top of each other in embedding space while differing on the one word that changes everything.
So I kept the similarity threshold tight and only enabled semantic caching for query types where "similar" had a real chance of meaning "same intent."
Similarity also wasn't enough.
Freshness mattered. Yesterday's answer can be wrong today if the underlying data changed.
And in a multi-tenant SaaS product, "same question" definitely doesn't mean "same answer" across customers.
So cache identity included the things that could change the result: tenant, relevant document or source version, and in some places prompt or model version.
That made caching less magical, but safe enough to use.
A cache miss costs another small model call.
A bad cache hit costs trust.
I'll take the miss.
Caching accounted for roughly the final 15%.
We stopped tracking just "LLM spend"
The $14,000 monthly number was useful because it scared someone, but it was a bad engineering metric.
If usage doubles and the bill doubles, that may be healthy.
If the bill doubles while successful tasks rise 20%, that is something else.
So we stopped looking only at monthly spend and started tracking cost per successful user workflow.
That matters once one button click can fan out into retrieval, routing, generation, retries, and fallbacks.
Each call can look cheap while the complete workflow quietly gets expensive.
Cost per successful task surfaces that much earlier than the invoice does.
What actually cut the bill
There was no single trick.
Roughly, the savings came from:
- about 30% from unnecessary input tokens and context
- about 25% from model routing
- about 15% from caching repeated work
The numbers don't separate perfectly because the changes affect each other.
A shorter prompt also makes routed requests cheaper. Caching means some requests never reach the router. Better retrieval changes downstream token usage.
The order mattered more than the percentages.
We measured first.
We went after the largest cost centers.
And after every change we checked quality before moving on — a discipline I picked up the hard way after the one time I didn't.
A few weeks later, the bill that had been just under $14,000 was hovering around $4,000.
Same product. Same quality bar. Very different economics.
If you take one thing from this
LLM cost optimization is an architecture problem more than a pricing problem.
Switching providers or negotiating rates might save something, but neither fixes an application that sends unnecessary context on every request, runs easy work through an expensive model, repeats work it's already done, or fires five calls behind one button click without measuring the full workflow.
Tokens are a resource.
Context is a resource.
Model capability is a resource.
Once an AI feature is in production, they need the same kind of instrumentation and budgeting you'd put around CPU, memory, database connections, or cloud spend.
The thing that changed our bill wasn't prompt compression, model routing, or caching.
It was visibility.
Before we instrumented the system, we had opinions about where the money went.
Afterward, we had traces.
The $14,000 bill had been climbing in plain sight for six months. By the time finance asked whether it was right, the expensive decisions had already been happening thousands of times every day.
We just couldn't see them.
And the one time I tried to cut without looking, I broke something instead.
If your AI feature is live and you can't say what one successful user workflow costs — and how much of that comes from input tokens, output tokens, retrieval, routing, retries, and repeated work — that's the number I'd find before optimizing anything else.