The Real Cost of Running LLMs in Production (And How to Cut It)

When teams budget for an LLM feature, they almost always start from the per-token price of the model they plan to use. That number matters, but in every cost audit we've run for clients, it's rarely the biggest line item once the system is actually in production. The real costs hide in the places nobody budgeted for at all.

Context is the silent multiplier

Retrieval-augmented systems in particular tend to over-provision context out of caution — pulling in more passages than a query actually needs, on the logic that more context can only help. It doesn't just cost more per call; past a certain point, extra irrelevant context measurably hurts answer quality, because the model has to work harder to identify what's actually relevant. Trimming retrieved context to what a query genuinely needs is one of the highest-leverage cost cuts available, and it usually improves accuracy at the same time.

Retries compound quietly

A naive retry policy — retry on any failure, up to N times, with the same prompt — turns a transient rate limit into a 3x-5x cost multiplier on the affected requests, and it does so invisibly because each individual retry looks cheap. The fix is boring but effective: exponential backoff, a hard cap on retries, and distinguishing between errors worth retrying (rate limits, timeouts) and errors that will just fail again (a malformed prompt, a genuinely unanswerable query).

• Cache aggressively for queries that repeat — FAQ-style traffic is far more common than teams expect.
• Route by task difficulty: a cheaper, smaller model for classification and extraction, a frontier model only where reasoning quality actually matters.
• Set explicit token budgets per feature and alert on overruns, the same way you'd alert on any other infra cost.
• Batch what can be batched instead of paying the latency and cost overhead of one-off calls.

The cost of not running evals

The least visible cost is the one that doesn't show up on an invoice: shipping a cheaper model without evaluating whether it actually holds up on your specific task, then quietly eating the cost of degraded output in support tickets, re-work, or churn. Running a real evaluation set against every model or prompt change is cheap compared to finding out in production that the switch degraded quality in a way nobody measured.

None of this requires exotic infrastructure. It requires treating LLM calls like any other expensive, fallible network dependency — measured, capped, cached, and evaluated — instead of a line item that gets a blank check because it's new.