How do you reduce LLM inference cost?

I start by measuring where the money goes. I break down spend by feature, model and token type, because output tokens usually cost several times more than input tokens, and I look at cost per successful task rather than per call, since a cheap model that needs three retries is not cheap.

The biggest lever is usually model choice. Many requests, such as classification, extraction or simple rewrites, work well on a small model, so a router sends those there and only escalates hard cases. Next is token volume: trim system prompts, retrieve five good chunks instead of twenty, cap output length and summarize long conversation history.

Then reuse: exact-match caching for repeated requests, prefix caching by putting static content first, and batch APIs, often discounted, for offline jobs such as nightly summarization. If we host our own models, quantization, continuous batching and keeping GPUs busy lower cost per token, and distillation can replace a large model on a narrow, high-volume task. Every change goes through the eval suite, because a cost cut that hurts task success can cost more in user trust.