How do you reduce LLM latency?

I split latency into parts before changing anything. Time to first token depends on queueing, network and prefill, which grows with prompt length. Generation time is the number of output tokens divided by decode speed. Around both sit retrieval, re-ranking and tool calls. A trace of a slow request usually shows one dominant part.

For user-facing chat, streaming is the first fix, because perceived latency is mostly time to first token. Then I cut output length, since each output token is a sequential decode step: ask for concise answers, set max tokens, and avoid generating long reasoning the user never sees. Shorter prompts help prefill, and prefix caching makes repeated system prompts and documents nearly free.

Next is architecture. I use small fast models for routing, classification and extraction, and reserve large models for the final answer. Independent steps such as retrieval from two sources or several tool calls run in parallel rather than one after another. On self-hosted models, quantization and speculative decoding can raise tokens per second. I track p50 and p95 separately, because tail latency is what users complain about.