How do you design caching for LLM applications?

I think of three layers. The simplest is an exact-match cache: hash the model name and version, the full prompt and the sampling parameters, and store the response. It is safe and works well for repeated requests such as the same document summarized by many users or batch jobs that are re-run.

The second is prompt or prefix caching. Many providers and inference servers can reuse the computation for a repeated prompt prefix, which cuts cost and time to first token. To benefit, I order prompts with the stable parts first, such as system instructions, tool definitions and shared documents, and the variable user input last.

The third is semantic caching, where a new query reuses the answer for a sufficiently similar past query. It can save a lot, but it is risky: two questions can be close in embedding space and need different answers, for example plans that differ by one word. I use it only for generic content, with a strict threshold, and never across users when answers depend on their data. Cache keys include tenant and permission scope, entries expire, and they are invalidated when the underlying documents change.