Practiced one out loud? Paste your answer into the free self-check and get a score in 30 seconds
What the interviewer is testing
Whether you understand autoregressive decoding well enough to explain what is cached and why, can estimate memory with the formula, and know that the cache, not only the weights, often limits batch size and context length.
Answer framework
- State the reason: during autoregressive decoding the keys and values of earlier tokens do not change.
- State the saving: store and reuse them, so each new token computes only its own Q, K and V instead of recomputing the whole prefix.
- State the cost: memory per sequence is roughly 2 x layers x kv_heads x head_dim x tokens x bytes per value.
- State the consequence: it often limits batch size and context length more than the weights do.
- Name the mitigations: grouped-query attention, KV quantization and paged allocation (as in vLLM's PagedAttention).
Sample answer
During autoregressive decoding the keys and values of earlier tokens do not change, so they are stored and reused; each new token computes only its own query, key and value. That turns per-step attention from recomputing the whole prefix into one pass over cached tensors.
The trade-off is memory. Per sequence it is roughly 2 x layers x kv_heads x head_dim x tokens x bytes per value. For a 32-layer model with 8 KV heads, head_dim 128, 8k tokens in fp16 that is 2 x 32 x 8 x 128 x 8192 x 2 bytes, about 1 GiB per sequence. That is why the KV cache often limits batch size and context length more than the weights do.
Grouped-query attention, KV quantization and paged allocation, as in vLLM's PagedAttention, exist to shrink or pack it.
Likely follow-ups
Common mistakes
- Saying queries are cached; only keys and values of earlier tokens are reused.
- Describing the speed benefit without the memory cost.
- Assuming model weights are the only memory constraint when serving long contexts or large batches.
- Not being able to do the back-of-envelope memory estimate.
- Confusing the KV cache with response caching or prompt caching for shared prefixes, which are application-level cost levers.
The 0 to 1 AI Engineer Interview Playbook covers the answer frameworks, system-design walkthroughs and an 8-week plan.
Get the playbook 40% off with code EARLYBF until Nov 3, 11:59 PM PTEARLYBF: 40% off until Nov 3Smaller start: Score your answers against 50 rubrics, $9. 50 AI engineer questions, one rubric each, PDF on Gumroad.
Interviewing for PM or TPM? See the full PM package, $59 on Gumroad: a resume system, a PM, TPM and PGM interview handbook, a 1:1 cheat sheet and a PM playbook. PM and TPM content, not AI engineer prep.