Practiced one out loud? Paste your answer into the free self-check and get a score in 30 seconds
What the interviewer is testing
Whether you can explain why fewer bits help serving (memory and bandwidth), name the common methods, and show the engineering discipline of checking quality on your own tasks afterwards.
Answer framework
- Define it: store weights, and sometimes activations or the KV cache, in fewer bits, such as 8-bit or 4-bit integers instead of 16-bit floats.
- Memory: roughly 2x smaller at 8-bit and 4x smaller at 4-bit for the weights.
- Speed: decoding is usually memory-bandwidth bound, so moving fewer bytes per token often raises throughput.
- Methods: post-training quantization such as GPTQ and AWQ keeps quality loss small at 4-bit for many models; QLoRA uses 4-bit weights for fine-tuning.
- Risk: quality loss is uneven, often showing first in math, code and long-context tasks, so re-run your evals.
Sample answer
Quantization means representing model weights with fewer bits. A 7-billion-parameter model in 16-bit needs about 14 GB just for weights; at 4-bit that drops to roughly 3.5 GB plus some overhead for scales. That can be the difference between needing a large GPU and fitting on a smaller one, or between one replica and several per GPU.
It often speeds things up too. During decoding each new token requires reading the weights from memory, so the step is usually limited by memory bandwidth rather than arithmetic. Fewer bytes per weight means more tokens per second, especially at small batch sizes.
For post-training quantization I would start with an established method such as AWQ or GPTQ at 4-bit, or 8-bit if quality is critical, and quantize the KV cache if long contexts are the bottleneck. Then I re-run our task evals, not just a general benchmark, because degradation is uneven: arithmetic, code and long-context retrieval tend to suffer first.
Likely follow-ups
Common mistakes
- Claiming quantization has no quality cost.
- Checking only perplexity or a public benchmark instead of task evals.
- Forgetting the KV cache, which can dominate memory at long context.
- Assuming 4-bit is always faster; it depends on kernels and hardware support.
The 0 to 1 AI Engineer Interview Playbook covers the answer frameworks, system-design walkthroughs and an 8-week plan.
Get the playbook 40% off with code EARLYBF until Nov 3, 11:59 PM PTEARLYBF: 40% off until Nov 3Smaller start: Score your answers against 50 rubrics, $9. 50 AI engineer questions, one rubric each, PDF on Gumroad.
Interviewing for PM or TPM? See the full PM package, $59 on Gumroad: a resume system, a PM, TPM and PGM interview handbook, a 1:1 cheat sheet and a PM playbook. PM and TPM content, not AI engineer prep.
Related questions
Based on question #10 in the full question bank.