AI Engineer Interview Questions / Questions
Mid

What is model distillation and when would you use it?

How to answer it in an AI engineer interview: what is being tested, a step-by-step framework, a sample answer and the mistakes to avoid.

Get the playbook Score my answer free

$19.99 on Gumroad, instant download40% off with code EARLYBF until Nov 3, 11:59 PM PTEARLYBF: 40% off until Nov 3

Smaller start: Score your answers against 50 rubrics, $9. 50 AI engineer questions, one rubric each, PDF on Gumroad.

Practiced one out loud? Paste your answer into the free self-check and get a score in 30 seconds

What the interviewer is testing

Whether you understand how distillation works for LLMs, can name the conditions where it pays off, and know the practical limits including licensing terms.

Answer framework

  1. Define it: train a smaller student model to imitate a larger teacher model's outputs.
  2. Methods: train on the teacher's generated responses, or match its probability distributions when you have access to logits.
  3. When to use it: a narrow, high-volume task where the large model works well but costs too much or is too slow.
  4. Process: generate teacher outputs on real inputs, filter for quality, fine-tune the student, compare on a held-out eval.
  5. Limits: the student rarely matches the teacher outside the task, and model terms of use may restrict training on outputs.

Sample answer

Distillation transfers behavior from a large teacher model to a smaller student. For LLMs the most common version is simple: run the teacher on a large set of realistic inputs, keep the good responses, and fine-tune the student on them. If you control the teacher you can also train the student to match its full token probability distributions, which carries more information than a single answer.

I would use it when a strong model handles a well-defined task, such as classifying support tickets or extracting fields from invoices, but serving it at our volume costs too much or adds too much latency. A distilled small model can often get close to the teacher on that one task at a fraction of the cost per request.

I would validate it on a held-out set scored against ground truth, not just against teacher agreement, because the student inherits the teacher's errors. I would also watch the edge cases, which small models handle worse, and check the teacher's license, since some providers restrict using outputs to train competing models.

Likely follow-ups

Distillation vs quantization? Quantization shrinks the same model's numeric precision; distillation trains a different, smaller model. They can be combined.

Common mistakes

The PlaybookWant the full method?

The 0 to 1 AI Engineer Interview Playbook covers the answer frameworks, system-design walkthroughs and an 8-week plan.

Get the playbook 40% off with code EARLYBF until Nov 3, 11:59 PM PTEARLYBF: 40% off until Nov 3

Smaller start: Score your answers against 50 rubrics, $9. 50 AI engineer questions, one rubric each, PDF on Gumroad.

Interviewing for PM or TPM? See the full PM package, $59 on Gumroad: a resume system, a PM, TPM and PGM interview handbook, a 1:1 cheat sheet and a PM playbook. PM and TPM content, not AI engineer prep.

Related questions

Based on question #56 in the full question bank.