Practiced one out loud? Paste your answer into the free self-check and get a score in 30 seconds
What the interviewer is testing
Whether you understand sparse activation, can separate total from active parameters, and know the systems costs that make MoE harder to train and serve than a dense model.
Answer framework
- Structure: replace a dense feed-forward block with many expert networks plus a router.
- Routing: for each token the router picks the top-k experts (often k = 2) and combines their outputs.
- Benefit: total parameters are large but active parameters per token are small, so quality per training FLOP improves.
- Costs: all experts must still be held in memory; load-balancing losses are needed so the router does not overuse a few experts.
- Serving: expert parallelism adds communication, and batching is less efficient at low traffic because tokens spread across experts.
Sample answer
In a Mixture of Experts layer the single feed-forward block of a transformer layer is replaced by many experts, each its own feed-forward network, and a small router. For every token the router scores the experts and sends the token to the top few, often two, then mixes their outputs by the router weights. Attention layers are usually still dense.
The appeal is that you get the capacity of a very large model while each token only pays for a fraction of it. A model can have several times more total parameters than active parameters, so for the same training compute it usually reaches better quality than a dense model.
The trade-offs are mostly systems problems. Memory is set by total parameters, because every expert has to be loaded somewhere. Training needs auxiliary load-balancing losses or capacity limits, otherwise a few experts get most tokens. In serving, experts are often split across GPUs, so tokens move between devices, and at low traffic each expert sees small batches, which wastes compute. MoE pays off most at high, steady volume.
Likely follow-ups
Common mistakes
- Saying each expert specializes in a human-readable topic; routing is learned and not necessarily interpretable.
- Comparing MoE and dense models by total parameters only.
- Ignoring memory: sparse compute does not mean small memory.
- Leaving out load balancing.
The 0 to 1 AI Engineer Interview Playbook covers the answer frameworks, system-design walkthroughs and an 8-week plan.
Get the playbook 40% off with code EARLYBF until Nov 3, 11:59 PM PTEARLYBF: 40% off until Nov 3Smaller start: Score your answers against 50 rubrics, $9. 50 AI engineer questions, one rubric each, PDF on Gumroad.
Interviewing for PM or TPM? See the full PM package, $59 on Gumroad: a resume system, a PM, TPM and PGM interview handbook, a 1:1 cheat sheet and a PM playbook. PM and TPM content, not AI engineer prep.
Related questions
Based on question #12 in the full question bank.