Practiced one out loud? Paste your answer into the free self-check and get a score in 30 seconds
What the interviewer is testing
Whether you know what each training stage contributes to a chat model, can explain why a pre-trained model alone is not a helpful assistant, and can compare RLHF with DPO without hand-waving.
Answer framework
- Pre-training: next-token prediction on a very large text corpus. It produces knowledge and fluency, not assistant behavior.
- Instruction tuning (supervised fine-tuning, SFT): train on prompt-response pairs so the model follows instructions and a chat format.
- Preference tuning: train on comparisons where one answer is better than another, to shape helpfulness, tone and refusals.
- Compare the two preference methods: RLHF trains a reward model and optimizes against it with RL (for example PPO) under a KL penalty; DPO optimizes directly on the preference pairs.
- Point out the data funnel: each stage uses far less data than the one before, but the data gets more curated.
Sample answer
I think of it as three stages with different jobs. Pre-training is next-token prediction over trillions of tokens. The model picks up facts, grammar and some reasoning, but if you ask it a question it may just continue the text, because nothing has taught it to behave like an assistant.
Instruction tuning fixes that. We fine-tune on curated prompt-response pairs in a chat format, so the model learns to follow instructions, answer in the expected shape and stop when done. Preference tuning then teaches it which of two plausible answers people prefer. In classic RLHF we fit a reward model on human comparisons and optimize the policy with PPO, with a KL penalty that keeps it close to the SFT model so it does not exploit the reward model. DPO reaches a similar goal with a single supervised-style loss on chosen and rejected pairs, so there is no separate reward model and no RL loop.
Moving down the stages, the datasets get smaller and more carefully curated: pre-training is web scale, SFT is often tens of thousands of examples, preference data is comparisons.
Likely follow-ups
Common mistakes
- Saying RLHF teaches the model new knowledge; it mostly shapes behavior that pre-training made possible.
- Forgetting the KL penalty and why it exists (reward hacking).
- Describing DPO as reinforcement learning; it is a classification-style loss on preference pairs.
- Treating instruction tuning and preference tuning as the same step.
The 0 to 1 AI Engineer Interview Playbook covers the answer frameworks, system-design walkthroughs and an 8-week plan.
Get the playbook 40% off with code EARLYBF until Nov 3, 11:59 PM PTEARLYBF: 40% off until Nov 3Smaller start: Score your answers against 50 rubrics, $9. 50 AI engineer questions, one rubric each, PDF on Gumroad.
Interviewing for PM or TPM? See the full PM package, $59 on Gumroad: a resume system, a PM, TPM and PGM interview handbook, a 1:1 cheat sheet and a PM playbook. PM and TPM content, not AI engineer prep.
Related questions
Based on question #4 in the full question bank.