Practiced one out loud? Paste your answer into the free self-check and get a score in 30 seconds
What the interviewer is testing
Whether you split evaluation into retrieval and generation, know concrete metrics for each, build a golden set from real questions, and can trace a failing answer to the stage that broke.
Answer framework
- Evaluate retrieval and generation separately.
- Retrieval: recall@k, precision@k, MRR or nDCG against labelled question-to-source pairs.
- Generation: faithfulness (claims supported by the retrieved context), answer relevance, correctness against a reference, citation accuracy.
- Data and grading: a golden set from real user questions, an LLM judge calibrated against human labels, plus latency and cost per answer.
- Debugging: trace a failing question stage by stage and fix the earliest failing stage.
Sample answer
I evaluate it in two parts. For retrieval I use recall@k, precision@k, MRR or nDCG against labelled question-to-source pairs. For generation I measure faithfulness, meaning claims supported by the retrieved context, plus answer relevance, correctness against a reference and citation accuracy.
I build the golden set from real user questions, use an LLM judge calibrated against human labels, and track latency and cost per answer. For faithfulness I break each answer into claims, check each against the sources and report the share supported, and I check that cited passages actually contain the claims. I also evaluate refusal precision, not only answer accuracy.
When an answer is bad, I trace it: was the right document ingested and parsed correctly, chunked sensibly, in the top-k, ranked high enough, included in the prompt, and used by the model? I fix the earliest failing stage. Most issues are parsing, chunking or retrieval, not the generator.
Likely follow-ups
Common mistakes
- Scoring only the final answer, so you cannot tell whether retrieval or generation failed.
- Building the eval set from invented questions instead of real user questions.
- Using an uncalibrated LLM judge as the source of truth.
- Ignoring latency and cost per answer.
- Changing the prompt or model before checking whether the right chunk was even retrieved.
The 0 to 1 AI Engineer Interview Playbook covers the answer frameworks, system-design walkthroughs and an 8-week plan.
Get the playbook 40% off with code EARLYBF until Nov 3, 11:59 PM PTEARLYBF: 40% off until Nov 3Smaller start: Score your answers against 50 rubrics, $9. 50 AI engineer questions, one rubric each, PDF on Gumroad.
Interviewing for PM or TPM? See the full PM package, $59 on Gumroad: a resume system, a PM, TPM and PGM interview handbook, a 1:1 cheat sheet and a PM playbook. PM and TPM content, not AI engineer prep.
Related questions
Based on question #31, #40, #68, #33 in the full question bank.