AI Engineer Interview Questions / Questions
Senior

How do you debug poor RAG answer quality?

How to answer it in an AI engineer interview: what is being tested, a step-by-step framework, a sample answer and the mistakes to avoid.

Get the playbook Score my answer free

$19.99 on Gumroad, instant download40% off with code EARLYBF until Nov 3, 11:59 PM PTEARLYBF: 40% off until Nov 3

Smaller start: Score your answers against 50 rubrics, $9. 50 AI engineer questions, one rubric each, PDF on Gumroad.

Practiced one out loud? Paste your answer into the free self-check and get a score in 30 seconds

What the interviewer is testing

Whether you debug systematically by stage instead of rewriting the prompt, and whether you know where RAG usually breaks.

Answer framework

  1. Collect failing questions and, for each, the document that should answer it.
  2. Ingestion and parsing: was the document indexed, and is its text intact (tables, PDFs, scans)?
  3. Chunking: is the answer inside one sensible chunk with its context?
  4. Retrieval and ranking: did the chunk appear in the top-k, and was it ranked high enough to reach the prompt?
  5. Generation: was it in the prompt and did the model use it? Fix the earliest failing stage, then add the case to the eval set.

Sample answer

I start by turning complaints into concrete cases: a set of questions with bad answers, and for each one the document and passage that should have answered it. Then I trace each case through the pipeline in order.

Was the document ingested at all, and does the parsed text look right? Broken PDF tables and scanned pages are common culprits. Is the answer in one chunk, or was it split from its heading? If I search the index for that chunk directly, does it appear in the top 50, and where does it land after re-ranking? Did it make it into the final prompt, and did the model actually use it or ignore it?

I fix the earliest stage that fails, because later stages cannot recover from it. In my experience most failures are parsing, chunking or retrieval, not the generator, so rewriting the prompt is usually the last step. Every fixed case goes into the regression set so the same failure does not return.

Likely follow-ups

The right chunk is retrieved but the answer is still wrong. What next? Check its position and the amount of competing context, tighten instructions to answer from sources, and evaluate faithfulness on that case.

Common mistakes

The PlaybookWant the full method?

The 0 to 1 AI Engineer Interview Playbook covers the answer frameworks, system-design walkthroughs and an 8-week plan.

Get the playbook 40% off with code EARLYBF until Nov 3, 11:59 PM PTEARLYBF: 40% off until Nov 3

Smaller start: Score your answers against 50 rubrics, $9. 50 AI engineer questions, one rubric each, PDF on Gumroad.

Interviewing for PM or TPM? See the full PM package, $59 on Gumroad: a resume system, a PM, TPM and PGM interview handbook, a 1:1 cheat sheet and a PM playbook. PM and TPM content, not AI engineer prep.

Related questions

Based on question #40 in the full question bank.