AI Engineer Interview Questions / Questions
Entry

What is RAG (retrieval-augmented generation)?

How to answer it in an AI engineer interview: what is being tested, a step-by-step framework, a sample answer and the mistakes to avoid.

Get the playbook Score my answer free

$19.99 on Gumroad, instant download40% off with code EARLYBF until Nov 3, 11:59 PM PTEARLYBF: 40% off until Nov 3

Smaller start: Score your answers against 50 rubrics, $9. 50 AI engineer questions, one rubric each, PDF on Gumroad.

Practiced one out loud? Paste your answer into the free self-check and get a score in 30 seconds

What the interviewer is testing

Whether you can describe both halves of a RAG system (indexing and query time), name the parts that decide quality (chunking, hybrid search, re-ranking, metadata), and know that retrieval, not the generator, is usually where it fails first.

Answer framework

  1. Define it in one line: retrieve relevant chunks from your own documents at query time and put them in the prompt, so the model answers from them and cites sources.
  2. Indexing path: parse documents, split them into chunks, embed each chunk, and store the vectors with metadata (source, date, permissions).
  3. Query path: optionally rewrite the query, embed it, retrieve top-k by vector and/or keyword search, apply metadata filters, then re-rank.
  4. Generation: put the best chunks in the prompt with instructions to answer only from them and cite sources; make declining an acceptable answer.
  5. Close with evaluation: measure retrieval and generation separately, because retrieval usually fails first.

Sample answer

RAG has two paths. At indexing time I parse documents, split them into chunks, embed each chunk and store the vectors with metadata such as source, date and permissions. At query time I optionally rewrite the query, embed it, retrieve the top-k chunks by vector and/or keyword search, apply metadata filters and re-rank, then put the best chunks in the prompt with instructions to answer only from them and cite sources.

For chunking I follow document structure (headings, paragraphs, code blocks, table rows) rather than fixed character counts, typically a few hundred tokens with modest overlap. I use hybrid search when queries contain exact identifiers, product codes or error messages that embeddings blur, and a cross-encoder re-ranker on a wide candidate set (for example top 50) to keep the best 5 to 10.

I evaluate retrieval and generation separately. When RAG fails, it usually fails first at retrieval: the wrong chunk or a missing chunk, not the generator.

Likely follow-ups

Where does RAG usually fail first? Retrieval: the wrong chunk or a missing chunk, not the generator.
How should a RAG system decide to say "I don't know"? Combine signals: low retrieval or re-ranker scores below a calibrated threshold, no chunk that answers the question, and a verifier step that checks the draft answer against sources. Tell the model that declining is acceptable.

Common mistakes

The PlaybookWant the full method?

The 0 to 1 AI Engineer Interview Playbook covers the answer frameworks, system-design walkthroughs and an 8-week plan.

Get the playbook 40% off with code EARLYBF until Nov 3, 11:59 PM PTEARLYBF: 40% off until Nov 3

Smaller start: Score your answers against 50 rubrics, $9. 50 AI engineer questions, one rubric each, PDF on Gumroad.

Interviewing for PM or TPM? See the full PM package, $59 on Gumroad: a resume system, a PM, TPM and PGM interview handbook, a 1:1 cheat sheet and a PM playbook. PM and TPM content, not AI engineer prep.

Related questions

Based on question #26, #27, #28, #29, #33 in the full question bank.