AI Engineer Interview Questions / Questions
Mid

How does re-ranking work in RAG and why use it?

How to answer it in an AI engineer interview: what is being tested, a step-by-step framework, a sample answer and the mistakes to avoid.

Get the playbook Score my answer free

$19.99 on Gumroad, instant download40% off with code EARLYBF until Nov 3, 11:59 PM PTEARLYBF: 40% off until Nov 3

Smaller start: Score your answers against 50 rubrics, $9. 50 AI engineer questions, one rubric each, PDF on Gumroad.

Practiced one out loud? Paste your answer into the free self-check and get a score in 30 seconds

What the interviewer is testing

Whether you understand the bi-encoder vs cross-encoder difference and can place a re-ranker in a pipeline with sensible candidate counts and latency.

Answer framework

  1. Retrieve wide and cheap: get a candidate set, for example the top 50, from vector, keyword or hybrid search.
  2. Re-score each candidate with a model that reads the query and the passage together, usually a cross-encoder or an LLM.
  3. Keep the top 5 to 10 for the prompt.
  4. Explain why: a cross-encoder sees query and passage jointly, so it judges relevance better than comparing two separately computed embeddings.
  5. Manage cost: it is too slow for the whole corpus, so tune the candidate count against latency.

Sample answer

Retrieval with embeddings uses a bi-encoder: the query and each document are embedded separately and compared with a similarity score. That is fast enough to search millions of chunks, but each vector is a compressed summary, so the ranking is approximate.

Re-ranking adds a second stage. I take a wide candidate set, say the top 50 from hybrid search, and score every query-passage pair with a cross-encoder, which reads both texts together and can notice whether the passage actually answers the question, including negations and specific conditions. Then I keep the best 5 to 10 for the prompt.

The cost is latency, since the cross-encoder runs once per candidate. I tune the candidate count by looking at recall at 50 versus 100 from the first stage and the re-ranker's latency on our hardware. Putting the strongest passages first also helps because models use the start of the context more reliably.

Likely follow-ups

Can you use an LLM as the re-ranker? Yes, it can be more accurate on hard queries, but it is slower and more expensive, so it is usually applied to a small candidate set.

Common mistakes

The PlaybookWant the full method?

The 0 to 1 AI Engineer Interview Playbook covers the answer frameworks, system-design walkthroughs and an 8-week plan.

Get the playbook 40% off with code EARLYBF until Nov 3, 11:59 PM PTEARLYBF: 40% off until Nov 3

Smaller start: Score your answers against 50 rubrics, $9. 50 AI engineer questions, one rubric each, PDF on Gumroad.

Interviewing for PM or TPM? See the full PM package, $59 on Gumroad: a resume system, a PM, TPM and PGM interview handbook, a 1:1 cheat sheet and a PM playbook. PM and TPM content, not AI engineer prep.

Related questions

Based on question #29 in the full question bank.