Practiced one out loud? Paste your answer into the free self-check and get a score in 30 seconds
What the interviewer is testing
Whether you choose chunking from the documents and the questions rather than defaults, and whether you can explain the size trade-off and how to measure it.
Answer framework
- Follow structure: split on headings, sections, paragraphs, code blocks and table rows rather than fixed character counts.
- Pick a size range: typically a few hundred tokens with modest overlap, then tune.
- Know the trade-off: small chunks match precisely but lose context; large chunks keep context but dilute similarity and cost more tokens.
- Use parent retrieval when needed: retrieve small chunks, then pass their parent section to the model.
- Decide by measurement: compare recall@k on real questions for each strategy.
Sample answer
I start from the documents. Policies, manuals and wikis have headings and paragraphs, so I split along that structure and keep the heading path as metadata, for example 'Billing > Refunds > Annual plans'. Code is split by function or class, and tables by row with the header repeated, so each chunk still makes sense when read alone.
Then I choose a size, usually a few hundred tokens with a small overlap. Small chunks give sharper matches because the embedding represents one idea, but the model may not get enough context to answer. Large chunks carry more context but blur the embedding and use more of the prompt. A common compromise is small-to-parent retrieval: match on small chunks, then send the enclosing section to the model.
I do not settle this by intuition. I build a set of real questions with the passages that answer them, index the corpus with two or three strategies, and compare recall@k and final answer quality. The winner is often different for different document types.
Likely follow-ups
Common mistakes
- Using a library's default chunk size without testing.
- Splitting tables and code blocks in the middle.
- Dropping headings and document titles that give chunks context.
- Judging chunking by answer quality alone instead of retrieval recall.
The 0 to 1 AI Engineer Interview Playbook covers the answer frameworks, system-design walkthroughs and an 8-week plan.
Get the playbook 40% off with code EARLYBF until Nov 3, 11:59 PM PTEARLYBF: 40% off until Nov 3Smaller start: Score your answers against 50 rubrics, $9. 50 AI engineer questions, one rubric each, PDF on Gumroad.
Interviewing for PM or TPM? See the full PM package, $59 on Gumroad: a resume system, a PM, TPM and PGM interview handbook, a 1:1 cheat sheet and a PM playbook. PM and TPM content, not AI engineer prep.