Practiced one out loud? Paste your answer into the free self-check and get a score in 30 seconds
What the interviewer is testing
Whether you read public benchmark scores critically and know how to protect your own evaluation from the same problem.
Answer framework
- Define it: benchmark questions or answers appear in a model's training data, so the score measures memory, not ability.
- How it happens: benchmarks are published online and web-scale crawls pick them up, directly or through paraphrases.
- Effect: inflated scores and misleading model comparisons.
- Detection: n-gram overlap checks, testing on rephrased or newly written items, and comparing with performance on fresh data.
- Protection: keep a private eval set built from your own tasks and refresh it over time.
Sample answer
Benchmark contamination means a model has seen the test during training. Popular benchmarks are posted on the web along with discussions of the answers, and large training crawls pick them up. Then a high score may partly reflect memorization rather than the skill the benchmark was designed to measure.
It matters because teams choose models based on those numbers. If one model's score is inflated by contamination, it can look better than another model that would actually perform better on new problems. Signs include a model doing much better on an old benchmark than on a freshly written one of similar difficulty, or getting worse when questions are lightly reworded.
In practice I use public benchmarks only to shortlist models. The decision is made on a private eval set built from our own tasks, ideally from real user inputs, which is not published anywhere. I keep it out of any data used for fine-tuning, add new cases from production regularly, and keep a frozen hold-out portion that is never used while tuning prompts so I am not slowly overfitting to it myself.
Likely follow-ups
Common mistakes
- Choosing a model purely from leaderboard scores.
- Publishing or sharing your private eval set.
- Tuning prompts on the same set used for the final decision.
- Assuming contamination only means exact copies; paraphrases count too.
The 0 to 1 AI Engineer Interview Playbook covers the answer frameworks, system-design walkthroughs and an 8-week plan.
Get the playbook 40% off with code EARLYBF until Nov 3, 11:59 PM PTEARLYBF: 40% off until Nov 3Smaller start: Score your answers against 50 rubrics, $9. 50 AI engineer questions, one rubric each, PDF on Gumroad.
Interviewing for PM or TPM? See the full PM package, $59 on Gumroad: a resume system, a PM, TPM and PGM interview handbook, a 1:1 cheat sheet and a PM playbook. PM and TPM content, not AI engineer prep.
Related questions
Based on question #67 in the full question bank.