Practiced one out loud? Paste your answer into the free self-check and get a score in 30 seconds
What the interviewer is testing
Whether you can connect offline evaluation with online experiments and choose metrics that reflect user value rather than model scores.
Answer framework
- Gate with offline evals first so only candidates that pass reach users.
- Pick one primary metric tied to user value, such as task success or resolution without escalation.
- Add guardrail metrics: latency, cost per request, complaints, safety flags.
- Randomize by user, size the test for the effect you care about, and fix the duration in advance.
- Analyze honestly: avoid peeking, check segments, and read sampled conversations alongside the numbers.
Sample answer
I treat it as two stages. Offline, the new prompt or model has to beat the current one on our eval suite without regressions on safety or format. Only then does it go to an online test, because offline scores do not fully predict what users do.
Online, I pick one primary metric that reflects real value. For a support assistant, that could be the share of conversations resolved without a human agent, or for a writing feature, the share of suggestions users accept and keep. Thumbs-up rates are useful but sparse and biased toward people who bother to click. I add guardrails: p95 latency, cost per request, escalation and complaint rates, and safety flags. A variant that wins on the primary metric but doubles cost or latency needs a separate decision.
For the mechanics, I randomize at the user level so people get a consistent experience, estimate the sample size needed to detect a meaningful lift, and set the duration before starting to avoid stopping on a lucky day. Afterwards I check key segments and read a sample of conversations from each arm, since numbers alone can hide why a variant won.
Likely follow-ups
Common mistakes
- Skipping offline evaluation and testing everything live.
- Using model scores as the primary online metric.
- Stopping the test as soon as it looks significant.
- Ignoring cost and latency guardrails.
The 0 to 1 AI Engineer Interview Playbook covers the answer frameworks, system-design walkthroughs and an 8-week plan.
Get the playbook 40% off with code EARLYBF until Nov 3, 11:59 PM PTEARLYBF: 40% off until Nov 3Smaller start: Score your answers against 50 rubrics, $9. 50 AI engineer questions, one rubric each, PDF on Gumroad.
Interviewing for PM or TPM? See the full PM package, $59 on Gumroad: a resume system, a PM, TPM and PGM interview handbook, a 1:1 cheat sheet and a PM playbook. PM and TPM content, not AI engineer prep.