AI Engineer Interview Questions / Questions
Entry

What do temperature and top-p do in an LLM?

How to answer it in an AI engineer interview: what is being tested, a step-by-step framework, a sample answer and the mistakes to avoid.

Get the playbook Score my answer free

$19.99 on Gumroad, instant download40% off with code EARLYBF until Nov 3, 11:59 PM PTEARLYBF: 40% off until Nov 3

Smaller start: Score your answers against 50 rubrics, $9. 50 AI engineer questions, one rubric each, PDF on Gumroad.

Practiced one out loud? Paste your answer into the free self-check and get a score in 30 seconds

What the interviewer is testing

Whether you understand sampling mechanically and can pick settings for a task, and whether you know the practical caveat that low temperature does not give strict determinism.

Answer framework

  1. Temperature: divide the logits by T before softmax; low T sharpens the distribution, high T flattens it.
  2. Top-p (nucleus sampling): sample only from the smallest set of tokens whose cumulative probability reaches p.
  3. Choosing: low temperature (0 to 0.3) for extraction, classification and code; moderate (around 0.7 to 1.0) for brainstorming and creative text.
  4. Tune one knob at a time; changing both makes behavior hard to reason about.
  5. Caveat: even at temperature 0 outputs can differ across runs because of batching and floating-point effects.

Sample answer

Both settings control how the next token is sampled from the model's probability distribution. Temperature rescales the logits before the softmax. With a temperature near zero the most likely token almost always wins, so output is focused and repeatable; with a higher temperature lower-probability tokens get more chance, so output becomes more varied and also more error-prone.

Top-p trims the distribution instead of reshaping it. With top-p of 0.9 the model keeps the smallest group of tokens that together cover 90% of the probability and samples only from those, which cuts off the long tail of unlikely tokens.

In practice I set temperature by task: close to zero for extraction, classification, SQL or code where there is one right answer, and higher for brainstorming or marketing copy. I usually adjust either temperature or top-p, not both. I also do not promise strict determinism at temperature 0, because batching and floating-point differences can still change outputs slightly; if I need exact repeatability I cache results.

Likely follow-ups

What is the difference between top-p and top-k? Top-k keeps a fixed number of tokens; top-p keeps a variable number whose probabilities add up to p, so it adapts to how confident the model is.

Common mistakes

The PlaybookWant the full method?

The 0 to 1 AI Engineer Interview Playbook covers the answer frameworks, system-design walkthroughs and an 8-week plan.

Get the playbook 40% off with code EARLYBF until Nov 3, 11:59 PM PTEARLYBF: 40% off until Nov 3

Smaller start: Score your answers against 50 rubrics, $9. 50 AI engineer questions, one rubric each, PDF on Gumroad.

Interviewing for PM or TPM? See the full PM package, $59 on Gumroad: a resume system, a PM, TPM and PGM interview handbook, a 1:1 cheat sheet and a PM playbook. PM and TPM content, not AI engineer prep.

Related questions

Based on question #8 in the full question bank.