How do you make LLM-as-a-judge evaluation reliable?

An LLM judge is useful because it scales, but it is a measurement tool, so I calibrate it like one. First, the rubric: instead of asking whether an answer is good, I ask narrow questions such as whether every claim is supported by the provided sources, yes or no, with examples of passing and failing answers. One criterion per judgment is more reliable than a combined score.

Second, the format. Binary or pairwise judgments are more consistent than a 1-to-10 scale. For pairwise comparisons I run each pair twice with the order swapped, because judges tend to favor one position, and count a win only when both orders agree. I also watch for length bias and for a judge favoring outputs from its own model family.

Third, calibration. A domain expert labels a sample of a few hundred cases, and I measure agreement between the judge and the expert, for example with Cohen's kappa. If agreement is low, I fix the rubric before trusting any numbers. I version the judge prompt and model, and re-run calibration whenever either changes.