LLM-as-judge disagrees with itself between runs

Short summary

An engineer shares a real incident where an LLM-as-judge faithfulness gate flapped between pass and fail on identical inputs due to sampling temperature, model version drift, vague rubrics, and tie-breaking noise. The fix combines temperature 0 with seeding, pinning exact model snapshots, averaging over k=5 judged samples, quantizing scores to a coarse grid, and versioning the judge prompt as code. The key conceptual shift is treating the judge's output as a distribution with a measured noise band rather than a single ground-truth verdict, so CI only fails when the mean drops below threshold by more than the observed jitter.

• •LLM-as-judge gates flap because of temperature, model alias drift, vague rubrics, and tie-breaking noise
• •Fix: temperature 0, pin model snapshots, average k=5 samples, quantize scores, version the rubric in repo
• •Gate on mean minus measured stdev vs threshold so CI red means real regression, not jitter

Generated with AI, which can make mistakes.