The headline is about LLM-as-judge inconsistency between runs. Let me rewrite this concisely.

Short summary

An LLM-as-judge evaluation gate that returns different scores on identical inputs is worse than no gate—it becomes a slot machine developers pull until it goes green. The author identifies four sources of judge variance (sampling temperature, model version drift, prompt ambiguity, tie-breaking) and shares five fixes including temperature 0, pinning model snapshots, averaging over k=5 samples, quantizing scores, and versioning judge prompts as code. The provided Python implementation reduced run-to-run metric swing from 0.03 to under 0.01, making CI failures meaningful again.

• •Four sources of LLM-judge variance: sampling temperature, model version drift, prompt ambiguity, and tie-breaking noise
• •Five fixes: temp 0 + seed, pin model snapshots, average over k=5 samples, quantize scores to coarse grid, version judge prompts in repo
• •Gate should only fail when mean falls below threshold by more than measured noise band, not on jitter

Generated with AI, which can make mistakes.