Back to feed
Dev.to
Dev.to
7/14/2026
Comparing Two Eval Runs by Their Average Pass Rate Is the Wrong Test

Comparing Two Eval Runs by Their Average Pass Rate Is the Wrong Test

Short summary

Comparing two LLM eval runs by their average pass rates is statistically wrong when both runs share the same test items. Because per-item outcomes are correlated, you need a paired analysis: McNemar's test for pass/fail results, or bootstrap over per-item deltas for graded scores. Unpaired comparisons either overestimate uncertainty (underpowered) or miss significant differences that overlapping marginal confidence intervals obscure.

  • •Same-item eval runs require paired analysis, not independent comparison of averages
  • •Use McNemar's test for pass/fail; bootstrap per-item deltas for graded scores
  • •Unpaired tests can hide regressions or miss real improvements due to ignored covariance

Generated with AI, which can make mistakes.

#ai-tools
Read full article at Dev.to

Is this a good recommendation for you?

Comments

Failed to load comments. Please try again.

Explore more

The headline is about regression detection in LLM eval sets being too small. Let me craft a concise headline.
Dev.to
How I A/B test LLM prompts without fooling myself
Dev.to
The original title is: "The Mean Is Lying to You: Benchmarks Hide the Variance That Breaks Prod"
Dev.to
The original title is "Your New Prompt 'Feels' Better. That's Not an Eval."
Dev.to