The headline is about regression detection in LLM eval sets being too small. Let me craft a concise headline.

Short summary

Most LLM eval sets are too small to reliably detect regressions. A 50-example set has only ~35% power to catch a drop from 0.90 to 0.85 pass rate, missing the regression two out of three times. The author provides a standard-library Python function to compute required sample sizes using two-proportion power analysis, and recommends reporting confidence intervals and using per-criterion binary labels instead of vague graded scores.

• •A 50-example eval set has ~35% power to detect a 5pp pass-rate drop — it misses the regression two times in three
• •You need ~253 examples for 80% power to catch a slide from 0.90 to 0.85
• •Report confidence intervals on each run and prefer per-criterion binary labels over graded scores

Generated with AI, which can make mistakes.