Back to feed
Dev.to
Dev.to
7/21/2026
Distinguishing wrong from absent

Distinguishing wrong from absent

Short summary

The model-drift eval board uses an exact-match grader on a frozen test suite, but this design conflates absent answers (rate limits, refusals, truncations) with wrong ones. A model appeared to drop from 69% to 3% overnight due to 429 rate-limit errors scored as regressions. The board mitigates this by excluding accuracy points only when a run's reliability falls below 50%, avoiding inflated scores on hard prompts that correlate with failures.

  • •Exact-match grading conflates absent answers (rate limits, refusals, truncations) with wrong ones
  • •A 69% to 3% overnight drop was caused by 429 rate limits, not a real regression
  • •Board excludes accuracy points only when run reliability drops below 50%, preventing difficulty-correlated bias

Generated with AI, which can make mistakes.

#ai-tools#open-source
Read full article at Dev.to

Is this a good recommendation for you?

Comments

Failed to load comments. Please try again.

Explore more

The headline is about regression detection in LLM eval sets being too small. Let me craft a concise headline.
Dev.to
LLM-as-judge disagrees with itself between runs
Dev.to
The headline is about LLM-as-judge inconsistency between runs. Let me rewrite this concisely.
Dev.to
Your AI isn't too weak. Your evals are missing.
Dev.to