Most Evals Measure the Wrong Thing

Short summary

Most AI agent benchmarks measure isolated capability rather than real-world reliability — how models handle malformed tool calls, unexpected errors, and ambiguous instructions. The author proposes evaluating graceful degradation: retry behavior, clarification requests, and garbage detection. Until such benchmarks exist, they recommend capturing production failures and replaying them as a practical eval strategy.

• •Lab evals measure capability; production evals need to measure reliability under messy conditions
• •Standard benchmarks miss real agent failures like malformed tool calls and error-loop stuck states
• •Author adds a failure-replay step to their eval pipeline: capture prod failures, inject into test harness

Generated with AI, which can make mistakes.