Zero Is Not a Score

Short summary

An engineer discovered their AI agent eval scores had been 0% for months due to a broken auth token in the eval harness, not because the skills were bad. The key insight: a metric with zero variance means the instrument broke, not that the system is failing. The author shares similar patterns across advertising dashboards, agent operations, and healthcare billing, and now treats flat numbers as red flags rather than reassurance.

• •0% eval scores across all agent skills were caused by a wrong auth token returning 401s, not by bad skills
• •A flat metric with no variance indicates a broken instrument, not a broken system — healthy metrics breathe
• •Author now requires any eval harness to prove it can pass a known-good case before trusting its scores

Generated with AI, which can make mistakes.