Your Agent Success Rate Counts Only the Survivors

Short summary

Dashboard success rates that count only clean pass/fail verdicts silently exclude timed-out, aborted, and hung runs, inflating reliability numbers. This survivorship bias means your metrics improve as more runs vanish—exactly the failure mode you should fear most. The fix: count runs that started, not runs that finished; one-line change drops 90% success to 72% in the author's 2,190 production runs.

• •Naive success rates (SUCCEEDED/(SUCCEEDED+FAILED)) delete hung and timed-out runs from the denominator
• •This survivorship bias rewards the worst failure mode: invisible hung processes
• •Fix: count started runs instead of finished runs—cuts reported 90% success rate to true 72%

Generated with AI, which can make mistakes.