AI agent passed all evals but CFO killed it over cost-per-resolution

Short summary

An AI agent passed every evaluation metric in a published eval harness but was ultimately killed by the CFO because the cost per successful resolution exceeded the cost of the human workers it was meant to replace. The piece highlights the gap between technical performance metrics and real-world financial viability for AI agents. It underscores that unit economics, not just accuracy, determines whether an AI agent ships.

• •AI agent passed all eval metrics but was killed by CFO over cost
• •Successful resolutions cost more than the humans it replaced
• •Technical accuracy alone is insufficient; unit economics drive deployment decisions

Generated with AI, which can make mistakes.