The agent evaluation gap: Enterprise AI organizations have a reality-alignment problem, not a coverage problem — and most are shipping to production anyway

Short summary

A VentureBeat survey of 157 enterprises reveals a critical agent evaluation gap: 50% have shipped AI agents that passed internal evaluations but then failed in production, and only 5% fully trust automated evaluation today. Despite this, 66% already allow or are engineering toward zero-human-in-the-loop deployment for low-risk agents. The core problem is not evaluation coverage but reality alignment — evaluations pass agents that fail real customers, and autonomy is scaling faster than assurance.

• •50% of enterprises shipped an agent that passed evals but failed a customer in production
• •Only 5% fully trust automated evaluation; top complaint is poor alignment with real-world outcomes (29%)
• •66% allow or are building toward zero-human-in-the-loop deployment despite immature evaluation tooling

Generated with AI, which can make mistakes.