I reviewed six "operator-ready" checklists for AI agents. None of them define the problem correctly.

Short summary

Most AI agent readiness frameworks measure static eval pass-rates, but operators face dynamic distribution shifts from real-world data, changing use cases, and unanticipated inputs. True operator-readiness requires testing on the operator's actual corpus (50+ docs), disaggregating failure modes beyond aggregate scores, and implementing ongoing drift monitoring. The gap between eval accuracy and real-world performance is structural and only closes with mandatory pre-handoff validation on the operator's own data.

• •Production readiness (eval pass-rate) ≠ operator readiness (maintained performance on real operator data)
• •Distribution shift is the default condition: operators add document types, use cases, and business context changes beyond your eval set
• •Pre-deployment gate: test on 50 operator-corpus documents; if accuracy drops >5 points vs. eval pass-rate, deployment isn't ready

Generated with AI, which can make mistakes.