Study of 8 AI agent instruction collections finds only 11% of rules are verifiable from repositories

Short summary

The author analyzed 1,332 instruction units across 8 public AI agent-instruction collections and found a median of only 11.1% of procedural instructions are checkable from the repository alone. Five collections mandate zero output artifacts, and of the 103 artifacts the other three require, 44% have never existed in any commit. The findings reveal two failure modes: collections making no verifiable claims, and collections whose claims fail at roughly coin-flip rates.

• •Only 11.1% median of procedural AI agent instructions are checkable from repository evidence
• •44% of mandated output artifacts across 3 collections have never existed in git history
• •Two failure modes: no verifiable claims at all, or claims failing at coin-flip rates

Generated with AI, which can make mistakes.