I tested 3 models as AI agent quality inspectors: the stronger the model, the more valid work it rejects

Short summary

When LLMs inspect agent output, stronger models eliminate false positives (catching all garbage) but introduce false negatives (rejecting 75% of valid work). This precision-recall tradeoff can't be solved by model strength alone. Quality gates work best as risk-transfer chains: phase gates catch nothing, small models filter obvious garbage, and humans review the edge cases the strong model flags.

• •Stronger LLM inspectors don't solve quality verification—they shift the problem from false positives to false negatives.
• •Small models (≤4B) achieve ~25% false positives but reject ~50% of valid work; strong models hit 0% false positives but reject ~75% of valid work.
• •Design quality gates as risk-transfer chains with human review at the end, not as deterministic solutions.

Generated with AI, which can make mistakes.