An alternative to LLM quality gates: deterministic routing + sampling

Short summary

The author argues that using LLMs to judge LLM output is fundamentally flawed, citing six experiments showing high misclassification and false-positive rates across model tiers. They propose a deterministic routing system that classifies tasks by risk type—fully automating verifiable outputs, blocking high-risk tasks, auto-releasing low-risk content, and using diff reviews for medium-risk work. The approach replaces semantic quality judgment with statistical process control for anomaly detection and fixed-rate sampling for zero-shot outputs, eliminating LLM-based quality inspectors entirely.

• •LLM-as-judge quality gates fail reproducibly: 50% misclassification, 70% consistency at temp 0
• •Deterministic routing by task type (verifiable/high-risk/low-risk/medium-risk) replaces semantic judgment
• •SPC anomaly detection on behavioral metadata plus fixed 5% sampling catches format issues without LLM cost

Generated with AI, which can make mistakes.