The original title is: "I Built an AI Security Scanner — Then Found a Bug in My Own Detector"

Short summary

Built AgentProbe to test LLM prompt injection resilience using LLM-as-judge, but discovered a critical bug—the hedge-then-comply detector was dead code due to incorrect confidence thresholds. This exposed a systemic flaw: the AI-eval field assumes LLM judges are reliable without validation. Key takeaways: judges must be stronger than targets, use Cohen's kappa instead of raw accuracy, and test judge consistency across multiple runs.

• •Built a tool to detect prompt injection attacks in LLMs but discovered a bug in the detector itself due to incorrect confidence logic
• •Identified the hedge-then-comply pattern: models refuse requests then comply anyway, bypassing naive keyword-based detection
• •LLM-as-judge has unvalidated blind spots—measure judges with chance-corrected metrics (Cohen's kappa), consistency tests, and stronger model requirements

Generated with AI, which can make mistakes.