Make the Wrong Answer Cheap

Short summary

An expert practitioner argues that AI agent confidence thresholds should be set by the cost of wrong answers, not by nervousness. An unsupervised inbox agent runs at 0.7 confidence because its writes are idempotent (upserts, skip-live-deals), making mistakes nearly free. A supervised agent runs at 0.8 because false positives cost real money. The key insight: build walls (idempotence, guards) instead of turning dials (thresholds).

• •Set agent confidence thresholds based on the cost of wrong answers, not anxiety
• •Idempotent writes (upserts, skip-live-records) make mistakes cheap, enabling autonomy without human review
• •A medical claims denial engine improved from 36.7% to 76.7% action accuracy by anchoring to a deterministic action table instead of tightening prompts

Generated with AI, which can make mistakes.