Your Agent's Confidence Score Is Not a Probability

Short summary

Self-reported LLM confidence scores are not calibrated probabilities — they are tokens from the same forward pass that generated the answer, making them maximally circular and corruptible as gating signals. The author proposes a three-tier evidence framework: Tier 1 uses unforgeable external checks (file exists, code compiles), Tier 2 uses statistical baselines the agent didn't author, and Tier 3 (model-as-judge) is offline-only. Self-confidence scores rank below all tiers and should never gate production routing.

• •Self-reported confidence is the model grading its own homework — same weights produce both answer and confidence
• •Three-tier evidence framework: unforgeable checks, statistical baselines, and offline model-as-judge
• •Gating on self-confidence is worse than gating on nothing because it provides false comfort

Generated with AI, which can make mistakes.