The original title is "Your New Prompt 'Feels' Better. That's Not an Eval."

Short summary

The post argues that eyeballing a few hand-picked examples to validate prompt changes is confirmation bias, not evaluation. It recommends building a 20-50 row eval set with input/expected-output pairs, scored automatically the same way each time, and re-run on every future change to catch regressions. The framing is aimed at AI interview prep, with heavy promotion of the author's paid prep sessions.

• •Hand-picked examples are confirmation bias, not evaluation — build a 20-50 row eval set
• •Score outputs automatically: exact match for structured output, similarity/grounding for open-ended text
• •Re-run evals on every future prompt or retrieval change to catch silent regressions

Generated with AI, which can make mistakes.