More Context Made My Classifier Worse: Building a Machine-Maintained Failure Taxonomy

Short summary

ML evaluation feedback loops break down when managed manually—failures become stale and patterns are missed. Author built a machine-maintained JSON taxonomy that links individual failures to persistent failure classes across evaluation runs, enabling automated pattern discovery and prioritization. Critical discovery: adding more context made the classifier perform worse, revealing the actual root cause was prompt design decisions that over-weighted adjacent signals, not insufficient data volume.

• •Manual failure tracking breaks down immediately; structured, machine-maintained JSON prevents notes from becoming stale
• •Single taxonomy across eval runs enables automated pattern detection and recurrence tracking without manual curation
• •Model reasoning traces are the richest diagnostic signal for identifying which evidence the model over-weights in wrong predictions

Generated with AI, which can make mistakes.