Give Your CI Pipeline an Error Budget Before Adding AI Triage

Short summary

Before adding AI triage to a CI pipeline, establish error budgets using four SLIs: queue time, execution time, infrastructure failure rate, and retry recovery rate. Split failures into categories (code, test_flake, infrastructure, etc.) and set concrete thresholds so that exceeding the budget triggers operational action. Only once structured evidence—revision, runner image, attempt, failure class—is consistently captured should an AI summary layer be added and evaluated on classification accuracy and false reassurance, not prose quality.

• •Define four SLIs: queue_seconds, execution_seconds, infra_failure_rate, retry_recovery_rate
• •Classify every failure and set budget thresholds; exceeding them should change priorities
• •Require structured evidence metadata before AI triage; evaluate the model on accuracy and dangerous false reassurance

Generated with AI, which can make mistakes.