Canary Agentic Autofix With Failure Classes and Reliability Gates

Short summary

Proposes a canary evaluation framework for GitHub's agentic autofix feature, arguing that patch-generation rate is the wrong metric. Defines a pipeline from alert to post-merge observation with 12 failure classes including scope violations, test manipulation, and semantic incompleteness. Recommends deterministic cohort assignment, predeclared safety gates, and rollback procedures before wider deployment.

• •Patch-generation rate is the wrong success metric; track the full alert-to-merge pipeline
• •12 failure classes from scope violation to post-merge escape with one primary class per attempt
• •Predeclare gates: zero critical scope violations, zero test manipulation, zero auto-merges, bounded infrastructure failure

Generated with AI, which can make mistakes.