I Spent a Week Fixing the Wrong Skill (And Other Lessons from Evaluating an AI PR Reviewer)

Short summary

A developer shares lessons from evaluating an AI PR review plugin built on Claude Opus across 43 scenarios. The baseline model already catches ~65% of textbook bugs, so the plugin's real value lies in false-positive suppression and risk classification, not raw bug detection. Key insights include the importance of eval weighting (one weight change widened the gap 9pp with zero code changes) and debugging upstream skills rather than rewriting prompts when false positives persist.

• •Baseline Claude Opus catches ~65% of textbook bugs without guidance; plugin value is in false-positive suppression and risk classification
• •Changing eval scoring weights (risk classification 5→10 points) widened the gap 9pp with zero code changes
• •A persistent false positive took four prompt rewrites to fix, but the actual fix was one line in an upstream skill

Generated with AI, which can make mistakes.