The original title is 12 words: "I picked a coding agent off a leaderboard. It flopped on our codebase."

Short summary

A developer shares lessons from picking a coding agent based on public benchmark scores that failed in production. The benchmarks measured memory, not problem-solving, used clean setups unlike real work, and had lower pass bars than actual code review. The solution: replay your last 50 merged PRs against the agent and score whether its output would pass your team's review process.

• •Public coding-agent benchmarks don't reflect real codebase habits, review standards, or messy workflows
• •Replay your last 50 merged PRs as a custom eval: roll back repo, give agent the issue, compare to merged diff
• •Score on behavior (would it pass review?) not text similarity; check tool-use order, test execution, and diff size

Generated with AI, which can make mistakes.