Stop Trusting Vibes: A Reproducible Harness for Comparing AI Coding Models on Your Own Codebase

Short summary

A reproducible harness for comparing AI coding models against your own codebase using three measurable axes: correctness, edit locality, and iteration cost. Tasks are mined from your git commit log — small, self-contained fixes that serve as realistic prompts. The runner checks out parent commits, applies model patches, runs your tests, and penalizes models that touch forbidden paths. Includes shell scripts for task extraction and a Python scorer with adjustable weights.

• •Mine eval tasks from git history with automated filtering for small self-contained commits
• •Score on correctness (tests pass), edit locality (no forbidden paths), and iteration cost
• •Includes extract_tasks.sh and score_run.py with adjustable scoring rubric

Generated with AI, which can make mistakes.