The original title is: "Stop Benchmarking Coding Models by Vibes: A Repeatable 20-Task Harness You Can Run Tonight"

Short summary

The author presents a minimal two-file harness (tasks.jsonl + run.py) for benchmarking coding models on 20 fixed tasks with deterministic unit-test scoring, rubric grading, and diff matching. The harness works against any OpenAI-compatible API endpoint and runs 3 samples per task to measure consistency. The article discloses sponsorship by MonkeyCode for free model access and server hosting, though the harness itself is provider-agnostic.

• •Two-file harness: 20 fixed coding tasks scored via unit tests, rubrics, and diff matching
• •Works with any OpenAI-compatible endpoint; runs 3 samples per task for consistency scoring
• •Disclosure: sponsored by MonkeyCode for free access, but harness is provider-agnostic

Generated with AI, which can make mistakes.