Which LLM should I actually code with? I built a small benchmark to find out

Short summary

A developer built a small benchmark (14 problems across Python, C#, and Bash) comparing three LLMs on pass@k, cost, and latency. All models achieved 100% pass@3 on problems they were allowed to answer, but one model had content filtering that reduced its coverage to 10 of 14 problems. The real differentiator was latency (6.6x spread) and cost (1.3x spread), with one model showing dramatic slowdowns on Bash specifically. The takeaway: pick coding LLMs by speed and price per language, since accuracy is no longer the deciding factor.

• •All three tested LLMs solved 100% of problems they were allowed to answer, making accuracy a non-differentiator
• •Latency varied 6.6x between fastest and slowest; cost varied only 1.3x
• •One model was fast on Python/C# but extremely slow on Bash (90-168s per problem), showing that single-number benchmarks are misleading

Generated with AI, which can make mistakes.