Are Bigger AI Models Actually Making Developers Faster?

Short summary

A developer reflects on whether bigger AI models actually improve real-world productivity, citing a METR study showing experienced developers were 19% slower on certain tasks when using AI tools. The article argues benchmarks measure capability, not productivity, and that time spent prompting, reviewing, and refactoring AI output can offset speed gains. The conclusion: AI is valuable when it assists thinking rather than replacing engineering judgment.

• •METR study found experienced devs were 19% slower using AI on studied tasks
• •Benchmarks measure capability, not real-world developer productivity
• •Skilled engineers with lightweight models can outperform unskilled users of flagship models

Generated with AI, which can make mistakes.