Benchmark shows code maps outperform model upgrades for AI agent dependency detection across 13 repos

Short summary

The author benchmarks AI coding agents across 13 Ruby repos and finds that structural code maps dramatically improve dependency detection — Claude Opus went from 2/11 to 11/11 correct dependents on Chatwoot, with gains replicated across five models from three families. The core argument is that code maps provide deterministic computed facts while LLMs provide probabilistic inferences, and this is a category difference that better models cannot close. Scaling inference makes answers more confident, not more correct, so tools that compute over a closed world remain essential regardless of model progress.

• •Benchmark across 13 Ruby repos shows code maps boost agent dependency detection from ~18% to ~90%+ accuracy
• •Gains replicated across Opus, GPT-5.5, Devstral, Qwen, and Kimi — not a single-model artifact
• •Computed facts vs inferred guesses is a category gap that model upgrades don't close

Generated with AI, which can make mistakes.