How much does context cost an AI coding agent? grep vs graph vs LSP, measured across 936 runs

Short summary

Benchmarked four code-context approaches (grep, structural graph, LSP, codegraph) across 936 runs on Claude models. Simple lookups show minimal cost difference (~3×), but complex tasks (blast radius, overload resolution) separate sharply—grep fails (0.71 accuracy, 6–24× cost), structural tools stay cheap and accurate. Optimal choice depends on task complexity and model size.

• •Measured 4 context-provisioning tools across 936 benchmark runs varying model (Haiku/Sonnet/Opus) and task type on apache/superset
• •Simple code lookups tie across all tools; complex analysis tasks expose large gaps—grep collapses while structural tools stay efficient
• •Best tool depends on task complexity and chosen model; no one-size-fits-all answer for large polyglot codebases

Generated with AI, which can make mistakes.