What Context Does a Coding Agent Actually Need to Act?

Short summary

This arXiv study isolates what context a coding agent truly needs when editing code on SWE-bench Verified. The finding is starkly minimal: natural-language summaries of code answer far fewer behavioral questions than source (4/45 vs 27/45), surrounding file context barely matters (UML skeletons ≈ deletion, p=0.75), and compressed context matches whole files at a third of the tokens (19K vs 94K). The authors also report a ~9% noise floor from temperature-0 API inference flips, undermining small-effect claims on this benchmark.

• •Coding agents need surprisingly minimal context: the signal lives in the code being edited itself, not summaries or surrounding files
• •Compressed context matches full-file performance at 19K vs 94K tokens per resolved issue
• •Temperature-0 API inference flips ~9% of outcomes between identical runs, establishing a noise floor for SWE-bench results

Generated with AI, which can make mistakes.