The original title is "Why did my benchmark stop at N=22? A debugging story in nine bugs"

Short summary

A developer recounts nine bugs found while debugging a multi-language A2A agent benchmark that mysteriously stopped at N=22. The root cause was CPython 3.11's 4300-digit int-to-string conversion limit crashing on the 24th Mersenne prime, compounded by decorative stringification, fragile LLM-prose regex parsing, and misleading chart labels conflating pipeline architecture with language performance. After fixing all nine bugs, the benchmark went from incomplete to 96/96 datapoints.

• •CPython 3.11's 4300-digit int→str limit silently crashed the Python agent at N=24, but the stringified output was never used
• •Benchmark relied on regex-matching LLM prose phrasing for timing data, which never matched Gemini's actual responses
• •Chart labeled as language comparison was actually a pipeline comparison — 400× gap between direct agents and Gemini-routed agents

Generated with AI, which can make mistakes.