Eval Harness Capture Bugs: How Infrastructure Failures Masquerade as Model Failures

Short summary

A debugging narrative about two capture-layer bugs in an AI agent eval harness that caused working model outputs from Claude Sonnet and local Llama 3.2 to be falsely reported as malformed. The first bug involved ANSI terminal control bytes corrupting subprocess output from Ollama CLI; the second was a parser rejecting valid JSON wrapped in markdown fences. The author used Sentry instrumentation to identify which layer broke before blaming the model.

• •Eval scoreboards can mask infrastructure bugs as model failures by compressing different error types into one label
• •CLI subprocess capture can corrupt model output with invisible ANSI terminal control sequences
• •Sentry instrumentation helps attribute failures to the correct layer (capture, parse, or model) instead of blaming the model

Generated with AI, which can make mistakes.