The Rubric Line That Sent Me Down a Three-Layer Debugging Rabbit Hole

Short summary

A developer shares a three-layer debugging journey in an adversarial multi-agent system (RedCouncil) built on Qwen-Max, where a broken evaluation metric was empty-by-definition, conflict detection assumed vocabulary overlap instead of genuine disagreement, and non-determinism across retry layers made results unreliable. Key lessons: eval code needs as much debugging as product code, metrics that never change are structurally broken, and detection logic must match the system's design assumptions.

• •Eval pipelines can fail silently — a metric that's always the same value is structurally broken
• •Conflict detection in multi-agent systems can't assume agents describe disagreements similarly
• •Non-determinism compounds across every unpinned layer; budget real time for evaluation code

Generated with AI, which can make mistakes.