Six Bugs Only a Live Model Could Teach Us

Short summary

When deploying agentic systems to production, live model behavior exposes critical bugs that offline tests alone cannot catch. This post details six real failures in a Qwen-based environmental-compliance agent—schema mismatches, empty plans, malformed citations, workflow ordering, repeated clarifications, and entity ambiguity—with code fixes for each. The core principle: let the LLM handle narration but keep structured outcomes deterministic and code-owned.

• •Offline tests miss critical failures from model output distribution variance
• •Six concrete bugs fixed via schema preprocessing, validation, and workflow hardening
• •Core pattern: let LLM narrate, keep structured outcomes (entity resolution, approvals, report assembly) deterministic and code-owned

Generated with AI, which can make mistakes.