The original title is: "Structured output broke on us three times. The third time taught us operator-ready."

Short summary

A contract-extraction agent taught three lessons about operator-readiness versus production-readiness. Schema validation catches structure but not wrong values—add confidence scoring; retry logic converts uncertainty into false confidence—flag content failures for human review; and testing on your data diverges from the operator's data—sample their actual corpus during pre-handoff validation to detect distribution shift.

• •Schema validation alone passes wrong values matching the structure—requires field-level confidence scoring to catch content errors
• •Retry logic can silently convert model uncertainty into plausible false values without alerting operators
• •Distribution shift from real operator data is invisible until deployment—mandatory operator corpus sampling (50 docs) before go-live

Generated with AI, which can make mistakes.