My agent dry-ran fine in staging 100 times — then wrecked production on the first real run

Short summary

After 6 months running agents in production, a developer discovered staging-to-production data bleeds from agent non-determinism (same input → different tool-call paths) and mock responses fooling tests about write success. The fix: propagate dry-run flags via KV across all writes in a runId. Critical edge case: hook failures cause fallthrough to real writes, requiring separate monitoring.

• •Agent non-determinism and mock staging responses mask production failures, causing orphaned data
• •Propagate dry-run flags via KV across all writes within a run to enforce atomicity
• •Hook failures cause fallthrough to real writes—monitor as separate alert from agent failures

Generated with AI, which can make mistakes.