Fixing dual-write inconsistency and missing idempotency in a 10M msg/day warehouse system

Short summary

A warehouse execution system processing 5-10M messages/day on ActiveMQ suffered duplicate order processing and silent failures due to dual-write inconsistency and lack of idempotency. The team migrated to NATS JetStream and implemented the Outbox pattern (writing business changes and event records in the same DB transaction) plus idempotent consumers that check for already-processed events. The broker swap was 20% of the work; the patterns built on top were the other 80%.

• •Dual-write inconsistency between DB and message broker caused duplicate orders and silent failures at 5-10M messages/day
• •Outbox pattern moves event publishing into the same DB transaction, eliminating an entire category of incidents
• •Idempotent consumers check processed-event records before applying side effects, handling at-least-once delivery safely

Generated with AI, which can make mistakes.