We hit an at-least-once delivery trap. Here is how we fixed the race conditions.

Short summary

Duplicate webhooks in an event-driven architecture bypassed application-level checks due to a concurrent race condition across container instances, corrupting database state. The fix implements a two-layer idempotent consumer pattern: Redis SETNX distributed locks as a fast first defense, plus a relational database unique constraint on event hashes as a bulletproof backup within the same transaction. Stress testing confirmed 100% data consistency under 10,000 concurrent duplicate requests per second.

• •Concurrent duplicate events bypassed application-level checks, causing database inconsistency
• •Two-layer fix: Redis SETNX locks (fast path) + DB unique constraints on event hashes (bulletproof backup)
• •Stress tested at 10,000 concurrent duplicates/sec with 100% data consistency and minimal overhead

Generated with AI, which can make mistakes.