The Acknowledgment Gap - How Event-Driven Systems Lose Messages Without Errors

Short summary

The article describes a subtle failure mode in event-driven systems where acknowledgments conflate 'accepted' with 'completed,' causing messages to be silently dropped without errors or alerts. The author traces the bug through offset-based consumer groups like Kafka, where committing an offset is a promise that all work up to that point is done. When a downstream handoff reports success prematurely, the consumer commits its offset and the original message is lost forever.

• •Acknowledgment gap: systems report success for accepted-but-not-completed work
• •Offset commits in Kafka are promises about the past—false commits cause silent data loss
• •Bug appears at async boundaries where handoffs use ambiguous status codes like 202

Generated with AI, which can make mistakes.