Our service discovery caught its own failure and switched itself off

Short summary

An auth service on managed Kubernetes relied on multicast peer discovery, but cloud networks drop multicast packets. Each replica thought it was alone and elected itself leader, causing redundant jobs to run three times silently. The fix: replace gossip with a blob lease stored in a strongly-consistent platform primitive. The lesson: verify network assumptions before porting distributed protocols to managed infrastructure.

• •Multicast-based gossip protocols fail silently on managed cloud platforms when assumptions about network behavior are wrong
• •Blob-lease leadership is simpler, more observable, and leverages platform primitives instead of consensus algorithms
• •Before porting distributed systems patterns, ask what assumptions the protocol makes and what the platform already provides

Generated with AI, which can make mistakes.