How Kubernetes Health Checks Brought Down a Payment Service

Short summary

A Tier 2 payment service went down because every pod entered a continuous restart cycle with spiked CPU and memory. The initial fix of provisioning more memory restored service but didn't address root cause. Investigation revealed downstream service health check failures preceded the outage, and Kubernetes probe configuration was the likely culprit. The article explains startup, liveness, and readiness probes and how misconfigured probes can cause cascading restart loops.

• •Payment service outage caused by pod restart loops, not genuine memory pressure
• •Downstream health check failures preceded the outage by 30 minutes
• •Kubernetes probe misconfiguration likely triggered cascading restarts

Generated with AI, which can make mistakes.