Back to feed
Dev.to
Dev.to
7/7/2026
The Pod That Lied: A Kubernetes Readiness Probe War Story

The Pod That Lied: A Kubernetes Readiness Probe War Story

Short summary

Two Kubernetes pods passed readiness checks and remained in the service's endpoint list while unable to serve requests, causing intermittent 500 errors for customers. Root cause: a database failover broke their connection pools, but the readiness probe only checked if the HTTP server was running. Key takeaway: readiness probes must verify actual application functionality against real dependencies, not just confirm the process is alive.

  • •Database failover broke connection pools; pods unable to query database but health check only verified HTTP server was running
  • •Pods remained in service rotation and received real traffic despite returning 500 errors intermittently
  • •Root issue: readiness probe tested liveness, not readiness—it did not touch the actual dependency (database)

Generated with AI, which can make mistakes.

Read full article at Dev.to

Is this a good recommendation for you?

Comments

Failed to load comments. Please try again.

Explore more

How Kubernetes Health Checks Brought Down a Payment Service
Dev.to
The headline needs to capture: identical Kubernetes health probes, 10-minute outage became 40 minutes, the cause (misconfiguration).
Dev.to
The Volume That Wouldn't Let Go: A Kubernetes PVC War Story
Dev.to
The original title is "Kubernetes Health Probes: Liveness, Readiness, and Startup Explained"
Dev.to