Fill a WAL Volume on Purpose Before ENOSPC Turns Your Health Check Green

Short summary

A service can report healthy while WAL writes silently fail due to ENOSPC, creating a false green health check. The post provides a container-based drill to intentionally fill a disk fixture, drive write workloads, and verify that readiness probes catch durable-write failures rather than just process liveness. It includes threshold-based actions, cleanup procedures, and integrity checks to ensure writes truly resume after space is restored.

• •Process-only liveness probes miss ENOSPC-related WAL write failures
• •Container drill fills a fixture volume to test readiness under disk exhaustion
• •Recovery requires integrity checks and a successful test write before readiness returns true

Generated with AI, which can make mistakes.