I Spent 40 Minutes at 11pm Debugging a Deploy That Wasn't Broken

Short summary

Deploy script repeatedly failed due to a race condition: database migrations ran before Postgres finished initializing. The fix is exponential backoff with jitter, only retrying transient failures (503s, timeouts) not deterministic ones (404s, 401s). Includes production-ready bash implementation and best practices guidance.

• •Race condition: migration started before database finished booting, causing intermittent deploy failures
• •Solution: exponential backoff retry function with jitter to handle transient failures gracefully
• •Best practice: distinguish what to retry (timeouts, 503s) from what to fail fast on (404s, syntax errors)

Generated with AI, which can make mistakes.