A Dead-Man's Switch for Scheduled Jobs (Because Silent Failures Rot Data)

Short summary

A solo operator describes building a dead-man's switch for scheduled jobs after a content curation job died silently for days. The pattern: the job stamps a timestamp on success, and a separate watcher on a different runtime checks the stamp's age. Key insight: alerting for 'things that stop happening' is a different problem from alerting for 'things that fail.' The threshold should match the content's natural rhythm, not the job's schedule — a daily curation job may only need a 30-day staleness alert.

• •Silent failures (job stops running) require different alerting than loud failures (job errors)
• •Dead-man's switch: job stamps success timestamp, separate watcher checks staleness
• •Set thresholds based on content rhythm, not job schedule — avoid alert fatigue

Generated with AI, which can make mistakes.