The Day I Realized "It Ran Successfully" Means Nothing in Databricks Production

Short summary

A senior engineer shipped a Databricks MERGE pipeline that ran successfully but scanned 3.8 TB without Z-ordering, costing $2,300 in eleven days for a job that should have cost $40. The article argues that production data engineering failures are silent—pipelines can corrupt data, inflate costs 50x, and drop records while showing green status. The mindset shift is about asking better questions before go-live: failure resilience, cost as correctness, and SLA awareness.

• •Pipelines can succeed technically while failing on cost, data quality, and SLAs
• •Scale exposes assumptions that dev testing cannot catch
• •Cost profile and graceful degradation are first-class production concerns

Generated with AI, which can make mistakes.