The original title is: "Our GCP daily cost dropped 80% in six weeks, here is every change we made"

Short summary

A team running a federated health data platform on GCP in Indonesia saw their daily cloud costs triple due to upstream data backfills causing a deduplication scheduler to scan an increasingly large FHIR datastore. By analyzing billing exports and BigQuery job metadata, they identified that a single scheduler query was responsible for most of the cost spike, then implemented table clustering, high-water mark routing, and scheduler pauses to cut BigQuery scan volume from 9.26 TiB/day down to 0.47 TiB/day. The post walks through each optimization with concrete before/after metrics and code snippets.

• •GCP costs tripled when upstream data backfills caused a deduplication scheduler to scan a growing 201 GiB FHIR store three times per hour
• •BigQuery INFORMATION_SCHEMA analysis revealed the scheduler was the top cost driver, with daily scans rising from 2.36 TiB to 9.26 TiB in five weeks
• •Fixes included CLUSTER BY on the ledger table, high-water mark routing to skip anti-joins when no backlog exists, and pausing redundant schedulers, reducing scans to 0.47 TiB/day

Generated with AI, which can make mistakes.