Back to feed
Dev.to
Dev.to
6/30/2026
Data Costs: Are You a Victim of Your Own Success?

Data Costs: Are You a Victim of Your Own Success?

Short summary

Frequent analytical queries against unpartitioned historical data create runaway costs: scanning 100x more data by day 100 for the same insight. BigQuery, Athena, and Snowflake charge per data scanned, making zero-ETL architectures prohibitively expensive at scale. Partition pruning or state management layers isolate new data only, keeping costs flat as history grows.

  • •Zero-ETL convenience hits limits when rescanning full historical datasets for 15-minute frequency queries: costs explode from $24/day to $2,400/day
  • •All major hyperscalers (BigQuery at $6.25/TiB, Athena at $5/TB, Snowflake via credits) charge per scan, penalizing architectural decisions
  • •Add state management or partition pruning to scan only new data (~520MB/run), maintaining Day 1 costs indefinitely

Generated with AI, which can make mistakes.

Read full article at Dev.to

Is this a good recommendation for you?

Comments

Failed to load comments. Please try again.

Explore more

The original title is: "Our GCP daily cost dropped 80% in six weeks, here is every change we made"
Dev.to
Cloud Bills Took a Decade to Get FinOps. AI Bills Won't Get That Long.
Dev.to
The Autoscaling Bug That Costs Companies Thousands Before Anyone Notices
Dev.to
The Day I Realized "It Ran Successfully" Means Nothing in Databricks Production
Dev.to