When does Iceberg beat Parquet+projection on AWS Glue, and when doesn't ?

Short summary

Author built a test project comparing Apache Iceberg vs Parquet+partition-projection on AWS Glue with the same streaming pipeline (Kinesis → Firehose → Athena), documenting which storage format wins under different conditions. Shares design decisions with rationale for read modes (DynamicFrame vs Spark vs Iceberg), trade-offs, and TDD-friendly architectural patterns. Includes practical guidance on partition projection constraints and cost implications.

• •Compares Iceberg vs Parquet+projection on identical AWS Glue streaming pipeline
• •Explains read mode trade-offs and cost/performance implications for each approach
• •Shares architectural patterns including partition projection constraints and TDD code organization

Generated with AI, which can make mistakes.