Apache Spark Query Optimization on Databricks: Catalyst, AQE, and Photon Engine

Short summary

Databricks compounds Apache Spark's native Catalyst optimizer with Adaptive Query Execution and the Photon C++ vectorized engine to dramatically improve query performance. The Catalyst pipeline—parsing, analysis, logical optimization, and physical planning—transforms human-written SQL into optimized execution plans using rules like predicate pushdown and column pruning. Understanding these three layers lets you write queries that cooperate with the engine.

• •Catalyst's 4-stage pipeline (parsing, analysis, logical optimization, physical planning) transforms SQL into optimized plans
• •AQE re-optimizes at runtime with actual statistics; Photon is a C++ vectorized executor replacing the JVM
• •Key optimization rules (predicate pushdown, column pruning, constant folding) reduce query latency dramatically

Generated with AI, which can make mistakes.