Using Adaptive Query Execution

Spark 3.0 and later includes an additional layer of optimization that is called Adaptive Query Execution (AQE). It optimizes queries based upon the metrics that are collected during query runtime. It uses the runtime statistics to pick the most efficient execution plan. This feature is enabled by default in Apache Spark version 3.2.0 and later. You can turn it off by setting the Spark configuration to spark.sql.adaptive.enabled=false. AQE has three primary features: coalescing post-shuffle partitions, converting sort-merge joins to broadcast joins, and optimizing skew joins.