[GLUTEN-3486][CH] Fix AQE cannot coalesce shuffle partitions #3941
Conversation
|
Run Gluten Clickhouse CI |
|
LGTM |
| import org.apache.spark.sql.catalyst.expressions.Expression | ||
| import org.apache.spark.sql.catalyst.plans.physical.HashPartitioning | ||
|
|
||
| class HashPartitioningWrapper( |
There was a problem hiding this comment.
Could we add some description for the functionality of this class?
| @@ -20,6 +20,7 @@ import io.glutenproject.GlutenConfig | |||
| import io.glutenproject.backendsapi.BackendsApiManager | |||
| import io.glutenproject.execution._ | |||
| import io.glutenproject.extension.{GlutenPlan, ValidationResult} | |||
| import io.glutenproject.sql.shims.SparkShimLoader | |||
| (projectExpressions.size, HashPartitioning(newExprs, numPartitions), project) | ||
| ( | ||
| projectExpressions.size, | ||
| new HashPartitioningWrapper(exprs, newExprs, numPartitions), |
There was a problem hiding this comment.
I'm not sure I get it. AQE won't touch the expressions in hash partitioning, then how can it affect AQE behavior ? Does it also affect range partitioning?
There was a problem hiding this comment.
https://github.com/apache/spark/blob/b9cc813adf0ba592f020bc9778b373800ad81b8f/sql/core/src/main/scala/org/apache/spark/sql/execution/adaptive/AdaptiveSparkPlanExec.scala#L180C30-L180C30 here ValidateRequirements.validate(applied, distribution) will validate failed due to shuffle hash expressions changed.
There was a problem hiding this comment.
I see, and these codes only be used in ch backend so Velox backend does not have this issue.
There was a problem hiding this comment.
I see, and these codes only be used in ch backend so Velox backend does not have this issue.
yes
|
Run Gluten Clickhouse CI |
|
Run Gluten Clickhouse CI |
|
===== Performance report for TPCH SF2000 with Velox backend, for reference only ====
|
What changes were proposed in this pull request?
When there are non-field expressions in shuffle hash expression, we add a pre-project before shuffle exchange operator and change shuffle hash expression to field expression
projected_partitioning_valuewhich prevents AQE'sCoalesceShufflePartitionsandOptimizeSkewedJoinrules from being effective. This pr introduces aHashPartitioningWrapperin order to remain original shuffle hash expressions.(Fixes: #3486)
How was this patch tested?
manual tests