[GLUTEN-3361] Support spark 3.4 in Gluten #3360

JkSelf · 2023-10-10T02:15:11Z

What changes were proposed in this pull request?

Follow up #3262.

Mainly changes:

In Spark 3.4, there was an API change in BatchScanExec. You can find the details here. A new table parameter has been added, so it needs to be included in the shim layer.
We have copied some code from vanilla Spark in FileSourceScanExecTransformer. You can find it here. In Spark 3.3 and 3.2, the copied code was private, but in Spark 3.4, it is protected. Therefore, it should be placed in the shim layer. Additionally, GlutenTimeMetric.scala and Arm.scala need to be placed in the shim layer to resolve compilation issues.
Offset.scala is newly introduced in Spark 3.4. Hence, it should be added to the Spark 3.2 and Spark 3.3 shim layers.
Empty2Null is in org.apache.spark.sql.execution.datasources.FileFormatWriter package in spark 3.2 and spark 3.3. So we also need add Empty2Null in org.apache.spark.sql.execution.datasources.FileFormatWriter package in Spark 3.4 to pass the compile issue.
The PartitionedFile API changed the filePath parameter from string to SparkPath object in 3.4.
In Spark 3.4, StatFunctions.scala is no longer needed in shim layer. Therefore, it has been moved from shim common to spark 3.3 and shim spark 3.2 shim layer.
PromotePrecision.scala has been deleted in Spark 3.4.
The unused DwrfScan code has been removed from the Spark 3.2 and Spark 3.3 shim layers.
Since DataSourceStrategyUtil.scala is not used in gluten, it has been deleted.
A new API, visitOffset(p: Offset), has been added to the LogicalPlanVisitor in Spark 3.4.

How was this patch tested?

Existing tests.

github-actions · 2023-10-10T02:15:28Z

Thanks for opening a pull request!

Could you open an issue for this pull request on Github Issues?

https://github.com/oap-project/gluten/issues

Then could you also rename commit message and pull request title in the following format?

[GLUTEN-${ISSUES_ID}][COMPONENT]feat/fix: ${detailed message}

See also:

Other pull requests

github-actions · 2023-10-10T02:15:45Z

Run Gluten Clickhouse CI

github-actions · 2023-10-10T02:17:50Z

#3361

github-actions · 2023-10-10T02:28:13Z

Run Gluten Clickhouse CI

github-actions · 2023-10-10T03:07:12Z

Run Gluten Clickhouse CI

github-actions · 2023-10-10T03:25:42Z

Run Gluten Clickhouse CI

github-actions · 2023-10-10T07:03:04Z

Run Gluten Clickhouse CI

github-actions · 2023-10-10T08:58:02Z

Run Gluten Clickhouse CI

github-actions · 2023-10-13T03:20:44Z

Run Gluten Clickhouse CI

github-actions · 2023-10-13T03:24:19Z

Run Gluten Clickhouse CI

github-actions · 2023-10-13T04:36:54Z

Run Gluten Clickhouse CI

github-actions · 2023-10-13T07:27:10Z

Run Gluten Clickhouse CI

github-actions · 2023-10-14T06:28:56Z

Run Gluten Clickhouse CI

JkSelf · 2023-10-16T01:12:30Z

@zzcclp Can you help to look the failed two unit tests in CH backend? Thanks.

github-actions · 2023-10-16T01:54:19Z

Run Gluten Clickhouse CI

github-actions · 2023-10-16T07:14:01Z

Run Gluten Clickhouse CI

github-actions · 2023-10-16T07:19:31Z

Run Gluten Clickhouse CI

github-actions · 2023-10-16T07:21:13Z

Run Gluten Clickhouse CI

github-actions · 2023-10-16T07:25:26Z

Run Gluten Clickhouse CI

github-actions · 2023-10-16T08:12:48Z

Run Gluten Clickhouse CI

github-actions · 2023-10-16T10:51:43Z

Run Gluten Clickhouse CI

github-actions · 2023-10-16T10:59:18Z

Run Gluten Clickhouse CI

github-actions · 2023-10-16T13:46:51Z

Run Gluten Clickhouse CI

shims/spark34/src/main/java/org/apache/spark/sql/execution/joins/HashJoin.scala.deprecated

shims/spark34/src/main/scala/org/apache/spark/sql/catalyst/expressions/PromotePrecision.scala

github-actions · 2023-10-17T06:44:27Z

Run Gluten Clickhouse CI

shims/spark34/src/main/scala/org/apache/spark/sql/execution/datasources/FileFormatWriter.scala

github-actions · 2023-10-18T01:26:24Z

Run Gluten Clickhouse CI

zhouyuan · 2023-10-18T09:15:31Z

gluten-core/src/main/scala/org/apache/spark/rpc/GlutenDriverEndpoint.scala

@@ -40,6 +40,8 @@ class GlutenDriverEndpoint extends IsolatedRpcEndpoint with Logging {
  private val driverEndpoint: RpcEndpointRef =
    rpcEnv.setupEndpoint(GlutenRpcConstants.GLUTEN_DRIVER_ENDPOINT_NAME, this)

+  // TODO(yuan): get thread cnt from spark context
+  override def threadCount(): Int = 1


@lwz9103 @zzcclp will this change threadCount() impact CK backend?

the default value is 1 , right ?

It seems ok to me.

zhouyuan

it's better to also update dev/buildbundle-veloxbe.sh to build spark 3.4 package

github-actions · 2023-10-19T02:11:33Z

Run Gluten Clickhouse CI

Signed-off-by: Yuan Zhou <[email protected]>

github-actions · 2023-10-19T06:38:24Z

Run Gluten Clickhouse CI

zzcclp · 2023-10-19T08:32:05Z

LGTM

zhouyuan

👍 works on my local TPC-H/DS

zhouyuan · 2023-10-20T05:57:28Z

notes on TODOs:

the UT for Spark 3.4 will be added in follow up commits.
The docker image in GHA needs to be updated with Spark 3.4 package.

JkSelf changed the title ~~Upgrade spark3.4~~ [GLUTEN-3361] Upgrade spark version to 3.4 in Gluten Oct 10, 2023

zhouyuan mentioned this pull request Oct 10, 2023

WIP: support spark-3.4 #3262

Closed

JkSelf changed the title ~~[GLUTEN-3361] Upgrade spark version to 3.4 in Gluten~~ [GLUTEN-3361][WIP] Upgrade spark version to 3.4 in Gluten Oct 10, 2023

JkSelf force-pushed the upgrade-spark3.4 branch from e508365 to dc879f4 Compare October 10, 2023 03:25

JkSelf force-pushed the upgrade-spark3.4 branch from dc879f4 to c1637c5 Compare October 10, 2023 07:02

JkSelf force-pushed the upgrade-spark3.4 branch from c1637c5 to 81681f2 Compare October 10, 2023 08:57

JkSelf changed the title ~~[GLUTEN-3361][WIP] Upgrade spark version to 3.4 in Gluten~~ [GLUTEN-3361]Upgrade spark version to 3.4 in Gluten Oct 10, 2023

PHILO-HE changed the title ~~[GLUTEN-3361]Upgrade spark version to 3.4 in Gluten~~ [GLUTEN-3361] Support spark 3.4 in Gluten Oct 11, 2023

JkSelf force-pushed the upgrade-spark3.4 branch from 81681f2 to 91028ad Compare October 13, 2023 03:20

JkSelf force-pushed the upgrade-spark3.4 branch from 91028ad to 18123f2 Compare October 13, 2023 03:23

JkSelf force-pushed the upgrade-spark3.4 branch from 18123f2 to e833b67 Compare October 13, 2023 04:36

JkSelf force-pushed the upgrade-spark3.4 branch from e833b67 to 39ddeee Compare October 13, 2023 07:26

JkSelf mentioned this pull request Oct 13, 2023

[GLUTEN-3154][FOLLOWUP] Add keyGroupedPartitioning in BatchScanExecTransformer for spark-3.3 #3184

Merged

JkSelf force-pushed the upgrade-spark3.4 branch from 39ddeee to af113bd Compare October 14, 2023 06:28

JkSelf force-pushed the upgrade-spark3.4 branch from af113bd to 2cd396d Compare October 16, 2023 01:53

JkSelf force-pushed the upgrade-spark3.4 branch from 80ed047 to 499f73e Compare October 16, 2023 07:18

JkSelf force-pushed the upgrade-spark3.4 branch from 793c085 to f53edc1 Compare October 16, 2023 08:12

JkSelf force-pushed the upgrade-spark3.4 branch from 209da1e to 8efe616 Compare October 16, 2023 10:58

ulysses-you reviewed Oct 17, 2023

View reviewed changes

shims/spark34/src/main/java/org/apache/spark/sql/execution/joins/HashJoin.scala.deprecated Outdated Show resolved Hide resolved

ulysses-you reviewed Oct 17, 2023

View reviewed changes

shims/spark34/src/main/scala/org/apache/spark/sql/catalyst/expressions/PromotePrecision.scala Show resolved Hide resolved

JkSelf force-pushed the upgrade-spark3.4 branch from db3a09d to 8a0d44e Compare October 17, 2023 06:43

ulysses-you reviewed Oct 17, 2023

View reviewed changes

shims/spark34/src/main/scala/org/apache/spark/sql/execution/datasources/FileFormatWriter.scala Outdated Show resolved Hide resolved

zhouyuan reviewed Oct 18, 2023

View reviewed changes

zhouyuan reviewed Oct 19, 2023

View reviewed changes

JkSelf force-pushed the upgrade-spark3.4 branch from 3c6c482 to afa806d Compare October 19, 2023 02:11

zhouyuan and others added 2 commits October 19, 2023 06:06

WIP: support spark-3.4

f7b7a33

Signed-off-by: Yuan Zhou <[email protected]>

Support spark 3.4 shim layer in gluten

47e1575

JkSelf force-pushed the upgrade-spark3.4 branch from afa806d to 47e1575 Compare October 19, 2023 06:37

zhouyuan approved these changes Oct 20, 2023

View reviewed changes

zhouyuan merged commit 3e73f32 into apache:main Oct 20, 2023
15 checks passed

zhouyuan mentioned this pull request Nov 16, 2023

Support Spark 3.4 in Gluten. #3361

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[GLUTEN-3361] Support spark 3.4 in Gluten #3360

[GLUTEN-3361] Support spark 3.4 in Gluten #3360

JkSelf commented Oct 10, 2023 •

edited

Loading

github-actions bot commented Oct 10, 2023

github-actions bot commented Oct 10, 2023

github-actions bot commented Oct 10, 2023

github-actions bot commented Oct 10, 2023

github-actions bot commented Oct 10, 2023

github-actions bot commented Oct 10, 2023

github-actions bot commented Oct 10, 2023

github-actions bot commented Oct 10, 2023

github-actions bot commented Oct 13, 2023

github-actions bot commented Oct 13, 2023

github-actions bot commented Oct 13, 2023

github-actions bot commented Oct 13, 2023

github-actions bot commented Oct 14, 2023

JkSelf commented Oct 16, 2023

github-actions bot commented Oct 16, 2023

github-actions bot commented Oct 16, 2023

github-actions bot commented Oct 16, 2023

github-actions bot commented Oct 16, 2023

github-actions bot commented Oct 16, 2023

github-actions bot commented Oct 16, 2023

github-actions bot commented Oct 16, 2023

github-actions bot commented Oct 16, 2023

github-actions bot commented Oct 16, 2023

github-actions bot commented Oct 17, 2023

github-actions bot commented Oct 18, 2023

zhouyuan Oct 18, 2023

zzcclp Oct 19, 2023

zzcclp Oct 19, 2023

zhouyuan left a comment

github-actions bot commented Oct 19, 2023

github-actions bot commented Oct 19, 2023

zzcclp commented Oct 19, 2023

zhouyuan left a comment

zhouyuan commented Oct 20, 2023

[GLUTEN-3361] Support spark 3.4 in Gluten #3360

[GLUTEN-3361] Support spark 3.4 in Gluten #3360

Conversation

JkSelf commented Oct 10, 2023 • edited Loading

What changes were proposed in this pull request?

How was this patch tested?

github-actions bot commented Oct 10, 2023

github-actions bot commented Oct 10, 2023

github-actions bot commented Oct 10, 2023

github-actions bot commented Oct 10, 2023

github-actions bot commented Oct 10, 2023

github-actions bot commented Oct 10, 2023

github-actions bot commented Oct 10, 2023

github-actions bot commented Oct 10, 2023

github-actions bot commented Oct 13, 2023

github-actions bot commented Oct 13, 2023

github-actions bot commented Oct 13, 2023

github-actions bot commented Oct 13, 2023

github-actions bot commented Oct 14, 2023

JkSelf commented Oct 16, 2023

github-actions bot commented Oct 16, 2023

github-actions bot commented Oct 16, 2023

github-actions bot commented Oct 16, 2023

github-actions bot commented Oct 16, 2023

github-actions bot commented Oct 16, 2023

github-actions bot commented Oct 16, 2023

github-actions bot commented Oct 16, 2023

github-actions bot commented Oct 16, 2023

github-actions bot commented Oct 16, 2023

github-actions bot commented Oct 17, 2023

github-actions bot commented Oct 18, 2023

zhouyuan Oct 18, 2023

Choose a reason for hiding this comment

zzcclp Oct 19, 2023

Choose a reason for hiding this comment

zzcclp Oct 19, 2023

Choose a reason for hiding this comment

zhouyuan left a comment

Choose a reason for hiding this comment

github-actions bot commented Oct 19, 2023

github-actions bot commented Oct 19, 2023

zzcclp commented Oct 19, 2023

zhouyuan left a comment

Choose a reason for hiding this comment

zhouyuan commented Oct 20, 2023

JkSelf commented Oct 10, 2023 •

edited

Loading