[GLUTEN-3475] Add spark35 scala213 #3476

holdenk · 2023-10-20T21:17:57Z

What changes were proposed in this pull request?

(Please fill in changes proposed in this fix)

Fixes: #3475

How was this patch tested?

In progress, existing tests and compiling

See https://www.youtube.com/watch?v=UbB7v2f7A6k

github-actions · 2023-10-20T21:18:17Z

Thanks for opening a pull request!

Could you open an issue for this pull request on Github Issues?

https://github.com/oap-project/gluten/issues

Then could you also rename commit message and pull request title in the following format?

[GLUTEN-${ISSUES_ID}][COMPONENT]feat/fix: ${detailed message}

See also:

Other pull requests

github-actions · 2023-10-20T21:18:31Z

Run Gluten Clickhouse CI

github-actions · 2023-10-20T21:29:00Z

Run Gluten Clickhouse CI

LuciferYang

Seems that some code has not been uploaded yet? I haven't been able to compile successfully locally yet

LuciferYang · 2023-11-08T07:11:42Z

dev/buildbundle-veloxbe.sh

@@ -7,3 +7,6 @@ cd $GLUTEN_DIR
 mvn clean package -Pbackends-velox -Prss -Pspark-3.2 -DskipTests
 mvn clean package -Pbackends-velox -Prss -Pspark-3.3 -DskipTests
 mvn clean package -Pbackends-velox -Prss -Pspark-3.4 -DskipTests
+mvn clean package -Pbackends-velox -Prss -Pspark-3.5 -DskipTests
+# No celbron Scala 2.13 support yet.


Typo: celeborn

LuciferYang · 2023-11-08T07:17:01Z

backends-clickhouse/pom.xml

@@ -79,7 +79,7 @@
    <dependency>
      <groupId>org.scalacheck</groupId>
      <artifactId>scalacheck_${scala.binary.version}</artifactId>
-      <version>1.13.5</version>
+      <version>1.14.3</version>


https://github.com/oap-project/gluten/blob/d843bd93b460d6988c5d654d57978d096b0a835a/backends-clickhouse/pom.xml#L106-L117

There are some related dependencies that use hard coding _ 2.12 in this pom.xml, they should also use ${scala.binary.version}, I suspect there are other similar cases

LuciferYang · 2023-11-08T07:21:03Z

gluten-core/src/main/scala/io/glutenproject/GlutenPlugin.scala

@@ -190,7 +190,8 @@ private[glutenproject] class GlutenDriverPlugin extends DriverPlugin with Loggin

 private[glutenproject] class GlutenExecutorPlugin extends ExecutorPlugin {
  private var executorEndpoint: GlutenExecutorEndpoint = _
-  private val taskListeners: Seq[TaskListener] = Array(TaskResources)
+  private val taskListeners: Seq[TaskListener] =
+    (immutable.ArraySeq.unsafeWrapArray(Array(TaskResources)))


immutable.ArraySeq.unsafeWrapArray is a Scala 2.13 only api, if this performance optimization is necessary, this file may need to be split into Scala 2.12 and Scala 2.13 versions

LuciferYang · 2023-11-08T07:22:47Z

gluten-core/src/main/scala/io/glutenproject/backendsapi/BackendsApiManager.scala

@@ -20,7 +20,7 @@ import io.glutenproject.GlutenPlugin

 import java.util.ServiceLoader

-import scala.collection.JavaConverters
+import scala.jdk.CollectionConverters._


IIRC, scala.jdk.CollectionConverters also Scala 2.13 only

Ah good point, looks like there is a backwards compact lib ( https://stackoverflow.com/questions/59197883/version-agnostic-way-to-convert-from-java-to-scala-collections-and-back ) I'll try for Scala 2.12 support

github-actions · 2023-11-11T02:03:44Z

Run Gluten Clickhouse CI

holdenk · 2023-11-11T02:05:00Z

Still WIP but made some more progress

github-actions · 2023-11-11T02:05:11Z

Run Gluten Clickhouse CI

github-actions · 2023-11-13T04:35:04Z

Run Gluten Clickhouse CI

github-actions · 2023-11-17T01:24:11Z

Run Gluten Clickhouse CI

github-actions · 2023-11-17T02:44:16Z

Run Gluten Clickhouse CI

github-actions · 2023-11-18T06:55:39Z

Run Gluten Clickhouse CI

holdenk · 2023-11-21T02:43:54Z

@ LuciferYang any chance I can get read access to Jenkins?

github-actions · 2023-11-21T02:51:06Z

Run Gluten Clickhouse CI

github-actions · 2023-11-21T05:37:48Z

Run Gluten Clickhouse CI

github-actions · 2023-11-21T06:49:08Z

#3475

github-actions · 2023-11-22T01:37:03Z

Run Gluten Clickhouse CI

github-actions · 2023-11-22T01:37:39Z

Run Gluten Clickhouse CI

github-actions · 2023-11-22T01:37:41Z

Run Gluten Clickhouse CI

holdenk · 2023-11-22T01:49:14Z

@LuciferYang any chance I can get read access to Jenkins?

LuciferYang · 2023-11-22T03:09:35Z

@LuciferYang any chance I can get read access to Jenkins?

Sorry, I'm not familiar with this matter.

cc @zhouyuan

github-actions · 2023-11-22T20:47:57Z

Run Gluten Clickhouse CI

…ack spotless plugin version.

…atch? idk.

github-actions · 2023-12-05T00:47:02Z

Run Gluten Clickhouse CI

holdenk · 2023-12-25T19:18:21Z

Hey y'all, just following up I'm happy to update the PR on top of main if someone can get me access to CI and has the cycles to review, otherwise I'll just leave it be.

JkSelf · 2023-12-26T01:17:27Z

@holdenk Thanks for your great work. I have approved the CI running. The changes in this PR are too extensive, making it difficult to review. I noticed that the Scala version in Spark 3.5 is still 2.12 here. So, can we enable only spark 3.5 in this PR and create a separate PR to upgrade the Scala version to 2.13 if necessary? Also, could you please help list some of the main modifications in the PR description to facilitate our review?

holdenk · 2023-12-26T01:25:08Z

@JkSelf thanks! Can I get a log in so I can see the CI results / logs?

As far as splitting out Scala 2.13 / Spark 3.5 changes certainly possible but given everything so far I'd want to know there was someone with merge permissions with the time and interest to do a review.

JkSelf · 2023-12-26T02:43:19Z

@holdenk You can click on the Details to view the CI log. I will follow up with the review later. Thanks.

holdenk · 2023-12-26T02:44:49Z

So the Jenkins instance requires a login, without that I'm not sure how to fully debug the issues which come up when I fix the GHA issues.

Also before I make another PR would like to know who to work with on review for Spark 3.5.

JkSelf · 2023-12-26T02:46:31Z

So the Jenkins instance requires a login, without that I'm not sure how to fully debug the issues which come up when I fix the GHA issues.

@zhztheplayer Do you have any input? Thanks.

zhztheplayer · 2023-12-26T02:51:36Z

The Jenkins CI is only for ClickHouse backend, username/password: gluten/hN2xX3uQ4m , see ref.

holdenk · 2023-12-26T02:53:38Z

Awesome, thanks for the login.

JkSelf · 2023-12-26T02:56:06Z

@holdenk It appears that the Velox backends jobs have also failed. Please ensure that all Velox and ClickHouse backends pass successfully. Thanks.

holdenk · 2023-12-26T02:58:09Z

So if you're down to review the change sure I'll make a Scala 2.12 PR once I've got some time (maybe later this week or next), but unless there's a reviewer available I'm not working on this anymore.

To be clear: I totally understand if folks don't have review bandwidth I know how much work that can be from Spark -- but it's the holidays in the US and I only have so much bandwidth.

JkSelf · 2023-12-26T03:03:12Z

Yes, Can we merge the Spark 3.5 PR first and then proceed with the Scala 2.13 PR?

holdenk · 2023-12-26T03:07:10Z

Rocking, I'll try to put together a Scala 2.12 Spark 3.5 PR for you to review for the first week of January :)

github-actions · 2024-02-13T01:44:11Z

This PR is stale because it has been open 45 days with no activity. Remove stale label or comment or this will be closed in 10 days.

github-actions · 2024-02-24T01:42:53Z

This PR was auto-closed because it has been stalled for 10 days with no activity. Please feel free to reopen if it is still valid. Thanks.

FelixYBW requested a review from JkSelf October 31, 2023 04:27

LuciferYang reviewed Nov 8, 2023

View reviewed changes

holdenk force-pushed the add-spark35-scala213 branch from 18f96d8 to 32f8cfb Compare November 13, 2023 04:34

holdenk force-pushed the add-spark35-scala213 branch from 31aa990 to 7e1916c Compare November 21, 2023 02:50

holdenk changed the title ~~[WIP][3475] Add spark35 scala213~~ [VL-3475] Add spark35 scala213 Nov 21, 2023

holdenk changed the title ~~[VL-3475] Add spark35 scala213~~ [3475] Add spark35 scala213 Nov 21, 2023

holdenk changed the title ~~[3475] Add spark35 scala213~~ [GLUTEN-3475] Add spark35 scala213 Nov 21, 2023

holdenk force-pushed the add-spark35-scala213 branch from 29cc597 to 19c02f8 Compare November 22, 2023 01:36

holdenk force-pushed the add-spark35-scala213 branch from 97e70f1 to 7fa97e9 Compare November 22, 2023 01:37

holdenk force-pushed the add-spark35-scala213 branch from c359501 to 852487c Compare November 22, 2023 21:26

holdenk added 11 commits December 4, 2023 16:22

Add an argline for the tests.

dd34fce

Add surefire config so we have happy netty for java tests too.

16b9fe1

Try and bump maven / mojo / surefire plugin versions.

90809dc

bump surefire too

3c7553e

The docker container used for building in CI only has JDK8 :'( roll b…

773acce

…ack spotless plugin version.

Try and make local testing semi-possible

e211bb9

Try and fix running locally.

a93a03b

Pin google java format back for JDK8.

3a328c5

Maybe we can list this as a plugin dep and that'll make the version m…

a33f388

…atch? idk.

oops

4b12977

More fix for the rebase

19951a9

holdenk force-pushed the add-spark35-scala213 branch from f23b456 to 19951a9 Compare December 5, 2023 00:46

holdenk mentioned this pull request Jan 17, 2024

[GLUTEN-4424] Explore adding Spark 35 w/ Scala 2.12 only (WIP) #4425

Closed

github-actions bot added the stale stale label Feb 13, 2024

github-actions bot closed this Feb 24, 2024

[GLUTEN-3475] Add spark35 scala213 #3476

[GLUTEN-3475] Add spark35 scala213 #3476

Conversation

holdenk commented Oct 20, 2023 • edited Loading

What changes were proposed in this pull request?

How was this patch tested?

github-actions bot commented Oct 20, 2023

github-actions bot commented Oct 20, 2023

github-actions bot commented Oct 20, 2023

LuciferYang left a comment

Choose a reason for hiding this comment

LuciferYang Nov 8, 2023

Choose a reason for hiding this comment

LuciferYang Nov 8, 2023

Choose a reason for hiding this comment

LuciferYang Nov 8, 2023

Choose a reason for hiding this comment

LuciferYang Nov 8, 2023

Choose a reason for hiding this comment

holdenk Nov 11, 2023

Choose a reason for hiding this comment

github-actions bot commented Nov 11, 2023

holdenk commented Nov 11, 2023

github-actions bot commented Nov 11, 2023

github-actions bot commented Nov 13, 2023

github-actions bot commented Nov 17, 2023

github-actions bot commented Nov 17, 2023

github-actions bot commented Nov 18, 2023

holdenk commented Nov 21, 2023

github-actions bot commented Nov 21, 2023

github-actions bot commented Nov 21, 2023

github-actions bot commented Nov 21, 2023

github-actions bot commented Nov 22, 2023

github-actions bot commented Nov 22, 2023

github-actions bot commented Nov 22, 2023

holdenk commented Nov 22, 2023

LuciferYang commented Nov 22, 2023

github-actions bot commented Nov 22, 2023

github-actions bot commented Dec 5, 2023

holdenk commented Dec 25, 2023

JkSelf commented Dec 26, 2023

holdenk commented Dec 26, 2023

JkSelf commented Dec 26, 2023

holdenk commented Dec 26, 2023 • edited Loading

JkSelf commented Dec 26, 2023

zhztheplayer commented Dec 26, 2023 • edited Loading

holdenk commented Dec 26, 2023

JkSelf commented Dec 26, 2023

holdenk commented Dec 26, 2023 • edited Loading

JkSelf commented Dec 26, 2023

holdenk commented Dec 26, 2023

github-actions bot commented Feb 13, 2024

github-actions bot commented Feb 24, 2024

holdenk commented Oct 20, 2023 •

edited

Loading

holdenk commented Dec 26, 2023 •

edited

Loading

zhztheplayer commented Dec 26, 2023 •

edited

Loading

holdenk commented Dec 26, 2023 •

edited

Loading