Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[GLUTEN-3475] Add spark35 scala213 #3476

Closed
wants to merge 29 commits into from
Closed

[GLUTEN-3475] Add spark35 scala213 #3476

wants to merge 29 commits into from

Commits on Dec 5, 2023

  1. WIP: Add Spark 3.5 and Scala 2.13 support

    @holdenk
    holdenk committed Dec 5, 2023
    Configuration menu
    Copy the full SHA
    7178e5e View commit details
    Browse the repository at this point in the history

  2. Continue.

    @holdenk
    holdenk committed Dec 5, 2023
    Configuration menu
    Copy the full SHA
    16e9d9c View commit details
    Browse the repository at this point in the history

  3. More progress for Scala 2.13

    @holdenk
    holdenk committed Dec 5, 2023
    Configuration menu
    Copy the full SHA
    62efcee View commit details
    Browse the repository at this point in the history

  4. Add the missing Spark 35 shims

    @holdenk
    holdenk committed Dec 5, 2023
    Configuration menu
    Copy the full SHA
    19e25bd View commit details
    Browse the repository at this point in the history

  5. Typo

    @holdenk
    holdenk committed Dec 5, 2023
    Configuration menu
    Copy the full SHA
    19aad6f View commit details
    Browse the repository at this point in the history

  6. Remove hard coded Scala 2.12 dependencies. 

    Update spotless

Add scala collection compat lib.

Add scala compat libs
    @holdenk
    holdenk committed Dec 5, 2023
    Configuration menu
    Copy the full SHA
    5a802ff View commit details
    Browse the repository at this point in the history

  7. spotless apply.

    @holdenk
    holdenk committed Dec 5, 2023
    Configuration menu
    Copy the full SHA
    ce5cf3a View commit details
    Browse the repository at this point in the history

  8. Make the 3.5 substrait based on the previous.

    @holdenk
    holdenk committed Dec 5, 2023
    Configuration menu
    Copy the full SHA
    239a618 View commit details
    Browse the repository at this point in the history

  9. Work on making basic versions function 

    hm conv to seq
    @holdenk
    holdenk committed Dec 5, 2023
    Configuration menu
    Copy the full SHA
    8b8824a View commit details
    Browse the repository at this point in the history

  10. Move around some files so they make more sense and update the build f… 

    …iles.
    @holdenk
    holdenk committed Dec 5, 2023
    Configuration menu
    Copy the full SHA
    fb0e23b View commit details
    Browse the repository at this point in the history

  11. Use simplified partiioned file API

    @holdenk
    holdenk committed Dec 5, 2023
    Configuration menu
    Copy the full SHA
    6114ebe View commit details
    Browse the repository at this point in the history

  12. Make some more progress towards 3.5 support.

    @holdenk
    holdenk committed Dec 5, 2023
    Configuration menu
    Copy the full SHA
    1223d84 View commit details
    Browse the repository at this point in the history

  13. Add shim for fromAttributes

    @holdenk
    holdenk committed Dec 5, 2023
    Configuration menu
    Copy the full SHA
    be5dcf0 View commit details
    Browse the repository at this point in the history

  14. Compiles in Spark 3.5 and Scala 2.13

    @holdenk
    holdenk committed Dec 5, 2023
    Configuration menu
    Copy the full SHA
    f92a99e View commit details
    Browse the repository at this point in the history

  15. Work a bit on getting 3.4 working again.

    @holdenk
    holdenk committed Dec 5, 2023
    Configuration menu
    Copy the full SHA
    8a5b9dc View commit details
    Browse the repository at this point in the history

  16. Fix import so we can get ArraySeq from scala.collection.immutable._ i… 

    …n Scala 2.13+ and collection.compat.immutable._ Scala 2.12-

Fix imports magic

Fix imports again

Fix style
    @holdenk
    holdenk committed Dec 5, 2023
    Configuration menu
    Copy the full SHA
    093ce48 View commit details
    Browse the repository at this point in the history

  17. Compiles with Spark 3.5 and Scala 2.12

    @holdenk
    holdenk committed Dec 5, 2023
    Configuration menu
    Copy the full SHA
    b7ca56c View commit details
    Browse the repository at this point in the history

  18. We don't need the toArray and with that it compiles for 3.4 and 3.5 w… 

    …/Scala 2.13
    @holdenk
    holdenk committed Dec 5, 2023
    Configuration menu
    Copy the full SHA
    09cbb6b View commit details
    Browse the repository at this point in the history

  19. Add an argline for the tests.

    @holdenk
    holdenk committed Dec 5, 2023
    Configuration menu
    Copy the full SHA
    dd34fce View commit details
    Browse the repository at this point in the history

  20. Add surefire config so we have happy netty for java tests too.

    @holdenk
    holdenk committed Dec 5, 2023
    Configuration menu
    Copy the full SHA
    16b9fe1 View commit details
    Browse the repository at this point in the history

  21. Try and bump maven / mojo / surefire plugin versions.

    @holdenk
    holdenk committed Dec 5, 2023
    Configuration menu
    Copy the full SHA
    90809dc View commit details
    Browse the repository at this point in the history

  22. bump surefire too

    @holdenk
    holdenk committed Dec 5, 2023
    Configuration menu
    Copy the full SHA
    3c7553e View commit details
    Browse the repository at this point in the history

  23. The docker container used for building in CI only has JDK8 :'( roll b… 

    …ack spotless plugin version.
    @holdenk
    holdenk committed Dec 5, 2023
    Configuration menu
    Copy the full SHA
    773acce View commit details
    Browse the repository at this point in the history

  24. Try and make local testing semi-possible

    @holdenk
    holdenk committed Dec 5, 2023
    Configuration menu
    Copy the full SHA
    e211bb9 View commit details
    Browse the repository at this point in the history

  25. Try and fix running locally.

    @holdenk
    holdenk committed Dec 5, 2023
    Configuration menu
    Copy the full SHA
    a93a03b View commit details
    Browse the repository at this point in the history

  26. Pin google java format back for JDK8.

    @holdenk
    holdenk committed Dec 5, 2023
    Configuration menu
    Copy the full SHA
    3a328c5 View commit details
    Browse the repository at this point in the history

  27. Maybe we can list this as a plugin dep and that'll make the version m… 

    …atch? idk.
    @holdenk
    holdenk committed Dec 5, 2023
    Configuration menu
    Copy the full SHA
    a33f388 View commit details
    Browse the repository at this point in the history

  28. oops

    @holdenk
    holdenk committed Dec 5, 2023
    Configuration menu
    Copy the full SHA
    4b12977 View commit details
    Browse the repository at this point in the history

  29. More fix for the rebase

    @holdenk
    holdenk committed Dec 5, 2023
    Configuration menu
    Copy the full SHA
    19951a9 View commit details
    Browse the repository at this point in the history