Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[GLUTEN-3715][VL] followup to add CentOS7 support for GCS #3789

Merged
merged 3 commits into from
Nov 22, 2023
Merged

[GLUTEN-3715][VL] followup to add CentOS7 support for GCS #3789

merged 3 commits into from
Nov 22, 2023

Conversation

zhouyuan
Copy link
Contributor

@zhouyuan zhouyuan commented Nov 21, 2023 •
edited
Loading

What changes were proposed in this pull request?

add GCS support on Centos7, also removed the unnecessary arrow components in static build job

How was this patch tested?

pass GHA

@github-actions GitHub Actions
Copy link

#3715

@github-actions GitHub Actions
Copy link

Run Gluten Clickhouse CI

@github-actions GitHub Actions
Copy link

Run Gluten Clickhouse CI

@zhouyuan zhouyuan changed the title [GLUTEN-3715][VL] followup to add CentOS7 support [GLUTEN-3715][VL] followup to add CentOS7 support for GCS Nov 21, 2023
@github-actions GitHub Actions
Copy link

Run Gluten Clickhouse CI

@github-actions GitHub Actions
Copy link

Run Gluten Clickhouse CI

@zhouyuan zhouyuan force-pushed the wip_gcs_followup branch 3 times, most recently from b09281f to 55b6b73 Compare November 21, 2023 09:02
@apache apache deleted a comment from github-actions bot Nov 21, 2023
@apache apache deleted a comment from github-actions bot Nov 21, 2023
@apache apache deleted a comment from github-actions bot Nov 21, 2023
@apache apache deleted a comment from github-actions bot Nov 21, 2023
@apache apache deleted a comment from github-actions bot Nov 21, 2023
@github-actions GitHub Actions
Copy link

Run Gluten Clickhouse CI

@github-actions GitHub Actions
Copy link

Run Gluten Clickhouse CI

@zhouyuan zhouyuan changed the title [GLUTEN-3715][VL] followup to add CentOS7 support for GCS [DNM][GLUTEN-3715][VL] followup to add CentOS7 support for GCS Nov 21, 2023
@github-actions GitHub Actions
Copy link

Run Gluten Clickhouse CI

@github-actions GitHub Actions
Copy link

Run Gluten Clickhouse CI

@github-actions GitHub Actions
Copy link

Run Gluten Clickhouse CI

@zhouyuan zhouyuan changed the title [DNM][GLUTEN-3715][VL] followup to add CentOS7 support for GCS [GLUTEN-3715][VL] followup to add CentOS7 support for GCS Nov 22, 2023
@zhouyuan
[skipci] rename gha job
e807e0e 
Signed-off-by: Yuan Zhou <[email protected]>
@github-actions GitHub Actions
Copy link

Run Gluten Clickhouse CI

zhouyuan
zhouyuan commented Nov 22, 2023
View reviewed changes
tools/gluten-te/centos/dockerfile-buildenv
# Build & install Spark 3.2.2
RUN cd /opt && git clone --depth 1 --branch v3.2.2 https://github.com/apache/spark.git spark322
RUN cd /opt/spark322 && ./build/mvn -Pyarn -DskipTests clean install
# # Build & install Spark 3.2.2
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

these are commented out as gluten only run spark unit tests on ubuntu

zhouyuan
zhouyuan commented Nov 22, 2023
View reviewed changes
dev/vcpkg/vcpkg.json
@@ -2,25 +2,8 @@
"$schema": "https://raw.githubusercontent.com/microsoft/vcpkg-tool/main/docs/vcpkg.schema.json",
"builtin-baseline": "a7b6122f6b6504d16d96117336a0562693579933",
"dependencies": ["jemalloc"],
"default-features": ["arrow", "velox", "velox-s3", "velox-gcs", "velox-hdfs"],
"default-features": ["velox", "velox-s3", "velox-gcs", "velox-hdfs"],
Copy link
Contributor Author

@zhouyuan zhouyuan Nov 22, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

not necessary now as gluten is using velox built in arrow

zhouyuan
zhouyuan commented Nov 22, 2023
View reviewed changes
.github/workflows/velox_be.yml
MOUNT_MAVEN_CACHE=OFF \
OS_IMAGE=centos:7 \
OS_VERSION=7 \
tools/gluten-te/centos/cbash.sh sleep 14400
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

using vanilla centos image here as the dependency is broken in pre-built image

PHILO-HE
PHILO-HE approved these changes Nov 22, 2023
View reviewed changes
Copy link
Contributor

@PHILO-HE PHILO-HE left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good!

@zhouyuan zhouyuan merged commit 16abf39 into apache:main Nov 22, 2023
17 checks passed
@GlutenPerfBot
Copy link
Contributor

===== Performance report for TPCH SF2000 with Velox backend, for reference only ====

query log/native_3789_time.csv log/native_master_11_21_2023_b40a5f094_time.csv difference percentage
q1 34.76 33.71 -1.045 96.99%
q2 24.50 24.92 0.420 101.71%
q3 37.71 37.63 -0.075 99.80%
q4 36.58 35.99 -0.591 98.39%
q5 71.05 70.48 -0.574 99.19%
q6 7.26 7.11 -0.149 97.94%
q7 83.13 84.08 0.949 101.14%
q8 87.68 87.02 -0.664 99.24%
q9 127.34 124.24 -3.103 97.56%
q10 43.88 46.06 2.182 104.97%
q11 19.39 19.24 -0.142 99.27%
q12 25.30 25.38 0.084 100.33%
q13 45.33 45.84 0.513 101.13%
q14 14.20 18.33 4.131 129.10%
q15 29.10 28.80 -0.305 98.95%
q16 15.38 15.37 -0.016 99.90%
q17 100.83 100.95 0.120 100.12%
q18 147.26 146.37 -0.890 99.40%
q19 13.96 12.89 -1.072 92.32%
q20 27.17 28.27 1.096 104.03%
q21 222.93 222.09 -0.835 99.63%
q22 13.10 12.99 -0.112 99.15%
total 1227.83 1227.75 -0.075 99.99%

@GlutenPerfBot
Copy link
Contributor

===== Performance report for TPCH SF2000 with Velox backend, for reference only ====

query log/native_master_11_22_2023_time.csv log/native_master_11_21_2023_b40a5f094_time.csv difference percentage
q1 34.52 33.71 -0.808 97.66%
q2 24.61 24.92 0.308 101.25%
q3 37.51 37.63 0.119 100.32%
q4 36.72 35.99 -0.730 98.01%
q5 70.57 70.48 -0.091 99.87%
q6 7.86 7.11 -0.756 90.38%
q7 84.69 84.08 -0.608 99.28%
q8 85.82 87.02 1.203 101.40%
q9 124.99 124.24 -0.756 99.40%
q10 43.50 46.06 2.567 105.90%
q11 19.62 19.24 -0.375 98.09%
q12 25.22 25.38 0.161 100.64%
q13 45.75 45.84 0.090 100.20%
q14 19.48 18.33 -1.148 94.11%
q15 27.08 28.80 1.720 106.35%
q16 15.52 15.37 -0.155 99.00%
q17 100.26 100.95 0.690 100.69%
q18 147.25 146.37 -0.886 99.40%
q19 13.16 12.89 -0.269 97.96%
q20 26.97 28.27 1.302 104.83%
q21 221.08 222.09 1.014 100.46%
q22 12.89 12.99 0.098 100.76%
total 1225.07 1227.75 2.689 100.22%

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@PHILO-HE PHILO-HE PHILO-HE approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@zhouyuan @GlutenPerfBot @PHILO-HE