Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[VL] enable 11-22 upstreaming velox rebase #3773

Merged
merged 9 commits into from
Nov 27, 2023
Merged

Conversation

JkSelf
Copy link
Contributor

@JkSelf JkSelf commented Nov 20, 2023

What changes were proposed in this pull request?

  1. Make some modifications in Gluten by incorporating the BridgeOptions parameter for updating the Arrow conversion API in Support different timestamp units in arrow bridge facebookincubator/velox#7625
  2. Upgrade DuckDB to 0.8.1 facebookincubator/velox#6725 upgrades DuckDB to version 0.8.1, requiring changes in Gluten to update its dependency on DuckDB as well.

How was this patch tested?

Existing UTs.

Copy link

Thanks for opening a pull request!

Could you open an issue for this pull request on Github Issues?

https://github.com/oap-project/gluten/issues

Then could you also rename commit message and pull request title in the following format?

[GLUTEN-${ISSUES_ID}][COMPONENT]feat/fix: ${detailed message}

See also:

@JkSelf JkSelf force-pushed the test-11-19 branch 9 times, most recently from 4d0540f to 02564bf Compare November 24, 2023 03:27
Copy link

Run Gluten Clickhouse CI

Copy link

Run Gluten Clickhouse CI

@JkSelf JkSelf force-pushed the test-11-19 branch 3 times, most recently from 9a6d8a4 to 4af499f Compare November 24, 2023 08:25
Copy link

Run Gluten Clickhouse CI

@JkSelf JkSelf force-pushed the test-11-19 branch 4 times, most recently from 082539a to b3ccc93 Compare November 24, 2023 22:29
@@ -120,11 +120,18 @@ function compile {
COMPILE_OPTION="$COMPILE_OPTION -DVELOX_ENABLE_GCS=ON"
fi

if [[ $ENABLE_TESTS == "ON" ]] || [[ $ENABLE_BENCHMARK == "ON" ]]; then
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this part necessary? I note VELOX_ENABLE_PARSE=ON if VELOX_BUILD_TESTING or VELXO_BUILD_BENCHMARKS is ON according to CMakeLists.txt in velox. Maybe, we can try to remove this part to see CI feedback. Thanks!

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@PHILO-HE This is necessary because in Velox, VELOX_ENABLE_PARSE is set to ON by default, while in Gluten, VELOX_ENABLE_DUCKDB is set to OFF by default. Therefore, if tests are not enabled, Velox compilation will exclude DuckDB but include Parse, resulting in a compilation issue where DuckDB files cannot be found.

if(BUILD_TESTS)
add_velox_dependency(parse::expression "${VELOX_COMPONENTS_PATH}/parse/libvelox_parse_expression.a")
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: we can just move this part into the above if branch.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@PHILO-HE Updated.

zhouyuan
zhouyuan previously approved these changes Nov 27, 2023
@zhouyuan zhouyuan changed the title [VL] Test 11-19 upstreaming velox rebase [VL] enable 11-22 upstreaming velox rebase Nov 27, 2023
@zhouyuan zhouyuan merged commit 1e65165 into apache:main Nov 27, 2023
14 of 16 checks passed
@GlutenPerfBot
Copy link
Contributor

===== Performance report for TPCH SF2000 with Velox backend, for reference only ====

query log/native_3773_time.csv log/native_master_11_26_2023_f456a7e86_time.csv difference percentage
q1 34.44 34.38 -0.063 99.82%
q2 24.96 24.62 -0.332 98.67%
q3 37.93 37.74 -0.189 99.50%
q4 35.92 36.67 0.752 102.09%
q5 72.02 68.82 -3.200 95.56%
q6 7.33 7.12 -0.209 97.15%
q7 83.60 82.01 -1.584 98.11%
q8 86.97 87.72 0.755 100.87%
q9 126.75 120.79 -5.960 95.30%
q10 47.28 44.22 -3.061 93.53%
q11 21.24 19.96 -1.288 93.94%
q12 25.82 28.56 2.737 110.60%
q13 46.24 46.61 0.368 100.80%
q14 18.58 14.35 -4.234 77.21%
q15 27.20 29.15 1.953 107.18%
q16 16.59 15.41 -1.182 92.87%
q17 103.07 100.98 -2.090 97.97%
q18 152.18 145.61 -6.577 95.68%
q19 12.96 12.88 -0.079 99.39%
q20 28.57 27.79 -0.779 97.27%
q21 224.28 223.56 -0.720 99.68%
q22 13.18 13.03 -0.143 98.91%
total 1247.11 1221.98 -25.126 97.99%

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants