Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[VL] Use value of spark.io.compression.codec as Velox's spill codec #2500

Merged
merged 1 commit into from
Oct 31, 2023

Conversation

jinchengchenghh
Copy link
Contributor

@jinchengchenghh jinchengchenghh commented Jul 28, 2023

Use Spark's existing option spark.io.compression.codec as Velox backend's spill codec.

@github-actions
Copy link

Thanks for opening a pull request!

Could you open an issue for this pull request on Github Issues?

https://github.com/oap-project/gluten/issues

Then could you also rename commit message and pull request title in the following format?

[GLUTEN-${ISSUES_ID}][COMPONENT]feat/fix: ${detailed message}

See also:

@github-actions
Copy link

Run Gluten Clickhouse CI

@FelixYBW FelixYBW changed the title Add the spill compress config [Velox] Add the spill compress config Jul 28, 2023
@jinchengchenghh jinchengchenghh added the pending velox rebase the patch will be merged after velox rebased with Meta main branch label Aug 16, 2023
@github-actions
Copy link

Run Gluten Clickhouse CI

@github-actions
Copy link

Run Gluten Clickhouse CI

@jinchengchenghh jinchengchenghh removed the pending velox rebase the patch will be merged after velox rebased with Meta main branch label Oct 30, 2023
@zhztheplayer
Copy link
Member

Two questions:

  1. What happens if unsupported codec type passed to Velox?
  2. Do spill files get compressed without this patch?

@zhztheplayer zhztheplayer changed the title [Velox] Add the spill compress config [VL] Add the spill compress config Oct 31, 2023
@zhztheplayer zhztheplayer changed the title [VL] Add the spill compress config [VL] Use value of spark.io.compression.codec as Velox's spill codec Oct 31, 2023
@zhztheplayer zhztheplayer merged commit 599d959 into apache:main Oct 31, 2023
18 checks passed
@GlutenPerfBot
Copy link
Contributor

===== Performance report for TPCH SF2000 with Velox backend, for reference only ====

query log/native_2500_time.csv log/native_master_10_30_2023_f4c419e1c_time.csv difference percentage
q1 33.19 34.74 1.548 104.66%
q2 25.08 25.08 0.002 100.01%
q3 38.19 40.40 2.204 105.77%
q4 37.74 37.88 0.140 100.37%
q5 71.34 70.91 -0.434 99.39%
q6 8.83 9.39 0.564 106.39%
q7 87.27 85.52 -1.745 98.00%
q8 88.20 83.86 -4.346 95.07%
q9 122.50 118.87 -3.624 97.04%
q10 55.85 53.35 -2.509 95.51%
q11 19.44 20.07 0.635 103.27%
q12 26.56 26.35 -0.220 99.17%
q13 48.56 49.81 1.243 102.56%
q14 17.68 19.49 1.806 110.21%
q15 34.86 35.98 1.117 103.20%
q16 16.31 16.33 0.028 100.17%
q17 101.59 100.70 -0.890 99.12%
q18 148.25 147.38 -0.869 99.41%
q19 17.20 16.78 -0.413 97.60%
q20 31.13 31.64 0.511 101.64%
q21 223.99 225.78 1.799 100.80%
q22 13.13 13.13 0.004 100.03%
total 1266.89 1263.44 -3.450 99.73%

@zhouyuan
Copy link
Contributor

zhouyuan commented Nov 1, 2023

Two questions:

  1. What happens if unsupported codec type passed to Velox?
  2. Do spill files get compressed without this patch?

for q2: it's not compressed w/o this patch AFAIK.

CC: @jackylee-ch

thanks, -yuan

@jinchengchenghh
Copy link
Contributor Author

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants