Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[GLUTEN-3459] Skip unnecessary local sort for static partition write #3460

Merged
merged 1 commit into from
Oct 23, 2023
Merged

[GLUTEN-3459] Skip unnecessary local sort for static partition write #3460

merged 1 commit into from
Oct 23, 2023

Conversation

lwz9103
Copy link
Contributor

@lwz9103 lwz9103 commented Oct 20, 2023

What changes were proposed in this pull request?

(Fixes: #3459)

@github-actions
Copy link

#3459

@github-actions
Copy link

Run Gluten Clickhouse CI

@zhouyuan
Copy link
Contributor

CC: @JkSelf

@lwz9103 lwz9103 changed the title [GLUTEN-3459] Avoid unnecessary sort in v1 write if it's not dynamic partition [GLUTEN-3459] Skip unnecessary local sort for static partition write Oct 20, 2023
@github-actions
Copy link

Run Gluten Clickhouse CI

JkSelf
JkSelf reviewed Oct 20, 2023
View reviewed changes
shims/spark33/src/main/scala/org/apache/spark/sql/execution/datasources/FileFormatWriter.scala
@@ -100,6 +100,32 @@ object FileFormatWriter extends Logging {
* @return
* The set of all partition paths that were updated during this write job.
*/

// scalastyle:off argcount
def write(
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@lwz9103 Can we also support in spark 32?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's a good suggestion, and it has been added. Thanks.

@github-actions
Copy link

Run Gluten Clickhouse CI

@binmahone binmahone mentioned this pull request Oct 23, 2023
Closed
JkSelf
JkSelf approved these changes Oct 23, 2023
View reviewed changes
Copy link
Contributor

@JkSelf JkSelf left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM.

@baibaichen baibaichen merged commit 19335ba into apache:main Oct 23, 2023
15 checks passed
@GlutenPerfBot
Copy link
Contributor

===== Performance report for TPCH SF2000 with Velox backend, for reference only ====

query log/native_3460_time.csv log/native_master_10_19_2023_a9324e620_time.csv difference percentage
q1 40.81 40.32 -0.493 98.79%
q2 24.62 24.36 -0.261 98.94%
q3 41.64 40.92 -0.726 98.26%
q4 37.55 33.60 -3.953 89.47%
q5 69.21 70.38 1.172 101.69%
q6 8.69 8.94 0.255 102.94%
q7 86.89 102.14 15.258 117.56%
q8 84.40 104.41 20.008 123.71%
q9 122.80 156.04 33.233 127.06%
q10 54.65 72.84 18.195 133.30%
q11 20.21 22.07 1.860 109.20%
q12 27.98 34.28 6.303 122.53%
q13 49.61 72.13 22.524 145.40%
q14 18.20 25.22 7.021 138.58%
q15 35.44 40.39 4.948 113.96%
q16 16.29 18.59 2.299 114.11%
q17 101.24 105.84 4.608 104.55%
q18 144.96 217.17 72.214 149.82%
q19 17.04 27.16 10.118 159.36%
q20 30.98 40.56 9.588 130.95%
q21 223.19 335.44 112.251 150.29%
q22 13.06 17.06 4.006 130.68%
total 1269.44 1609.87 340.429 126.82%

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@JkSelf JkSelf JkSelf approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Skip unnecessary local sort for static partition write. From [SPARK-37194]
5 participants
@lwz9103 @zhouyuan @GlutenPerfBot @JkSelf @baibaichen