Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[chore](dependencies)upgrade hive catalog to 2.0.0 #33943

Merged
merged 1 commit into from
Apr 22, 2024
Merged

[chore](dependencies)upgrade hive catalog to 2.0.0 #33943

merged 1 commit into from
Apr 22, 2024

Conversation

CalvinKirs
Copy link
Member

@CalvinKirs CalvinKirs commented Apr 22, 2024 •
edited
Loading

  • Version 2.0.0 of the shade package eliminates potential jar conflicts, resolves dependency component issues, and significantly reduces package size.

  • Utilize the directly-dependent guava library instead of relying on transitively included libraries.

  • Adapt paimon 0.7.0

  • 2.0 branch change: [chore](dependencies)upgrade hive catalog to 2.0.0 #33424

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@CalvinKirs
Copy link
Member Author

run buildall

@CalvinKirs
Adapt paimon 0.6.0
93e9b95 
Version 2.0.0 of the shade package eliminates potential jar conflicts, resolves dependency component issues, and significantly reduces package size.
Utilize the directly-dependent guava library instead of relying on transitively included libraries.
@CalvinKirs
Copy link
Member Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 38628 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 93e9b95990ffe160ed18269ea806af7f547b3926, data reload: false

------ Round 1 ----------------------------------
q1	17623	4933	4221	4221
q2	2012	181	181	181
q3	10469	1119	1190	1119
q4	10187	780	871	780
q5	7518	2779	2678	2678
q6	220	132	135	132
q7	1013	604	577	577
q8	9215	2076	2055	2055
q9	7552	6592	6558	6558
q10	8614	3549	3562	3549
q11	487	234	229	229
q12	429	222	215	215
q13	17767	2921	2945	2921
q14	266	225	235	225
q15	525	478	488	478
q16	547	382	374	374
q17	982	720	750	720
q18	7465	6761	6734	6734
q19	7260	1554	1537	1537
q20	665	309	306	306
q21	3443	2738	2986	2738
q22	368	303	301	301
Total cold run time: 114627 ms
Total hot run time: 38628 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4402	4236	4259	4236
q2	383	276	267	267
q3	2935	2787	2748	2748
q4	1886	1589	1587	1587
q5	5398	5361	5335	5335
q6	216	123	128	123
q7	2275	1897	1880	1880
q8	3231	3692	3375	3375
q9	8691	8591	8795	8591
q10	4119	3955	3933	3933
q11	605	507	499	499
q12	815	637	653	637
q13	15880	3196	3167	3167
q14	335	297	294	294
q15	512	496	497	496
q16	499	451	436	436
q17	1858	1551	1499	1499
q18	7993	7907	8165	7907
q19	1694	1557	1586	1557
q20	1988	1853	1881	1853
q21	10155	4915	4961	4915
q22	554	469	468	468
Total cold run time: 76424 ms
Total hot run time: 55803 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 184768 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 93e9b95990ffe160ed18269ea806af7f547b3926, data reload: false

query1	883	374	355	355
query2	6440	3338	2464	2464
query3	6669	203	206	203
query4	22810	21301	21399	21301
query5	4131	411	418	411
query6	281	184	199	184
query7	4592	295	284	284
query8	246	180	175	175
query9	8575	2305	2296	2296
query10	414	236	242	236
query11	14774	14148	14170	14148
query12	135	91	85	85
query13	1627	359	353	353
query14	9297	7185	7641	7185
query15	225	190	188	188
query16	7854	268	264	264
query17	1894	581	571	571
query18	1944	278	279	278
query19	208	159	157	157
query20	96	84	85	84
query21	202	127	125	125
query22	5099	4828	4792	4792
query23	33721	33148	33263	33148
query24	7100	3016	3092	3016
query25	486	373	398	373
query26	689	160	156	156
query27	2162	356	382	356
query28	3976	2074	2062	2062
query29	866	623	632	623
query30	264	184	190	184
query31	983	754	757	754
query32	73	52	53	52
query33	475	245	241	241
query34	881	484	500	484
query35	820	723	733	723
query36	1027	920	919	919
query37	108	82	75	75
query38	3521	3395	3363	3363
query39	1075	1033	1051	1033
query40	176	129	126	126
query41	45	42	45	42
query42	101	95	97	95
query43	604	547	537	537
query44	1112	751	776	751
query45	300	274	262	262
query46	1100	741	752	741
query47	2012	1935	1920	1920
query48	373	303	303	303
query49	742	383	407	383
query50	784	394	400	394
query51	6967	6924	6856	6856
query52	112	87	96	87
query53	350	284	274	274
query54	261	258	224	224
query55	74	74	75	74
query56	260	232	230	230
query57	1283	1208	1183	1183
query58	223	211	202	202
query59	3599	3190	3272	3190
query60	248	228	231	228
query61	90	85	88	85
query62	549	435	439	435
query63	301	279	274	274
query64	4159	3903	4026	3903
query65	3037	3043	3030	3030
query66	741	326	331	326
query67	15574	15288	14900	14900
query68	5319	536	555	536
query69	465	301	307	301
query70	1244	1145	1180	1145
query71	1377	1268	1265	1265
query72	6598	2648	2424	2424
query73	720	320	320	320
query74	6804	6505	6388	6388
query75	3306	2629	2588	2588
query76	2629	970	1013	970
query77	383	265	260	260
query78	10872	10245	10279	10245
query79	8089	517	520	517
query80	1843	443	426	426
query81	511	259	240	240
query82	1466	95	94	94
query83	257	158	161	158
query84	267	80	82	80
query85	1641	264	256	256
query86	538	290	285	285
query87	3507	3280	3303	3280
query88	5343	2326	2347	2326
query89	491	362	367	362
query90	1944	184	183	183
query91	126	96	96	96
query92	61	50	48	48
query93	6672	512	503	503
query94	1083	178	227	178
query95	383	299	291	291
query96	619	268	257	257
query97	3149	2938	2950	2938
query98	234	221	221	221
query99	1181	831	865	831
Total cold run time: 280922 ms
Total hot run time: 184768 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.61 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 93e9b95990ffe160ed18269ea806af7f547b3926, data reload: false

query1	0.04	0.03	0.03
query2	0.08	0.04	0.04
query3	0.22	0.05	0.05
query4	1.68	0.06	0.07
query5	0.50	0.50	0.52
query6	1.47	0.73	0.73
query7	0.02	0.02	0.01
query8	0.05	0.04	0.04
query9	0.57	0.50	0.49
query10	0.55	0.55	0.54
query11	0.16	0.12	0.12
query12	0.15	0.12	0.11
query13	0.61	0.59	0.59
query14	0.75	0.78	0.77
query15	0.81	0.81	0.81
query16	0.36	0.36	0.37
query17	0.94	0.98	0.96
query18	0.20	0.27	0.21
query19	1.73	1.78	1.68
query20	0.01	0.01	0.01
query21	15.40	0.65	0.65
query22	4.27	7.11	2.21
query23	18.31	1.37	1.29
query24	1.94	0.23	0.20
query25	0.14	0.09	0.08
query26	0.26	0.17	0.16
query27	0.08	0.09	0.09
query28	13.36	1.01	0.97
query29	12.58	3.28	3.24
query30	0.25	0.08	0.06
query31	2.84	0.38	0.38
query32	3.28	0.46	0.46
query33	2.79	2.83	2.83
query34	17.23	4.40	4.42
query35	4.54	4.50	4.44
query36	0.65	0.48	0.45
query37	0.18	0.15	0.16
query38	0.15	0.15	0.14
query39	0.05	0.03	0.04
query40	0.17	0.15	0.15
query41	0.09	0.05	0.05
query42	0.05	0.05	0.04
query43	0.04	0.03	0.04
Total cold run time: 109.55 s
Total hot run time: 30.61 s

morningman
morningman approved these changes Apr 22, 2024
View reviewed changes
Copy link
Contributor

@morningman morningman left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Apr 22, 2024
@github-actions GitHub Actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions GitHub Actions
Copy link
Contributor

PR approved by anyone and no changes requested.

@CalvinKirs CalvinKirs merged commit fe1ca68 into apache:master Apr 22, 2024
26 of 29 checks passed
@CalvinKirs CalvinKirs deleted the job-info-storage branch April 22, 2024 10:36
morningman pushed a commit to morningman/doris that referenced this pull request Apr 24, 2024
@CalvinKirs @morningman
Adapt paimon 0.6.0 (apache#33943)
d6a55c0 
Version 2.0.0 of the shade package eliminates potential jar conflicts, resolves dependency component issues, and significantly reduces package size.
Utilize the directly-dependent guava library instead of relying on transitively included libraries.
@morningman morningman mentioned this pull request Apr 24, 2024
Merged
yiguolei pushed a commit that referenced this pull request Apr 24, 2024
@morningman @CalvinKirs
[upgrade](hive-shade)(paimon) upgrade hive shade to 2.0.0 and paimon …
a18576b 
…to 0.7 (#34085)

* Adapt paimon 0.6.0 (#33943)

Version 2.0.0 of the shade package eliminates potential jar conflicts, resolves dependency component issues, and significantly reduces package size.
Utilize the directly-dependent guava library instead of relying on transitively included libraries.

* [chore](dependencies)Upgrade paimon to 0.7.0 (#33987)

---------

Co-authored-by: Calvin Kirs <[email protected]>
yiguolei pushed a commit that referenced this pull request Apr 25, 2024
@morningman @CalvinKirs @Doris-Extras
[upgrade](hive-shade)(paimon) upgrade hive shade to 2.0.0 and paimon …
fbd2c9d 
…to 0.7 (#34085)

* Adapt paimon 0.6.0 (#33943)

Version 2.0.0 of the shade package eliminates potential jar conflicts, resolves dependency component issues, and significantly reduces package size.
Utilize the directly-dependent guava library instead of relying on transitively included libraries.

* [chore](dependencies)Upgrade paimon to 0.7.0 (#33987)

---------

Co-authored-by: Calvin Kirs <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@morningman morningman morningman approved these changes

@JNSimba JNSimba JNSimba approved these changes

Assignees
No one assigned
Labels
approved Indicates a PR has been approved by one committer. dev/2.1.3-merged reviewed
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@CalvinKirs @doris-robot @morningman @JNSimba @yiguolei