Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[bugfix](hive)Remove redundant split operations #40683

Merged
merged 1 commit into from
Sep 18, 2024

Conversation

wuwenchi
Copy link
Contributor

@wuwenchi wuwenchi commented Sep 11, 2024

Proposed changes

In explain, we don't need to get all the splits.

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@wuwenchi
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 38310 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit f5bd6780014e2a5e7a17dc0fdcde54d9716fe588, data reload: false

------ Round 1 ----------------------------------
q1	8239	4600	4313	4313
q2	1109	189	182	182
q3	3677	1093	1166	1093
q4	4326	833	727	727
q5	3428	2853	3043	2853
q6	236	140	138	138
q7	986	619	619	619
q8	4060	2086	2091	2086
q9	6775	7047	6606	6606
q10	2584	2254	2199	2199
q11	408	244	247	244
q12	395	223	219	219
q13	18197	3115	3093	3093
q14	291	240	243	240
q15	535	495	490	490
q16	517	441	409	409
q17	985	692	733	692
q18	7292	6904	6869	6869
q19	1237	1032	1010	1010
q20	606	343	331	331
q21	3798	2949	2890	2890
q22	1134	1007	1033	1007
Total cold run time: 70815 ms
Total hot run time: 38310 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4417	4325	4356	4325
q2	386	276	280	276
q3	2940	2714	2684	2684
q4	1947	1683	1674	1674
q5	5482	5416	5433	5416
q6	227	132	131	131
q7	2062	1733	1783	1733
q8	3232	3390	3356	3356
q9	8578	8504	8527	8504
q10	3464	3247	3213	3213
q11	599	496	505	496
q12	790	593	594	593
q13	4823	3140	3131	3131
q14	306	280	282	280
q15	529	480	488	480
q16	528	482	509	482
q17	1789	1510	1507	1507
q18	7756	7665	7574	7574
q19	1703	1571	1460	1460
q20	2069	1844	1833	1833
q21	5335	5349	5383	5349
q22	1127	1046	1023	1023
Total cold run time: 60089 ms
Total hot run time: 55520 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 193145 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit f5bd6780014e2a5e7a17dc0fdcde54d9716fe588, data reload: false

query1	922	380	382	380
query2	6506	1920	1902	1902
query3	6644	211	221	211
query4	34253	23412	23285	23285
query5	4132	514	509	509
query6	252	182	168	168
query7	4586	304	296	296
query8	300	228	220	220
query9	8489	2489	2489	2489
query10	440	279	288	279
query11	17456	15115	15114	15114
query12	154	107	97	97
query13	1647	378	384	378
query14	10185	6879	6835	6835
query15	284	178	173	173
query16	8103	461	447	447
query17	1568	561	553	553
query18	2147	285	279	279
query19	330	144	144	144
query20	132	110	111	110
query21	209	101	103	101
query22	4646	4063	4171	4063
query23	34225	33528	33453	33453
query24	11185	2909	2781	2781
query25	627	371	386	371
query26	1168	155	151	151
query27	2310	289	305	289
query28	7255	2063	2046	2046
query29	831	403	401	401
query30	306	158	151	151
query31	1014	769	805	769
query32	97	58	57	57
query33	752	294	278	278
query34	984	480	468	468
query35	872	718	727	718
query36	1088	923	912	912
query37	164	99	88	88
query38	4049	3936	3921	3921
query39	1442	1429	1415	1415
query40	206	121	118	118
query41	49	48	46	46
query42	118	98	100	98
query43	499	465	480	465
query44	1191	761	747	747
query45	201	170	198	170
query46	1095	701	728	701
query47	1856	1788	1837	1788
query48	376	303	288	288
query49	1086	442	440	440
query50	835	413	408	408
query51	7184	6989	6916	6916
query52	105	88	88	88
query53	256	184	190	184
query54	983	454	457	454
query55	75	78	79	78
query56	278	266	251	251
query57	1213	1065	1049	1049
query58	243	259	229	229
query59	2975	2788	2804	2788
query60	306	269	271	269
query61	136	103	102	102
query62	846	652	658	652
query63	233	198	192	192
query64	5220	674	657	657
query65	3211	3156	3194	3156
query66	1454	338	337	337
query67	16000	15340	15411	15340
query68	3109	857	845	845
query69	418	323	329	323
query70	1172	1111	1186	1111
query71	362	342	338	338
query72	6076	3646	3533	3533
query73	584	575	579	575
query74	8953	8794	8947	8794
query75	3116	2982	2926	2926
query76	1875	861	847	847
query77	482	404	398	398
query78	9429	10009	9904	9904
query79	920	896	850	850
query80	826	802	794	794
query81	448	260	267	260
query82	265	267	261	261
query83	193	193	199	193
query84	233	111	106	106
query85	666	448	443	443
query86	309	323	304	304
query87	4367	4275	4389	4275
query88	4288	4069	4054	4054
query89	378	368	365	365
query90	1336	328	327	327
query91	137	136	139	136
query92	81	80	77	77
query93	915	918	915	915
query94	458	388	363	363
query95	457	427	431	427
query96	473	471	473	471
query97	3153	3168	3124	3124
query98	235	229	235	229
query99	1414	1264	1268	1264
Total cold run time: 287112 ms
Total hot run time: 193145 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 31.38 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit f5bd6780014e2a5e7a17dc0fdcde54d9716fe588, data reload: false

query1	0.05	0.04	0.04
query2	0.07	0.04	0.04
query3	0.22	0.05	0.06
query4	1.65	0.09	0.08
query5	0.52	0.50	0.51
query6	1.12	0.73	0.73
query7	0.02	0.01	0.01
query8	0.05	0.04	0.04
query9	0.54	0.50	0.49
query10	0.55	0.56	0.54
query11	0.17	0.12	0.11
query12	0.15	0.13	0.12
query13	0.62	0.59	0.59
query14	1.40	1.44	1.45
query15	0.84	0.83	0.81
query16	0.38	0.38	0.37
query17	1.02	1.07	1.07
query18	0.17	0.17	0.18
query19	1.85	1.79	1.80
query20	0.01	0.01	0.02
query21	15.39	0.66	0.66
query22	4.71	7.16	1.74
query23	18.27	1.43	1.26
query24	2.04	0.24	0.23
query25	0.15	0.08	0.08
query26	0.26	0.18	0.18
query27	0.08	0.07	0.08
query28	13.20	1.02	0.99
query29	12.57	3.33	3.32
query30	0.24	0.05	0.05
query31	2.87	0.41	0.39
query32	3.25	0.48	0.47
query33	2.93	3.04	3.05
query34	17.00	4.38	4.42
query35	4.49	4.48	4.44
query36	0.66	0.48	0.49
query37	0.18	0.15	0.16
query38	0.15	0.15	0.16
query39	0.05	0.04	0.04
query40	0.15	0.13	0.13
query41	0.09	0.06	0.04
query42	0.06	0.05	0.05
query43	0.04	0.04	0.04
Total cold run time: 110.23 s
Total hot run time: 31.38 s

Copy link
Contributor

@morningman morningman left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Sep 11, 2024
Copy link
Contributor

PR approved by at least one committer and no changes requested.

Copy link
Contributor

PR approved by anyone and no changes requested.

@morningman morningman merged commit b39b24a into apache:master Sep 18, 2024
27 of 30 checks passed
wuwenchi added a commit to wuwenchi/doris_new that referenced this pull request Sep 20, 2024
## Proposed changes

In explain, we don't need to get all the splits.
morningman pushed a commit that referenced this pull request Sep 20, 2024
wuwenchi added a commit to wuwenchi/doris_new that referenced this pull request Sep 27, 2024
## Proposed changes

In explain, we don't need to get all the splits.
gavinchou pushed a commit that referenced this pull request Sep 28, 2024
## Proposed changes

In explain, we don't need to get all the splits.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by one committer. dev/2.1.7-merged dev/3.0.2-merged p0_b reviewed
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants