Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[opt](nereids) refine expression estimation #40698

Merged
merged 2 commits into from
Sep 24, 2024
Merged

[opt](nereids) refine expression estimation #40698

merged 2 commits into from
Sep 24, 2024

Conversation

xzj7019
Copy link
Contributor

@xzj7019 xzj7019 commented Sep 11, 2024 •
edited
Loading

Stats deriving refinement step 2: refine expression estimation(part I)

a. refine casewhen/if/literal's avgDataSize/numNull/ndv info.
b. search column statistics cache at first during expression visiting.
c. fix StringType's width() returning -1.

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@xzj7019
Copy link
Contributor Author

xzj7019 commented Sep 11, 2024

run buildall

1 similar comment
@xzj7019
Copy link
Contributor Author

xzj7019 commented Sep 11, 2024

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 38717 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 1a6b92e3bcbb5083d5c7266c40d128796399405c, data reload: false

------ Round 1 ----------------------------------
q1	18004	4643	4344	4344
q2	2024	183	187	183
q3	10508	1109	1159	1109
q4	10212	759	747	747
q5	7771	2916	2877	2877
q6	222	137	135	135
q7	994	623	598	598
q8	9330	2129	2070	2070
q9	7147	6613	6620	6613
q10	7063	2221	2252	2221
q11	499	243	244	243
q12	522	224	221	221
q13	17776	3121	3055	3055
q14	288	242	237	237
q15	526	484	473	473
q16	558	444	437	437
q17	1008	700	776	700
q18	7760	7107	6998	6998
q19	1384	1112	1118	1112
q20	690	348	333	333
q21	4009	3167	3012	3012
q22	1133	999	1020	999
Total cold run time: 109428 ms
Total hot run time: 38717 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4367	4328	4351	4328
q2	389	275	272	272
q3	2920	2673	2711	2673
q4	1898	1677	1683	1677
q5	5438	5466	5482	5466
q6	227	131	127	127
q7	2168	1724	1769	1724
q8	3261	3426	3379	3379
q9	8572	8500	8540	8500
q10	3482	3196	3220	3196
q11	596	511	500	500
q12	773	604	630	604
q13	9820	3115	3099	3099
q14	311	273	276	273
q15	537	482	483	482
q16	543	503	477	477
q17	1812	1482	1498	1482
q18	7897	7680	7721	7680
q19	1695	1545	1472	1472
q20	2055	1812	1829	1812
q21	5567	5220	5253	5220
q22	1136	1043	1030	1030
Total cold run time: 65464 ms
Total hot run time: 55473 ms

@xzj7019
Copy link
Contributor Author

xzj7019 commented Sep 11, 2024

run buildall

1 similar comment
@xzj7019
Copy link
Contributor Author

xzj7019 commented Sep 11, 2024

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 39038 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit e9b64c4f793e069b5754eae1e7d9d3ed00201c73, data reload: false

------ Round 1 ----------------------------------
q1	17618	4489	4349	4349
q2	2023	188	185	185
q3	10960	1207	1125	1125
q4	10514	845	834	834
q5	8068	2927	2822	2822
q6	233	139	140	139
q7	984	631	626	626
q8	9491	2129	2087	2087
q9	7413	6695	6637	6637
q10	7088	2262	2277	2262
q11	462	243	246	243
q12	421	231	230	230
q13	18881	3097	3109	3097
q14	283	236	248	236
q15	549	480	478	478
q16	558	429	430	429
q17	989	742	751	742
q18	7399	7079	7173	7079
q19	1395	1029	1090	1029
q20	687	340	331	331
q21	3907	3113	3074	3074
q22	1104	1004	1019	1004
Total cold run time: 111027 ms
Total hot run time: 39038 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4431	4330	4293	4293
q2	387	281	260	260
q3	2957	2746	2703	2703
q4	1943	1683	1668	1668
q5	5414	5457	5471	5457
q6	222	129	130	129
q7	2109	1778	1795	1778
q8	3227	3359	3352	3352
q9	8484	8526	8506	8506
q10	3461	3241	3188	3188
q11	610	495	506	495
q12	791	596	589	589
q13	12915	3118	3089	3089
q14	299	283	281	281
q15	525	480	485	480
q16	545	461	473	461
q17	1796	1499	1477	1477
q18	8059	7780	7613	7613
q19	1676	1463	1579	1463
q20	2031	1852	1819	1819
q21	5451	5107	5161	5107
q22	1125	1053	1021	1021
Total cold run time: 68458 ms
Total hot run time: 55229 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 192387 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit e9b64c4f793e069b5754eae1e7d9d3ed00201c73, data reload: false

query1	909	383	388	383
query2	6499	1960	1927	1927
query3	6655	204	219	204
query4	30144	23314	23225	23225
query5	4162	513	495	495
query6	267	182	176	176
query7	4579	292	294	292
query8	287	233	238	233
query9	8491	2515	2504	2504
query10	425	283	276	276
query11	17559	15021	15132	15021
query12	154	102	102	102
query13	1644	391	365	365
query14	10082	7116	7198	7116
query15	255	169	170	169
query16	8136	448	449	448
query17	1624	553	543	543
query18	2134	290	287	287
query19	330	150	155	150
query20	121	110	113	110
query21	215	108	106	106
query22	4496	4153	4253	4153
query23	34490	33680	33600	33600
query24	11063	2874	2792	2792
query25	617	384	390	384
query26	1163	151	160	151
query27	2749	275	277	275
query28	7599	2026	2030	2026
query29	810	400	405	400
query30	286	162	150	150
query31	971	746	817	746
query32	99	52	63	52
query33	758	298	293	293
query34	962	471	469	469
query35	864	735	733	733
query36	1080	936	939	936
query37	166	86	85	85
query38	3945	3937	3930	3930
query39	1449	1399	1411	1399
query40	200	114	110	110
query41	48	49	45	45
query42	108	95	95	95
query43	505	496	472	472
query44	1274	756	748	748
query45	196	167	168	167
query46	1089	725	747	725
query47	1922	1783	1822	1783
query48	372	288	293	288
query49	1045	428	436	428
query50	800	415	410	410
query51	7026	6916	6738	6738
query52	102	85	86	85
query53	255	185	187	185
query54	1034	440	452	440
query55	75	75	78	75
query56	267	258	251	251
query57	1221	1090	1066	1066
query58	249	228	252	228
query59	2971	2855	2860	2855
query60	292	263	268	263
query61	103	98	102	98
query62	836	651	659	651
query63	219	189	185	185
query64	5306	706	643	643
query65	3237	3181	3191	3181
query66	1457	331	329	329
query67	15928	15412	15319	15319
query68	3112	560	547	547
query69	395	256	254	254
query70	1200	1133	1107	1107
query71	328	271	267	267
query72	6143	4046	3959	3959
query73	730	321	330	321
query74	9273	8838	8821	8821
query75	3409	2684	2689	2684
query76	1921	1033	1008	1008
query77	483	315	319	315
query78	9926	9362	9300	9300
query79	936	882	877	877
query80	894	805	817	805
query81	456	265	265	265
query82	282	267	271	267
query83	193	192	192	192
query84	230	112	106	106
query85	692	395	379	379
query86	325	316	321	316
query87	4272	4357	4396	4357
query88	4598	4044	4044	4044
query89	383	366	362	362
query90	1790	309	315	309
query91	120	122	124	122
query92	76	72	73	72
query93	909	923	923	923
query94	667	361	364	361
query95	420	407	408	407
query96	477	473	470	470
query97	3084	3129	3095	3095
query98	237	237	219	219
query99	1399	1287	1287	1287
Total cold run time: 285736 ms
Total hot run time: 192387 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 31.89 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit e9b64c4f793e069b5754eae1e7d9d3ed00201c73, data reload: false

query1	0.04	0.04	0.04
query2	0.08	0.04	0.04
query3	0.22	0.05	0.05
query4	1.67	0.08	0.08
query5	0.51	0.48	0.50
query6	1.13	0.74	0.73
query7	0.02	0.02	0.01
query8	0.05	0.04	0.05
query9	0.54	0.50	0.48
query10	0.54	0.56	0.55
query11	0.16	0.12	0.11
query12	0.14	0.12	0.12
query13	0.60	0.59	0.59
query14	1.39	1.39	1.43
query15	0.85	0.82	0.82
query16	0.38	0.37	0.38
query17	1.05	1.06	1.00
query18	0.20	0.19	0.20
query19	1.86	1.88	1.79
query20	0.01	0.01	0.01
query21	15.40	0.66	0.65
query22	4.15	6.90	2.36
query23	18.33	1.36	1.31
query24	2.19	0.21	0.21
query25	0.17	0.08	0.09
query26	0.25	0.19	0.18
query27	0.07	0.07	0.08
query28	13.22	1.03	1.00
query29	12.60	3.31	3.31
query30	0.24	0.06	0.05
query31	2.89	0.40	0.40
query32	3.23	0.48	0.47
query33	2.96	3.06	3.01
query34	17.00	4.44	4.40
query35	4.47	4.43	4.43
query36	0.66	0.47	0.49
query37	0.18	0.16	0.15
query38	0.16	0.15	0.15
query39	0.04	0.04	0.04
query40	0.16	0.13	0.13
query41	0.09	0.05	0.04
query42	0.06	0.05	0.05
query43	0.05	0.05	0.04
Total cold run time: 110.01 s
Total hot run time: 31.89 s

@xzj7019 xzj7019 marked this pull request as draft September 13, 2024 09:57
@xzj7019
Copy link
Contributor Author

xzj7019 commented Sep 19, 2024

run buildall

1 similar comment
@xzj7019
Copy link
Contributor Author

xzj7019 commented Sep 19, 2024

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 41568 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 18f227d0506cbe8f6a81821ea0f7a9f39f07126d, data reload: false

------ Round 1 ----------------------------------
q1	14922	7359	7315	7315
q2	1323	166	161	161
q3	4357	1214	1305	1214
q4	4328	781	695	695
q5	3107	3102	3086	3086
q6	235	149	152	149
q7	1008	615	612	612
q8	3375	2020	2038	2020
q9	6469	6399	6420	6399
q10	2675	2295	2316	2295
q11	442	250	248	248
q12	404	223	221	221
q13	17816	2958	2984	2958
q14	239	226	213	213
q15	573	516	528	516
q16	672	628	615	615
q17	991	841	819	819
q18	7581	6633	6812	6633
q19	1352	1113	988	988
q20	596	289	284	284
q21	3953	3195	3122	3122
q22	1113	1005	1018	1005
Total cold run time: 77531 ms
Total hot run time: 41568 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7296	7233	7240	7233
q2	319	231	237	231
q3	2939	2801	2764	2764
q4	1942	1695	1745	1695
q5	5444	5462	5426	5426
q6	227	138	140	138
q7	2062	1724	1689	1689
q8	3157	3322	3348	3322
q9	8439	8480	8464	8464
q10	3386	3334	3366	3334
q11	579	476	467	467
q12	761	547	590	547
q13	10574	3011	3006	3006
q14	299	253	259	253
q15	568	505	515	505
q16	740	670	699	670
q17	1769	1556	1549	1549
q18	7785	7433	7455	7433
q19	1695	1563	1616	1563
q20	2060	1792	1801	1792
q21	5440	5238	5274	5238
q22	1134	1003	1022	1003
Total cold run time: 68615 ms
Total hot run time: 58322 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 195653 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 18f227d0506cbe8f6a81821ea0f7a9f39f07126d, data reload: false

query1	982	377	367	367
query2	6528	2135	2080	2080
query3	6706	210	224	210
query4	34455	23431	23815	23431
query5	5490	470	465	465
query6	262	177	161	161
query7	4617	292	296	292
query8	269	225	227	225
query9	9353	2743	2721	2721
query10	503	276	277	276
query11	18331	15321	15198	15198
query12	152	99	98	98
query13	1620	414	397	397
query14	10491	7507	7778	7507
query15	276	177	182	177
query16	7643	466	496	466
query17	1637	583	559	559
query18	1394	313	306	306
query19	360	146	150	146
query20	119	110	112	110
query21	211	108	104	104
query22	4606	4336	4353	4336
query23	34680	33965	33985	33965
query24	11173	2893	2929	2893
query25	635	406	399	399
query26	1129	164	157	157
query27	2326	289	300	289
query28	8106	2477	2468	2468
query29	739	428	427	427
query30	322	167	154	154
query31	995	774	793	774
query32	100	57	62	57
query33	771	295	298	295
query34	946	494	488	488
query35	856	746	724	724
query36	1094	916	943	916
query37	154	95	92	92
query38	4062	3892	3931	3892
query39	1446	1400	1414	1400
query40	214	102	98	98
query41	52	52	49	49
query42	113	97	97	97
query43	527	482	504	482
query44	1291	853	802	802
query45	194	167	169	167
query46	1154	770	771	770
query47	1896	1759	1797	1759
query48	465	364	369	364
query49	1138	421	422	421
query50	825	403	412	403
query51	7107	6907	6929	6907
query52	98	86	87	86
query53	257	182	189	182
query54	1160	462	477	462
query55	82	76	83	76
query56	296	279	268	268
query57	1189	1083	1045	1045
query58	244	238	238	238
query59	3394	3157	3092	3092
query60	297	282	277	277
query61	149	104	101	101
query62	848	662	661	661
query63	225	193	191	191
query64	4212	657	627	627
query65	3261	3217	3196	3196
query66	946	297	294	294
query67	16017	15492	15532	15492
query68	3113	864	860	860
query69	450	343	340	340
query70	1201	1180	1202	1180
query71	332	340	322	322
query72	5974	3487	3371	3371
query73	588	577	585	577
query74	9292	9011	8982	8982
query75	3080	2919	2853	2853
query76	1921	885	873	873
query77	408	387	363	363
query78	10609	9247	9286	9247
query79	910	902	897	897
query80	589	577	572	572
query81	465	258	253	253
query82	230	234	237	234
query83	163	166	158	158
query84	239	109	106	106
query85	672	384	358	358
query86	306	318	323	318
query87	4424	4392	4346	4346
query88	4767	4092	4081	4081
query89	375	368	366	366
query90	1286	317	322	317
query91	173	165	167	165
query92	77	74	73	73
query93	921	918	913	913
query94	557	370	382	370
query95	430	423	411	411
query96	485	497	491	491
query97	3110	3111	3156	3111
query98	226	223	216	216
query99	1442	1301	1300	1300
Total cold run time: 292005 ms
Total hot run time: 195653 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 32.56 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 18f227d0506cbe8f6a81821ea0f7a9f39f07126d, data reload: false

query1	0.04	0.04	0.04
query2	0.06	0.03	0.03
query3	0.23	0.07	0.06
query4	1.65	0.10	0.10
query5	0.50	0.49	0.49
query6	1.14	0.72	0.72
query7	0.02	0.02	0.02
query8	0.04	0.03	0.03
query9	0.56	0.50	0.49
query10	0.56	0.58	0.55
query11	0.15	0.11	0.10
query12	0.15	0.11	0.11
query13	0.61	0.59	0.59
query14	3.00	2.98	3.01
query15	0.87	0.82	0.82
query16	0.38	0.38	0.39
query17	0.98	1.01	1.05
query18	0.23	0.21	0.21
query19	1.89	1.94	1.96
query20	0.00	0.01	0.01
query21	15.35	0.59	0.59
query22	3.72	2.99	1.64
query23	17.41	0.82	0.70
query24	2.58	1.29	1.00
query25	0.30	0.09	0.11
query26	0.43	0.13	0.14
query27	0.03	0.04	0.04
query28	10.73	1.10	1.07
query29	12.56	3.29	3.26
query30	0.25	0.06	0.06
query31	2.88	0.38	0.37
query32	3.29	0.46	0.45
query33	3.01	2.98	3.06
query34	16.88	4.32	4.45
query35	4.44	4.42	4.40
query36	0.68	0.50	0.49
query37	0.08	0.06	0.05
query38	0.04	0.03	0.04
query39	0.04	0.02	0.02
query40	0.17	0.12	0.13
query41	0.08	0.03	0.02
query42	0.03	0.03	0.02
query43	0.03	0.03	0.03
Total cold run time: 108.07 s
Total hot run time: 32.56 s

@xzj7019 xzj7019 force-pushed the refine_stats_derive2 branch 2 times, most recently from 0c0bc72 to de5d517 Compare September 19, 2024 13:37
@xzj7019
Copy link
Contributor Author

xzj7019 commented Sep 19, 2024

run buildall

@xzj7019 xzj7019 force-pushed the refine_stats_derive2 branch 2 times, most recently from f05e9f1 to 9b94470 Compare September 20, 2024 04:06
@xzj7019
Copy link
Contributor Author

xzj7019 commented Sep 20, 2024

run buildall

@xzj7019
Copy link
Contributor Author

xzj7019 commented Sep 20, 2024

run buildall

@xzj7019 xzj7019 marked this pull request as ready for review September 20, 2024 06:37
@xzj7019
Copy link
Contributor Author

xzj7019 commented Sep 20, 2024

run buildall

@xzj7019
Copy link
Contributor Author

xzj7019 commented Sep 23, 2024

run buildall

@xzj7019
Copy link
Contributor Author

xzj7019 commented Sep 23, 2024

run buildall

@xzj7019
Copy link
Contributor Author

xzj7019 commented Sep 23, 2024

run p0

@xzj7019
Copy link
Contributor Author

xzj7019 commented Sep 24, 2024

run buildall

@xzj7019
Copy link
Contributor Author

xzj7019 commented Sep 24, 2024

run cloud_p0

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Sep 24, 2024
@github-actions GitHub Actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions GitHub Actions
Copy link
Contributor

PR approved by anyone and no changes requested.

@morrySnow morrySnow merged commit 7a7bc6a into apache:master Sep 24, 2024
25 of 28 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@englefly englefly englefly approved these changes

@morrySnow morrySnow morrySnow approved these changes

Assignees
No one assigned
Labels
approved Indicates a PR has been approved by one committer. dev/3.0.x reviewed
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@xzj7019 @doris-robot @englefly @morrySnow