Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[opt](nereids)stats derive for min()/max() agg function #40126

Merged
merged 2 commits into from
Sep 6, 2024

Conversation

englefly
Copy link
Contributor

@englefly englefly commented Aug 29, 2024

Proposed changes

the column stats for min(A) agg function is estimated as column stats of A.
in current version, min(A).ndv is 1, this is error-prone.

Issue Number: close #xxx

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@englefly
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 37865 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit f15d57ca8202229519345f4911fe24181a45ba47, data reload: false

------ Round 1 ----------------------------------
q1	18736	4792	4338	4338
q2	2013	182	172	172
q3	10485	1116	1093	1093
q4	10147	833	724	724
q5	7728	2916	2756	2756
q6	221	139	138	138
q7	936	619	599	599
q8	9329	2058	2025	2025
q9	7259	6514	6503	6503
q10	6998	2254	2215	2215
q11	444	245	245	245
q12	398	237	232	232
q13	17756	3029	3046	3029
q14	288	234	245	234
q15	523	485	491	485
q16	581	498	509	498
q17	976	659	640	640
q18	7485	6727	6850	6727
q19	1394	936	1028	936
q20	686	340	350	340
q21	3932	2961	2939	2939
q22	1135	1036	997	997
Total cold run time: 109450 ms
Total hot run time: 37865 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4327	4288	4264	4264
q2	378	284	283	283
q3	2912	2704	2674	2674
q4	1972	1715	1646	1646
q5	5436	5393	5378	5378
q6	217	129	129	129
q7	2121	1802	1731	1731
q8	3188	3351	3352	3351
q9	8454	8469	8436	8436
q10	3411	3181	3198	3181
q11	604	502	496	496
q12	793	625	638	625
q13	14564	3049	3081	3049
q14	323	279	275	275
q15	530	482	477	477
q16	631	564	570	564
q17	1779	1510	1494	1494
q18	7865	7565	7637	7565
q19	1672	1594	1502	1502
q20	2073	1819	1835	1819
q21	5557	5425	5086	5086
q22	1139	1016	1049	1016
Total cold run time: 69946 ms
Total hot run time: 55041 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 187945 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit f15d57ca8202229519345f4911fe24181a45ba47, data reload: false

query1	902	381	366	366
query2	6465	2067	1936	1936
query3	6640	224	221	221
query4	33704	23228	23142	23142
query5	4178	515	493	493
query6	247	165	162	162
query7	4589	305	295	295
query8	260	210	219	210
query9	8754	2480	2476	2476
query10	439	267	265	265
query11	16217	15315	15279	15279
query12	154	104	101	101
query13	1634	403	389	389
query14	10085	7553	6366	6366
query15	309	176	183	176
query16	7515	444	439	439
query17	1671	576	570	570
query18	2007	301	293	293
query19	338	153	148	148
query20	118	112	115	112
query21	205	110	107	107
query22	4425	4167	4125	4125
query23	34211	33658	33180	33180
query24	11249	2906	2852	2852
query25	656	405	400	400
query26	986	165	164	164
query27	2685	296	286	286
query28	7494	2138	2113	2113
query29	800	438	436	436
query30	302	162	153	153
query31	978	784	787	784
query32	99	60	60	60
query33	776	305	299	299
query34	934	486	498	486
query35	865	750	730	730
query36	1064	941	947	941
query37	158	100	96	96
query38	3990	3994	3787	3787
query39	1462	1385	1388	1385
query40	207	123	121	121
query41	51	52	48	48
query42	118	96	98	96
query43	503	462	474	462
query44	1237	763	761	761
query45	204	170	170	170
query46	1106	751	769	751
query47	1889	1804	1811	1804
query48	387	304	300	300
query49	1024	429	439	429
query50	821	409	424	409
query51	7159	7116	7140	7116
query52	101	89	87	87
query53	255	192	183	183
query54	1009	467	464	464
query55	80	80	88	80
query56	288	266	274	266
query57	1204	1085	1070	1070
query58	249	229	233	229
query59	3157	2915	2787	2787
query60	307	282	279	279
query61	127	121	125	121
query62	861	667	681	667
query63	231	189	186	186
query64	5415	696	707	696
query65	3231	3139	3127	3127
query66	1426	340	341	340
query67	15588	15194	15247	15194
query68	4755	581	577	577
query69	433	269	279	269
query70	1111	1057	1125	1057
query71	435	280	276	276
query72	6656	4052	4041	4041
query73	758	335	344	335
query74	9309	8832	8958	8832
query75	3573	2672	2665	2665
query76	2815	1093	972	972
query77	505	316	323	316
query78	11440	9447	9118	9118
query79	2138	543	547	543
query80	848	513	526	513
query81	591	236	234	234
query82	617	150	151	150
query83	217	151	157	151
query84	239	78	77	77
query85	1208	288	321	288
query86	481	298	288	288
query87	4481	4311	4310	4310
query88	4087	2347	2346	2346
query89	386	285	287	285
query90	1857	195	189	189
query91	124	97	100	97
query92	75	50	51	50
query93	1208	567	553	553
query94	730	309	299	299
query95	353	266	270	266
query96	592	272	282	272
query97	3207	3067	3088	3067
query98	221	227	197	197
query99	1530	1280	1246	1246
Total cold run time: 294780 ms
Total hot run time: 187945 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 31.93 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit f15d57ca8202229519345f4911fe24181a45ba47, data reload: false

query1	0.04	0.04	0.04
query2	0.08	0.04	0.04
query3	0.22	0.06	0.06
query4	1.68	0.08	0.08
query5	0.48	0.50	0.49
query6	1.12	0.73	0.75
query7	0.02	0.01	0.01
query8	0.05	0.04	0.04
query9	0.54	0.47	0.47
query10	0.52	0.54	0.54
query11	0.16	0.12	0.11
query12	0.15	0.12	0.12
query13	0.63	0.59	0.59
query14	2.10	2.05	2.12
query15	0.83	0.83	0.82
query16	0.37	0.38	0.39
query17	1.05	1.06	1.03
query18	0.20	0.22	0.21
query19	1.84	1.85	1.73
query20	0.01	0.00	0.01
query21	15.40	0.68	0.67
query22	4.92	6.15	1.76
query23	18.32	1.41	1.31
query24	2.07	0.22	0.23
query25	0.16	0.08	0.09
query26	0.26	0.17	0.17
query27	0.08	0.08	0.08
query28	13.26	1.03	0.99
query29	12.62	3.31	3.32
query30	0.24	0.06	0.06
query31	2.87	0.40	0.39
query32	3.26	0.48	0.49
query33	3.01	3.01	3.02
query34	16.93	4.37	4.43
query35	4.48	4.41	4.44
query36	0.65	0.48	0.48
query37	0.19	0.16	0.16
query38	0.16	0.15	0.17
query39	0.04	0.04	0.03
query40	0.16	0.12	0.12
query41	0.09	0.05	0.05
query42	0.05	0.05	0.05
query43	0.05	0.04	0.04
Total cold run time: 111.36 s
Total hot run time: 31.93 s

@englefly
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 38449 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 8b4948c1c1ad443b8b7df7c32bf2e3ff196adbab, data reload: false

------ Round 1 ----------------------------------
q1	17653	4425	4317	4317
q2	2025	182	170	170
q3	10502	1096	1127	1096
q4	10131	808	701	701
q5	7720	2874	2870	2870
q6	233	145	143	143
q7	1000	637	615	615
q8	9555	2061	2019	2019
q9	7269	6523	6542	6523
q10	6996	2227	2210	2210
q11	450	247	240	240
q12	396	227	228	227
q13	17773	3063	3041	3041
q14	270	242	240	240
q15	509	488	481	481
q16	585	517	520	517
q17	978	714	754	714
q18	7404	6945	6849	6849
q19	1403	1111	1117	1111
q20	711	336	328	328
q21	3986	3085	3032	3032
q22	1103	1043	1005	1005
Total cold run time: 108652 ms
Total hot run time: 38449 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4324	4298	4243	4243
q2	384	287	268	268
q3	2869	2626	2703	2626
q4	1900	1642	1679	1642
q5	5417	5412	5423	5412
q6	226	131	131	131
q7	2092	1778	1761	1761
q8	3204	3339	3320	3320
q9	8437	8404	8478	8404
q10	3453	3216	3146	3146
q11	611	498	512	498
q12	790	630	601	601
q13	12777	3051	3065	3051
q14	299	291	276	276
q15	533	477	487	477
q16	614	595	588	588
q17	1799	1530	1496	1496
q18	7760	7441	7457	7441
q19	1679	1569	1478	1478
q20	2050	1834	1834	1834
q21	5458	5142	5195	5142
q22	1122	1053	1028	1028
Total cold run time: 67798 ms
Total hot run time: 54863 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 188835 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 8b4948c1c1ad443b8b7df7c32bf2e3ff196adbab, data reload: false

query1	916	388	380	380
query2	6456	2029	2009	2009
query3	6644	218	235	218
query4	31293	23477	23313	23313
query5	4202	518	497	497
query6	250	173	178	173
query7	4599	292	290	290
query8	249	205	210	205
query9	8441	2492	2487	2487
query10	425	274	273	273
query11	15570	14985	15051	14985
query12	151	105	98	98
query13	1623	379	378	378
query14	9708	7244	7153	7153
query15	276	180	186	180
query16	7929	463	493	463
query17	1583	566	542	542
query18	1995	288	287	287
query19	283	150	141	141
query20	117	109	108	108
query21	212	104	102	102
query22	4367	4130	4140	4130
query23	34171	33502	33491	33491
query24	11131	2944	2863	2863
query25	629	388	374	374
query26	1151	153	162	153
query27	2462	283	282	282
query28	7139	2135	2096	2096
query29	799	414	428	414
query30	308	164	160	160
query31	985	771	774	771
query32	103	57	58	57
query33	753	296	281	281
query34	985	483	485	483
query35	892	737	728	728
query36	1071	960	932	932
query37	170	99	101	99
query38	3987	3875	3901	3875
query39	1454	1400	1401	1400
query40	206	118	116	116
query41	48	48	46	46
query42	110	97	98	97
query43	514	457	482	457
query44	1212	763	746	746
query45	197	171	166	166
query46	1105	741	737	737
query47	1879	1788	1829	1788
query48	380	298	306	298
query49	1107	447	506	447
query50	802	421	414	414
query51	7284	7179	7277	7179
query52	101	87	85	85
query53	255	198	184	184
query54	938	453	457	453
query55	82	76	77	76
query56	289	262	251	251
query57	1210	1075	1044	1044
query58	259	227	240	227
query59	3060	2883	2854	2854
query60	294	272	270	270
query61	108	100	105	100
query62	836	621	665	621
query63	213	185	181	181
query64	4483	679	661	661
query65	3226	3168	3136	3136
query66	1396	346	352	346
query67	15634	15275	15153	15153
query68	3537	602	587	587
query69	409	326	283	283
query70	1168	1155	1107	1107
query71	331	277	288	277
query72	6375	4151	4047	4047
query73	760	333	348	333
query74	9316	8859	8759	8759
query75	3411	2660	2746	2660
query76	2158	1040	1002	1002
query77	521	320	313	313
query78	9711	9035	9584	9035
query79	1070	539	545	539
query80	785	526	507	507
query81	562	245	249	245
query82	571	148	143	143
query83	240	152	159	152
query84	229	80	78	78
query85	1034	295	345	295
query86	295	295	309	295
query87	4408	4306	4311	4306
query88	3092	2365	2335	2335
query89	385	287	369	287
query90	1882	201	193	193
query91	131	107	105	105
query92	62	52	51	51
query93	1041	544	532	532
query94	575	308	287	287
query95	356	262	266	262
query96	595	268	270	268
query97	3220	3099	3099	3099
query98	221	202	203	202
query99	1577	1256	1247	1247
Total cold run time: 282735 ms
Total hot run time: 188835 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 32.25 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 8b4948c1c1ad443b8b7df7c32bf2e3ff196adbab, data reload: false

query1	0.05	0.04	0.04
query2	0.09	0.04	0.04
query3	0.23	0.05	0.05
query4	1.67	0.08	0.08
query5	0.49	0.48	0.49
query6	1.14	0.74	0.72
query7	0.02	0.01	0.01
query8	0.06	0.05	0.05
query9	0.55	0.50	0.49
query10	0.54	0.56	0.54
query11	0.16	0.11	0.12
query12	0.16	0.12	0.12
query13	0.60	0.58	0.58
query14	2.06	2.03	2.05
query15	0.90	0.83	0.82
query16	0.37	0.38	0.37
query17	0.98	0.99	0.98
query18	0.22	0.21	0.19
query19	1.81	1.77	1.85
query20	0.01	0.01	0.01
query21	15.42	0.70	0.67
query22	3.83	7.65	2.06
query23	18.24	1.32	1.35
query24	2.12	0.23	0.23
query25	0.14	0.08	0.08
query26	0.26	0.18	0.17
query27	0.09	0.07	0.09
query28	13.24	1.03	1.01
query29	12.67	3.35	3.32
query30	0.24	0.06	0.05
query31	2.86	0.40	0.39
query32	3.26	0.49	0.48
query33	2.95	3.01	3.05
query34	16.89	4.38	4.43
query35	4.53	4.49	4.44
query36	0.65	0.50	0.49
query37	0.20	0.16	0.16
query38	0.16	0.15	0.17
query39	0.05	0.04	0.04
query40	0.16	0.13	0.14
query41	0.10	0.05	0.04
query42	0.06	0.04	0.05
query43	0.05	0.04	0.05
Total cold run time: 110.28 s
Total hot run time: 32.25 s

@englefly
Copy link
Contributor Author

englefly commented Sep 4, 2024

run buildall

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Sep 4, 2024
Copy link
Contributor

github-actions bot commented Sep 4, 2024

PR approved by at least one committer and no changes requested.

Copy link
Contributor

github-actions bot commented Sep 4, 2024

PR approved by anyone and no changes requested.

@doris-robot
Copy link

TPC-H: Total hot run time: 38205 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 736a5d4332daa025b8551ec5f8542b19cfff4589, data reload: false

------ Round 1 ----------------------------------
q1	17615	4788	4271	4271
q2	2006	181	175	175
q3	11891	964	1156	964
q4	10531	670	680	670
q5	7762	2866	2807	2807
q6	233	140	137	137
q7	951	612	604	604
q8	9316	2048	2061	2048
q9	7277	6579	6571	6571
q10	6993	2362	2230	2230
q11	438	241	245	241
q12	402	218	228	218
q13	19025	3053	3071	3053
q14	279	239	241	239
q15	544	492	481	481
q16	599	508	503	503
q17	987	726	735	726
q18	7168	6829	6845	6829
q19	1388	970	1080	970
q20	688	328	328	328
q21	4370	3128	3132	3128
q22	1129	1025	1012	1012
Total cold run time: 111592 ms
Total hot run time: 38205 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4393	4308	4290	4290
q2	373	274	278	274
q3	2913	2645	2738	2645
q4	1976	1664	1689	1664
q5	5689	5673	5803	5673
q6	226	136	145	136
q7	2223	1837	1882	1837
q8	3319	3469	3489	3469
q9	8888	8884	8806	8806
q10	3623	3404	3375	3375
q11	589	503	507	503
q12	843	660	672	660
q13	15255	3255	3303	3255
q14	334	289	296	289
q15	542	489	492	489
q16	616	585	580	580
q17	1864	1563	1530	1530
q18	8077	7804	7818	7804
q19	1721	1481	1697	1481
q20	2144	1910	1938	1910
q21	5719	5677	5639	5639
q22	1144	1062	1104	1062
Total cold run time: 72471 ms
Total hot run time: 57371 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 192719 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 736a5d4332daa025b8551ec5f8542b19cfff4589, data reload: false

query1	1247	880	864	864
query2	6479	1958	1994	1958
query3	10713	4067	4117	4067
query4	60012	26607	23183	23183
query5	5394	510	528	510
query6	398	164	174	164
query7	5825	309	297	297
query8	290	208	214	208
query9	9010	2525	2523	2523
query10	503	277	282	277
query11	18215	15041	15530	15041
query12	149	108	100	100
query13	1553	395	380	380
query14	11249	6866	6525	6525
query15	246	189	172	172
query16	7549	470	448	448
query17	1143	579	589	579
query18	2019	301	308	301
query19	307	170	144	144
query20	126	112	110	110
query21	213	104	106	104
query22	4639	4475	4663	4475
query23	34425	33340	33493	33340
query24	5995	2910	2921	2910
query25	525	383	372	372
query26	684	153	157	153
query27	1799	278	275	275
query28	3834	2053	2038	2038
query29	687	400	408	400
query30	246	151	157	151
query31	915	765	712	712
query32	84	52	55	52
query33	473	297	288	288
query34	864	485	475	475
query35	819	740	731	731
query36	1059	930	930	930
query37	149	95	84	84
query38	4060	3938	3903	3903
query39	1473	1400	1389	1389
query40	203	113	118	113
query41	49	48	47	47
query42	117	98	99	98
query43	517	480	485	480
query44	1063	739	739	739
query45	201	168	169	168
query46	1088	780	727	727
query47	1899	1825	1810	1810
query48	380	307	305	305
query49	775	445	444	444
query50	819	422	425	422
query51	7050	6885	6903	6885
query52	101	86	91	86
query53	262	185	180	180
query54	570	463	450	450
query55	74	74	76	74
query56	290	265	260	260
query57	1235	1096	1098	1096
query58	232	243	244	243
query59	3049	2813	2848	2813
query60	304	280	295	280
query61	129	121	122	121
query62	766	662	662	662
query63	222	185	189	185
query64	2955	764	747	747
query65	3211	3194	3146	3146
query66	685	356	359	356
query67	15390	15243	15320	15243
query68	3169	592	594	592
query69	399	292	287	287
query70	1171	1132	1067	1067
query71	370	285	281	281
query72	6490	4054	4054	4054
query73	753	332	326	326
query74	9110	8912	8840	8840
query75	3336	2687	2639	2639
query76	1424	1058	1005	1005
query77	552	326	309	309
query78	9636	9140	9796	9140
query79	1646	538	531	531
query80	831	519	508	508
query81	541	238	234	234
query82	380	146	146	146
query83	167	148	152	148
query84	265	76	74	74
query85	814	279	276	276
query86	317	303	283	283
query87	4359	4353	4291	4291
query88	2995	2348	2340	2340
query89	404	287	286	286
query90	1888	195	188	188
query91	122	99	99	99
query92	68	50	50	50
query93	1106	532	531	531
query94	829	287	300	287
query95	338	256	247	247
query96	590	264	265	264
query97	3230	3107	3107	3107
query98	218	206	204	204
query99	1761	1272	1289	1272
Total cold run time: 308285 ms
Total hot run time: 192719 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 32.45 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 736a5d4332daa025b8551ec5f8542b19cfff4589, data reload: false

query1	0.05	0.04	0.04
query2	0.08	0.04	0.04
query3	0.23	0.05	0.06
query4	1.67	0.07	0.07
query5	0.51	0.50	0.49
query6	1.13	0.74	0.73
query7	0.02	0.01	0.02
query8	0.05	0.04	0.04
query9	0.55	0.50	0.49
query10	0.54	0.54	0.54
query11	0.16	0.12	0.12
query12	0.15	0.11	0.12
query13	0.59	0.59	0.59
query14	2.10	2.05	2.12
query15	0.92	0.82	0.83
query16	0.38	0.37	0.38
query17	0.96	0.96	1.00
query18	0.23	0.22	0.21
query19	1.91	1.70	1.79
query20	0.01	0.01	0.00
query21	15.39	0.68	0.67
query22	4.50	6.50	2.20
query23	18.28	1.35	1.27
query24	2.15	0.25	0.22
query25	0.16	0.08	0.08
query26	0.27	0.19	0.19
query27	0.08	0.08	0.07
query28	13.19	1.04	1.02
query29	12.65	3.44	3.41
query30	0.24	0.06	0.06
query31	2.87	0.41	0.40
query32	3.25	0.48	0.47
query33	3.01	3.04	3.05
query34	16.94	4.41	4.46
query35	4.49	4.44	4.48
query36	0.66	0.50	0.47
query37	0.19	0.15	0.16
query38	0.16	0.15	0.15
query39	0.05	0.04	0.04
query40	0.16	0.12	0.12
query41	0.10	0.05	0.05
query42	0.06	0.05	0.05
query43	0.04	0.04	0.04
Total cold run time: 111.13 s
Total hot run time: 32.45 s

@englefly
Copy link
Contributor Author

englefly commented Sep 6, 2024

run cloud_p0

@englefly englefly merged commit 744a1ec into apache:master Sep 6, 2024
25 of 28 checks passed
@englefly englefly deleted the agg-min-max-stats branch September 6, 2024 04:03
dataroaring pushed a commit that referenced this pull request Oct 9, 2024
## Proposed changes
the column stats for min(A) agg function is estimated as column stats of A.
in current version, min(A).ndv is 1, this is error-prone.

Issue Number: close #xxx

<!--Describe your changes.-->
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by one committer. dev/3.0.3-merged reviewed
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants