Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Enhancement](partition) Refine some auto partition behaviours #32737

Merged
merged 6 commits into from
Apr 1, 2024

Conversation

zclllyybb
Copy link
Contributor

@zclllyybb zclllyybb commented Mar 24, 2024

Proposed changes

Issue Number: close #xxx

  1. fix legacy planner grammer
  2. fix nereids planner parsing
  3. fix cases
  4. forbid auto range partition with null column
  5. fix CreateTableStmt with auto partition and some partition items.

1 and 2 are about #31585

doc pr: apache/doris-website#488

Further comments

If this is a relatively large or complex change, kick off the discussion at [email protected] by explaining why you chose the solution you did and what alternatives you considered, etc...

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@zclllyybb
Copy link
Contributor Author

run buildall

Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@zclllyybb
Copy link
Contributor Author

run buildall

Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

TPC-H: Total hot run time: 37797 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit de50443b1982be3c89d6854e403c0eb68e061668, data reload: false

------ Round 1 ----------------------------------
q1	17654	4187	4086	4086
q2	2126	163	152	152
q3	10580	1161	1184	1161
q4	10229	822	778	778
q5	7484	2974	2997	2974
q6	203	122	123	122
q7	1019	578	556	556
q8	9338	1997	1989	1989
q9	7556	6597	6554	6554
q10	8486	3441	3546	3441
q11	432	229	224	224
q12	412	201	197	197
q13	17815	2824	2857	2824
q14	237	200	197	197
q15	515	461	460	460
q16	492	378	377	377
q17	941	517	617	517
q18	7153	6437	6383	6383
q19	1757	1443	1476	1443
q20	560	265	244	244
q21	3580	2915	2813	2813
q22	349	305	312	305
Total cold run time: 108918 ms
Total hot run time: 37797 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4192	4099	4045	4045
q2	331	228	233	228
q3	2983	2850	2816	2816
q4	1818	1602	1615	1602
q5	5293	5287	5339	5287
q6	192	117	116	116
q7	2217	1846	1851	1846
q8	3174	3283	3261	3261
q9	8648	8691	8635	8635
q10	3799	3747	3768	3747
q11	543	452	449	449
q12	717	515	555	515
q13	16937	2844	2865	2844
q14	291	249	252	249
q15	489	456	456	456
q16	478	447	427	427
q17	1706	1526	1454	1454
q18	7439	7181	7012	7012
q19	1609	1569	1481	1481
q20	1929	1721	1716	1716
q21	4796	4691	4673	4673
q22	508	466	455	455
Total cold run time: 70089 ms
Total hot run time: 53314 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 186910 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit de50443b1982be3c89d6854e403c0eb68e061668, data reload: false

query1	942	365	357	357
query2	7348	2000	1970	1970
query3	6702	214	206	206
query4	31904	21299	21304	21299
query5	4364	485	400	400
query6	265	180	172	172
query7	4643	296	284	284
query8	246	170	169	169
query9	9143	2269	2272	2269
query10	562	243	261	243
query11	14793	14485	14413	14413
query12	140	93	87	87
query13	1634	414	410	410
query14	11835	11217	11653	11217
query15	356	195	205	195
query16	7539	252	253	252
query17	2085	558	540	540
query18	1929	280	276	276
query19	365	158	163	158
query20	89	88	89	88
query21	204	134	126	126
query22	4981	4795	4786	4786
query23	33118	32874	32691	32691
query24	10900	2849	2856	2849
query25	615	392	397	392
query26	1215	157	169	157
query27	2390	346	362	346
query28	6703	1884	1860	1860
query29	870	657	631	631
query30	301	151	152	151
query31	979	729	732	729
query32	90	57	59	57
query33	771	259	262	259
query34	1036	496	479	479
query35	829	614	622	614
query36	1002	888	869	869
query37	120	77	79	77
query38	3531	3400	3436	3400
query39	1483	1456	1417	1417
query40	206	116	115	115
query41	54	50	48	48
query42	105	97	100	97
query43	466	474	447	447
query44	1148	727	746	727
query45	266	254	261	254
query46	1101	695	678	678
query47	1893	1838	1825	1825
query48	433	350	344	344
query49	1099	349	342	342
query50	762	367	372	367
query51	6696	6631	6637	6631
query52	106	91	94	91
query53	344	281	280	280
query54	321	251	249	249
query55	98	80	77	77
query56	247	231	236	231
query57	1215	1112	1124	1112
query58	234	201	208	201
query59	2810	2711	2639	2639
query60	279	247	261	247
query61	116	118	116	116
query62	645	446	461	446
query63	305	274	275	274
query64	5860	4164	4069	4069
query65	3088	3022	3011	3011
query66	818	383	383	383
query67	15653	14869	14839	14839
query68	8894	527	527	527
query69	641	374	368	368
query70	1414	1103	1157	1103
query71	518	291	284	284
query72	6567	2711	2532	2532
query73	1604	315	321	315
query74	7513	6648	6653	6648
query75	4505	2965	3031	2965
query76	5572	885	859	859
query77	667	266	251	251
query78	10884	10182	10198	10182
query79	10907	516	519	516
query80	1780	389	383	383
query81	535	212	220	212
query82	356	206	204	204
query83	216	143	141	141
query84	288	77	84	77
query85	1088	332	317	317
query86	364	321	291	291
query87	3866	3512	3542	3512
query88	5023	2268	2254	2254
query89	478	371	370	370
query90	2071	173	178	173
query91	169	137	136	136
query92	69	47	47	47
query93	6178	498	473	473
query94	1345	181	171	171
query95	420	327	319	319
query96	612	280	261	261
query97	3030	2898	2890	2890
query98	229	218	208	208
query99	1155	886	927	886
Total cold run time: 313348 ms
Total hot run time: 186910 ms

@doris-robot
Copy link

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit de50443b1982be3c89d6854e403c0eb68e061668 with default session variables
Stream load json:         18 seconds loaded 2358488459 Bytes, about 124 MB/s
Stream load orc:          59 seconds loaded 1101869774 Bytes, about 17 MB/s
Stream load parquet:      32 seconds loaded 861443392 Bytes, about 25 MB/s
Insert into select:       22.1 seconds inserted 10000000 Rows, about 452K ops/s

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 35.26% (8733/24769)
Line Coverage: 27.06% (71518/264328)
Region Coverage: 26.29% (37107/141125)
Branch Coverage: 23.19% (18976/81828)
Coverage Report: http://coverage.selectdb-in.cc/coverage/de50443b1982be3c89d6854e403c0eb68e061668_de50443b1982be3c89d6854e403c0eb68e061668/report/index.html

@zclllyybb
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 38342 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit de50443b1982be3c89d6854e403c0eb68e061668, data reload: false

------ Round 1 ----------------------------------
q1	17613	4484	4123	4123
q2	2113	155	147	147
q3	11225	1164	1272	1164
q4	10606	782	835	782
q5	7815	3081	3000	3000
q6	203	126	126	126
q7	1058	631	628	628
q8	9429	2102	2058	2058
q9	7221	6670	6670	6670
q10	8380	3437	3595	3437
q11	432	225	221	221
q12	372	200	196	196
q13	17807	2896	2877	2877
q14	251	202	205	202
q15	503	462	455	455
q16	494	368	375	368
q17	968	519	587	519
q18	7305	6567	6475	6475
q19	3916	1441	1411	1411
q20	553	248	246	246
q21	3626	2949	2940	2940
q22	347	297	298	297
Total cold run time: 112237 ms
Total hot run time: 38342 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4126	4070	4069	4069
q2	337	228	229	228
q3	2978	2850	2841	2841
q4	1855	1554	1561	1554
q5	5312	5347	5339	5339
q6	197	114	115	114
q7	2248	1883	1835	1835
q8	3177	3312	3286	3286
q9	8719	8723	8763	8723
q10	3849	3813	3733	3733
q11	552	440	449	440
q12	726	538	544	538
q13	16893	2910	2860	2860
q14	273	249	255	249
q15	503	451	461	451
q16	496	422	429	422
q17	1884	1532	1483	1483
q18	7427	7273	7134	7134
q19	1615	1514	1553	1514
q20	1916	1729	1693	1693
q21	4835	4686	4727	4686
q22	529	446	446	446
Total cold run time: 70447 ms
Total hot run time: 53638 ms

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 35.25% (8732/24769)
Line Coverage: 27.06% (71528/264341)
Region Coverage: 26.29% (37109/141132)
Branch Coverage: 23.19% (18978/81834)
Coverage Report: http://coverage.selectdb-in.cc/coverage/de50443b1982be3c89d6854e403c0eb68e061668_de50443b1982be3c89d6854e403c0eb68e061668/report/index.html

@doris-robot
Copy link

TPC-DS: Total hot run time: 186564 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit de50443b1982be3c89d6854e403c0eb68e061668, data reload: false

query1	919	387	360	360
query2	7497	2078	1979	1979
query3	6730	206	214	206
query4	31858	21535	21352	21352
query5	4295	502	409	409
query6	272	187	169	169
query7	4625	290	295	290
query8	223	170	177	170
query9	9200	2274	2246	2246
query10	572	245	248	245
query11	15690	14319	14402	14319
query12	134	93	86	86
query13	1624	399	406	399
query14	12017	11033	11419	11033
query15	258	218	196	196
query16	8171	264	259	259
query17	1984	599	547	547
query18	2095	291	285	285
query19	353	162	160	160
query20	97	93	89	89
query21	207	136	133	133
query22	4928	4719	4772	4719
query23	33465	32731	32710	32710
query24	11863	2880	2766	2766
query25	627	359	364	359
query26	1773	149	158	149
query27	2962	345	350	345
query28	7479	1860	1844	1844
query29	996	629	609	609
query30	316	149	149	149
query31	948	743	728	728
query32	93	58	56	56
query33	771	249	330	249
query34	970	474	470	470
query35	824	614	614	614
query36	1010	888	889	888
query37	177	84	75	75
query38	3606	3460	3444	3444
query39	1477	1448	1430	1430
query40	285	111	112	111
query41	48	47	45	45
query42	106	95	100	95
query43	463	438	457	438
query44	1130	725	714	714
query45	277	259	255	255
query46	1113	713	692	692
query47	1922	1838	1824	1824
query48	427	361	351	351
query49	1232	325	323	323
query50	762	371	372	371
query51	6752	6606	6649	6606
query52	108	90	92	90
query53	351	273	275	273
query54	321	237	247	237
query55	85	77	86	77
query56	250	226	220	220
query57	1225	1153	1154	1153
query58	253	210	215	210
query59	2804	2694	2654	2654
query60	273	250	274	250
query61	115	116	117	116
query62	670	444	456	444
query63	313	282	281	281
query64	6961	4137	4163	4137
query65	3131	3042	3024	3024
query66	1442	388	381	381
query67	15221	14978	14696	14696
query68	5329	504	519	504
query69	561	394	391	391
query70	1237	1143	1202	1143
query71	436	286	285	285
query72	6329	2897	2611	2611
query73	730	324	307	307
query74	7026	6657	6662	6657
query75	3723	2930	2968	2930
query76	3509	875	911	875
query77	415	270	253	253
query78	10942	10157	10213	10157
query79	8734	517	521	517
query80	2169	389	367	367
query81	557	218	210	210
query82	1381	196	197	196
query83	277	152	155	152
query84	297	77	79	77
query85	2000	333	304	304
query86	495	313	302	302
query87	3742	3587	3515	3515
query88	5035	2308	2270	2270
query89	542	357	368	357
query90	2058	177	173	173
query91	164	135	151	135
query92	60	48	47	47
query93	6800	494	477	477
query94	1267	172	171	171
query95	433	328	327	327
query96	608	269	263	263
query97	3044	2877	2902	2877
query98	223	212	205	205
query99	1234	905	936	905
Total cold run time: 312503 ms
Total hot run time: 186564 ms

@doris-robot
Copy link

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit de50443b1982be3c89d6854e403c0eb68e061668 with default session variables
Stream load json:         18 seconds loaded 2358488459 Bytes, about 124 MB/s
Stream load orc:          59 seconds loaded 1101869774 Bytes, about 17 MB/s
Stream load parquet:      31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select:       22.5 seconds inserted 10000000 Rows, about 444K ops/s

@zclllyybb
Copy link
Contributor Author

not do this now

@zclllyybb zclllyybb closed this Mar 28, 2024
@zclllyybb zclllyybb reopened this Mar 28, 2024
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@zclllyybb zclllyybb changed the title [Feature](partition) Support auto create range partition for NULL value [bugfix](partition) Fix auto range partition with null value Mar 28, 2024
@zclllyybb
Copy link
Contributor Author

run buildall

Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 35.25% (8741/24796)
Line Coverage: 27.02% (71556/264783)
Region Coverage: 26.28% (37136/141334)
Branch Coverage: 23.17% (18987/81934)
Coverage Report: http://coverage.selectdb-in.cc/coverage/d36efe4d8d214f98a1cb659a713ee22e2a4466d2_d36efe4d8d214f98a1cb659a713ee22e2a4466d2/report/index.html

@doris-robot
Copy link

TPC-H: Total hot run time: 37711 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit d36efe4d8d214f98a1cb659a713ee22e2a4466d2, data reload: false

------ Round 1 ----------------------------------
q1	17607	4244	4074	4074
q2	2116	155	156	155
q3	10578	1177	1201	1177
q4	10220	780	747	747
q5	7537	2934	2969	2934
q6	202	125	124	124
q7	1046	580	600	580
q8	9343	1980	1955	1955
q9	7071	6629	6552	6552
q10	8407	3444	3589	3444
q11	445	236	216	216
q12	419	211	201	201
q13	17797	2876	2855	2855
q14	230	211	202	202
q15	520	471	450	450
q16	494	376	377	376
q17	939	582	567	567
q18	7203	6527	6353	6353
q19	1962	1363	1451	1363
q20	536	252	256	252
q21	3543	2903	2834	2834
q22	347	304	300	300
Total cold run time: 108562 ms
Total hot run time: 37711 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4088	4134	4075	4075
q2	321	234	227	227
q3	3015	2833	2864	2833
q4	1842	1580	1584	1580
q5	5267	5339	5330	5330
q6	195	117	118	117
q7	2240	1857	1900	1857
q8	3151	3292	3278	3278
q9	8750	8737	8622	8622
q10	3812	3779	3752	3752
q11	560	466	453	453
q12	731	555	550	550
q13	16931	2865	2880	2865
q14	282	253	272	253
q15	505	463	469	463
q16	485	428	428	428
q17	1719	1498	1465	1465
q18	7627	7183	7162	7162
q19	1631	1570	1528	1528
q20	1936	1731	1725	1725
q21	5048	4860	4798	4798
q22	533	469	465	465
Total cold run time: 70669 ms
Total hot run time: 53826 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 182535 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit d36efe4d8d214f98a1cb659a713ee22e2a4466d2, data reload: false

query1	938	380	349	349
query2	7224	1970	1881	1881
query3	6700	223	221	221
query4	32465	21949	21860	21860
query5	4335	398	407	398
query6	720	185	185	185
query7	6324	294	306	294
query8	1098	180	178	178
query9	12897	2316	2318	2316
query10	562	246	268	246
query11	15420	14242	14194	14194
query12	136	89	89	89
query13	1615	420	431	420
query14	10287	7920	7592	7592
query15	312	194	192	192
query16	8237	261	262	261
query17	2170	587	555	555
query18	2112	287	285	285
query19	352	167	157	157
query20	95	92	91	91
query21	203	129	123	123
query22	4961	4796	4808	4796
query23	33680	32793	32894	32793
query24	10629	2904	2901	2901
query25	626	394	401	394
query26	1212	163	162	162
query27	2462	344	344	344
query28	7148	1903	1873	1873
query29	885	634	634	634
query30	302	155	154	154
query31	989	726	724	724
query32	93	58	57	57
query33	771	255	259	255
query34	1073	482	508	482
query35	829	607	597	597
query36	999	866	881	866
query37	125	65	73	65
query38	3504	3522	3439	3439
query39	1475	1460	1413	1413
query40	210	116	112	112
query41	50	50	47	47
query42	104	97	99	97
query43	466	437	451	437
query44	1124	725	744	725
query45	283	261	281	261
query46	1102	715	720	715
query47	1893	1828	1855	1828
query48	430	362	357	357
query49	1090	342	322	322
query50	756	364	371	364
query51	6697	6685	6656	6656
query52	104	91	89	89
query53	337	269	274	269
query54	312	261	224	224
query55	85	80	82	80
query56	244	222	216	216
query57	1197	1127	1118	1118
query58	237	205	199	199
query59	2732	2627	2502	2502
query60	260	232	242	232
query61	98	93	91	91
query62	656	444	467	444
query63	306	273	276	273
query64	5701	4081	3956	3956
query65	3069	3029	3051	3029
query66	856	374	366	366
query67	15269	14879	15083	14879
query68	5473	530	519	519
query69	574	386	380	380
query70	1259	1151	1165	1151
query71	424	279	262	262
query72	6362	2704	2665	2665
query73	705	325	319	319
query74	7782	6464	6557	6464
query75	3036	2262	2237	2237
query76	3518	840	888	840
query77	448	266	261	261
query78	10843	10299	10264	10264
query79	8340	516	520	516
query80	1942	387	384	384
query81	547	225	213	213
query82	1669	89	86	86
query83	340	150	147	147
query84	295	88	81	81
query85	1705	383	343	343
query86	486	285	294	285
query87	3668	3553	3518	3518
query88	5078	2308	2303	2303
query89	545	367	364	364
query90	2034	173	173	173
query91	169	139	142	139
query92	65	47	49	47
query93	6519	507	487	487
query94	1226	175	174	174
query95	447	334	321	321
query96	602	263	263	263
query97	2680	2469	2498	2469
query98	221	212	217	212
query99	1204	942	924	924
Total cold run time: 311839 ms
Total hot run time: 182535 ms

@doris-robot
Copy link

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit d36efe4d8d214f98a1cb659a713ee22e2a4466d2 with default session variables
Stream load json:         19 seconds loaded 2358488459 Bytes, about 118 MB/s
Stream load orc:          59 seconds loaded 1101869774 Bytes, about 17 MB/s
Stream load parquet:      32 seconds loaded 861443392 Bytes, about 25 MB/s
Insert into select:       14.0 seconds inserted 10000000 Rows, about 714K ops/s

@zclllyybb
Copy link
Contributor Author

run buildall

Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 35.56% (8840/24862)
Line Coverage: 27.30% (72492/265583)
Region Coverage: 26.51% (37523/141558)
Branch Coverage: 23.32% (19131/82054)
Coverage Report: http://coverage.selectdb-in.cc/coverage/a5b4c519b7bd28ec7690e60270517ee41e9e8de3_a5b4c519b7bd28ec7690e60270517ee41e9e8de3/report/index.html

@zclllyybb
Copy link
Contributor Author

run buildall

Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 35.55% (8839/24862)
Line Coverage: 27.29% (72489/265583)
Region Coverage: 26.51% (37521/141558)
Branch Coverage: 23.32% (19132/82054)
Coverage Report: http://coverage.selectdb-in.cc/coverage/8c5fdbfdd22db3875bc0a4179b1e1f5c809b5dcf_8c5fdbfdd22db3875bc0a4179b1e1f5c809b5dcf/report/index.html

@doris-robot
Copy link

TPC-H: Total hot run time: 37988 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 8c5fdbfdd22db3875bc0a4179b1e1f5c809b5dcf, data reload: false

------ Round 1 ----------------------------------
q1	17631	4346	4233	4233
q2	2077	195	185	185
q3	10453	1095	1140	1095
q4	10183	714	718	714
q5	7412	2784	2686	2686
q6	219	128	129	128
q7	1012	602	594	594
q8	9204	1990	2006	1990
q9	7411	6572	6524	6524
q10	8532	3460	3496	3460
q11	468	237	231	231
q12	493	209	209	209
q13	17767	2954	2959	2954
q14	265	235	230	230
q15	523	482	449	449
q16	512	373	382	373
q17	950	541	607	541
q18	7302	6708	6719	6708
q19	5457	1433	1460	1433
q20	661	315	304	304
q21	3401	2646	2679	2646
q22	352	302	301	301
Total cold run time: 112285 ms
Total hot run time: 37988 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4316	4183	4208	4183
q2	380	278	268	268
q3	2990	2717	2639	2639
q4	1861	1593	1537	1537
q5	5288	5218	5174	5174
q6	213	122	122	122
q7	2182	1820	1863	1820
q8	3166	3273	3342	3273
q9	8556	8523	8511	8511
q10	3939	3822	3851	3822
q11	591	494	489	489
q12	794	655	609	609
q13	17661	3334	3194	3194
q14	330	302	295	295
q15	519	482	496	482
q16	516	455	463	455
q17	1802	1487	1503	1487
q18	8071	7745	7667	7667
q19	1675	1575	1586	1575
q20	1951	1753	1818	1753
q21	5219	4953	4927	4927
q22	526	465	462	462
Total cold run time: 72546 ms
Total hot run time: 54744 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 181973 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 8c5fdbfdd22db3875bc0a4179b1e1f5c809b5dcf, data reload: false

query1	877	353	1114	353
query2	6491	2105	1891	1891
query3	6661	221	214	214
query4	36462	21457	21235	21235
query5	4249	407	390	390
query6	240	186	170	170
query7	4005	294	294	294
query8	231	170	178	170
query9	5692	2282	2282	2282
query10	361	236	246	236
query11	14675	14202	14166	14166
query12	143	93	89	89
query13	978	375	367	367
query14	9024	6925	7015	6925
query15	209	175	181	175
query16	7444	267	266	266
query17	1782	602	568	568
query18	1772	302	287	287
query19	214	156	164	156
query20	98	86	95	86
query21	198	127	133	127
query22	4951	4856	4815	4815
query23	33275	32729	32915	32729
query24	11092	2959	2926	2926
query25	577	414	404	404
query26	708	166	155	155
query27	3053	360	380	360
query28	6465	1915	1909	1909
query29	914	642	657	642
query30	298	168	165	165
query31	972	764	747	747
query32	66	55	55	55
query33	548	258	261	258
query34	1120	503	516	503
query35	875	729	711	711
query36	1010	927	934	927
query37	111	71	73	71
query38	3671	3644	3653	3644
query39	1705	1632	1608	1608
query40	190	131	131	131
query41	50	55	46	46
query42	109	104	107	104
query43	500	473	455	455
query44	1183	734	734	734
query45	293	277	289	277
query46	1114	728	705	705
query47	1997	1953	1990	1953
query48	396	304	303	303
query49	870	365	373	365
query50	795	415	401	401
query51	6846	6840	6830	6830
query52	107	95	94	94
query53	356	286	291	286
query54	259	235	240	235
query55	85	80	81	80
query56	246	234	235	234
query57	1316	1242	1208	1208
query58	221	220	211	211
query59	3005	2772	2501	2501
query60	263	230	229	229
query61	94	92	94	92
query62	581	452	435	435
query63	307	277	278	277
query64	4560	4227	3997	3997
query65	3088	3010	3029	3010
query66	737	338	336	336
query67	15668	15124	14966	14966
query68	10447	548	561	548
query69	619	311	302	302
query70	1223	1197	1151	1151
query71	504	279	271	271
query72	6658	2642	2428	2428
query73	1257	325	327	325
query74	7065	6415	6362	6362
query75	3588	2351	2338	2338
query76	6263	952	1076	952
query77	587	273	249	249
query78	10992	10199	10039	10039
query79	10911	518	517	517
query80	1985	432	429	429
query81	539	1067	217	217
query82	183	87	86	86
query83	220	170	168	168
query84	263	82	79	79
query85	1047	280	265	265
query86	367	304	300	300
query87	3691	3543	3552	3543
query88	5067	2388	2378	2378
query89	478	375	374	374
query90	2067	172	169	169
query91	125	94	94	94
query92	63	48	47	47
query93	5792	506	504	504
query94	1403	177	179	177
query95	398	311	302	302
query96	603	271	265	265
query97	2644	2490	2513	2490
query98	229	217	223	217
query99	1057	816	834	816
Total cold run time: 306788 ms
Total hot run time: 181973 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.59 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 8c5fdbfdd22db3875bc0a4179b1e1f5c809b5dcf, data reload: false

query1	0.04	0.04	0.03
query2	0.08	0.04	0.04
query3	0.22	0.05	0.04
query4	1.68	0.07	0.07
query5	0.49	0.47	0.48
query6	1.13	0.66	0.67
query7	0.02	0.01	0.02
query8	0.05	0.05	0.04
query9	0.56	0.51	0.48
query10	0.54	0.55	0.54
query11	0.15	0.09	0.11
query12	0.14	0.11	0.10
query13	0.58	0.59	0.58
query14	0.76	0.79	0.77
query15	0.83	0.80	0.81
query16	0.36	0.37	0.37
query17	0.97	0.99	1.01
query18	0.24	0.24	0.24
query19	1.77	1.73	1.67
query20	0.01	0.01	0.01
query21	15.53	0.65	0.65
query22	2.84	2.61	2.68
query23	17.15	0.89	0.87
query24	1.14	0.25	0.21
query25	0.09	0.08	0.08
query26	0.24	0.17	0.17
query27	0.07	0.07	0.07
query28	14.06	0.94	0.92
query29	12.57	3.32	3.32
query30	0.29	0.08	0.08
query31	2.84	0.37	0.38
query32	3.27	0.45	0.45
query33	2.80	2.92	2.83
query34	16.56	4.46	4.41
query35	4.52	4.49	4.46
query36	0.60	0.48	0.47
query37	0.18	0.16	0.16
query38	0.14	0.14	0.14
query39	0.05	0.04	0.04
query40	0.18	0.14	0.15
query41	0.09	0.05	0.05
query42	0.06	0.05	0.05
query43	0.04	0.04	0.04
Total cold run time: 105.93 s
Total hot run time: 30.59 s

@doris-robot
Copy link

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit 8c5fdbfdd22db3875bc0a4179b1e1f5c809b5dcf with default session variables
Stream load json:         19 seconds loaded 2358488459 Bytes, about 118 MB/s
Stream load orc:          59 seconds loaded 1101869774 Bytes, about 17 MB/s
Stream load parquet:      31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select:       13.7 seconds inserted 10000000 Rows, about 729K ops/s

Copy link
Contributor

@HappenLee HappenLee left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Apr 1, 2024
Copy link
Contributor

github-actions bot commented Apr 1, 2024

PR approved by at least one committer and no changes requested.

Copy link
Contributor

github-actions bot commented Apr 1, 2024

PR approved by anyone and no changes requested.

@zhangstar333 zhangstar333 merged commit 0afa283 into apache:master Apr 1, 2024
25 of 30 checks passed
@zclllyybb zclllyybb deleted the null_range branch April 1, 2024 09:29
zclllyybb added a commit to zclllyybb/doris that referenced this pull request Apr 1, 2024
…e#32737)

fix legacy planner grammer
fix nereids planner parsing
fix cases
forbid auto range partition with null column
fix CreateTableStmt with auto partition and some partition items.
1 and 2 are about apache#31585
doc pr: apache/doris-website#488
zclllyybb added a commit to zclllyybb/doris that referenced this pull request Apr 9, 2024
…e#32737)

fix legacy planner grammer
fix nereids planner parsing
fix cases
forbid auto range partition with null column
fix CreateTableStmt with auto partition and some partition items.
1 and 2 are about apache#31585
doc pr: apache/doris-website#488
yiguolei pushed a commit that referenced this pull request Apr 9, 2024
… (#33412)

fix legacy planner grammer
fix nereids planner parsing
fix cases
forbid auto range partition with null column
fix CreateTableStmt with auto partition and some partition items.
1 and 2 are about #31585
doc pr: apache/doris-website#488
zclllyybb added a commit to zclllyybb/doris that referenced this pull request Apr 11, 2024
zclllyybb added a commit to zclllyybb/doris that referenced this pull request Apr 11, 2024
…e#32737)

fix legacy planner grammer
fix nereids planner parsing
fix cases
forbid auto range partition with null column
fix CreateTableStmt with auto partition and some partition items.
1 and 2 are about apache#31585
doc pr: apache/doris-website#488
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by one committer. dev/2.1.2-merged kind/behavior-changed reviewed
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants