Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[enhance](S3) Use allocator for s3 buffer's allocation #38519

Draft
wants to merge 1 commit into
base: branch-2.0
Choose a base branch
from
Draft

[enhance](S3) Use allocator for s3 buffer's allocation #38519

wants to merge 1 commit into from

Conversation

ByteYue
Copy link
Contributor

@ByteYue ByteYue commented Jul 30, 2024

Proposed changes

Remove the static buffer pool for S3 upload. Use the dynamic allocator to control the memory usage for S3 upload which is more adaptive and easy to use. Remove all the useless config plus.

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@ByteYue
Copy link
Contributor Author

ByteYue commented Jul 30, 2024

run buildall

github-actions[bot]
github-actions bot reviewed Jul 30, 2024
View reviewed changes
Copy link
Contributor

@github-actions github-actions bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

clang-tidy made some suggestions

be/src/io/fs/s3_file_write_bufferpool.cpp Show resolved Hide resolved
@ByteYue
Copy link
Contributor Author

ByteYue commented Jul 30, 2024

run buildall

1 similar comment
@ByteYue
Copy link
Contributor Author

ByteYue commented Jul 30, 2024

run buildall

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 37.82% (8120/21471)
Line Coverage: 29.49% (66596/225862)
Region Coverage: 28.97% (34336/118522)
Branch Coverage: 24.84% (17640/71002)
Coverage Report: http://coverage.selectdb-in.cc/coverage/d801554ad811d8b7ef56ecb2162ba8ff0813e675_d801554ad811d8b7ef56ecb2162ba8ff0813e675/report/index.html

@doris-robot
Copy link

TPC-H: Total hot run time: 50407 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit d801554ad811d8b7ef56ecb2162ba8ff0813e675, data reload: false

------ Round 1 ----------------------------------
q1	17523	4420	4353	4353
q2	2073	157	145	145
q3	10266	1909	1939	1909
q4	10098	1250	1356	1250
q5	8482	3940	3955	3940
q6	236	129	126	126
q7	2099	1629	1598	1598
q8	9288	2778	2753	2753
q9	10626	11033	10617	10617
q10	8689	3546	3532	3532
q11	423	255	237	237
q12	488	309	310	309
q13	18363	3992	4029	3992
q14	363	338	319	319
q15	509	459	464	459
q16	674	573	588	573
q17	1142	1007	971	971
q18	7289	6860	6865	6860
q19	1791	1648	1644	1644
q20	546	289	321	289
q21	4473	4174	4089	4089
q22	543	442	459	442
Total cold run time: 115984 ms
Total hot run time: 50407 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4400	4350	4326	4326
q2	330	230	220	220
q3	4178	4212	4198	4198
q4	2793	2779	2779	2779
q5	7166	7075	7035	7035
q6	242	123	123	123
q7	3293	2796	2844	2796
q8	4394	4474	4511	4474
q9	16973	16958	16893	16893
q10	4282	4292	4281	4281
q11	766	684	689	684
q12	1017	857	867	857
q13	5896	3771	3737	3737
q14	445	446	416	416
q15	507	457	455	455
q16	736	703	678	678
q17	3886	3913	3889	3889
q18	8936	8922	8972	8922
q19	1758	1746	1695	1695
q20	2416	2168	2124	2124
q21	8592	8591	8516	8516
q22	1051	999	972	972
Total cold run time: 84057 ms
Total hot run time: 80070 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 205193 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit d801554ad811d8b7ef56ecb2162ba8ff0813e675, data reload: false

query1	928	428	391	391
query2	6521	2897	2872	2872
query3	6924	210	200	200
query4	20307	18085	18108	18085
query5	19733	6545	6548	6545
query6	291	221	240	221
query7	4170	298	305	298
query8	407	435	464	435
query9	3108	2670	2606	2606
query10	431	313	306	306
query11	11220	10810	10687	10687
query12	126	80	75	75
query13	5599	707	684	684
query14	17433	13797	13649	13649
query15	368	249	244	244
query16	6482	287	265	265
query17	1705	1454	886	886
query18	2313	426	413	413
query19	213	148	156	148
query20	78	80	82	80
query21	194	100	99	99
query22	5179	5078	4946	4946
query23	32765	32088	32225	32088
query24	6957	6572	6533	6533
query25	531	440	425	425
query26	526	173	160	160
query27	1856	301	291	291
query28	6139	2351	2309	2309
query29	2957	2675	2681	2675
query30	245	167	166	166
query31	928	733	737	733
query32	71	61	61	61
query33	405	251	246	246
query34	866	479	480	479
query35	1133	975	941	941
query36	1395	1257	1215	1215
query37	94	60	61	60
query38	3099	2947	2986	2947
query39	1390	1325	1354	1325
query40	201	99	97	97
query41	47	45	44	44
query42	85	80	88	80
query43	809	661	717	661
query44	1140	723	732	723
query45	248	241	234	234
query46	1218	959	962	959
query47	1780	1818	1717	1717
query48	1008	717	701	701
query49	630	369	389	369
query50	879	621	633	621
query51	4831	4714	4703	4703
query52	99	86	96	86
query53	462	331	331	331
query54	2667	2492	2505	2492
query55	83	86	84	84
query56	233	214	203	203
query57	1237	1110	1114	1110
query58	224	194	206	194
query59	4096	3914	3901	3901
query60	230	211	202	202
query61	96	94	94	94
query62	817	483	500	483
query63	491	340	346	340
query64	2406	1527	1421	1421
query65	3704	3593	3585	3585
query66	840	378	381	378
query67	17330	16023	16031	16023
query68	7644	645	645	645
query69	558	347	349	347
query70	1514	1475	1430	1430
query71	392	309	322	309
query72	6507	3481	3510	3481
query73	730	318	326	318
query74	6336	5795	5911	5795
query75	4504	3670	3797	3670
query76	4519	1138	1166	1138
query77	545	245	263	245
query78	12534	12221	11525	11525
query79	9266	658	636	636
query80	2424	409	405	405
query81	533	238	232	232
query82	1346	102	98	98
query83	188	134	135	134
query84	258	72	72	72
query85	1308	334	327	327
query86	371	305	293	293
query87	3288	2991	3100	2991
query88	5193	2272	2293	2272
query89	380	287	286	286
query90	1733	219	215	215
query91	188	152	147	147
query92	59	53	54	53
query93	5871	596	554	554
query94	869	207	211	207
query95	1138	1079	1072	1072
query96	668	323	322	322
query97	6459	6337	6372	6337
query98	195	182	176	176
query99	2854	847	836	836
Total cold run time: 313350 ms
Total hot run time: 205193 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 31.33 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit d801554ad811d8b7ef56ecb2162ba8ff0813e675, data reload: false

query1	0.02	0.02	0.02
query2	0.07	0.03	0.02
query3	0.24	0.04	0.05
query4	1.79	0.06	0.06
query5	0.54	0.51	0.52
query6	1.24	0.61	0.61
query7	0.02	0.01	0.01
query8	0.03	0.03	0.02
query9	0.52	0.48	0.48
query10	0.54	0.54	0.53
query11	0.12	0.09	0.09
query12	0.11	0.10	0.09
query13	0.62	0.61	0.62
query14	0.78	0.78	0.79
query15	0.78	0.77	0.76
query16	0.38	0.36	0.37
query17	0.98	1.02	1.01
query18	0.23	0.26	0.23
query19	1.89	1.85	1.83
query20	0.01	0.01	0.01
query21	15.45	0.57	0.56
query22	1.93	2.21	2.08
query23	17.66	1.11	0.97
query24	4.03	1.04	2.04
query25	0.39	0.10	0.04
query26	0.54	0.15	0.17
query27	0.05	0.04	0.04
query28	8.17	0.72	0.72
query29	12.74	2.32	2.29
query30	0.62	0.54	0.54
query31	2.82	0.39	0.37
query32	3.39	0.50	0.49
query33	3.06	3.05	3.13
query34	15.65	4.81	4.81
query35	4.83	4.86	4.85
query36	1.06	1.01	1.01
query37	0.06	0.04	0.04
query38	0.03	0.02	0.02
query39	0.03	0.01	0.02
query40	0.16	0.14	0.14
query41	0.06	0.01	0.01
query42	0.02	0.01	0.01
query43	0.03	0.02	0.02
Total cold run time: 103.69 s
Total hot run time: 31.33 s

@doris-robot
Copy link

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit d801554ad811d8b7ef56ecb2162ba8ff0813e675 with default session variables
Stream load json:         20 seconds loaded 2358488459 Bytes, about 112 MB/s
Stream load orc:          59 seconds loaded 1101869774 Bytes, about 17 MB/s
Stream load parquet:      31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select:       21.6 seconds inserted 10000000 Rows, about 462K ops/s

@ByteYue
Copy link
Contributor Author

ByteYue commented Jul 30, 2024

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 49921 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 7aa70d7780d123c8f52de7108aa4b8536b5ffd72, data reload: false

------ Round 1 ----------------------------------
q1	17619	4437	4359	4359
q2	2106	151	150	150
q3	10288	1924	1934	1924
q4	10340	1270	1351	1270
q5	8457	3912	3845	3845
q6	254	124	124	124
q7	2061	1617	1611	1611
q8	9323	2753	2739	2739
q9	10743	10333	10227	10227
q10	8659	3470	3529	3470
q11	414	249	252	249
q12	471	303	299	299
q13	18328	3960	4001	3960
q14	351	325	326	325
q15	503	453	448	448
q16	690	579	576	576
q17	1113	981	966	966
q18	7248	6935	7033	6935
q19	1789	1587	1593	1587
q20	512	324	301	301
q21	4424	4122	4119	4119
q22	539	454	437	437
Total cold run time: 116232 ms
Total hot run time: 49921 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4339	4294	4311	4294
q2	321	226	218	218
q3	4141	4162	4114	4114
q4	2730	2717	2734	2717
q5	7088	7057	7052	7052
q6	235	121	120	120
q7	3232	2815	2870	2815
q8	4419	4422	4452	4422
q9	16841	16595	16703	16595
q10	4149	4281	4374	4281
q11	750	696	706	696
q12	1018	848	864	848
q13	7672	3749	3736	3736
q14	449	419	415	415
q15	522	456	458	456
q16	730	672	670	670
q17	3860	3851	3861	3851
q18	8823	8844	8851	8844
q19	1734	1733	1683	1683
q20	2377	2086	2139	2086
q21	8397	8471	8497	8471
q22	1067	963	987	963
Total cold run time: 84894 ms
Total hot run time: 79347 ms

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 37.82% (8120/21472)
Line Coverage: 29.49% (66604/225887)
Region Coverage: 28.97% (34338/118526)
Branch Coverage: 24.84% (17640/71006)
Coverage Report: http://coverage.selectdb-in.cc/coverage/7aa70d7780d123c8f52de7108aa4b8536b5ffd72_7aa70d7780d123c8f52de7108aa4b8536b5ffd72/report/index.html

@doris-robot
Copy link

TPC-DS: Total hot run time: 202617 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 7aa70d7780d123c8f52de7108aa4b8536b5ffd72, data reload: false

query1	963	391	380	380
query2	6531	2770	2599	2599
query3	6922	206	199	199
query4	20248	18036	17958	17958
query5	19733	6466	6488	6466
query6	277	223	230	223
query7	4151	301	313	301
query8	421	424	460	424
query9	3107	2644	2590	2590
query10	411	298	289	289
query11	11281	10829	10588	10588
query12	131	76	75	75
query13	5602	667	705	667
query14	18115	13758	13656	13656
query15	365	242	247	242
query16	6475	285	265	265
query17	1713	1471	884	884
query18	2302	413	407	407
query19	205	147	160	147
query20	81	75	84	75
query21	193	98	95	95
query22	5262	5098	4997	4997
query23	32594	31951	31895	31895
query24	6984	6514	6538	6514
query25	524	424	425	424
query26	530	161	167	161
query27	1884	292	288	288
query28	6129	2399	2325	2325
query29	2869	2937	2606	2606
query30	254	170	169	169
query31	898	736	762	736
query32	68	64	62	62
query33	417	256	252	252
query34	855	464	490	464
query35	1136	920	910	910
query36	1334	1267	1106	1106
query37	87	61	58	58
query38	3118	2928	2925	2925
query39	1384	1333	1323	1323
query40	205	97	95	95
query41	47	50	44	44
query42	83	84	85	84
query43	711	668	637	637
query44	1128	719	718	718
query45	248	240	238	238
query46	1234	956	973	956
query47	2129	1700	1791	1700
query48	1041	705	706	705
query49	620	386	367	367
query50	858	604	608	604
query51	4765	4626	4681	4626
query52	92	79	87	79
query53	461	318	314	314
query54	2652	2448	2444	2444
query55	101	74	87	74
query56	252	217	241	217
query57	1182	1115	1131	1115
query58	216	187	215	187
query59	4187	3798	4137	3798
query60	224	212	213	212
query61	97	97	99	97
query62	837	467	507	467
query63	481	339	350	339
query64	2477	1531	1332	1332
query65	3581	3523	3534	3523
query66	816	386	392	386
query67	15698	15397	14885	14885
query68	9756	656	635	635
query69	587	340	357	340
query70	1639	1530	1399	1399
query71	420	306	324	306
query72	6604	3510	3488	3488
query73	739	320	328	320
query74	6293	5859	5802	5802
query75	5384	3770	3681	3681
query76	6132	1143	1198	1143
query77	1046	259	253	253
query78	12476	11786	11480	11480
query79	6936	645	633	633
query80	1102	394	406	394
query81	499	241	237	237
query82	1486	102	97	97
query83	172	134	130	130
query84	258	72	69	69
query85	887	332	336	332
query86	347	300	298	298
query87	3210	2983	3028	2983
query88	4769	2302	2307	2302
query89	373	290	299	290
query90	1954	214	211	211
query91	175	141	156	141
query92	62	53	53	53
query93	4806	574	596	574
query94	700	204	201	201
query95	1112	1061	1059	1059
query96	635	324	323	323
query97	6472	6423	6382	6382
query98	189	181	172	172
query99	2921	985	901	901
Total cold run time: 312118 ms
Total hot run time: 202617 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 31.34 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 7aa70d7780d123c8f52de7108aa4b8536b5ffd72, data reload: false

query1	0.03	0.02	0.02
query2	0.07	0.02	0.02
query3	0.24	0.04	0.05
query4	1.78	0.07	0.07
query5	0.53	0.52	0.53
query6	1.23	0.62	0.63
query7	0.01	0.01	0.01
query8	0.03	0.03	0.02
query9	0.53	0.50	0.49
query10	0.55	0.52	0.52
query11	0.11	0.09	0.09
query12	0.12	0.09	0.09
query13	0.62	0.61	0.61
query14	0.79	0.78	0.81
query15	0.78	0.76	0.76
query16	0.37	0.40	0.38
query17	1.00	1.00	1.01
query18	0.20	0.27	0.25
query19	1.95	1.86	1.86
query20	0.02	0.01	0.01
query21	15.46	0.56	0.55
query22	2.39	2.28	2.15
query23	17.08	0.93	0.86
query24	5.10	0.98	1.26
query25	0.39	0.11	0.06
query26	0.54	0.14	0.16
query27	0.04	0.04	0.03
query28	7.73	0.78	0.73
query29	12.77	2.31	2.33
query30	0.62	0.54	0.52
query31	2.80	0.39	0.37
query32	3.38	0.50	0.49
query33	3.08	3.09	3.07
query34	15.28	4.82	4.78
query35	4.84	4.86	4.88
query36	1.05	1.01	1.01
query37	0.06	0.05	0.05
query38	0.04	0.02	0.02
query39	0.02	0.01	0.02
query40	0.17	0.14	0.14
query41	0.07	0.02	0.01
query42	0.02	0.02	0.02
query43	0.03	0.02	0.02
Total cold run time: 103.92 s
Total hot run time: 31.34 s

@doris-robot
Copy link

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit 7aa70d7780d123c8f52de7108aa4b8536b5ffd72 with default session variables
Stream load json:         19 seconds loaded 2358488459 Bytes, about 118 MB/s
Stream load orc:          59 seconds loaded 1101869774 Bytes, about 17 MB/s
Stream load parquet:      31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select:       21.4 seconds inserted 10000000 Rows, about 467K ops/s

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Jul 30, 2024
@github-actions GitHub Actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions GitHub Actions
Copy link
Contributor

PR approved by anyone and no changes requested.

@ByteYue
Copy link
Contributor Author

ByteYue commented Jul 31, 2024

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 50056 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 6db5c2f1ceb5ce7bc954d6bd27850f002fa1cd30, data reload: false

------ Round 1 ----------------------------------
q1	17965	4417	4372	4372
q2	2083	163	151	151
q3	10502	1887	1926	1887
q4	10341	1267	1350	1267
q5	8590	3977	3991	3977
q6	259	140	128	128
q7	2065	1605	1577	1577
q8	9611	2752	2721	2721
q9	13851	10491	10250	10250
q10	8630	3534	3525	3525
q11	416	249	250	249
q12	475	306	305	305
q13	18373	3989	4075	3989
q14	352	326	333	326
q15	520	470	459	459
q16	675	571	570	570
q17	1138	968	885	885
q18	7212	6940	6959	6940
q19	1788	1642	1624	1624
q20	542	301	312	301
q21	4449	4118	4110	4110
q22	531	453	443	443
Total cold run time: 120368 ms
Total hot run time: 50056 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4353	4281	4337	4281
q2	317	227	223	223
q3	4203	4199	4182	4182
q4	2759	2762	2762	2762
q5	7179	7143	7112	7112
q6	241	122	120	120
q7	3255	2839	2861	2839
q8	4359	4508	4529	4508
q9	16917	16897	16853	16853
q10	4231	4280	4254	4254
q11	737	702	682	682
q12	1027	838	890	838
q13	6747	3741	3760	3741
q14	458	417	413	413
q15	498	466	471	466
q16	742	693	675	675
q17	3977	3835	3869	3835
q18	8785	8776	8759	8759
q19	1727	1695	1700	1695
q20	2387	2079	2079	2079
q21	8540	8560	8516	8516
q22	1066	985	985	985
Total cold run time: 84505 ms
Total hot run time: 79818 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 203707 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 6db5c2f1ceb5ce7bc954d6bd27850f002fa1cd30, data reload: false

query1	916	424	378	378
query2	6553	2859	2321	2321
query3	6922	207	206	206
query4	21284	18059	17902	17902
query5	19736	6461	6518	6461
query6	279	217	242	217
query7	4168	300	320	300
query8	461	453	427	427
query9	3101	2649	2619	2619
query10	415	302	294	294
query11	11229	10759	10709	10709
query12	126	78	77	77
query13	5602	691	686	686
query14	17329	13120	13171	13120
query15	353	243	260	243
query16	6479	288	260	260
query17	1695	1449	872	872
query18	2339	410	407	407
query19	201	146	156	146
query20	83	79	80	79
query21	194	97	94	94
query22	5234	5134	5027	5027
query23	32544	31962	31728	31728
query24	7034	6513	6441	6441
query25	510	449	424	424
query26	611	167	160	160
query27	2009	291	298	291
query28	6133	2359	2330	2330
query29	2788	2725	2659	2659
query30	240	164	164	164
query31	891	770	736	736
query32	70	61	58	58
query33	404	254	250	250
query34	851	470	481	470
query35	1123	908	962	908
query36	1362	1237	1087	1087
query37	86	60	61	60
query38	3038	2943	2913	2913
query39	1362	1324	1325	1324
query40	216	94	92	92
query41	52	45	44	44
query42	90	93	82	82
query43	723	768	686	686
query44	1201	713	721	713
query45	250	237	234	234
query46	1223	930	981	930
query47	1823	1671	1660	1660
query48	1013	712	719	712
query49	621	375	369	369
query50	866	591	606	591
query51	4730	4623	4670	4623
query52	94	85	90	85
query53	446	320	313	313
query54	2659	2465	2464	2464
query55	94	79	89	79
query56	255	224	206	206
query57	1191	1123	1039	1039
query58	211	204	185	185
query59	4076	3734	4193	3734
query60	250	193	206	193
query61	111	95	99	95
query62	795	513	473	473
query63	478	345	351	345
query64	2624	1520	1513	1513
query65	3616	3555	3538	3538
query66	808	375	365	365
query67	17148	17812	16737	16737
query68	4950	652	649	649
query69	531	361	335	335
query70	1466	1565	1593	1565
query71	369	328	311	311
query72	6344	3478	3490	3478
query73	717	323	321	321
query74	6188	5815	5809	5809
query75	4289	3755	3722	3722
query76	2964	1154	1193	1154
query77	418	249	253	249
query78	12454	11498	11735	11498
query79	6936	613	629	613
query80	3250	413	397	397
query81	555	245	235	235
query82	1397	108	94	94
query83	240	136	134	134
query84	225	72	72	72
query85	1598	337	338	337
query86	439	288	325	288
query87	3227	3009	3033	3009
query88	4556	2282	2293	2282
query89	366	292	285	285
query90	1601	208	207	207
query91	173	158	140	140
query92	77	61	52	52
query93	2180	561	579	561
query94	915	210	213	210
query95	1142	1058	1037	1037
query96	631	323	323	323
query97	6514	6397	6348	6348
query98	210	184	178	178
query99	2688	866	915	866
Total cold run time: 303029 ms
Total hot run time: 203707 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.09 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 6db5c2f1ceb5ce7bc954d6bd27850f002fa1cd30, data reload: false

query1	0.03	0.03	0.02
query2	0.07	0.03	0.02
query3	0.24	0.04	0.04
query4	1.79	0.07	0.07
query5	0.52	0.52	0.52
query6	1.23	0.62	0.62
query7	0.02	0.01	0.01
query8	0.03	0.03	0.02
query9	0.51	0.49	0.48
query10	0.53	0.53	0.52
query11	0.12	0.08	0.08
query12	0.11	0.09	0.10
query13	0.63	0.62	0.62
query14	0.77	0.79	0.79
query15	0.79	0.76	0.78
query16	0.36	0.37	0.36
query17	1.01	1.01	1.03
query18	0.23	0.26	0.25
query19	1.88	1.86	1.85
query20	0.01	0.01	0.01
query21	15.47	0.56	0.54
query22	2.07	2.44	1.54
query23	16.59	1.13	0.90
query24	5.86	0.64	0.91
query25	0.39	0.11	0.05
query26	0.55	0.14	0.16
query27	0.05	0.03	0.04
query28	7.68	0.77	0.72
query29	12.61	2.32	2.12
query30	0.55	0.51	0.49
query31	2.80	0.40	0.36
query32	3.36	0.50	0.49
query33	3.09	3.07	3.05
query34	15.27	4.79	4.78
query35	4.87	4.88	4.85
query36	1.07	1.02	1.00
query37	0.06	0.05	0.04
query38	0.03	0.02	0.02
query39	0.02	0.02	0.02
query40	0.16	0.14	0.14
query41	0.07	0.02	0.01
query42	0.02	0.02	0.01
query43	0.02	0.02	0.01
Total cold run time: 103.54 s
Total hot run time: 30.09 s

@doris-robot
Copy link

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit 6db5c2f1ceb5ce7bc954d6bd27850f002fa1cd30 with default session variables
Stream load json:         20 seconds loaded 2358488459 Bytes, about 112 MB/s
Stream load orc:          59 seconds loaded 1101869774 Bytes, about 17 MB/s
Stream load parquet:      32 seconds loaded 861443392 Bytes, about 25 MB/s
Insert into select:       21.7 seconds inserted 10000000 Rows, about 460K ops/s

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 37.81% (8118/21471)
Line Coverage: 29.48% (66581/225880)
Region Coverage: 28.96% (34325/118527)
Branch Coverage: 24.83% (17630/71004)
Coverage Report: http://coverage.selectdb-in.cc/coverage/6db5c2f1ceb5ce7bc954d6bd27850f002fa1cd30_6db5c2f1ceb5ce7bc954d6bd27850f002fa1cd30/report/index.html

@ByteYue
pass benchmark tool
397818a 
no lambda

check

init
@ByteYue
Copy link
Contributor Author

ByteYue commented Jul 31, 2024

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 50080 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 397818ae14a60fab13ce2e8e9a3d982a0199aa2c, data reload: false

------ Round 1 ----------------------------------
q1	17765	4377	4303	4303
q2	2059	155	148	148
q3	10296	1937	1924	1924
q4	10342	1257	1331	1257
q5	8787	3912	3959	3912
q6	230	126	123	123
q7	2062	1582	1584	1582
q8	9332	2752	2697	2697
q9	10874	10714	10629	10629
q10	8644	3526	3529	3526
q11	420	245	248	245
q12	470	298	308	298
q13	18340	3945	4008	3945
q14	353	325	331	325
q15	521	453	459	453
q16	672	576	567	567
q17	1116	975	949	949
q18	7291	6771	6990	6771
q19	1792	1660	1611	1611
q20	515	307	307	307
q21	4421	4090	4123	4090
q22	526	418	439	418
Total cold run time: 116828 ms
Total hot run time: 50080 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4341	4297	4320	4297
q2	317	223	225	223
q3	4175	4141	4115	4115
q4	2751	2743	2748	2743
q5	7203	7103	7141	7103
q6	237	122	120	120
q7	3221	2807	2813	2807
q8	4427	4486	4546	4486
q9	17399	17046	16927	16927
q10	4241	4216	4230	4216
q11	746	710	687	687
q12	1017	866	844	844
q13	6641	3723	3745	3723
q14	460	410	434	410
q15	505	466	465	465
q16	727	676	690	676
q17	3821	3785	3829	3785
q18	8848	8726	8796	8726
q19	1701	1688	1639	1639
q20	2374	2093	2107	2093
q21	8483	8452	8539	8452
q22	1012	988	946	946
Total cold run time: 84647 ms
Total hot run time: 79483 ms

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 37.80% (8116/21472)
Line Coverage: 29.47% (66562/225884)
Region Coverage: 28.95% (34311/118528)
Branch Coverage: 24.83% (17627/71004)
Coverage Report: http://coverage.selectdb-in.cc/coverage/397818ae14a60fab13ce2e8e9a3d982a0199aa2c_397818ae14a60fab13ce2e8e9a3d982a0199aa2c/report/index.html

@doris-robot
Copy link

TPC-DS: Total hot run time: 203833 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 397818ae14a60fab13ce2e8e9a3d982a0199aa2c, data reload: false

query1	938	423	375	375
query2	6563	2667	2586	2586
query3	6917	208	199	199
query4	21111	18074	17934	17934
query5	19737	6510	6570	6510
query6	282	210	221	210
query7	4175	303	320	303
query8	407	388	373	373
query9	3108	2681	2612	2612
query10	433	301	301	301
query11	11325	10712	10741	10712
query12	121	83	74	74
query13	5595	697	691	691
query14	18014	13759	13362	13362
query15	373	242	245	242
query16	6449	282	263	263
query17	1751	1425	881	881
query18	2314	413	409	409
query19	215	148	152	148
query20	72	77	82	77
query21	189	99	95	95
query22	5280	5042	4892	4892
query23	32436	32119	32107	32107
query24	6934	6444	6495	6444
query25	528	436	439	436
query26	527	162	159	159
query27	1878	292	291	291
query28	6159	2345	2344	2344
query29	2986	2737	2776	2737
query30	248	169	171	169
query31	939	764	737	737
query32	70	62	63	62
query33	414	263	257	257
query34	847	485	467	467
query35	1145	912	935	912
query36	1266	1222	1101	1101
query37	87	63	61	61
query38	3045	2893	2925	2893
query39	1393	1319	1338	1319
query40	216	96	95	95
query41	46	43	44	43
query42	90	81	78	78
query43	703	773	813	773
query44	1118	722	728	722
query45	254	242	236	236
query46	1226	968	962	962
query47	1806	1855	1815	1815
query48	1036	712	703	703
query49	636	363	372	363
query50	866	618	586	586
query51	4789	4681	4730	4681
query52	96	85	82	82
query53	453	331	324	324
query54	2674	2467	2482	2467
query55	84	75	79	75
query56	243	233	238	233
query57	1222	1097	1008	1008
query58	217	216	199	199
query59	3917	4167	4069	4069
query60	214	200	222	200
query61	100	98	98	98
query62	688	509	481	481
query63	511	343	344	343
query64	2625	1575	1583	1575
query65	3620	3615	3553	3553
query66	771	395	398	395
query67	15570	15188	15357	15188
query68	8812	660	640	640
query69	592	353	348	348
query70	1458	1358	1344	1344
query71	417	312	318	312
query72	6587	3509	3529	3509
query73	742	322	317	317
query74	6359	5934	5863	5863
query75	5353	3661	3737	3661
query76	5400	1129	1209	1129
query77	914	262	254	254
query78	12857	11726	11507	11507
query79	8370	641	633	633
query80	1524	409	415	409
query81	481	239	231	231
query82	1659	100	100	100
query83	174	133	134	133
query84	263	73	73	73
query85	912	335	341	335
query86	342	338	293	293
query87	3208	3022	3034	3022
query88	5106	2315	2294	2294
query89	390	293	283	283
query90	1949	211	216	211
query91	174	141	144	141
query92	61	52	62	52
query93	5978	572	576	572
query94	716	214	207	207
query95	1118	1073	1063	1063
query96	648	334	333	333
query97	6523	6453	6453	6453
query98	192	179	178	178
query99	2943	841	936	841
Total cold run time: 314284 ms
Total hot run time: 203833 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 31.44 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 397818ae14a60fab13ce2e8e9a3d982a0199aa2c, data reload: false

query1	0.02	0.02	0.03
query2	0.08	0.02	0.02
query3	0.25	0.04	0.04
query4	1.80	0.06	0.06
query5	0.53	0.52	0.51
query6	1.25	0.64	0.61
query7	0.01	0.01	0.01
query8	0.03	0.02	0.02
query9	0.52	0.50	0.49
query10	0.55	0.53	0.52
query11	0.13	0.08	0.09
query12	0.11	0.08	0.09
query13	0.62	0.61	0.62
query14	0.79	0.79	0.79
query15	0.78	0.75	0.75
query16	0.36	0.36	0.37
query17	1.01	0.99	0.99
query18	0.19	0.29	0.24
query19	1.95	1.81	1.86
query20	0.01	0.01	0.00
query21	15.46	0.54	0.55
query22	2.02	2.28	2.07
query23	17.33	0.98	0.91
query24	6.10	1.71	1.26
query25	0.34	0.10	0.05
query26	0.79	0.16	0.15
query27	0.04	0.03	0.04
query28	6.11	0.78	0.78
query29	12.59	2.31	2.41
query30	0.63	0.53	0.59
query31	2.79	0.40	0.37
query32	4.03	0.50	0.51
query33	3.09	3.06	3.04
query34	15.25	4.84	4.79
query35	4.89	4.85	4.88
query36	1.05	1.01	1.00
query37	0.06	0.05	0.04
query38	0.03	0.02	0.02
query39	0.02	0.02	0.02
query40	0.16	0.13	0.14
query41	0.07	0.01	0.01
query42	0.02	0.01	0.01
query43	0.02	0.02	0.02
Total cold run time: 103.88 s
Total hot run time: 31.44 s

@doris-robot
Copy link

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit 397818ae14a60fab13ce2e8e9a3d982a0199aa2c with default session variables
Stream load json:         19 seconds loaded 2358488459 Bytes, about 118 MB/s
Stream load orc:          59 seconds loaded 1101869774 Bytes, about 17 MB/s
Stream load parquet:      32 seconds loaded 861443392 Bytes, about 25 MB/s
Insert into select:       21.2 seconds inserted 10000000 Rows, about 471K ops/s

@xiaokang xiaokang marked this pull request as draft July 31, 2024 14:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@github-actions github-actions[bot] github-actions[bot] left review comments

@gavinchou gavinchou gavinchou approved these changes

Assignees
No one assigned
Labels
approved Indicates a PR has been approved by one committer. reviewed
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@ByteYue @doris-robot @gavinchou