[chore](config) Add config to control BufferedReader and S3FileWriter's thread pool's min max nums #33974

ByteYue · 2024-04-22T09:18:40Z

Proposed changes

Issue Number: close #xxx

This PR is mainly to provide more granular control over the thread pool size for remote IO. When import prefetching finds that the prefetch speed is too fast or too slow, you can adjust buffered_reader_prefetch_thread_pool_{min,max}thread_num to control the corresponding number of prefetch threads. When the upload speed to S3 is too fast or too slow, you can adjust s3_file_upload_thread_pool{min,max}_thread_num.

Further comments

If this is a relatively large or complex change, kick off the discussion at [email protected] by explaining why you chose the solution you did and what alternatives you considered, etc...

doris-robot · 2024-04-22T09:18:45Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

ByteYue · 2024-04-22T09:19:53Z

run buildall

github-actions · 2024-04-22T09:23:53Z

clang-tidy review says "All clean, LGTM! 👍"

morningman

PLEASE explain detail about what are these config for and when should we set it.

doris-robot · 2024-04-22T10:27:53Z

TPC-H: Total hot run time: 38930 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit d0ea24c5728139c4d09de12505011887db2c7427, data reload: false

------ Round 1 ----------------------------------
q1	17881	4802	4398	4398
q2	2509	195	200	195
q3	11668	1197	1212	1197
q4	10720	807	840	807
q5	8039	2807	2675	2675
q6	221	131	133	131
q7	1008	603	593	593
q8	9223	2074	2064	2064
q9	7409	6590	6528	6528
q10	8456	3544	3509	3509
q11	457	235	230	230
q12	397	216	218	216
q13	19306	2964	2933	2933
q14	267	228	239	228
q15	536	484	485	484
q16	520	380	380	380
q17	966	691	678	678
q18	7439	6746	6761	6746
q19	7083	1546	1494	1494
q20	665	326	307	307
q21	3449	2827	2878	2827
q22	375	310	323	310
Total cold run time: 118594 ms
Total hot run time: 38930 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4328	4239	4211	4211
q2	377	264	276	264
q3	3050	2768	2726	2726
q4	1890	1549	1541	1541
q5	5359	5377	5327	5327
q6	217	121	122	121
q7	2258	1889	1908	1889
q8	3225	3370	3348	3348
q9	8612	8535	8622	8535
q10	3870	3703	3762	3703
q11	591	486	480	480
q12	749	582	619	582
q13	16499	2971	2980	2971
q14	316	278	281	278
q15	519	472	468	468
q16	489	425	427	425
q17	1781	1493	1472	1472
q18	7547	7627	7408	7408
q19	1658	1550	1559	1550
q20	1938	1768	1728	1728
q21	6582	4733	4827	4733
q22	541	475	484	475
Total cold run time: 72396 ms
Total hot run time: 54235 ms

doris-robot · 2024-04-22T10:39:32Z

TPC-DS: Total hot run time: 184032 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit d0ea24c5728139c4d09de12505011887db2c7427, data reload: false

query1	915	379	368	368
query2	6539	2403	2428	2403
query3	6642	215	216	215
query4	23085	21315	21411	21315
query5	4229	410	420	410
query6	276	174	175	174
query7	4791	290	285	285
query8	239	191	199	191
query9	8453	2259	2248	2248
query10	596	264	270	264
query11	14747	14251	14265	14251
query12	141	93	89	89
query13	1655	389	360	360
query14	10104	7915	7436	7436
query15	270	183	185	183
query16	8260	265	262	262
query17	1989	601	568	568
query18	2104	302	274	274
query19	323	150	157	150
query20	94	86	86	86
query21	199	134	135	134
query22	4975	4834	4866	4834
query23	33625	33075	33218	33075
query24	11710	3005	2963	2963
query25	714	364	362	362
query26	1750	152	144	144
query27	3007	309	319	309
query28	7638	1951	1952	1951
query29	1031	601	592	592
query30	282	175	174	174
query31	979	731	738	731
query32	95	51	53	51
query33	744	246	242	242
query34	1111	485	478	478
query35	848	705	688	688
query36	1036	878	879	878
query37	286	70	67	67
query38	3322	3208	3220	3208
query39	1599	1521	1529	1521
query40	269	123	120	120
query41	44	43	44	43
query42	104	93	92	92
query43	569	551	544	544
query44	1290	719	739	719
query45	293	252	268	252
query46	1091	720	712	712
query47	1986	1846	1852	1846
query48	366	310	294	294
query49	1130	383	385	383
query50	758	384	381	381
query51	6727	6562	6621	6562
query52	101	85	86	85
query53	344	291	283	283
query54	312	235	222	222
query55	88	70	71	70
query56	237	223	219	219
query57	1249	1120	1162	1120
query58	222	199	195	195
query59	3292	3134	3084	3084
query60	247	227	237	227
query61	117	105	88	88
query62	660	470	442	442
query63	302	283	277	277
query64	6209	3875	3722	3722
query65	3143	3066	3037	3037
query66	1377	340	326	326
query67	15288	15270	14906	14906
query68	5248	534	527	527
query69	513	302	296	296
query70	1246	1144	1203	1144
query71	1402	1261	1274	1261
query72	6577	2617	2434	2434
query73	720	323	317	317
query74	6872	6491	6309	6309
query75	3601	2589	2614	2589
query76	3773	951	965	951
query77	656	258	259	258
query78	10930	10399	10174	10174
query79	3692	532	517	517
query80	1928	444	421	421
query81	525	242	246	242
query82	1377	95	93	93
query83	337	175	170	170
query84	266	88	87	87
query85	1616	265	292	265
query86	479	295	316	295
query87	3480	3305	3222	3222
query88	4801	2427	2390	2390
query89	485	371	367	367
query90	1983	177	178	177
query91	123	101	95	95
query92	56	47	45	45
query93	4856	495	491	491
query94	1250	182	175	175
query95	389	299	282	282
query96	597	270	263	263
query97	3154	2920	2975	2920
query98	238	218	218	218
query99	1241	852	853	852
Total cold run time: 292629 ms
Total hot run time: 184032 ms

doris-robot · 2024-04-22T10:45:03Z

ClickBench: Total hot run time: 30.49 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit d0ea24c5728139c4d09de12505011887db2c7427, data reload: false

query1	0.04	0.03	0.03
query2	0.07	0.04	0.04
query3	0.22	0.05	0.06
query4	1.66	0.07	0.07
query5	0.50	0.50	0.52
query6	1.48	0.72	0.72
query7	0.02	0.02	0.01
query8	0.05	0.04	0.04
query9	0.55	0.50	0.48
query10	0.54	0.55	0.55
query11	0.16	0.11	0.12
query12	0.14	0.11	0.12
query13	0.60	0.59	0.58
query14	0.75	0.77	0.75
query15	0.83	0.80	0.80
query16	0.37	0.37	0.35
query17	1.00	0.96	0.93
query18	0.22	0.24	0.24
query19	1.74	1.70	1.66
query20	0.01	0.02	0.01
query21	15.40	0.66	0.65
query22	4.49	6.85	2.28
query23	18.24	1.23	1.29
query24	1.81	0.27	0.21
query25	0.14	0.09	0.09
query26	0.25	0.16	0.16
query27	0.08	0.08	0.08
query28	13.28	1.00	0.97
query29	12.56	3.27	3.23
query30	0.25	0.06	0.06
query31	2.88	0.38	0.37
query32	3.27	0.46	0.47
query33	2.78	2.79	2.84
query34	17.14	4.37	4.40
query35	4.47	4.49	4.46
query36	0.64	0.46	0.46
query37	0.18	0.16	0.16
query38	0.15	0.14	0.14
query39	0.05	0.04	0.04
query40	0.17	0.13	0.14
query41	0.09	0.04	0.05
query42	0.05	0.05	0.05
query43	0.04	0.03	0.03
Total cold run time: 109.36 s
Total hot run time: 30.49 s

doris-robot · 2024-04-22T10:45:46Z

TeamCity be ut coverage result:
Function Coverage: 35.36% (8914/25209)
Line Coverage: 27.09% (73300/270595)
Region Coverage: 26.23% (37866/144370)
Branch Coverage: 23.03% (19276/83698)
Coverage Report: http://coverage.selectdb-in.cc/coverage/d0ea24c5728139c4d09de12505011887db2c7427_d0ea24c5728139c4d09de12505011887db2c7427/report/index.html

ByteYue · 2024-04-22T12:30:12Z

run buildall

github-actions · 2024-04-22T12:31:05Z

clang-tidy review says "All clean, LGTM! 👍"

github-actions · 2024-04-22T12:35:14Z

clang-tidy review says "All clean, LGTM! 👍"

doris-robot · 2024-04-22T13:55:14Z

TeamCity be ut coverage result:
Function Coverage: 35.36% (8915/25209)
Line Coverage: 27.09% (73320/270612)
Region Coverage: 26.24% (37876/144362)
Branch Coverage: 23.04% (19286/83696)
Coverage Report: http://coverage.selectdb-in.cc/coverage/72db3d3d923df2893677badee2be7b06ce55cb7b_72db3d3d923df2893677badee2be7b06ce55cb7b/report/index.html

doris-robot · 2024-04-22T13:56:34Z

TPC-H: Total hot run time: 38944 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 72db3d3d923df2893677badee2be7b06ce55cb7b, data reload: false

------ Round 1 ----------------------------------
q1	17595	4649	4481	4481
q2	1996	179	184	179
q3	10465	1155	1220	1155
q4	10190	763	756	756
q5	7502	2754	2699	2699
q6	213	132	133	132
q7	1014	625	599	599
q8	9242	2091	2087	2087
q9	7392	6668	6597	6597
q10	8576	3573	3565	3565
q11	462	228	227	227
q12	529	222	214	214
q13	17771	2930	2920	2920
q14	270	232	227	227
q15	521	475	476	475
q16	536	418	381	381
q17	977	667	680	667
q18	7472	6935	6729	6729
q19	8165	1534	1538	1534
q20	647	293	293	293
q21	3758	2717	2757	2717
q22	376	313	310	310
Total cold run time: 115669 ms
Total hot run time: 38944 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4333	4243	4204	4204
q2	376	270	279	270
q3	3030	2747	2769	2747
q4	1867	1615	1588	1588
q5	5385	5370	5319	5319
q6	213	123	124	123
q7	2236	1869	1877	1869
q8	3250	3356	3380	3356
q9	8589	8626	8587	8587
q10	3928	3670	3695	3670
q11	579	489	491	489
q12	783	568	596	568
q13	16359	2960	2986	2960
q14	314	279	272	272
q15	511	479	474	474
q16	469	423	424	423
q17	1777	1484	1457	1457
q18	8163	7986	8063	7986
q19	1713	1604	1543	1543
q20	2119	1913	1794	1794
q21	10866	5134	4942	4942
q22	565	504	473	473
Total cold run time: 77425 ms
Total hot run time: 55114 ms

github-actions · 2024-04-23T04:11:49Z

PR approved by at least one committer and no changes requested.

github-actions · 2024-04-23T04:11:50Z

PR approved by anyone and no changes requested.

dataroaring

LGTM

github-actions · 2024-04-23T06:01:23Z

clang-tidy review says "All clean, LGTM! 👍"

ByteYue · 2024-04-23T06:39:12Z

run buildall

github-actions · 2024-04-23T06:44:21Z

clang-tidy review says "All clean, LGTM! 👍"

ByteYue · 2024-04-23T07:15:22Z

run buildall

github-actions · 2024-04-23T07:17:39Z

clang-tidy review says "All clean, LGTM! 👍"

doris-robot · 2024-04-23T08:27:31Z

ClickBench: Total hot run time: 31.11 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 227472d8eb02b22dd1446f838fc371b3046fba2c, data reload: false

query1	0.04	0.04	0.03
query2	0.09	0.05	0.04
query3	0.23	0.05	0.04
query4	1.68	0.06	0.07
query5	0.48	0.50	0.49
query6	1.27	0.92	0.81
query7	0.02	0.02	0.01
query8	0.06	0.04	0.04
query9	0.49	0.45	0.45
query10	0.51	0.52	0.51
query11	0.14	0.11	0.11
query12	0.14	0.10	0.11
query13	0.64	0.64	0.63
query14	0.95	0.99	1.03
query15	0.86	0.86	0.83
query16	0.38	0.35	0.38
query17	1.04	0.99	1.01
query18	0.20	0.24	0.21
query19	1.88	1.81	1.74
query20	0.01	0.01	0.01
query21	15.40	0.65	0.64
query22	4.44	7.41	1.99
query23	18.33	1.29	1.26
query24	1.46	0.37	0.26
query25	0.16	0.09	0.10
query26	0.28	0.17	0.17
query27	0.11	0.10	0.09
query28	13.42	1.02	1.02
query29	12.63	3.43	3.29
query30	0.28	0.07	0.06
query31	2.84	0.39	0.41
query32	3.25	0.49	0.49
query33	2.80	2.79	2.71
query34	17.28	4.49	4.51
query35	4.54	4.52	4.54
query36	0.65	0.48	0.47
query37	0.21	0.18	0.18
query38	0.20	0.19	0.19
query39	0.05	0.05	0.05
query40	0.20	0.15	0.16
query41	0.10	0.06	0.06
query42	0.07	0.05	0.05
query43	0.05	0.05	0.05
Total cold run time: 109.86 s
Total hot run time: 31.11 s

doris-robot · 2024-04-23T08:41:49Z

TeamCity be ut coverage result:
Function Coverage: 35.23% (8916/25306)
Line Coverage: 27.00% (73341/271665)
Region Coverage: 26.17% (37886/144747)
Branch Coverage: 22.99% (19292/83930)
Coverage Report: http://coverage.selectdb-in.cc/coverage/227472d8eb02b22dd1446f838fc371b3046fba2c_227472d8eb02b22dd1446f838fc371b3046fba2c/report/index.html

be/src/common/config.cpp

be/src/runtime/exec_env_init.cpp

github-actions · 2024-04-23T13:52:39Z

clang-tidy review says "All clean, LGTM! 👍"

ByteYue · 2024-04-23T15:00:42Z

run buildall

github-actions · 2024-04-23T15:05:53Z

clang-tidy review says "All clean, LGTM! 👍"

doris-robot · 2024-04-23T16:10:10Z

ClickBench: Total hot run time: 31.12 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 204d0bdcba88f7b995f46974e24e63102ea4a2e4, data reload: false

query1	0.04	0.04	0.02
query2	0.08	0.04	0.03
query3	0.23	0.05	0.05
query4	1.68	0.07	0.07
query5	0.51	0.52	0.49
query6	1.33	0.86	0.83
query7	0.02	0.01	0.01
query8	0.05	0.04	0.04
query9	0.50	0.45	0.45
query10	0.50	0.51	0.50
query11	0.15	0.10	0.11
query12	0.13	0.11	0.11
query13	0.65	0.63	0.64
query14	0.89	0.99	0.98
query15	0.86	0.84	0.85
query16	0.38	0.37	0.37
query17	1.05	1.06	1.00
query18	0.21	0.23	0.24
query19	1.90	1.74	1.78
query20	0.02	0.01	0.02
query21	15.41	0.66	0.66
query22	4.89	6.36	1.79
query23	18.31	1.35	1.28
query24	1.97	0.27	0.23
query25	0.15	0.11	0.09
query26	0.28	0.18	0.17
query27	0.09	0.09	0.10
query28	13.31	1.04	1.01
query29	12.70	3.40	3.31
query30	0.25	0.07	0.06
query31	2.83	0.40	0.40
query32	3.25	0.49	0.49
query33	2.90	2.93	2.77
query34	16.93	4.45	4.76
query35	4.44	4.82	4.64
query36	0.67	0.47	0.49
query37	0.21	0.17	0.17
query38	0.20	0.20	0.18
query39	0.05	0.05	0.05
query40	0.20	0.16	0.17
query41	0.11	0.06	0.06
query42	0.06	0.06	0.05
query43	0.05	0.04	0.05
Total cold run time: 110.44 s
Total hot run time: 31.12 s

doris-robot · 2024-04-23T16:26:08Z

TeamCity be ut coverage result:
Function Coverage: 35.24% (8916/25304)
Line Coverage: 26.99% (73336/271743)
Region Coverage: 26.15% (37867/144821)
Branch Coverage: 22.96% (19282/83966)
Coverage Report: http://coverage.selectdb-in.cc/coverage/204d0bdcba88f7b995f46974e24e63102ea4a2e4_204d0bdcba88f7b995f46974e24e63102ea4a2e4/report/index.html

be/src/runtime/exec_env_init.cpp

ByteYue · 2024-04-24T03:20:43Z

run buildall

github-actions · 2024-04-24T03:26:05Z

clang-tidy review says "All clean, LGTM! 👍"

doris-robot · 2024-04-24T05:02:21Z

TeamCity be ut coverage result:
Function Coverage: 35.21% (8916/25323)
Line Coverage: 26.98% (73340/271856)
Region Coverage: 26.15% (37882/144875)
Branch Coverage: 22.96% (19286/83984)
Coverage Report: http://coverage.selectdb-in.cc/coverage/a9965e8b641cc096ab8b381379365bf7a1fd5428_a9965e8b641cc096ab8b381379365bf7a1fd5428/report/index.html

ByteYue changed the title ~~[chore](config) Add config to control BufferedReader and S3FileWriter's thread pool~~ [chore](config) Add config to control BufferedReader and S3FileWriter's thread pool's min max nums Apr 22, 2024

morningman requested changes Apr 22, 2024

View reviewed changes

ByteYue requested a review from morningman April 22, 2024 12:30

morningman previously approved these changes Apr 23, 2024

View reviewed changes

github-actions bot added the approved Indicates a PR has been approved by one committer. label Apr 23, 2024

github-actions bot added the reviewed label Apr 23, 2024

dataroaring previously approved these changes Apr 23, 2024

View reviewed changes

ByteYue added 3 commits April 23, 2024 13:55

add config

e2ac5c8

add config

6a7f90b

format

24f639b

ByteYue dismissed stale reviews from dataroaring and morningman via 24f639b April 23, 2024 05:55

ByteYue force-pushed the remove_useless_s3_conf branch from 72db3d3 to 24f639b Compare April 23, 2024 05:55

github-actions bot removed the approved Indicates a PR has been approved by one committer. label Apr 23, 2024

consider cpu cores

8917616

correct

227472d

ByteYue requested review from morningman and dataroaring April 23, 2024 07:11

gavinchou requested changes Apr 23, 2024

View reviewed changes

be/src/common/config.cpp Outdated Show resolved Hide resolved

be/src/common/config.cpp Outdated Show resolved Hide resolved

be/src/runtime/exec_env_init.cpp Outdated Show resolved Hide resolved

resolve comment

010b8d1

resolve comment

204d0bd

gavinchou reviewed Apr 23, 2024

View reviewed changes

be/src/runtime/exec_env_init.cpp Show resolved Hide resolved

prevent zero

a9965e8

ByteYue requested a review from gavinchou April 24, 2024 12:10

gavinchou approved these changes Apr 24, 2024

View reviewed changes

platoneko approved these changes Apr 24, 2024

View reviewed changes

dataroaring merged commit 3034ac3 into apache:master Apr 25, 2024
25 of 28 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[chore](config) Add config to control BufferedReader and S3FileWriter's thread pool's min max nums #33974

[chore](config) Add config to control BufferedReader and S3FileWriter's thread pool's min max nums #33974

ByteYue commented Apr 22, 2024 •

edited

Loading

doris-robot commented Apr 22, 2024

ByteYue commented Apr 22, 2024

github-actions bot commented Apr 22, 2024

morningman left a comment

doris-robot commented Apr 22, 2024

doris-robot commented Apr 22, 2024

doris-robot commented Apr 22, 2024

doris-robot commented Apr 22, 2024

ByteYue commented Apr 22, 2024

github-actions bot commented Apr 22, 2024

github-actions bot commented Apr 22, 2024

doris-robot commented Apr 22, 2024

doris-robot commented Apr 22, 2024

github-actions bot commented Apr 23, 2024

github-actions bot commented Apr 23, 2024

dataroaring left a comment

github-actions bot commented Apr 23, 2024

ByteYue commented Apr 23, 2024

github-actions bot commented Apr 23, 2024

ByteYue commented Apr 23, 2024

github-actions bot commented Apr 23, 2024

doris-robot commented Apr 23, 2024

doris-robot commented Apr 23, 2024

github-actions bot commented Apr 23, 2024

ByteYue commented Apr 23, 2024

github-actions bot commented Apr 23, 2024

doris-robot commented Apr 23, 2024

doris-robot commented Apr 23, 2024

ByteYue commented Apr 24, 2024

github-actions bot commented Apr 24, 2024

doris-robot commented Apr 24, 2024

[chore](config) Add config to control BufferedReader and S3FileWriter's thread pool's min max nums #33974

[chore](config) Add config to control BufferedReader and S3FileWriter's thread pool's min max nums #33974

Conversation

ByteYue commented Apr 22, 2024 • edited Loading

Proposed changes

Further comments

doris-robot commented Apr 22, 2024

ByteYue commented Apr 22, 2024

github-actions bot commented Apr 22, 2024

morningman left a comment

Choose a reason for hiding this comment

doris-robot commented Apr 22, 2024

doris-robot commented Apr 22, 2024

doris-robot commented Apr 22, 2024

doris-robot commented Apr 22, 2024

ByteYue commented Apr 22, 2024

github-actions bot commented Apr 22, 2024

github-actions bot commented Apr 22, 2024

doris-robot commented Apr 22, 2024

doris-robot commented Apr 22, 2024

github-actions bot commented Apr 23, 2024

github-actions bot commented Apr 23, 2024

dataroaring left a comment

Choose a reason for hiding this comment

github-actions bot commented Apr 23, 2024

ByteYue commented Apr 23, 2024

github-actions bot commented Apr 23, 2024

ByteYue commented Apr 23, 2024

github-actions bot commented Apr 23, 2024

doris-robot commented Apr 23, 2024

doris-robot commented Apr 23, 2024

github-actions bot commented Apr 23, 2024

ByteYue commented Apr 23, 2024

github-actions bot commented Apr 23, 2024

doris-robot commented Apr 23, 2024

doris-robot commented Apr 23, 2024

ByteYue commented Apr 24, 2024

github-actions bot commented Apr 24, 2024

doris-robot commented Apr 24, 2024

ByteYue commented Apr 22, 2024 •

edited

Loading