[enhancement](compaction) optimize the cpu consumption of the compaction task producer thread #40152

luwei16 · 2024-08-29T15:25:23Z

No description provided.

…ion task producer thread

doris-robot · 2024-08-29T15:25:29Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

luwei16 · 2024-08-29T15:27:22Z

run buildall

github-actions · 2024-08-29T15:32:14Z

clang-tidy review says "All clean, LGTM! 👍"

github-actions · 2024-08-29T15:34:05Z

clang-tidy review says "All clean, LGTM! 👍"

zclllyybb · 2024-08-29T16:10:27Z

改了config默认值，记得提文档pr

doris-robot · 2024-08-29T16:12:10Z

TPC-H: Total hot run time: 38796 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 498cd2a778d41c9610f6b1897c01d9bdf0a1b944, data reload: false

------ Round 1 ----------------------------------
q1	17993	4608	4386	4386
q2	2030	193	176	176
q3	10466	1142	1172	1142
q4	10150	698	795	698
q5	7751	2958	2865	2865
q6	230	138	140	138
q7	977	615	596	596
q8	9334	2123	2109	2109
q9	7192	6624	6626	6624
q10	7025	2248	2221	2221
q11	513	240	245	240
q12	412	223	221	221
q13	18034	3081	3133	3081
q14	283	242	231	231
q15	538	494	499	494
q16	572	510	516	510
q17	993	692	702	692
q18	7461	6991	6959	6959
q19	1408	1095	1122	1095
q20	699	342	340	340
q21	4087	2968	3194	2968
q22	1094	1029	1010	1010
Total cold run time: 109242 ms
Total hot run time: 38796 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4420	4348	4334	4334
q2	383	284	275	275
q3	2947	2661	2709	2661
q4	1944	1638	1619	1619
q5	5525	5474	5537	5474
q6	228	133	131	131
q7	2176	1781	1741	1741
q8	3251	3402	3405	3402
q9	8588	8559	8476	8476
q10	3479	3314	3251	3251
q11	623	504	487	487
q12	789	619	611	611
q13	10499	3126	3087	3087
q14	309	272	292	272
q15	536	482	478	478
q16	645	558	550	550
q17	1815	1497	1481	1481
q18	8064	7565	7625	7565
q19	1695	1486	1564	1486
q20	2072	1831	1858	1831
q21	5488	5266	5341	5266
q22	1143	1040	1038	1038
Total cold run time: 66619 ms
Total hot run time: 55516 ms

doris-robot · 2024-08-29T16:23:53Z

TPC-DS: Total hot run time: 188729 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 498cd2a778d41c9610f6b1897c01d9bdf0a1b944, data reload: false

query1	910	380	372	372
query2	6454	1998	1930	1930
query3	6651	224	229	224
query4	34278	23254	23279	23254
query5	4181	491	485	485
query6	249	168	176	168
query7	4580	298	290	290
query8	254	197	195	195
query9	8463	2504	2490	2490
query10	438	271	267	267
query11	17843	15130	15074	15074
query12	146	103	100	100
query13	1621	398	371	371
query14	9779	7181	6839	6839
query15	273	168	171	168
query16	7902	489	431	431
query17	1608	585	577	577
query18	1928	302	302	302
query19	206	151	152	151
query20	120	110	118	110
query21	216	105	110	105
query22	4468	4340	4287	4287
query23	34176	33600	33563	33563
query24	11230	2934	2864	2864
query25	637	378	402	378
query26	1119	156	157	156
query27	2537	282	285	282
query28	7444	2147	2124	2124
query29	793	412	396	396
query30	305	152	157	152
query31	971	755	804	755
query32	98	59	58	58
query33	770	292	286	286
query34	1001	476	535	476
query35	840	728	728	728
query36	1080	953	933	933
query37	157	96	85	85
query38	4029	3808	3844	3808
query39	1465	1401	1389	1389
query40	196	118	116	116
query41	47	50	43	43
query42	117	95	99	95
query43	517	477	455	455
query44	1215	765	756	756
query45	199	168	169	168
query46	1102	770	736	736
query47	1892	1831	1810	1810
query48	378	314	295	295
query49	1094	426	432	426
query50	821	410	407	407
query51	7202	7116	7097	7097
query52	126	87	85	85
query53	257	193	180	180
query54	981	451	470	451
query55	76	74	78	74
query56	292	261	253	253
query57	1173	1075	1099	1075
query58	233	230	229	229
query59	3043	2781	2992	2781
query60	298	263	270	263
query61	108	96	102	96
query62	835	656	663	656
query63	227	185	186	185
query64	4771	681	658	658
query65	3205	3128	3142	3128
query66	1396	332	335	332
query67	15776	15581	15464	15464
query68	3300	591	579	579
query69	388	291	283	283
query70	1210	1138	1131	1131
query71	328	276	275	275
query72	6457	4049	3917	3917
query73	760	335	337	335
query74	9191	8794	8938	8794
query75	3399	2693	2700	2693
query76	1884	1014	982	982
query77	499	327	333	327
query78	9605	9176	9093	9093
query79	1030	545	534	534
query80	689	515	504	504
query81	452	233	238	233
query82	249	154	143	143
query83	174	156	152	152
query84	223	81	75	75
query85	669	288	277	277
query86	304	301	250	250
query87	4409	4291	4366	4291
query88	2948	2374	2463	2374
query89	381	294	282	282
query90	1793	189	193	189
query91	125	101	98	98
query92	64	50	54	50
query93	1038	553	543	543
query94	696	310	302	302
query95	342	263	262	262
query96	594	268	278	268
query97	3229	3079	3100	3079
query98	220	204	211	204
query99	1473	1281	1273	1273
Total cold run time: 286920 ms
Total hot run time: 188729 ms

doris-robot · 2024-08-29T16:29:24Z

ClickBench: Total hot run time: 31.81 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 498cd2a778d41c9610f6b1897c01d9bdf0a1b944, data reload: false

query1	0.05	0.04	0.04
query2	0.08	0.04	0.04
query3	0.23	0.06	0.05
query4	1.68	0.08	0.07
query5	0.50	0.48	0.49
query6	1.13	0.74	0.72
query7	0.02	0.02	0.02
query8	0.06	0.05	0.05
query9	0.53	0.49	0.48
query10	0.55	0.53	0.53
query11	0.15	0.12	0.12
query12	0.15	0.12	0.12
query13	0.61	0.58	0.58
query14	2.09	2.05	2.06
query15	0.89	0.83	0.81
query16	0.36	0.34	0.38
query17	1.05	0.98	1.00
query18	0.21	0.20	0.20
query19	1.82	1.71	1.74
query20	0.01	0.01	0.01
query21	15.40	0.67	0.66
query22	4.40	7.13	1.84
query23	18.28	1.38	1.34
query24	2.10	0.22	0.22
query25	0.15	0.09	0.08
query26	0.28	0.19	0.18
query27	0.08	0.07	0.08
query28	13.24	1.02	1.00
query29	12.61	3.33	3.36
query30	0.24	0.05	0.06
query31	2.87	0.40	0.39
query32	3.27	0.49	0.48
query33	2.97	2.96	3.00
query34	17.10	4.36	4.35
query35	4.50	4.40	4.44
query36	0.67	0.47	0.47
query37	0.19	0.16	0.15
query38	0.15	0.16	0.15
query39	0.04	0.03	0.04
query40	0.15	0.12	0.13
query41	0.10	0.05	0.05
query42	0.06	0.05	0.05
query43	0.05	0.04	0.04
Total cold run time: 111.07 s
Total hot run time: 31.81 s

…-compaction-task

luwei16 · 2024-09-05T14:57:47Z

run buildall

…-compaction-task

dataroaring · 2024-09-10T16:04:03Z

run buildall

…-compaction-task

luwei16 · 2024-09-11T09:45:07Z

run buildall

doris-robot · 2024-09-11T11:25:11Z

TeamCity be ut coverage result:
Function Coverage: 36.89% (9455/25629)
Line Coverage: 28.25% (77764/275307)
Region Coverage: 27.64% (40135/145191)
Branch Coverage: 24.26% (20398/84076)
Coverage Report: http://coverage.selectdb-in.cc/coverage/48680fe800a66dedcbdf78aa1c9bebd02718c752_48680fe800a66dedcbdf78aa1c9bebd02718c752/report/index.html

doris-robot · 2024-09-11T11:26:05Z

TPC-H: Total hot run time: 38645 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 48680fe800a66dedcbdf78aa1c9bebd02718c752, data reload: false

------ Round 1 ----------------------------------
q1	18284	4856	4397	4397
q2	2045	189	188	188
q3	11373	993	1125	993
q4	10314	694	697	694
q5	7750	2872	2871	2871
q6	234	141	138	138
q7	962	623	596	596
q8	9327	2134	2122	2122
q9	7401	6635	6681	6635
q10	7032	2264	2219	2219
q11	482	242	236	236
q12	405	228	226	226
q13	18984	3127	3144	3127
q14	292	230	246	230
q15	533	517	481	481
q16	553	436	421	421
q17	1016	784	824	784
q18	7464	7087	6932	6932
q19	1402	1117	1117	1117
q20	703	324	332	324
q21	4422	3190	2885	2885
q22	1137	1036	1029	1029
Total cold run time: 112115 ms
Total hot run time: 38645 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4449	4413	4328	4328
q2	385	285	279	279
q3	2906	2722	2688	2688
q4	1922	1692	1707	1692
q5	5747	5747	5853	5747
q6	235	133	134	133
q7	2261	1824	1839	1824
q8	3316	3540	3553	3540
q9	8946	8927	8936	8927
q10	3697	3430	3390	3390
q11	620	520	516	516
q12	836	660	646	646
q13	13234	3160	3337	3160
q14	311	300	292	292
q15	544	510	483	483
q16	553	489	492	489
q17	1851	1537	1551	1537
q18	8434	7884	7911	7884
q19	1805	1631	1553	1553
q20	2166	1969	1931	1931
q21	5820	5742	5842	5742
q22	1096	1051	1036	1036
Total cold run time: 71134 ms
Total hot run time: 57817 ms

doris-robot · 2024-09-11T11:38:26Z

TPC-DS: Total hot run time: 197981 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 48680fe800a66dedcbdf78aa1c9bebd02718c752, data reload: false

query1	1262	893	856	856
query2	6364	1951	1974	1951
query3	10669	4259	3822	3822
query4	59765	27733	23472	23472
query5	5033	518	505	505
query6	388	166	165	165
query7	5636	303	292	292
query8	322	224	226	224
query9	7540	2505	2507	2505
query10	426	286	277	277
query11	17519	15170	15512	15170
query12	187	108	103	103
query13	1444	412	415	412
query14	10725	7125	7719	7125
query15	212	182	176	176
query16	6768	470	466	466
query17	1108	563	582	563
query18	1478	292	292	292
query19	203	151	146	146
query20	120	118	112	112
query21	223	110	107	107
query22	4708	4624	4786	4624
query23	34284	33574	33547	33547
query24	6029	2919	2899	2899
query25	502	381	387	381
query26	602	150	153	150
query27	1593	276	288	276
query28	3589	2075	2042	2042
query29	639	404	414	404
query30	224	158	147	147
query31	936	762	775	762
query32	67	55	51	51
query33	435	284	292	284
query34	891	475	471	471
query35	860	751	715	715
query36	1085	926	956	926
query37	142	88	85	85
query38	4118	3987	3932	3932
query39	1469	1391	1407	1391
query40	192	116	114	114
query41	46	47	45	45
query42	113	93	93	93
query43	523	470	476	470
query44	1099	764	761	761
query45	197	171	171	171
query46	1141	744	739	739
query47	1908	1809	1830	1809
query48	370	290	304	290
query49	767	444	441	441
query50	851	423	419	419
query51	7071	6961	7004	6961
query52	98	86	85	85
query53	250	182	177	177
query54	569	454	448	448
query55	78	73	73	73
query56	280	267	263	263
query57	1203	1073	1098	1073
query58	226	225	241	225
query59	3236	2884	2755	2755
query60	297	275	264	264
query61	105	101	97	97
query62	756	640	697	640
query63	226	186	188	186
query64	1369	679	684	679
query65	3265	3134	3192	3134
query66	623	334	332	332
query67	15850	15585	15272	15272
query68	1419	551	533	533
query69	391	262	273	262
query70	1240	1138	1013	1013
query71	343	283	274	274
query72	4620	4005	4047	4005
query73	763	328	333	328
query74	9294	9038	8949	8949
query75	3381	2694	2761	2694
query76	1349	971	955	955
query77	531	314	316	314
query78	9854	9703	9334	9334
query79	973	888	865	865
query80	990	819	814	814
query81	528	260	255	255
query82	1367	267	267	267
query83	248	192	188	188
query84	253	106	104	104
query85	702	394	379	379
query86	322	304	320	304
query87	4412	4455	4375	4375
query88	4590	4231	4241	4231
query89	392	376	369	369
query90	1914	317	319	317
query91	128	126	120	120
query92	77	75	78	75
query93	950	943	928	928
query94	704	379	441	379
query95	440	413	407	407
query96	480	482	481	481
query97	3149	3120	3164	3120
query98	234	241	218	218
query99	1495	1280	1316	1280
Total cold run time: 299807 ms
Total hot run time: 197981 ms

doris-robot · 2024-09-11T11:43:57Z

ClickBench: Total hot run time: 31.56 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 48680fe800a66dedcbdf78aa1c9bebd02718c752, data reload: false

query1	0.05	0.04	0.04
query2	0.07	0.04	0.04
query3	0.22	0.05	0.05
query4	1.68	0.07	0.07
query5	0.48	0.51	0.50
query6	1.12	0.73	0.72
query7	0.02	0.01	0.02
query8	0.06	0.04	0.05
query9	0.55	0.48	0.49
query10	0.54	0.55	0.55
query11	0.16	0.11	0.11
query12	0.15	0.13	0.12
query13	0.61	0.59	0.59
query14	1.43	1.42	1.44
query15	0.85	0.83	0.83
query16	0.37	0.37	0.38
query17	1.07	1.05	1.06
query18	0.19	0.17	0.18
query19	1.94	1.83	1.83
query20	0.02	0.01	0.00
query21	15.39	0.67	0.66
query22	4.02	6.82	1.90
query23	18.27	1.33	1.30
query24	2.07	0.22	0.22
query25	0.15	0.09	0.07
query26	0.25	0.18	0.18
query27	0.08	0.07	0.08
query28	13.30	1.04	1.02
query29	12.63	3.35	3.35
query30	0.24	0.06	0.06
query31	2.86	0.39	0.40
query32	3.26	0.48	0.48
query33	3.00	3.02	3.02
query34	17.09	4.38	4.39
query35	4.50	4.42	4.46
query36	0.66	0.47	0.48
query37	0.19	0.15	0.17
query38	0.16	0.14	0.15
query39	0.05	0.04	0.04
query40	0.17	0.13	0.13
query41	0.10	0.04	0.06
query42	0.06	0.05	0.05
query43	0.05	0.03	0.04
Total cold run time: 110.13 s
Total hot run time: 31.56 s

luwei16 · 2024-09-13T08:17:14Z

run cloud_p0

…-compaction-task

luwei16 · 2024-09-24T08:58:32Z

run buildall

doris-robot · 2024-09-24T10:55:53Z

TeamCity be ut coverage result:
Function Coverage: 37.31% (9624/25792)
Line Coverage: 28.72% (79647/277315)
Region Coverage: 28.17% (41186/146219)
Branch Coverage: 24.80% (20989/84640)
Coverage Report: http://coverage.selectdb-in.cc/coverage/e36cf49509bf0640373163861038cb6cd3d705b6_e36cf49509bf0640373163861038cb6cd3d705b6/report/index.html

dataroaring

LGTM

github-actions · 2024-09-25T03:21:05Z

PR approved by at least one committer and no changes requested.

github-actions · 2024-09-25T03:21:07Z

PR approved by anyone and no changes requested.

luwei16 · 2024-09-25T09:53:00Z

run cloud_p0

dataroaring · 2024-09-26T14:53:58Z

run buildall

dataroaring

LGTM

doris-robot · 2024-09-26T16:43:46Z

TeamCity be ut coverage result:
Function Coverage: 37.31% (9637/25828)
Line Coverage: 28.72% (79750/277726)
Region Coverage: 28.13% (41215/146533)
Branch Coverage: 24.76% (20991/84786)
Coverage Report: http://coverage.selectdb-in.cc/coverage/d969ee2e5462f6e380980b7d164cbd111414a1a8_d969ee2e5462f6e380980b7d164cbd111414a1a8/report/index.html

…-compaction-task-new

luwei16 · 2024-10-09T09:30:47Z

run buildall

github-actions

clang-tidy made some suggestions

github-actions · 2024-10-09T09:34:25Z

be/test/olap/compaction_score_test.cpp

+// specific language governing permissions and limitations
+// under the License.
+
+#include <gmock/gmock-actions.h>


warning: 'gmock/gmock-actions.h' file not found [clang-diagnostic-error]

#include <gmock/gmock-actions.h> ^

doris-robot · 2024-10-09T11:24:46Z

TeamCity be ut coverage result:
Function Coverage: 37.29% (9636/25840)
Line Coverage: 28.69% (79951/278630)
Region Coverage: 28.11% (41299/146945)
Branch Coverage: 24.73% (21047/85094)
Coverage Report: http://coverage.selectdb-in.cc/coverage/0b568ae65263216b52c4a3a718a3531f28fcc088_0b568ae65263216b52c4a3a718a3531f28fcc088/report/index.html

luwei16 · 2024-10-10T07:16:46Z

run p0

dataroaring

LGTM

github-actions · 2024-10-10T14:24:05Z

PR approved by at least one committer and no changes requested.

[enhancement](compaction) optimize the CPU consumption of the compact…

230426f

…ion task producer thread

fix

498cd2a

luwei16 added 4 commits September 4, 2024 21:16

Merge remote-tracking branch 'doris-origin/master' into luwei/improve…

5a7c5dc

…-compaction-task

Merge remote-tracking branch 'doris-origin/master' into luwei/improve…

8b45bb0

…-compaction-task

fix

06fbaf1

fix code style

7a72197

luwei16 changed the title ~~[enhancement](compaction) optimize the CPU consumption of the compaction task producer thread~~ [enhancement](compaction) optimize the cpu consumption of the compaction task producer thread Sep 10, 2024

Merge remote-tracking branch 'doris-origin/master' into luwei/improve…

ba03c96

…-compaction-task

luwei16 added 2 commits September 11, 2024 17:41

fix

e09c7d6

Merge remote-tracking branch 'doris-origin/master' into luwei/improve…

48680fe

…-compaction-task

luwei16 added 4 commits September 22, 2024 22:30

Merge remote-tracking branch 'doris-origin/master' into luwei/improve…

755982c

…-compaction-task

fix

9dcce8e

fix

05b8ad6

add ut

3cd5c11

fix

e36cf49

dataroaring previously approved these changes Sep 25, 2024

View reviewed changes

github-actions bot added the approved Indicates a PR has been approved by one committer. label Sep 25, 2024

github-actions bot added the reviewed label Sep 25, 2024

Yukang-Lian approved these changes Sep 25, 2024

View reviewed changes

Merge branch 'master' into luwei/improve-compaction-task

d969ee2

dataroaring dismissed their stale review via d969ee2 September 26, 2024 14:53

dataroaring previously approved these changes Sep 26, 2024

View reviewed changes

Merge remote-tracking branch 'doris-origin/master' into luwei/improve…

0b568ae

…-compaction-task-new

luwei16 dismissed dataroaring’s stale review via 0b568ae October 9, 2024 09:29

github-actions bot removed the approved Indicates a PR has been approved by one committer. label Oct 9, 2024

github-actions bot reviewed Oct 9, 2024

View reviewed changes

dataroaring approved these changes Oct 10, 2024

View reviewed changes

github-actions bot added the approved Indicates a PR has been approved by one committer. label Oct 10, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[enhancement](compaction) optimize the cpu consumption of the compaction task producer thread #40152

[enhancement](compaction) optimize the cpu consumption of the compaction task producer thread #40152

luwei16 commented Aug 29, 2024

doris-robot commented Aug 29, 2024

luwei16 commented Aug 29, 2024

github-actions bot commented Aug 29, 2024

github-actions bot commented Aug 29, 2024

zclllyybb commented Aug 29, 2024

doris-robot commented Aug 29, 2024

doris-robot commented Aug 29, 2024

doris-robot commented Aug 29, 2024

luwei16 commented Sep 5, 2024

dataroaring commented Sep 10, 2024

luwei16 commented Sep 11, 2024

doris-robot commented Sep 11, 2024

doris-robot commented Sep 11, 2024

doris-robot commented Sep 11, 2024

doris-robot commented Sep 11, 2024

luwei16 commented Sep 13, 2024

luwei16 commented Sep 24, 2024

doris-robot commented Sep 24, 2024

dataroaring left a comment

github-actions bot commented Sep 25, 2024

github-actions bot commented Sep 25, 2024

luwei16 commented Sep 25, 2024

dataroaring commented Sep 26, 2024

dataroaring left a comment

doris-robot commented Sep 26, 2024

luwei16 commented Oct 9, 2024

github-actions bot left a comment

github-actions bot Oct 9, 2024

doris-robot commented Oct 9, 2024

luwei16 commented Oct 10, 2024

dataroaring left a comment

github-actions bot commented Oct 10, 2024

[enhancement](compaction) optimize the cpu consumption of the compaction task producer thread #40152

Are you sure you want to change the base?

[enhancement](compaction) optimize the cpu consumption of the compaction task producer thread #40152

Conversation

luwei16 commented Aug 29, 2024

doris-robot commented Aug 29, 2024

luwei16 commented Aug 29, 2024

github-actions bot commented Aug 29, 2024

github-actions bot commented Aug 29, 2024

zclllyybb commented Aug 29, 2024

doris-robot commented Aug 29, 2024

doris-robot commented Aug 29, 2024

doris-robot commented Aug 29, 2024

luwei16 commented Sep 5, 2024

dataroaring commented Sep 10, 2024

luwei16 commented Sep 11, 2024

doris-robot commented Sep 11, 2024

doris-robot commented Sep 11, 2024

doris-robot commented Sep 11, 2024

doris-robot commented Sep 11, 2024

luwei16 commented Sep 13, 2024

luwei16 commented Sep 24, 2024

doris-robot commented Sep 24, 2024

dataroaring left a comment

Choose a reason for hiding this comment

github-actions bot commented Sep 25, 2024

github-actions bot commented Sep 25, 2024

luwei16 commented Sep 25, 2024

dataroaring commented Sep 26, 2024

dataroaring left a comment

Choose a reason for hiding this comment

doris-robot commented Sep 26, 2024

luwei16 commented Oct 9, 2024

github-actions bot left a comment

Choose a reason for hiding this comment

github-actions bot Oct 9, 2024

Choose a reason for hiding this comment

doris-robot commented Oct 9, 2024

luwei16 commented Oct 10, 2024

dataroaring left a comment

Choose a reason for hiding this comment

github-actions bot commented Oct 10, 2024