Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Fix]set workload group for load channel #40749

Merged
merged 1 commit into from
Sep 12, 2024

Conversation

wangbo
Copy link
Contributor

@wangbo wangbo commented Sep 12, 2024

Proposed changes

Set workload group for load channel, make its memory managed by Workload Group.

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@wangbo
Copy link
Contributor Author

wangbo commented Sep 12, 2024

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 43255 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit d6e20188d4b05566a042a12d6c276b7cefed0c68, data reload: false

------ Round 1 ----------------------------------
q1	19245	7475	7259	7259
q2	2044	191	202	191
q3	10463	1406	1421	1406
q4	10151	934	1020	934
q5	7719	3220	3183	3183
q6	246	155	157	155
q7	1054	630	634	630
q8	9468	2024	2053	2024
q9	6899	6323	6345	6323
q10	7033	2527	2526	2526
q11	432	268	263	263
q12	419	234	234	234
q13	17758	3083	3067	3067
q14	289	245	252	245
q15	593	560	532	532
q16	528	432	434	432
q17	1011	959	968	959
q18	7399	6714	6763	6714
q19	1386	1247	1242	1242
q20	617	362	341	341
q21	3980	3602	3597	3597
q22	1073	998	1038	998
Total cold run time: 109807 ms
Total hot run time: 43255 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7192	7228	7205	7205
q2	337	230	228	228
q3	2944	2930	2927	2927
q4	2011	1990	1976	1976
q5	5510	5429	5407	5407
q6	237	148	150	148
q7	2071	1683	1701	1683
q8	3258	3315	3370	3315
q9	8464	8473	8482	8473
q10	3419	3454	3457	3454
q11	581	475	473	473
q12	798	577	568	568
q13	6957	3076	3062	3062
q14	306	275	284	275
q15	574	521	522	521
q16	496	446	461	446
q17	1760	1718	1708	1708
q18	7999	7542	7527	7527
q19	1739	1700	1698	1698
q20	2059	1809	1800	1800
q21	5729	5574	5451	5451
q22	1104	994	1005	994
Total cold run time: 65545 ms
Total hot run time: 59339 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 195647 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit d6e20188d4b05566a042a12d6c276b7cefed0c68, data reload: false

query1	925	379	403	379
query2	6532	1794	1838	1794
query3	6663	212	220	212
query4	27944	24305	23840	23840
query5	5153	577	527	527
query6	271	183	174	174
query7	4590	302	303	302
query8	296	241	231	231
query9	8487	2610	2602	2602
query10	449	282	281	281
query11	16152	15639	15652	15639
query12	167	105	102	102
query13	1687	395	380	380
query14	11772	6883	7206	6883
query15	217	177	174	174
query16	7619	473	470	470
query17	1520	571	553	553
query18	2096	299	282	282
query19	200	150	147	147
query20	118	119	110	110
query21	212	102	105	102
query22	4274	4218	4156	4156
query23	34386	33822	33675	33675
query24	9319	3100	3091	3091
query25	642	383	386	383
query26	1111	156	160	156
query27	2323	284	288	284
query28	6499	2118	2097	2097
query29	887	418	409	409
query30	299	161	153	153
query31	992	781	802	781
query32	104	55	60	55
query33	722	299	297	297
query34	928	491	484	484
query35	903	722	739	722
query36	1042	940	918	918
query37	172	84	84	84
query38	4038	4017	4010	4010
query39	1463	1419	1410	1410
query40	210	115	112	112
query41	48	47	45	45
query42	126	99	98	98
query43	497	437	451	437
query44	1244	807	779	779
query45	198	167	171	167
query46	1120	834	828	828
query47	1917	1792	1796	1792
query48	364	282	286	282
query49	1113	448	453	448
query50	917	450	458	450
query51	7204	6866	6916	6866
query52	101	89	89	89
query53	262	192	191	191
query54	793	463	459	459
query55	78	78	77	77
query56	283	272	301	272
query57	1195	1114	1104	1104
query58	244	250	234	234
query59	2865	2668	2732	2668
query60	321	267	277	267
query61	107	99	105	99
query62	936	685	689	685
query63	228	192	199	192
query64	4545	673	664	664
query65	3270	3201	3185	3185
query66	1422	296	321	296
query67	16165	15637	15416	15416
query68	3168	866	856	856
query69	440	336	350	336
query70	1191	1160	1165	1160
query71	366	351	356	351
query72	6035	3523	3548	3523
query73	597	592	587	587
query74	9334	9154	9069	9069
query75	3196	3033	3026	3026
query76	1971	868	860	860
query77	452	420	420	420
query78	9305	9288	9354	9288
query79	923	901	900	900
query80	898	868	858	858
query81	454	271	273	271
query82	273	270	272	270
query83	196	201	202	201
query84	238	119	112	112
query85	714	462	457	457
query86	322	307	312	307
query87	4530	4326	4551	4326
query88	4192	4115	4077	4077
query89	382	377	381	377
query90	1278	324	321	321
query91	124	128	126	126
query92	91	75	75	75
query93	1065	1066	1058	1058
query94	618	384	393	384
query95	454	416	424	416
query96	475	475	477	475
query97	3144	3101	3159	3101
query98	232	234	230	230
query99	1566	1278	1322	1278
Total cold run time: 279171 ms
Total hot run time: 195647 ms

Copy link
Contributor

@yiguolei yiguolei left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@yiguolei
Copy link
Contributor

run buildall

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Sep 12, 2024
Copy link
Contributor

PR approved by at least one committer and no changes requested.

Copy link
Contributor

PR approved by anyone and no changes requested.

@doris-robot
Copy link

TPC-H: Total hot run time: 43377 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 9a6bfb2a04b6666f72212d72edaf3e0cd1e87e73, data reload: false

------ Round 1 ----------------------------------
q1	18013	7494	7404	7404
q2	2342	189	191	189
q3	10470	1392	1432	1392
q4	10157	985	985	985
q5	7726	3219	3168	3168
q6	243	155	160	155
q7	1056	627	626	626
q8	9474	2069	2043	2043
q9	6916	6317	6330	6317
q10	7048	2724	2555	2555
q11	432	261	258	258
q12	419	237	236	236
q13	17755	3043	3020	3020
q14	287	257	262	257
q15	578	530	544	530
q16	531	434	429	429
q17	1008	958	961	958
q18	7717	6830	6705	6705
q19	1382	1252	1244	1244
q20	626	349	324	324
q21	3919	3578	3556	3556
q22	1069	1026	1034	1026
Total cold run time: 109168 ms
Total hot run time: 43377 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7220	7238	7228	7228
q2	344	232	238	232
q3	2947	2918	2924	2918
q4	1986	1991	2016	1991
q5	5506	5428	5416	5416
q6	235	153	154	153
q7	2054	1695	1655	1655
q8	3259	3369	3346	3346
q9	8444	8480	8518	8480
q10	3423	3483	3467	3467
q11	571	475	482	475
q12	783	571	576	571
q13	6640	3075	3034	3034
q14	304	274	271	271
q15	567	529	524	524
q16	511	461	461	461
q17	1780	1729	1734	1729
q18	7938	7545	7672	7545
q19	1739	1716	1712	1712
q20	2042	1787	1833	1787
q21	5756	5428	5515	5428
q22	1058	1004	1009	1004
Total cold run time: 65107 ms
Total hot run time: 59427 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 196021 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 9a6bfb2a04b6666f72212d72edaf3e0cd1e87e73, data reload: false

query1	912	381	394	381
query2	6545	1841	1815	1815
query3	6672	213	223	213
query4	26235	24325	24012	24012
query5	5381	551	543	543
query6	286	178	181	178
query7	4610	302	309	302
query8	303	228	239	228
query9	8508	2617	2562	2562
query10	478	283	288	283
query11	16276	15749	15540	15540
query12	164	100	104	100
query13	1682	427	387	387
query14	11021	7364	7175	7175
query15	220	169	179	169
query16	7642	483	473	473
query17	1577	559	553	553
query18	2109	284	292	284
query19	191	145	143	143
query20	131	113	113	113
query21	212	107	103	103
query22	4616	4417	4193	4193
query23	34590	33805	33559	33559
query24	9545	3091	3078	3078
query25	658	387	400	387
query26	1155	157	180	157
query27	2200	277	283	277
query28	6466	2109	2064	2064
query29	908	411	414	411
query30	296	167	157	157
query31	988	777	784	777
query32	97	55	56	55
query33	732	301	295	295
query34	917	484	488	484
query35	886	757	734	734
query36	1066	936	887	887
query37	179	81	83	81
query38	4022	3946	3901	3901
query39	1443	1458	1403	1403
query40	210	114	115	114
query41	49	47	46	46
query42	120	98	100	98
query43	502	463	450	450
query44	1256	815	773	773
query45	200	168	173	168
query46	1116	849	795	795
query47	1908	1808	1824	1808
query48	368	296	297	296
query49	1117	448	441	441
query50	943	433	438	433
query51	7026	6910	6920	6910
query52	104	87	91	87
query53	262	185	183	183
query54	840	461	469	461
query55	78	80	76	76
query56	293	293	279	279
query57	1194	1099	1094	1094
query58	251	243	253	243
query59	3053	2701	2630	2630
query60	300	287	287	287
query61	130	101	102	101
query62	926	668	679	668
query63	227	188	187	187
query64	5312	696	651	651
query65	3234	3165	3164	3164
query66	1400	303	319	303
query67	16068	15533	15471	15471
query68	3184	874	857	857
query69	431	326	333	326
query70	1156	1154	1143	1143
query71	361	351	350	350
query72	5917	3520	3487	3487
query73	612	597	591	591
query74	9166	9213	9243	9213
query75	3144	3056	3050	3050
query76	1897	865	857	857
query77	454	415	413	413
query78	9574	9229	9263	9229
query79	928	892	886	886
query80	936	867	857	857
query81	454	273	274	273
query82	274	272	267	267
query83	205	198	203	198
query84	238	113	113	113
query85	723	473	461	461
query86	312	337	318	318
query87	4562	4441	4325	4325
query88	4283	4215	4227	4215
query89	387	393	396	393
query90	1304	327	327	327
query91	143	150	271	150
query92	80	76	74	74
query93	1090	1088	1066	1066
query94	610	392	401	392
query95	482	419	434	419
query96	489	480	480	480
query97	3116	3092	3167	3092
query98	236	231	225	225
query99	1595	1315	1300	1300
Total cold run time: 278749 ms
Total hot run time: 196021 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.64 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 9a6bfb2a04b6666f72212d72edaf3e0cd1e87e73, data reload: false

query1	0.05	0.05	0.04
query2	0.08	0.04	0.04
query3	0.22	0.04	0.04
query4	1.69	0.06	0.07
query5	0.50	0.49	0.50
query6	1.14	0.73	0.72
query7	0.02	0.01	0.02
query8	0.06	0.06	0.05
query9	0.56	0.52	0.50
query10	0.57	0.56	0.56
query11	0.16	0.12	0.12
query12	0.16	0.12	0.12
query13	0.62	0.61	0.60
query14	1.46	1.46	1.45
query15	0.90	0.86	0.86
query16	0.37	0.37	0.36
query17	1.04	1.05	1.02
query18	0.21	0.21	0.20
query19	1.95	1.86	1.81
query20	0.01	0.01	0.01
query21	15.40	0.66	0.65
query22	4.35	7.93	1.01
query23	17.80	1.27	1.27
query24	2.28	0.23	0.22
query25	0.19	0.08	0.08
query26	0.29	0.19	0.18
query27	0.07	0.07	0.07
query28	13.19	1.14	1.10
query29	12.57	3.33	3.35
query30	0.24	0.06	0.05
query31	2.88	0.42	0.41
query32	3.23	0.49	0.49
query33	3.04	3.03	3.10
query34	15.46	4.33	4.30
query35	4.36	4.32	4.32
query36	0.68	0.49	0.47
query37	0.18	0.16	0.16
query38	0.16	0.14	0.15
query39	0.04	0.04	0.04
query40	0.16	0.14	0.13
query41	0.10	0.05	0.05
query42	0.06	0.04	0.05
query43	0.05	0.04	0.04
Total cold run time: 108.55 s
Total hot run time: 30.64 s

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 36.89% (9463/25649)
Line Coverage: 28.26% (77820/275386)
Region Coverage: 27.66% (40182/145261)
Branch Coverage: 24.27% (20421/84128)
Coverage Report: http://coverage.selectdb-in.cc/coverage/9a6bfb2a04b6666f72212d72edaf3e0cd1e87e73_9a6bfb2a04b6666f72212d72edaf3e0cd1e87e73/report/index.html

Copy link
Contributor

@xinyiZzz xinyiZzz left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@yiguolei yiguolei merged commit d8f911e into apache:master Sep 12, 2024
23 of 28 checks passed
dataroaring pushed a commit that referenced this pull request Oct 9, 2024
## Proposed changes
Set workload group for load channel, make its memory managed by Workload
Group.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by one committer. reviewed
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants