Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Feature](executor)Insert select limited by WorkloadGroup #30610

Merged
merged 1 commit into from
Jan 31, 2024

Conversation

wangbo
Copy link
Contributor

@wangbo wangbo commented Jan 31, 2024

Proposed changes

Insert select limited by WorkloadGroup;
It works only when enable_memtable_on_sink_node = true.

Test

sql:
set enable_memtable_on_sink_node = true; (required)
set experimental_enable_nereids_dml_with_pipeline = true;
set experimental_enable_pipeline_engine = true;

insert into hits_10w_dst select * from hits;

Test in a 96 core machine.
no limit cpu usage:
img_v3_027k_96ea0473-1a82-4bcd-a071-a6acd25098eg

set cpu hard limit to 10%
28a7c2f2-cb99-4395-8cc0-cc372c685e70

@wangbo
Copy link
Contributor Author

wangbo commented Jan 31, 2024

run buildall

@wangbo wangbo changed the title [Feature](executpr)Insert select limited by WorkloadGroup [Feature](executor)Insert select limited by WorkloadGroup Jan 31, 2024
Copy link
Contributor

@github-actions github-actions bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

clang-tidy made some suggestions

be/src/olap/memtable_flush_executor.cpp Show resolved Hide resolved
@doris-robot
Copy link

TPC-H: Total hot run time: 37173 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 51515e383b842a687c35220bc041a6cd15482af2, data reload: false

------ Round 1 ----------------------------------
q1	17698	4576	4444	4444
q2	2042	134	130	130
q3	10602	953	910	910
q4	4656	796	777	777
q5	7677	2958	2951	2951
q6	184	119	119	119
q7	1070	723	717	717
q8	9275	2038	2005	2005
q9	7286	6366	6305	6305
q10	8115	2447	2399	2399
q11	414	209	207	207
q12	744	275	274	274
q13	18037	3301	3339	3301
q14	266	248	256	248
q15	538	497	489	489
q16	493	412	403	403
q17	951	486	511	486
q18	7051	6475	6040	6040
q19	1590	1473	1371	1371
q20	616	323	329	323
q21	6905	2990	3127	2990
q22	798	302	284	284
Total cold run time: 107008 ms
Total hot run time: 37173 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4505	4463	4356	4356
q2	333	240	238	238
q3	3004	2854	2846	2846
q4	1857	1696	1646	1646
q5	5228	5225	5228	5225
q6	191	113	115	113
q7	2164	1709	1800	1709
q8	3097	3256	3279	3256
q9	8336	8379	8315	8315
q10	5944	3602	3571	3571
q11	562	460	460	460
q12	735	602	580	580
q13	11902	3079	3102	3079
q14	275	249	255	249
q15	535	489	493	489
q16	507	469	485	469
q17	1891	1666	1672	1666
q18	8114	7620	7614	7614
q19	8612	1524	1535	1524
q20	2137	1913	1906	1906
q21	5032	4537	4638	4537
q22	544	474	499	474
Total cold run time: 75505 ms
Total hot run time: 54322 ms

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 36.19% (8626/23836)
Line Coverage: 28.25% (70554/249750)
Region Coverage: 27.25% (36412/133611)
Branch Coverage: 24.04% (18646/77578)
Coverage Report: http://coverage.selectdb-in.cc/coverage/51515e383b842a687c35220bc041a6cd15482af2_51515e383b842a687c35220bc041a6cd15482af2/report/index.html

@doris-robot
Copy link

TPC-DS: Total hot run time: 173009 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 51515e383b842a687c35220bc041a6cd15482af2, data reload: false

query1	939	330	332	330
query2	6563	1945	1972	1945
query3	6700	212	205	205
query4	27332	22085	22060	22060
query5	4446	366	418	366
query6	246	155	164	155
query7	4604	264	258	258
query8	241	190	174	174
query9	9090	2266	2247	2247
query10	404	213	197	197
query11	16814	15471	15466	15466
query12	126	70	64	64
query13	1677	370	372	370
query14	9230	6615	6557	6557
query15	224	180	185	180
query16	6478	256	248	248
query17	1904	473	473	473
query18	1899	255	247	247
query19	180	131	134	131
query20	71	79	72	72
query21	209	135	130	130
query22	4913	4679	4420	4420
query23	31220	30374	30380	30374
query24	10695	2796	2788	2788
query25	541	308	317	308
query26	839	134	137	134
query27	2804	280	283	280
query28	5987	1836	1812	1812
query29	943	616	614	614
query30	284	136	137	136
query31	929	701	732	701
query32	88	54	57	54
query33	574	211	208	208
query34	833	445	455	445
query35	845	791	762	762
query36	1270	1190	1139	1139
query37	90	59	60	59
query38	3297	3185	3149	3149
query39	1320	1252	1261	1252
query40	196	82	90	82
query41	40	35	36	35
query42	92	82	84	82
query43	506	488	491	488
query44	1085	674	681	674
query45	196	174	174	174
query46	1031	650	660	650
query47	1614	1522	1523	1522
query48	380	295	303	295
query49	1116	291	290	290
query50	686	308	313	308
query51	5203	5162	5190	5162
query52	95	78	80	78
query53	320	262	256	256
query54	252	183	186	183
query55	84	79	74	74
query56	177	161	172	161
query57	956	916	899	899
query58	195	153	164	153
query59	2407	2219	2348	2219
query60	199	180	179	179
query61	85	85	83	83
query62	613	365	360	360
query63	280	271	255	255
query64	4051	1775	1734	1734
query65	3296	3266	3218	3218
query66	1129	318	318	318
query67	14234	14199	13967	13967
query68	4040	488	478	478
query69	435	322	304	304
query70	1510	1479	1492	1479
query71	268	211	217	211
query72	3777	2855	2835	2835
query73	699	324	312	312
query74	6631	6278	6280	6278
query75	2857	2271	2326	2271
query76	2967	975	990	975
query77	367	233	227	227
query78	9131	8686	8665	8665
query79	2522	496	493	493
query80	1372	330	320	320
query81	532	203	202	202
query82	830	80	79	79
query83	220	117	119	117
query84	225	75	73	73
query85	1321	346	340	340
query86	496	393	357	357
query87	3432	3289	3266	3266
query88	3776	2267	2242	2242
query89	427	373	354	354
query90	1975	198	193	193
query91	164	121	117	117
query92	57	44	46	44
query93	1927	412	412	412
query94	1316	161	161	161
query95	503	473	453	453
query96	621	328	331	328
query97	4247	4120	4124	4120
query98	204	189	181	181
query99	1073	719	713	713
Total cold run time: 267289 ms
Total hot run time: 173009 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 31.2 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 51515e383b842a687c35220bc041a6cd15482af2, data reload: false

query1	0.03	0.03	0.02
query2	0.06	0.03	0.03
query3	0.23	0.06	0.06
query4	1.68	0.10	0.09
query5	0.53	0.52	0.52
query6	1.22	0.64	0.65
query7	0.02	0.02	0.01
query8	0.04	0.02	0.02
query9	0.56	0.49	0.49
query10	0.55	0.55	0.56
query11	0.12	0.08	0.08
query12	0.11	0.09	0.09
query13	0.61	0.61	0.62
query14	0.79	0.79	0.82
query15	0.81	0.78	0.77
query16	0.39	0.39	0.39
query17	1.03	1.03	1.00
query18	0.25	0.25	0.27
query19	1.88	1.81	1.79
query20	0.01	0.01	0.02
query21	15.40	0.61	0.58
query22	2.71	2.15	2.28
query23	17.56	0.77	0.79
query24	2.51	1.55	0.78
query25	0.24	0.15	0.13
query26	0.63	0.15	0.13
query27	0.05	0.05	0.05
query28	11.16	0.85	0.86
query29	12.55	3.25	3.08
query30	0.61	0.60	0.53
query31	2.78	0.36	0.34
query32	3.37	0.48	0.49
query33	3.15	3.24	3.19
query34	15.82	4.37	4.34
query35	4.35	4.27	4.30
query36	1.12	1.06	1.07
query37	0.07	0.05	0.05
query38	0.04	0.03	0.03
query39	0.02	0.02	0.01
query40	0.17	0.13	0.12
query41	0.07	0.02	0.01
query42	0.03	0.02	0.02
query43	0.02	0.02	0.02
Total cold run time: 105.35 s
Total hot run time: 31.2 s

@doris-robot
Copy link

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit 51515e383b842a687c35220bc041a6cd15482af2 with default session variables
Stream load json:         18 seconds loaded 2358488459 Bytes, about 124 MB/s
Stream load orc:          58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet:      31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select:       13.7 seconds inserted 10000000 Rows, about 729K ops/s

Copy link
Contributor

@yiguolei yiguolei left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Jan 31, 2024
Copy link
Contributor

PR approved by at least one committer and no changes requested.

Copy link
Contributor

PR approved by anyone and no changes requested.

@yiguolei yiguolei merged commit 77e0ab1 into apache:master Jan 31, 2024
27 of 31 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by one committer. reviewed workload-group
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants