[Improment]Add workload group resource usage #39177

wangbo · 2024-08-09T13:32:26Z

Proposed changes

mysql [information_schema]>select BE_ID,WORKLOAD_GROUP_ID,CPU_USAGE,LOCAL_SCAN_BYTES_PER_SECOND/1024/1024 as scan_io_mb, MEMORY_USAGE_BYTES/1024/1024 mem_mb from workload_group_resource_usage;
+-------+-------------------+-----------+--------------------+-------------------+
| BE_ID | WORKLOAD_GROUP_ID | CPU_USAGE | scan_io_mb         | mem_mb            |
+-------+-------------------+-----------+--------------------+-------------------+
| 10005 |             62053 | 61.41%    | 1516.4589414596558 | 27970.84313774109 |
| 10005 |                 1 | 0.00%     |                  0 |                 0 |
+-------+-------------------+-----------+--------------------+-------------------+

doris-robot · 2024-08-09T13:32:32Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

github-actions

clang-tidy made some suggestions

be/src/exec/schema_scanner/schema_workload_group_resource_usage_scanner.cpp

be/src/runtime/workload_group/workload_group.h

be/src/runtime/workload_group/workload_group_manager.cpp

be/src/runtime/workload_management/io_throttle.cpp

be/src/runtime/workload_management/io_throttle.h

yiguolei · 2024-08-10T00:13:09Z

be/src/exec/schema_scanner/schema_workload_group_resource_usage_scanner.cpp

+        {"BE_ID", TYPE_BIGINT, sizeof(int64_t), false},
+        {"WORKLOAD_GROUP_ID", TYPE_BIGINT, sizeof(int64_t), false},
+        {"MEMORY_USAGE_BYTES", TYPE_BIGINT, sizeof(int64_t), false},
+        {"CPU_USAGE", TYPE_VARCHAR, sizeof(StringRef), false},


这个可能不行，不行就把百分号去掉。否则用户没法排序

be/src/runtime/workload_management/io_throttle.cpp

yiguolei · 2024-08-10T00:31:54Z

be/src/olap/delta_writer_v2.cpp

@@ -163,6 +163,13 @@ Status DeltaWriterV2::write(const vectorized::Block* block, const std::vector<ui
        }
    }
    SCOPED_RAW_TIMER(&_write_memtable_time);
+    ThreadCpuStopWatch cpu_time_stop_watch;


我觉得这个得加到async result writer 那个线程那里，这样所有的都被cover 住了

yiguolei · 2024-08-10T00:38:43Z

fe/fe-core/src/main/java/org/apache/doris/catalog/SchemaTable.java

+                            builder().column("BE_ID", ScalarType.createType(PrimitiveType.BIGINT))
+                                    .column("WORKLOAD_GROUP_ID", ScalarType.createType(PrimitiveType.BIGINT))
+                                    .column("MEMORY_USAGE_BYTES", ScalarType.createType(PrimitiveType.BIGINT))
+                                    .column("CPU_USAGE", ScalarType.createVarchar(256))


要不把这列的类型定义为整数，这样用户可以排序。
然后把列名改成CPU_USAGE(%)这样

be/src/runtime/workload_management/io_throttle.h

be/src/runtime/thread_context.h

yiguolei · 2024-08-10T00:46:11Z

be/src/runtime/workload_management/io_throttle.cpp

    }
-    _next_io_time_micros += ret < 1 ? static_cast<int64_t>(1) : static_cast<int64_t>(ret);
+    (*_io_adder) << io_bytes;


这里我感觉local 的也统计了，那么之前的就没必要再那个宏里修改了，直接都用这里的统计

github-actions

clang-tidy made some suggestions

be/src/runtime/workload_management/io_throttle.cpp

github-actions

clang-tidy made some suggestions

be/src/runtime/workload_group/workload_group_manager.cpp

be/src/runtime/workload_management/io_throttle.cpp

github-actions · 2024-08-10T09:56:17Z

PR approved by at least one committer and no changes requested.

github-actions · 2024-08-10T09:56:19Z

PR approved by anyone and no changes requested.

yiguolei · 2024-08-10T14:59:59Z

run buildall

doris-robot · 2024-08-10T16:51:47Z

TPC-H: Total hot run time: 39938 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 3bffef30f6b06fea87b0b810e4d64481cb16f6a6, data reload: false

------ Round 1 ----------------------------------
q1	17710	4665	4397	4397
q2	2022	177	171	171
q3	10524	1261	1173	1173
q4	10163	788	692	692
q5	7493	2524	2489	2489
q6	226	138	144	138
q7	990	600	614	600
q8	9318	1929	1952	1929
q9	8759	6632	6572	6572
q10	7074	2194	2214	2194
q11	495	253	248	248
q12	398	229	229	229
q13	17766	3017	3037	3017
q14	290	243	241	241
q15	536	503	491	491
q16	514	392	394	392
q17	984	722	682	682
q18	8281	7610	7451	7451
q19	4464	978	1072	978
q20	746	343	325	325
q21	5393	4710	4512	4512
q22	1130	1047	1017	1017
Total cold run time: 115276 ms
Total hot run time: 39938 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4547	4291	4325	4291
q2	383	276	274	274
q3	2845	2672	2769	2672
q4	2053	1804	1726	1726
q5	5671	5563	5507	5507
q6	226	136	139	136
q7	2136	1813	1795	1795
q8	3326	3471	3522	3471
q9	8808	8815	8852	8815
q10	3551	3317	3170	3170
q11	607	502	493	493
q12	841	663	660	660
q13	16571	3173	3181	3173
q14	323	301	290	290
q15	538	516	512	512
q16	506	432	434	432
q17	1847	1527	1554	1527
q18	8183	8020	7927	7927
q19	1788	1612	1658	1612
q20	2275	1919	1896	1896
q21	8677	5523	5274	5274
q22	1155	1068	1061	1061
Total cold run time: 76857 ms
Total hot run time: 56714 ms

doris-robot · 2024-08-10T17:04:27Z

TPC-DS: Total hot run time: 204326 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 3bffef30f6b06fea87b0b810e4d64481cb16f6a6, data reload: false

query1	908	376	371	371
query2	6456	1935	1874	1874
query3	6631	210	217	210
query4	34204	23137	23079	23079
query5	3651	501	518	501
query6	284	175	174	174
query7	4593	298	300	298
query8	249	209	205	205
query9	8555	2519	2467	2467
query10	545	470	432	432
query11	17409	14898	15208	14898
query12	137	123	98	98
query13	1631	372	359	359
query14	10268	7704	7881	7704
query15	262	206	256	206
query16	7410	477	490	477
query17	1312	579	559	559
query18	1709	295	294	294
query19	218	143	152	143
query20	125	117	115	115
query21	209	105	110	105
query22	4504	4315	4451	4315
query23	34400	33638	33786	33638
query24	11340	2668	2628	2628
query25	549	375	371	371
query26	702	155	151	151
query27	2174	282	276	276
query28	5885	2042	2025	2025
query29	777	414	411	411
query30	255	152	150	150
query31	980	760	723	723
query32	100	56	60	56
query33	626	281	283	281
query34	861	470	482	470
query35	962	858	843	843
query36	1087	927	939	927
query37	143	79	83	79
query38	4280	4232	4213	4213
query39	1455	1410	1395	1395
query40	206	124	121	121
query41	52	46	46	46
query42	119	101	98	98
query43	555	481	490	481
query44	1117	774	727	727
query45	238	205	202	202
query46	1099	743	795	743
query47	1848	1747	1752	1747
query48	373	298	303	298
query49	837	417	432	417
query50	805	415	421	415
query51	6778	6597	6564	6564
query52	112	95	95	95
query53	263	182	192	182
query54	892	456	452	452
query55	75	78	74	74
query56	276	242	245	242
query57	1162	1048	1081	1048
query58	239	232	253	232
query59	2953	2762	2922	2762
query60	299	267	266	266
query61	101	110	132	110
query62	792	640	637	637
query63	211	186	181	181
query64	9192	2256	1736	1736
query65	3231	3152	3138	3138
query66	723	336	329	329
query67	15192	14863	14698	14698
query68	4477	550	565	550
query69	445	385	383	383
query70	1203	1181	1153	1153
query71	377	293	284	284
query72	19530	18097	17891	17891
query73	767	330	329	329
query74	8973	8802	8829	8802
query75	3343	2651	2702	2651
query76	2156	1021	984	984
query77	521	318	325	318
query78	11211	10016	9573	9573
query79	2824	532	529	529
query80	2062	509	505	505
query81	627	225	231	225
query82	661	144	137	137
query83	290	147	147	147
query84	274	80	80	80
query85	1390	292	273	273
query86	432	296	311	296
query87	4805	4617	4609	4609
query88	3720	2502	2468	2468
query89	408	305	288	288
query90	1926	199	203	199
query91	125	95	95	95
query92	66	53	51	51
query93	3924	545	538	538
query94	842	301	297	297
query95	417	269	263	263
query96	620	276	271	271
query97	3235	3091	3139	3091
query98	218	196	191	191
query99	1592	1293	1299	1293
Total cold run time: 311047 ms
Total hot run time: 204326 ms

doris-robot · 2024-08-10T17:10:01Z

ClickBench: Total hot run time: 30.93 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 3bffef30f6b06fea87b0b810e4d64481cb16f6a6, data reload: false

query1	0.04	0.04	0.04
query2	0.07	0.04	0.04
query3	0.23	0.05	0.06
query4	1.67	0.09	0.08
query5	0.48	0.48	0.50
query6	1.14	0.74	0.73
query7	0.02	0.01	0.02
query8	0.05	0.05	0.05
query9	0.56	0.49	0.49
query10	0.54	0.54	0.54
query11	0.15	0.12	0.12
query12	0.15	0.12	0.12
query13	0.62	0.60	0.58
query14	0.75	0.78	0.78
query15	0.86	0.82	0.82
query16	0.35	0.36	0.36
query17	1.01	0.97	0.96
query18	0.24	0.23	0.22
query19	1.77	1.69	1.72
query20	0.01	0.00	0.01
query21	15.45	0.78	0.66
query22	4.05	7.48	2.20
query23	18.25	1.31	1.24
query24	2.06	0.24	0.21
query25	0.14	0.07	0.08
query26	0.30	0.22	0.22
query27	0.46	0.23	0.23
query28	13.33	1.03	1.00
query29	12.63	3.32	3.30
query30	0.24	0.05	0.06
query31	2.87	0.40	0.38
query32	3.28	0.50	0.48
query33	2.90	2.91	2.96
query34	16.87	4.36	4.34
query35	4.37	4.40	4.40
query36	0.65	0.48	0.48
query37	0.18	0.15	0.15
query38	0.16	0.15	0.15
query39	0.04	0.04	0.04
query40	0.14	0.12	0.13
query41	0.09	0.06	0.05
query42	0.06	0.05	0.05
query43	0.05	0.04	0.04
Total cold run time: 109.28 s
Total hot run time: 30.93 s

zclllyybb

LGTM. please add apache/doris-website's pr link in your pr content.

## Proposed changes ``` mysql [information_schema]>select BE_ID,WORKLOAD_GROUP_ID,CPU_USAGE,LOCAL_SCAN_BYTES_PER_SECOND/1024/1024 as scan_io_mb, MEMORY_USAGE_BYTES/1024/1024 mem_mb from workload_group_resource_usage; +-------+-------------------+-----------+--------------------+-------------------+ | BE_ID | WORKLOAD_GROUP_ID | CPU_USAGE | scan_io_mb | mem_mb | +-------+-------------------+-----------+--------------------+-------------------+ | 10005 | 62053 | 61.41% | 1516.4589414596558 | 27970.84313774109 | | 10005 | 1 | 0.00% | 0 | 0 | +-------+-------------------+-----------+--------------------+-------------------+ ```

``` mysql [information_schema]>select BE_ID,WORKLOAD_GROUP_ID,CPU_USAGE,LOCAL_SCAN_BYTES_PER_SECOND/1024/1024 as scan_io_mb, MEMORY_USAGE_BYTES/1024/1024 mem_mb from workload_group_resource_usage; +-------+-------------------+-----------+--------------------+-------------------+ | BE_ID | WORKLOAD_GROUP_ID | CPU_USAGE | scan_io_mb | mem_mb | +-------+-------------------+-----------+--------------------+-------------------+ | 10005 | 62053 | 61.41% | 1516.4589414596558 | 27970.84313774109 | | 10005 | 1 | 0.00% | 0 | 0 | +-------+-------------------+-----------+--------------------+-------------------+ ```

github-actions bot added doing meta-change labels Aug 9, 2024

Add workload group resource usage

f2037d0

wangbo force-pushed the 0809_wg_monitor branch from 415aca4 to f2037d0 Compare August 9, 2024 13:34

github-actions bot reviewed Aug 9, 2024

View reviewed changes

yiguolei reviewed Aug 10, 2024

View reviewed changes

be/src/runtime/workload_management/io_throttle.cpp Show resolved Hide resolved

yiguolei reviewed Aug 10, 2024

View reviewed changes

be/src/runtime/workload_management/io_throttle.h Show resolved Hide resolved

yiguolei reviewed Aug 10, 2024

View reviewed changes

be/src/runtime/thread_context.h Show resolved Hide resolved

yiguolei reviewed Aug 10, 2024

View reviewed changes

wangbo force-pushed the 0809_wg_monitor branch from 01770f9 to 01a500a Compare August 10, 2024 07:04

github-actions bot reviewed Aug 10, 2024

View reviewed changes

be/src/runtime/workload_management/io_throttle.cpp Show resolved Hide resolved

wangbo force-pushed the 0809_wg_monitor branch from 185b865 to 8d689c7 Compare August 10, 2024 07:08

github-actions bot reviewed Aug 10, 2024

View reviewed changes

be/src/runtime/workload_group/workload_group_manager.cpp Show resolved Hide resolved

be/src/runtime/workload_management/io_throttle.cpp Show resolved Hide resolved

be/src/runtime/workload_management/io_throttle.cpp Show resolved Hide resolved

wangbo force-pushed the 0809_wg_monitor branch 2 times, most recently from 7725bb5 to f3f42f0 Compare August 10, 2024 07:34

+1

3bffef3

wangbo force-pushed the 0809_wg_monitor branch from f3f42f0 to 3bffef3 Compare August 10, 2024 07:52

yiguolei approved these changes Aug 10, 2024

View reviewed changes

github-actions bot added the approved Indicates a PR has been approved by one committer. label Aug 10, 2024

github-actions bot added the reviewed label Aug 10, 2024

zclllyybb approved these changes Aug 11, 2024

View reviewed changes

yiguolei merged commit c77bca3 into apache:master Aug 12, 2024
29 of 32 checks passed

dataroaring added the dev/3.0.2-merged label Aug 17, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Improment]Add workload group resource usage #39177

[Improment]Add workload group resource usage #39177

wangbo commented Aug 9, 2024

doris-robot commented Aug 9, 2024

github-actions bot left a comment

yiguolei Aug 10, 2024

yiguolei Aug 10, 2024

yiguolei Aug 10, 2024

yiguolei Aug 10, 2024

github-actions bot left a comment

github-actions bot left a comment

github-actions bot commented Aug 10, 2024

github-actions bot commented Aug 10, 2024

yiguolei commented Aug 10, 2024

doris-robot commented Aug 10, 2024

doris-robot commented Aug 10, 2024

doris-robot commented Aug 10, 2024

zclllyybb left a comment

[Improment]Add workload group resource usage #39177

[Improment]Add workload group resource usage #39177

Conversation

wangbo commented Aug 9, 2024

Proposed changes

doris-robot commented Aug 9, 2024

github-actions bot left a comment

Choose a reason for hiding this comment

yiguolei Aug 10, 2024

Choose a reason for hiding this comment

yiguolei Aug 10, 2024

Choose a reason for hiding this comment

yiguolei Aug 10, 2024

Choose a reason for hiding this comment

yiguolei Aug 10, 2024

Choose a reason for hiding this comment

github-actions bot left a comment

Choose a reason for hiding this comment

github-actions bot left a comment

Choose a reason for hiding this comment

github-actions bot commented Aug 10, 2024

github-actions bot commented Aug 10, 2024

yiguolei commented Aug 10, 2024

doris-robot commented Aug 10, 2024

doris-robot commented Aug 10, 2024

doris-robot commented Aug 10, 2024

zclllyybb left a comment

Choose a reason for hiding this comment