[feature](backend) Add cpu core count and memory usage to "show backends" command. #33885

Mryange · 2024-04-19T06:49:49Z

Proposed changes

mysql [(none)]>show backends\G;
*************************** 1. row ***************************
              BackendId: 10016
                   Host: 127.0.0.1
          HeartbeatPort: 9229
                 BePort: 9239
               HttpPort: 8219
               BrpcPort: 8239
     ArrowFlightSqlPort: -1
          LastStartTime: 2024-04-19 15:50:26
          LastHeartbeat: 2024-04-19 15:50:30
                  Alive: true
   SystemDecommissioned: false
              TabletNum: 17247
       DataUsedCapacity: 31.558 GB
     TrashUsedCapcacity: 0.000 
          AvailCapacity: 387.957 GB
          TotalCapacity: 3.437 TB
                UsedPct: 88.98 %
         MaxDiskUsedPct: 88.98 %
     RemoteUsedCapacity: 0.000 
                    Tag: {"location" : "default"}
                 ErrMsg: 
                Version: doris-0.0.0-trunk-049729a361
                 Status: {"lastSuccessReportTabletsTime":"2024-04-19 15:50:31","lastStreamLoadTime":-1,"isQueryDisabled":false,"isLoadDisabled":false,"isActive":true,"currentFragmentNum":0,"lastFragmentUpdateTime":1713513015892}
HeartbeatFailureCounter: 0
               NodeRole: mix
               CpuCores: 96
                 Memory: 375.81 GB
1 row in set (0.04 sec)

Further comments

If this is a relatively large or complex change, kick off the discussion at [email protected] by explaining why you chose the solution you did and what alternatives you considered, etc...

doris-robot · 2024-04-19T06:49:53Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

Mryange · 2024-04-19T06:51:35Z

run buildall

github-actions · 2024-04-19T06:55:40Z

clang-tidy review says "All clean, LGTM! 👍"

be/src/agent/heartbeat_server.cpp

Mryange · 2024-04-19T07:05:38Z

run buildall

HappenLee · 2024-04-19T07:09:26Z

be/src/agent/heartbeat_server.cpp

@@ -88,6 +89,7 @@ void HeartbeatServer::heartbeat(THeartbeatResult& heartbeat_result,
                get_fragment_executing_count());
        heartbeat_result.backend_info.__set_fragment_last_active_time(
                get_fragment_last_active_time());
+        heartbeat_result.backend_info.__set_be_mem(PerfCounters::get_vm_rss());


use mem_info _s_physical_mem not this

github-actions · 2024-04-19T07:16:01Z

clang-tidy review says "All clean, LGTM! 👍"

doris-robot · 2024-04-19T08:35:36Z

TeamCity be ut coverage result:
Function Coverage: 35.42% (8922/25188)
Line Coverage: 27.15% (73323/270057)
Region Coverage: 26.26% (37856/144146)
Branch Coverage: 23.09% (19298/83590)
Coverage Report: http://coverage.selectdb-in.cc/coverage/9cc2c3c3475df1bc7a9a50a3752fe1f42af59da7_9cc2c3c3475df1bc7a9a50a3752fe1f42af59da7/report/index.html

Mryange · 2024-04-19T08:37:59Z

run buildall

github-actions · 2024-04-19T08:53:00Z

clang-tidy review says "All clean, LGTM! 👍"

doris-robot · 2024-04-19T10:03:37Z

TeamCity be ut coverage result:
Function Coverage: 35.42% (8922/25189)
Line Coverage: 27.12% (73283/270207)
Region Coverage: 26.26% (37867/144194)
Branch Coverage: 23.06% (19282/83606)
Coverage Report: http://coverage.selectdb-in.cc/coverage/2300d34697e8618ea67ecdf85f54a16f4c5a1346_2300d34697e8618ea67ecdf85f54a16f4c5a1346/report/index.html

HappenLee

LGTM

github-actions · 2024-04-19T10:05:56Z

PR approved by at least one committer and no changes requested.

github-actions · 2024-04-19T10:05:58Z

PR approved by anyone and no changes requested.

Mryange · 2024-04-19T12:25:06Z

run buildall

doris-robot · 2024-04-19T13:25:32Z

TeamCity be ut coverage result:
Function Coverage: 35.41% (8920/25189)
Line Coverage: 27.11% (73260/270208)
Region Coverage: 26.25% (37853/144194)
Branch Coverage: 23.05% (19273/83606)
Coverage Report: http://coverage.selectdb-in.cc/coverage/2300d34697e8618ea67ecdf85f54a16f4c5a1346_2300d34697e8618ea67ecdf85f54a16f4c5a1346/report/index.html

doris-robot · 2024-04-19T13:40:35Z

TPC-H: Total hot run time: 38402 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 2300d34697e8618ea67ecdf85f54a16f4c5a1346, data reload: false

------ Round 1 ----------------------------------
q1	17620	4301	4230	4230
q2	2008	188	179	179
q3	10468	1158	1185	1158
q4	10192	810	831	810
q5	7508	2709	2676	2676
q6	221	131	134	131
q7	1020	608	586	586
q8	9223	2070	2052	2052
q9	7414	6645	6531	6531
q10	8527	3535	3558	3535
q11	469	236	226	226
q12	454	222	214	214
q13	17772	2946	2953	2946
q14	265	229	241	229
q15	517	503	475	475
q16	520	377	383	377
q17	975	644	757	644
q18	7492	6834	6590	6590
q19	7552	1570	1478	1478
q20	661	330	303	303
q21	3469	2724	2879	2724
q22	360	313	308	308
Total cold run time: 114707 ms
Total hot run time: 38402 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4325	4257	4279	4257
q2	362	259	261	259
q3	3040	2721	2726	2721
q4	1862	1548	1557	1548
q5	5390	5361	5309	5309
q6	213	121	122	121
q7	2241	1948	1878	1878
q8	3257	3352	3363	3352
q9	8582	8605	8836	8605
q10	4076	3913	4000	3913
q11	610	489	482	482
q12	838	649	629	629
q13	17104	3250	3034	3034
q14	346	284	297	284
q15	510	488	481	481
q16	502	445	456	445
q17	1844	1552	1520	1520
q18	8169	8187	7822	7822
q19	1683	1630	1606	1606
q20	2060	1847	1832	1832
q21	5144	5022	4993	4993
q22	539	470	476	470
Total cold run time: 72697 ms
Total hot run time: 55561 ms

doris-robot · 2024-04-19T13:52:23Z

TPC-DS: Total hot run time: 185615 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 2300d34697e8618ea67ecdf85f54a16f4c5a1346, data reload: false

query1	874	385	377	377
query2	6188	2693	2431	2431
query3	6653	211	207	207
query4	24751	21342	21337	21337
query5	4198	412	422	412
query6	275	183	199	183
query7	4585	289	284	284
query8	243	193	184	184
query9	8638	2320	2291	2291
query10	415	240	257	240
query11	14763	14276	14221	14221
query12	130	93	85	85
query13	1638	364	364	364
query14	9264	8111	7649	7649
query15	260	178	183	178
query16	8215	286	271	271
query17	1920	594	585	585
query18	2102	290	276	276
query19	339	159	154	154
query20	96	88	84	84
query21	201	130	124	124
query22	5003	4800	4758	4758
query23	33932	33088	33260	33088
query24	11196	3104	3025	3025
query25	615	404	408	404
query26	714	162	166	162
query27	2245	376	378	376
query28	6012	2084	2117	2084
query29	887	633	640	633
query30	300	194	187	187
query31	979	789	816	789
query32	88	53	54	53
query33	675	253	254	253
query34	903	486	503	486
query35	861	707	714	707
query36	1093	944	961	944
query37	119	73	71	71
query38	3448	3371	3269	3269
query39	1624	1613	1592	1592
query40	189	128	127	127
query41	44	40	40	40
query42	108	100	99	99
query43	557	565	531	531
query44	1080	746	735	735
query45	274	271	247	247
query46	1110	769	743	743
query47	2071	1969	1974	1969
query48	374	312	305	305
query49	832	401	390	390
query50	798	399	400	399
query51	6897	6780	6771	6771
query52	97	87	89	87
query53	346	275	277	275
query54	301	228	248	228
query55	77	71	73	71
query56	246	219	217	217
query57	1201	1119	1093	1093
query58	212	191	198	191
query59	3329	3225	3230	3225
query60	273	221	228	221
query61	88	89	84	84
query62	596	435	438	435
query63	302	285	271	271
query64	4769	3879	3907	3879
query65	3039	3025	3026	3025
query66	743	335	338	335
query67	15560	14988	14977	14977
query68	8538	534	543	534
query69	540	303	308	303
query70	1228	1141	1100	1100
query71	1454	1266	1261	1261
query72	6511	2594	2437	2437
query73	749	317	315	315
query74	6769	6445	6354	6354
query75	4055	2627	2660	2627
query76	4675	958	972	958
query77	614	261	258	258
query78	10932	10190	10133	10133
query79	10222	572	528	528
query80	2078	451	430	430
query81	520	240	241	240
query82	825	96	94	94
query83	201	160	162	160
query84	257	84	85	84
query85	1272	273	256	256
query86	416	302	316	302
query87	3556	3218	3217	3217
query88	5148	2380	2388	2380
query89	530	378	371	371
query90	2009	177	177	177
query91	123	93	98	93
query92	58	48	45	45
query93	7336	510	491	491
query94	1159	175	176	175
query95	382	298	302	298
query96	605	261	258	258
query97	3122	2899	3007	2899
query98	237	217	213	213
query99	1243	836	840	836
Total cold run time: 299319 ms
Total hot run time: 185615 ms

doris-robot · 2024-04-19T13:57:52Z

ClickBench: Total hot run time: 30.42 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 2300d34697e8618ea67ecdf85f54a16f4c5a1346, data reload: false

query1	0.04	0.03	0.03
query2	0.08	0.04	0.04
query3	0.23	0.06	0.06
query4	1.66	0.08	0.07
query5	0.51	0.50	0.48
query6	1.47	0.72	0.72
query7	0.02	0.01	0.01
query8	0.05	0.04	0.05
query9	0.54	0.50	0.49
query10	0.54	0.55	0.54
query11	0.17	0.11	0.12
query12	0.15	0.12	0.13
query13	0.62	0.59	0.58
query14	0.76	0.77	0.79
query15	0.82	0.81	0.79
query16	0.36	0.35	0.36
query17	0.99	0.93	0.93
query18	0.23	0.22	0.25
query19	1.78	1.69	1.67
query20	0.02	0.01	0.01
query21	15.40	0.65	0.64
query22	4.50	6.84	2.13
query23	18.29	1.33	1.24
query24	1.68	0.28	0.20
query25	0.16	0.08	0.09
query26	0.25	0.16	0.15
query27	0.08	0.08	0.08
query28	13.38	1.01	0.99
query29	12.57	3.23	3.23
query30	0.25	0.06	0.06
query31	2.88	0.38	0.38
query32	3.28	0.46	0.46
query33	2.86	2.82	2.85
query34	16.94	4.47	4.41
query35	4.48	4.48	4.44
query36	0.66	0.46	0.46
query37	0.19	0.16	0.17
query38	0.14	0.14	0.14
query39	0.04	0.04	0.03
query40	0.17	0.15	0.15
query41	0.10	0.05	0.05
query42	0.06	0.05	0.05
query43	0.04	0.04	0.04
Total cold run time: 109.44 s
Total hot run time: 30.42 s

doris-robot · 2024-04-19T14:00:57Z

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit 2300d34697e8618ea67ecdf85f54a16f4c5a1346 with default session variables
Stream load json:         19 seconds loaded 2358488459 Bytes, about 118 MB/s
Stream load orc:          58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet:      33 seconds loaded 861443392 Bytes, about 24 MB/s
Insert into select:       13.6 seconds inserted 10000000 Rows, about 735K ops/s

github-actions bot added the meta-change label Apr 19, 2024

HappenLee reviewed Apr 19, 2024

View reviewed changes

be/src/agent/heartbeat_server.cpp Outdated Show resolved Hide resolved

Mryange force-pushed the show-backends branch from d7b826d to 9cc2c3c Compare April 19, 2024 07:04

HappenLee reviewed Apr 19, 2024

View reviewed changes

upd

2300d34

Mryange force-pushed the show-backends branch from 9cc2c3c to 2300d34 Compare April 19, 2024 08:35

HappenLee approved these changes Apr 19, 2024

View reviewed changes

github-actions bot added the approved Indicates a PR has been approved by one committer. label Apr 19, 2024

github-actions bot added the reviewed label Apr 19, 2024

Gabriel39 approved these changes Apr 22, 2024

View reviewed changes

HappenLee merged commit 440b2d0 into apache:master Apr 22, 2024
25 of 29 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[feature](backend) Add cpu core count and memory usage to "show backends" command. #33885

[feature](backend) Add cpu core count and memory usage to "show backends" command. #33885

Mryange commented Apr 19, 2024 •

edited

Loading

doris-robot commented Apr 19, 2024

Mryange commented Apr 19, 2024

github-actions bot commented Apr 19, 2024

Mryange commented Apr 19, 2024

HappenLee Apr 19, 2024

github-actions bot commented Apr 19, 2024

doris-robot commented Apr 19, 2024

Mryange commented Apr 19, 2024

github-actions bot commented Apr 19, 2024

doris-robot commented Apr 19, 2024

HappenLee left a comment

github-actions bot commented Apr 19, 2024

github-actions bot commented Apr 19, 2024

Mryange commented Apr 19, 2024

doris-robot commented Apr 19, 2024

doris-robot commented Apr 19, 2024

doris-robot commented Apr 19, 2024

doris-robot commented Apr 19, 2024

doris-robot commented Apr 19, 2024

[feature](backend) Add cpu core count and memory usage to "show backends" command. #33885

[feature](backend) Add cpu core count and memory usage to "show backends" command. #33885

Conversation

Mryange commented Apr 19, 2024 • edited Loading

Proposed changes

Further comments

doris-robot commented Apr 19, 2024

Mryange commented Apr 19, 2024

github-actions bot commented Apr 19, 2024

Mryange commented Apr 19, 2024

HappenLee Apr 19, 2024

Choose a reason for hiding this comment

github-actions bot commented Apr 19, 2024

doris-robot commented Apr 19, 2024

Mryange commented Apr 19, 2024

github-actions bot commented Apr 19, 2024

doris-robot commented Apr 19, 2024

HappenLee left a comment

Choose a reason for hiding this comment

github-actions bot commented Apr 19, 2024

github-actions bot commented Apr 19, 2024

Mryange commented Apr 19, 2024

doris-robot commented Apr 19, 2024

doris-robot commented Apr 19, 2024

doris-robot commented Apr 19, 2024

doris-robot commented Apr 19, 2024

doris-robot commented Apr 19, 2024

Mryange commented Apr 19, 2024 •

edited

Loading