Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[fix](profile) Fix content missing of brokerload profile #33969

Merged
merged 4 commits into from
Apr 23, 2024

Conversation

zhiqiang-hhhh
Copy link
Contributor

@zhiqiang-hhhh zhiqiang-hhhh commented Apr 22, 2024

Broker load does not set enable_profile for its load task, makes profile empty.

BTW: profile of broker load can not be fetch during task executing, since it only register profile after load job is finished.

before:
image

after:
image

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@zhiqiang-hhhh
Copy link
Contributor Author

run buildall

@zhiqiang-hhhh
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 38631 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 9475c1d039f44da163ce2db44d8efd3ccc2fc37f, data reload: false

------ Round 1 ----------------------------------
q1	17606	4419	4274	4274
q2	2013	187	192	187
q3	10451	1196	1188	1188
q4	10188	828	763	763
q5	7534	2783	2626	2626
q6	221	133	135	133
q7	1046	603	597	597
q8	9242	2071	2056	2056
q9	7561	6619	6505	6505
q10	8542	3492	3505	3492
q11	447	238	242	238
q12	453	220	221	220
q13	17772	2950	3018	2950
q14	276	232	228	228
q15	522	489	487	487
q16	535	385	376	376
q17	973	758	690	690
q18	7388	6813	6811	6811
q19	7759	1557	1434	1434
q20	639	299	300	299
q21	3466	2771	2914	2771
q22	368	306	310	306
Total cold run time: 115002 ms
Total hot run time: 38631 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4298	4207	4231	4207
q2	374	259	268	259
q3	2996	2747	2775	2747
q4	1899	1587	1582	1582
q5	5359	5355	5316	5316
q6	212	122	130	122
q7	2215	1855	1843	1843
q8	3214	3374	3365	3365
q9	8621	8612	8770	8612
q10	4075	3902	3960	3902
q11	586	518	493	493
q12	827	638	617	617
q13	15891	3346	3163	3163
q14	327	270	270	270
q15	524	505	494	494
q16	492	447	443	443
q17	1819	1544	1493	1493
q18	8050	7823	7943	7823
q19	1696	1569	1579	1569
q20	1985	1848	1852	1848
q21	10348	4875	4945	4875
q22	552	481	480	480
Total cold run time: 76360 ms
Total hot run time: 55523 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 185743 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 9475c1d039f44da163ce2db44d8efd3ccc2fc37f, data reload: false

query1	909	370	356	356
query2	6727	2492	2407	2407
query3	6663	219	218	218
query4	23129	21334	21370	21334
query5	4164	424	409	409
query6	275	193	203	193
query7	4600	281	280	280
query8	230	191	177	177
query9	8432	2255	2244	2244
query10	405	246	256	246
query11	14829	14167	14243	14167
query12	136	89	83	83
query13	1645	377	370	370
query14	9856	7993	7667	7667
query15	272	186	185	185
query16	8233	290	273	273
query17	1933	569	560	560
query18	2104	282	272	272
query19	332	158	156	156
query20	91	84	86	84
query21	196	131	124	124
query22	5047	4870	4808	4808
query23	34108	33185	33500	33185
query24	8450	3100	3040	3040
query25	610	429	407	407
query26	702	160	155	155
query27	2330	370	358	358
query28	5886	2015	2025	2015
query29	887	635	637	635
query30	286	181	180	180
query31	965	751	780	751
query32	91	54	58	54
query33	639	261	257	257
query34	927	484	488	484
query35	809	714	711	711
query36	1083	929	945	929
query37	106	74	72	72
query38	3434	3291	3291	3291
query39	1646	1592	1587	1587
query40	169	129	124	124
query41	45	43	45	43
query42	106	102	97	97
query43	605	546	530	530
query44	1097	771	746	746
query45	288	284	260	260
query46	1113	760	709	709
query47	2020	1919	1925	1919
query48	380	308	305	305
query49	835	423	400	400
query50	793	410	407	407
query51	6825	6758	6739	6739
query52	102	88	85	85
query53	343	272	278	272
query54	283	225	228	225
query55	75	71	73	71
query56	257	218	224	218
query57	1199	1160	1110	1110
query58	212	203	194	194
query59	3304	3212	3231	3212
query60	246	225	245	225
query61	88	84	84	84
query62	573	433	450	433
query63	300	275	275	275
query64	4546	4084	3763	3763
query65	3081	2998	3036	2998
query66	735	326	329	326
query67	15453	15082	15319	15082
query68	5146	539	534	534
query69	524	291	301	291
query70	1142	1165	1160	1160
query71	1377	1261	1269	1261
query72	6471	2634	2463	2463
query73	718	320	322	320
query74	6746	6551	6484	6484
query75	3381	2625	2651	2625
query76	3561	959	994	959
query77	577	265	279	265
query78	10970	10239	10177	10177
query79	8884	527	520	520
query80	1983	439	428	428
query81	551	237	239	237
query82	1197	94	93	93
query83	320	165	160	160
query84	260	82	83	82
query85	1759	262	295	262
query86	457	308	327	308
query87	3485	3308	3208	3208
query88	5185	2412	2403	2403
query89	565	374	369	369
query90	1891	181	178	178
query91	123	96	92	92
query92	55	46	48	46
query93	6843	499	492	492
query94	1146	177	174	174
query95	382	292	307	292
query96	615	259	263	259
query97	3139	3017	2964	2964
query98	234	210	223	210
query99	1222	875	851	851
Total cold run time: 289149 ms
Total hot run time: 185743 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.73 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 9475c1d039f44da163ce2db44d8efd3ccc2fc37f, data reload: false

query1	0.03	0.02	0.03
query2	0.08	0.04	0.04
query3	0.24	0.06	0.07
query4	1.64	0.10	0.10
query5	0.51	0.49	0.55
query6	1.43	0.73	0.72
query7	0.02	0.01	0.01
query8	0.06	0.05	0.05
query9	0.54	0.48	0.49
query10	0.53	0.54	0.54
query11	0.16	0.12	0.11
query12	0.15	0.12	0.12
query13	0.61	0.60	0.58
query14	0.76	0.77	0.78
query15	0.83	0.81	0.80
query16	0.36	0.37	0.38
query17	1.02	1.02	1.01
query18	0.20	0.25	0.22
query19	1.89	1.81	1.78
query20	0.02	0.01	0.01
query21	15.41	0.65	0.65
query22	4.42	6.90	1.93
query23	18.27	1.44	1.34
query24	1.41	0.38	0.26
query25	0.15	0.08	0.09
query26	0.27	0.16	0.16
query27	0.07	0.08	0.08
query28	13.31	1.07	1.05
query29	12.63	3.25	3.24
query30	0.26	0.07	0.06
query31	2.85	0.39	0.38
query32	3.26	0.47	0.47
query33	2.79	2.88	2.80
query34	17.06	4.43	4.45
query35	4.51	4.51	4.49
query36	0.64	0.46	0.47
query37	0.18	0.16	0.15
query38	0.15	0.14	0.14
query39	0.05	0.04	0.03
query40	0.18	0.12	0.14
query41	0.10	0.05	0.05
query42	0.06	0.05	0.05
query43	0.05	0.04	0.03
Total cold run time: 109.16 s
Total hot run time: 30.73 s

@zhiqiang-hhhh
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 38079 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit f4a460b48796eff04c14e46448560b5c0245a642, data reload: false

------ Round 1 ----------------------------------
q1	17619	4280	4214	4214
q2	2014	190	193	190
q3	10423	1293	1132	1132
q4	10178	863	738	738
q5	7514	2650	2639	2639
q6	218	132	135	132
q7	997	600	581	581
q8	9225	2050	2015	2015
q9	7508	6586	6486	6486
q10	8459	3469	3523	3469
q11	439	235	241	235
q12	424	217	213	213
q13	17786	2926	2927	2926
q14	266	223	235	223
q15	529	495	468	468
q16	539	387	386	386
q17	947	657	601	601
q18	7411	6711	6666	6666
q19	7134	1561	1509	1509
q20	648	310	305	305
q21	3477	2637	2837	2637
q22	379	314	320	314
Total cold run time: 114134 ms
Total hot run time: 38079 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4444	4253	4219	4219
q2	381	273	270	270
q3	3042	2786	2755	2755
q4	1891	1597	1606	1597
q5	5434	5405	5383	5383
q6	212	124	125	124
q7	2199	1877	1910	1877
q8	3210	3353	3343	3343
q9	8695	8618	8759	8618
q10	4244	4084	4145	4084
q11	624	521	549	521
q12	846	674	659	659
q13	17281	3372	3253	3253
q14	327	288	290	288
q15	530	488	483	483
q16	503	458	438	438
q17	1821	1507	1528	1507
q18	8278	7949	7860	7860
q19	1749	1556	1589	1556
q20	2030	1757	1842	1757
q21	14588	4857	4932	4857
q22	574	495	481	481
Total cold run time: 82903 ms
Total hot run time: 55930 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 185152 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit f4a460b48796eff04c14e46448560b5c0245a642, data reload: false

query1	896	361	348	348
query2	6199	2791	2446	2446
query3	6652	204	206	204
query4	22959	21543	21295	21295
query5	4189	399	398	398
query6	274	179	173	173
query7	4590	289	284	284
query8	249	187	199	187
query9	8464	2264	2267	2264
query10	416	244	247	244
query11	14745	14245	14117	14117
query12	136	89	84	84
query13	1655	372	372	372
query14	8816	8113	7289	7289
query15	242	183	184	183
query16	8006	255	260	255
query17	1941	562	549	549
query18	1600	270	263	263
query19	203	145	143	143
query20	96	84	85	84
query21	194	122	123	122
query22	5021	4775	4809	4775
query23	33716	33013	33318	33013
query24	11125	3080	2990	2990
query25	648	381	377	377
query26	1080	158	152	152
query27	2648	371	384	371
query28	7382	2037	2006	2006
query29	858	612	598	598
query30	333	181	175	175
query31	1007	748	748	748
query32	93	51	52	51
query33	726	241	235	235
query34	1273	499	484	484
query35	854	730	736	730
query36	1076	950	953	950
query37	147	71	74	71
query38	3597	3392	3380	3380
query39	1668	1568	1596	1568
query40	176	124	133	124
query41	45	42	47	42
query42	104	99	96	96
query43	609	541	538	538
query44	1205	750	745	745
query45	275	272	262	262
query46	1136	755	746	746
query47	2025	1927	1937	1927
query48	374	318	312	312
query49	922	395	400	395
query50	766	407	408	407
query51	6832	6724	6832	6724
query52	100	91	92	91
query53	357	276	277	276
query54	309	236	231	231
query55	79	73	72	72
query56	247	223	226	223
query57	1299	1206	1204	1204
query58	224	204	204	204
query59	3481	3367	3315	3315
query60	254	227	222	222
query61	90	88	103	88
query62	605	436	457	436
query63	302	281	277	277
query64	5032	4062	3946	3946
query65	3055	3097	3041	3041
query66	755	330	330	330
query67	15277	14970	14863	14863
query68	5238	537	541	537
query69	490	302	297	297
query70	1149	1188	1154	1154
query71	1428	1269	1264	1264
query72	6583	2607	2484	2484
query73	705	319	325	319
query74	6831	6423	6371	6371
query75	3398	2615	2653	2615
query76	3279	1055	995	995
query77	386	259	257	257
query78	11131	10229	10273	10229
query79	5254	523	514	514
query80	1792	424	416	416
query81	525	244	248	244
query82	849	92	97	92
query83	290	158	158	158
query84	261	85	82	82
query85	2019	266	257	257
query86	508	275	271	271
query87	3485	3263	3263	3263
query88	4989	2403	2380	2380
query89	488	364	358	358
query90	1935	177	174	174
query91	187	95	97	95
query92	56	47	53	47
query93	5103	491	484	484
query94	1135	177	173	173
query95	392	288	285	285
query96	611	265	258	258
query97	3159	2943	2884	2884
query98	246	214	212	212
query99	1220	868	876	868
Total cold run time: 286831 ms
Total hot run time: 185152 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.04 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit f4a460b48796eff04c14e46448560b5c0245a642, data reload: false

query1	0.04	0.03	0.03
query2	0.08	0.04	0.04
query3	0.23	0.05	0.05
query4	1.67	0.08	0.10
query5	0.49	0.50	0.50
query6	1.45	0.72	0.72
query7	0.02	0.02	0.01
query8	0.05	0.04	0.04
query9	0.55	0.49	0.49
query10	0.55	0.57	0.55
query11	0.15	0.12	0.11
query12	0.14	0.11	0.12
query13	0.58	0.58	0.59
query14	0.75	0.81	0.76
query15	0.82	0.80	0.80
query16	0.36	0.36	0.37
query17	0.95	0.98	0.95
query18	0.22	0.26	0.21
query19	1.85	1.73	1.65
query20	0.02	0.01	0.01
query21	15.40	0.66	0.65
query22	4.86	7.33	1.77
query23	18.29	1.31	1.25
query24	1.33	0.37	0.28
query25	0.15	0.08	0.08
query26	0.27	0.17	0.16
query27	0.07	0.08	0.07
query28	13.38	0.99	0.97
query29	12.66	3.24	3.27
query30	0.26	0.06	0.05
query31	2.85	0.36	0.36
query32	3.29	0.45	0.45
query33	2.77	2.78	2.82
query34	17.05	4.39	4.35
query35	4.45	4.50	4.47
query36	0.65	0.47	0.46
query37	0.18	0.16	0.15
query38	0.15	0.15	0.15
query39	0.05	0.04	0.05
query40	0.16	0.15	0.14
query41	0.09	0.04	0.04
query42	0.06	0.05	0.05
query43	0.04	0.04	0.03
Total cold run time: 109.43 s
Total hot run time: 30.04 s

Copy link
Contributor

@yiguolei yiguolei left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Apr 22, 2024
Copy link
Contributor

PR approved by at least one committer and no changes requested.

Copy link
Contributor

PR approved by anyone and no changes requested.

Copy link
Contributor

@liaoxin01 liaoxin01 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@liaoxin01 liaoxin01 merged commit a7221f2 into apache:master Apr 23, 2024
27 of 30 checks passed
@zhiqiang-hhhh zhiqiang-hhhh deleted the fix-brockerload-profile branch April 23, 2024 01:57
yiguolei pushed a commit that referenced this pull request May 14, 2024
* Fix compile

* fix style

* [fix](profile) Fix content missing of brokerload profile (#33969)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by one committer. dev/2.1.3-merged dev/3.0.0-merged reviewed
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants