Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[enhancement](mem-tracker) Use thread local mem tracker to track s3 file buffer memory usage #40597

Merged
merged 7 commits into from
Sep 26, 2024
Merged

[enhancement](mem-tracker) Use thread local mem tracker to track s3 file buffer memory usage #40597

merged 7 commits into from
Sep 26, 2024

Conversation

TangSiyang2001
Copy link
Collaborator

Proposed changes

Track s3 file buffer memory usage with thread local tracker, so that memory usage will be specified to detail.

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@TangSiyang2001
Copy link
Collaborator Author

run buildall

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 36.83% (9395/25508)
Line Coverage: 28.23% (77493/274521)
Region Coverage: 27.64% (40016/144801)
Branch Coverage: 24.26% (20355/83894)
Coverage Report: http://coverage.selectdb-in.cc/coverage/0e4d72ea783a7509f9ed383864fdf4e597d75cc2_0e4d72ea783a7509f9ed383864fdf4e597d75cc2/report/index.html

yiguolei
yiguolei previously approved these changes Sep 10, 2024
View reviewed changes
Copy link
Contributor

@yiguolei yiguolei left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Sep 10, 2024
@github-actions GitHub Actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions GitHub Actions
Copy link
Contributor

PR approved by anyone and no changes requested.

@doris-robot
Copy link

TPC-H: Total hot run time: 39643 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 0e4d72ea783a7509f9ed383864fdf4e597d75cc2, data reload: false

------ Round 1 ----------------------------------
q1	17901	5017	4503	4503
q2	2032	199	193	193
q3	11729	1018	1259	1018
q4	10538	773	864	773
q5	7795	2971	2974	2971
q6	249	147	144	144
q7	1046	648	633	633
q8	9349	2205	2200	2200
q9	7349	6831	6848	6831
q10	7098	2283	2264	2264
q11	470	255	249	249
q12	444	233	231	231
q13	17955	3109	3168	3109
q14	314	242	250	242
q15	567	515	502	502
q16	541	437	439	437
q17	1056	709	780	709
q18	7459	6967	6989	6967
q19	1431	1193	1103	1103
q20	713	342	356	342
q21	4241	3223	3305	3223
q22	1138	1018	999	999
Total cold run time: 111415 ms
Total hot run time: 39643 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4893	4501	4532	4501
q2	405	271	280	271
q3	2989	2772	2771	2771
q4	2117	1794	1769	1769
q5	5857	5902	5761	5761
q6	258	136	137	136
q7	2304	1829	1828	1828
q8	3563	3799	3805	3799
q9	8959	8968	9046	8968
q10	3785	3451	3520	3451
q11	666	517	534	517
q12	840	611	618	611
q13	14783	3346	3344	3344
q14	333	295	292	292
q15	572	505	503	503
q16	557	512	528	512
q17	2001	1650	1650	1650
q18	8289	8031	7931	7931
q19	1913	1661	1692	1661
q20	2183	1951	1966	1951
q21	6058	5756	5732	5732
q22	1147	1007	1049	1007
Total cold run time: 74472 ms
Total hot run time: 58966 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 198283 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 0e4d72ea783a7509f9ed383864fdf4e597d75cc2, data reload: false

query1	1277	889	934	889
query2	6457	1974	1893	1893
query3	10611	4019	4111	4019
query4	58387	25477	23667	23667
query5	4921	511	505	505
query6	382	174	174	174
query7	5562	295	292	292
query8	328	234	219	219
query9	7391	2521	2512	2512
query10	408	288	264	264
query11	16417	15143	15227	15143
query12	167	108	103	103
query13	1422	422	406	406
query14	10600	7444	7455	7444
query15	207	177	178	177
query16	6762	489	496	489
query17	1100	573	577	573
query18	1473	300	284	284
query19	192	144	147	144
query20	119	116	108	108
query21	214	102	104	102
query22	4724	4621	4326	4326
query23	34364	33808	33897	33808
query24	5968	2921	2832	2832
query25	495	378	380	378
query26	597	152	149	149
query27	1585	284	280	280
query28	3866	2074	2068	2068
query29	654	411	415	411
query30	241	151	157	151
query31	956	782	786	782
query32	68	55	52	52
query33	459	290	294	290
query34	888	472	468	468
query35	843	708	747	708
query36	1049	905	941	905
query37	143	89	84	84
query38	4017	3912	3934	3912
query39	1441	1420	1404	1404
query40	205	112	112	112
query41	45	46	45	45
query42	123	96	93	93
query43	516	493	463	463
query44	1087	762	749	749
query45	194	168	167	167
query46	1096	748	736	736
query47	1893	1780	1820	1780
query48	374	290	317	290
query49	769	450	463	450
query50	821	405	412	405
query51	6990	6945	6883	6883
query52	100	84	86	84
query53	246	176	180	176
query54	569	458	456	456
query55	75	77	74	74
query56	274	245	263	245
query57	1226	1071	1104	1071
query58	219	231	236	231
query59	2998	2871	2767	2767
query60	315	263	262	262
query61	103	97	102	97
query62	755	641	659	641
query63	219	190	183	183
query64	1356	679	668	668
query65	3219	3136	3165	3136
query66	651	337	346	337
query67	15939	15205	15202	15202
query68	1395	866	860	860
query69	423	318	319	318
query70	1185	1123	1205	1123
query71	350	341	330	330
query72	4658	3498	3505	3498
query73	596	574	578	574
query74	9191	8959	8847	8847
query75	2966	2942	3017	2942
query76	999	844	839	839
query77	507	409	402	402
query78	9310	9440	9186	9186
query79	898	883	866	866
query80	792	798	784	784
query81	450	260	263	260
query82	272	263	264	263
query83	192	189	188	188
query84	187	107	106	106
query85	628	393	389	389
query86	313	314	318	314
query87	4363	4406	4263	4263
query88	4327	4048	4066	4048
query89	372	356	362	356
query90	1432	310	309	309
query91	121	129	124	124
query92	79	83	80	80
query93	909	903	900	900
query94	528	376	374	374
query95	424	414	418	414
query96	473	463	462	462
query97	3112	3158	3144	3144
query98	231	231	230	230
query99	1296	1272	1281	1272
Total cold run time: 292111 ms
Total hot run time: 198283 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 31.75 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 0e4d72ea783a7509f9ed383864fdf4e597d75cc2, data reload: false

query1	0.04	0.04	0.04
query2	0.07	0.04	0.03
query3	0.22	0.06	0.07
query4	1.64	0.10	0.08
query5	0.51	0.50	0.50
query6	1.12	0.73	0.74
query7	0.02	0.01	0.02
query8	0.05	0.04	0.04
query9	0.55	0.49	0.50
query10	0.54	0.53	0.53
query11	0.16	0.13	0.12
query12	0.16	0.13	0.12
query13	0.62	0.59	0.59
query14	1.37	1.43	1.45
query15	0.86	0.83	0.82
query16	0.41	0.38	0.36
query17	1.05	0.97	0.98
query18	0.21	0.19	0.20
query19	1.80	1.79	1.79
query20	0.01	0.01	0.02
query21	15.41	0.66	0.67
query22	4.36	7.09	2.09
query23	18.28	1.44	1.34
query24	2.08	0.23	0.22
query25	0.16	0.09	0.09
query26	0.26	0.19	0.18
query27	0.08	0.08	0.07
query28	13.24	1.01	1.00
query29	12.58	3.36	3.36
query30	0.24	0.05	0.06
query31	2.86	0.41	0.40
query32	3.24	0.49	0.48
query33	3.00	2.97	3.06
query34	17.00	4.34	4.46
query35	4.46	4.49	4.51
query36	0.66	0.50	0.50
query37	0.19	0.17	0.16
query38	0.15	0.14	0.15
query39	0.06	0.04	0.04
query40	0.15	0.13	0.13
query41	0.09	0.04	0.04
query42	0.05	0.06	0.05
query43	0.04	0.04	0.04
Total cold run time: 110.05 s
Total hot run time: 31.75 s

@TangSiyang2001
Copy link
Collaborator Author

run buildall

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 36.83% (9396/25510)
Line Coverage: 28.24% (77517/274536)
Region Coverage: 27.63% (40012/144808)
Branch Coverage: 24.26% (20352/83900)
Coverage Report: http://coverage.selectdb-in.cc/coverage/6c64b0f1f985cc59f8878ff67656541bb60207c4_6c64b0f1f985cc59f8878ff67656541bb60207c4/report/index.html

@doris-robot
Copy link

TPC-H: Total hot run time: 37984 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 6c64b0f1f985cc59f8878ff67656541bb60207c4, data reload: false

------ Round 1 ----------------------------------
q1	17623	4358	4292	4292
q2	2017	188	183	183
q3	11682	953	1165	953
q4	10495	804	677	677
q5	7749	2859	2808	2808
q6	225	139	138	138
q7	949	618	614	614
q8	9314	2049	2047	2047
q9	7288	6539	6534	6534
q10	7009	2205	2161	2161
q11	449	244	242	242
q12	392	218	225	218
q13	17759	3098	3090	3090
q14	273	236	233	233
q15	536	502	488	488
q16	542	436	435	435
q17	985	678	778	678
q18	7260	6893	6932	6893
q19	1394	1007	1090	1007
q20	673	343	335	335
q21	3896	2947	2958	2947
q22	1103	1023	1011	1011
Total cold run time: 109613 ms
Total hot run time: 37984 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4357	4275	4282	4275
q2	390	277	258	258
q3	2844	2614	2687	2614
q4	1942	1635	1678	1635
q5	5603	5733	5736	5733
q6	225	152	150	150
q7	2217	1830	1857	1830
q8	3303	3425	3415	3415
q9	8886	8925	8838	8838
q10	3548	3374	3400	3374
q11	619	525	526	525
q12	849	639	684	639
q13	12698	3313	3214	3214
q14	312	291	295	291
q15	545	480	506	480
q16	552	489	479	479
q17	1819	1518	1550	1518
q18	8231	8003	7837	7837
q19	1746	1660	1656	1656
q20	2148	1913	1927	1913
q21	5833	5473	5641	5473
q22	1139	1010	999	999
Total cold run time: 69806 ms
Total hot run time: 57146 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 196957 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 6c64b0f1f985cc59f8878ff67656541bb60207c4, data reload: false

query1	1257	870	875	870
query2	6346	1950	1881	1881
query3	10579	3900	3904	3900
query4	60276	29152	23221	23221
query5	5115	501	507	501
query6	398	166	165	165
query7	5648	308	296	296
query8	319	223	222	222
query9	7713	2514	2519	2514
query10	410	275	268	268
query11	17260	15163	15405	15163
query12	151	103	109	103
query13	1475	396	389	389
query14	10621	7276	7115	7115
query15	207	177	164	164
query16	6561	500	508	500
query17	1086	587	586	586
query18	1486	296	292	292
query19	197	150	146	146
query20	130	109	107	107
query21	208	102	107	102
query22	4638	4531	4562	4531
query23	34287	33470	33616	33470
query24	6025	2916	2905	2905
query25	490	384	381	381
query26	608	152	154	152
query27	1611	279	281	279
query28	3772	2031	2006	2006
query29	646	407	409	407
query30	221	156	153	153
query31	920	730	773	730
query32	67	53	51	51
query33	431	289	285	285
query34	869	462	484	462
query35	841	710	735	710
query36	1063	943	907	907
query37	143	87	85	85
query38	3996	3975	3871	3871
query39	1429	1427	1395	1395
query40	201	114	117	114
query41	48	49	46	46
query42	121	101	98	98
query43	501	481	481	481
query44	1102	763	751	751
query45	195	168	166	166
query46	1096	752	756	752
query47	1929	1806	1814	1806
query48	379	301	304	301
query49	785	451	456	451
query50	850	416	430	416
query51	7082	6839	6963	6839
query52	97	89	88	88
query53	254	181	183	181
query54	580	487	470	470
query55	80	74	77	74
query56	292	266	262	262
query57	1216	1055	1073	1055
query58	220	291	221	221
query59	3035	2835	2744	2744
query60	286	273	268	268
query61	104	100	101	100
query62	755	663	676	663
query63	215	185	182	182
query64	1323	670	695	670
query65	3243	3150	3155	3150
query66	661	336	357	336
query67	15740	15665	15147	15147
query68	1910	552	552	552
query69	402	269	272	269
query70	1143	1128	1140	1128
query71	332	277	267	267
query72	4620	4094	3921	3921
query73	756	328	332	328
query74	9171	8892	8906	8892
query75	3301	2768	2709	2709
query76	1356	1001	922	922
query77	519	330	326	326
query78	9789	9279	9212	9212
query79	944	902	860	860
query80	854	817	806	806
query81	532	256	261	256
query82	268	264	266	264
query83	192	186	188	186
query84	266	105	105	105
query85	759	404	402	402
query86	319	311	309	309
query87	4352	4460	4400	4400
query88	4848	4135	4144	4135
query89	371	366	363	363
query90	2015	309	312	309
query91	125	128	120	120
query92	79	73	75	73
query93	911	912	912	912
query94	723	359	382	359
query95	448	417	413	413
query96	472	473	473	473
query97	3109	3132	3092	3092
query98	231	241	235	235
query99	1447	1263	1242	1242
Total cold run time: 298454 ms
Total hot run time: 196957 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 31.72 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 6c64b0f1f985cc59f8878ff67656541bb60207c4, data reload: false

query1	0.04	0.04	0.05
query2	0.07	0.04	0.04
query3	0.22	0.05	0.05
query4	1.66	0.08	0.09
query5	0.50	0.50	0.50
query6	1.13	0.74	0.73
query7	0.02	0.01	0.02
query8	0.05	0.05	0.05
query9	0.53	0.50	0.48
query10	0.55	0.57	0.54
query11	0.15	0.12	0.11
query12	0.15	0.13	0.12
query13	0.60	0.59	0.58
query14	1.39	1.39	1.39
query15	0.85	0.85	0.82
query16	0.35	0.36	0.38
query17	0.99	0.96	1.03
query18	0.22	0.21	0.21
query19	1.84	1.75	1.82
query20	0.02	0.01	0.01
query21	15.40	0.66	0.66
query22	4.72	6.49	2.18
query23	18.33	1.41	1.29
query24	2.12	0.23	0.22
query25	0.15	0.09	0.09
query26	0.26	0.18	0.18
query27	0.07	0.08	0.08
query28	13.23	1.03	1.00
query29	12.59	3.38	3.33
query30	0.24	0.06	0.05
query31	2.87	0.40	0.38
query32	3.27	0.48	0.48
query33	2.97	2.99	2.98
query34	17.10	4.43	4.48
query35	4.44	4.46	4.48
query36	0.65	0.47	0.47
query37	0.19	0.16	0.15
query38	0.16	0.15	0.15
query39	0.05	0.04	0.04
query40	0.16	0.13	0.13
query41	0.10	0.05	0.05
query42	0.06	0.05	0.05
query43	0.05	0.04	0.04
Total cold run time: 110.51 s
Total hot run time: 31.72 s

xinyiZzz
xinyiZzz previously approved these changes Sep 11, 2024
View reviewed changes
Copy link
Contributor

@xinyiZzz xinyiZzz left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@xinyiZzz
Copy link
Contributor

image

@TangSiyang2001
Copy link
Collaborator Author

run buildall

@github-actions github-actions bot removed the approved Indicates a PR has been approved by one committer. label Sep 11, 2024
@TangSiyang2001
Copy link
Collaborator Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 42161 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 68aeb8f3aab502d49fd44ff2a6066b7c5f8d9fdf, data reload: false

------ Round 1 ----------------------------------
q1	17604	7332	7164	7164
q2	2041	179	151	151
q3	10685	1132	1307	1132
q4	10551	853	918	853
q5	7766	3136	3183	3136
q6	230	144	143	143
q7	1003	616	592	592
q8	9423	2041	2036	2036
q9	6858	6426	6448	6426
q10	7058	2316	2354	2316
q11	451	250	263	250
q12	403	215	214	214
q13	17794	3005	3039	3005
q14	238	210	236	210
q15	575	515	530	515
q16	520	426	414	414
q17	995	922	921	921
q18	7389	6789	6772	6772
q19	1405	1232	1228	1228
q20	569	299	288	288
q21	3953	3406	3416	3406
q22	1128	989	1005	989
Total cold run time: 108639 ms
Total hot run time: 42161 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7138	7130	7144	7130
q2	319	224	222	222
q3	3104	3072	3010	3010
q4	2063	2039	1993	1993
q5	5608	5639	5717	5639
q6	241	146	140	140
q7	2174	1752	1790	1752
q8	3332	3414	3401	3401
q9	8798	8943	8784	8784
q10	3500	3517	3559	3517
q11	585	486	487	486
q12	859	612	589	589
q13	10315	3171	3122	3122
q14	306	272	273	272
q15	576	538	521	521
q16	512	474	473	473
q17	1792	1759	1730	1730
q18	8483	7990	8278	7990
q19	1770	1760	1752	1752
q20	2132	1871	1858	1858
q21	5563	5085	5418	5085
q22	1148	1051	1027	1027
Total cold run time: 70318 ms
Total hot run time: 60493 ms

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 37.32% (9577/25662)
Line Coverage: 28.70% (79128/275720)
Region Coverage: 28.18% (40968/145375)
Branch Coverage: 24.81% (20883/84188)
Coverage Report: http://coverage.selectdb-in.cc/coverage/68aeb8f3aab502d49fd44ff2a6066b7c5f8d9fdf_68aeb8f3aab502d49fd44ff2a6066b7c5f8d9fdf/report/index.html

@doris-robot
Copy link

TPC-DS: Total hot run time: 198315 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 68aeb8f3aab502d49fd44ff2a6066b7c5f8d9fdf, data reload: false

query1	1312	875	885	875
query2	6304	1789	1763	1763
query3	10767	4063	3770	3770
query4	55438	27011	23667	23667
query5	5245	523	482	482
query6	417	166	160	160
query7	5772	320	298	298
query8	272	216	216	216
query9	7340	2563	2563	2563
query10	455	285	278	278
query11	16709	15680	15392	15392
query12	147	100	95	95
query13	1487	462	412	412
query14	10550	7329	7171	7171
query15	196	189	189	189
query16	7100	514	477	477
query17	1180	604	576	576
query18	1870	315	313	313
query19	210	160	162	160
query20	123	110	117	110
query21	212	106	104	104
query22	4585	4691	4517	4517
query23	34802	33818	33753	33753
query24	6094	3189	3103	3103
query25	480	375	387	375
query26	635	160	165	160
query27	1694	282	284	282
query28	2860	2075	2057	2057
query29	659	396	401	396
query30	232	146	144	144
query31	949	761	780	761
query32	70	49	52	49
query33	437	278	295	278
query34	889	481	466	466
query35	847	725	754	725
query36	1023	898	914	898
query37	135	73	74	73
query38	4078	4018	3944	3944
query39	1455	1406	1411	1406
query40	200	95	91	91
query41	46	45	44	44
query42	115	94	94	94
query43	491	461	447	447
query44	1171	776	769	769
query45	194	165	167	165
query46	1092	825	805	805
query47	1906	1804	1846	1804
query48	453	341	356	341
query49	700	395	402	395
query50	912	431	430	430
query51	7071	6905	6848	6848
query52	98	85	89	85
query53	250	178	172	172
query54	553	447	452	447
query55	78	75	81	75
query56	267	250	259	250
query57	1221	1104	1093	1093
query58	221	228	236	228
query59	2759	2611	2568	2568
query60	282	268	290	268
query61	104	100	99	99
query62	768	652	665	652
query63	227	183	178	178
query64	1582	619	605	605
query65	3224	3138	3163	3138
query66	684	290	293	290
query67	15906	15534	15661	15534
query68	1974	860	853	853
query69	485	336	340	336
query70	1152	1165	1148	1148
query71	334	334	329	329
query72	4391	3326	3308	3308
query73	583	587	583	583
query74	9201	8986	9044	8986
query75	3059	2931	2999	2931
query76	917	841	839	839
query77	474	348	347	347
query78	9383	9263	9273	9263
query79	917	885	860	860
query80	585	579	566	566
query81	444	238	232	232
query82	199	195	198	195
query83	156	162	160	160
query84	215	97	101	97
query85	641	351	363	351
query86	307	330	318	318
query87	4378	4334	4397	4334
query88	4233	4109	4111	4109
query89	367	361	359	359
query90	1289	306	299	299
query91	160	163	160	160
query92	72	73	74	73
query93	1021	1014	1015	1014
query94	506	386	354	354
query95	451	418	413	413
query96	474	476	473	473
query97	3175	3142	3121	3121
query98	228	231	232	231
query99	1321	1300	1322	1300
Total cold run time: 290697 ms
Total hot run time: 198315 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 31.11 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 68aeb8f3aab502d49fd44ff2a6066b7c5f8d9fdf, data reload: false

query1	0.05	0.04	0.04
query2	0.06	0.02	0.02
query3	0.22	0.06	0.06
query4	1.65	0.10	0.10
query5	0.51	0.49	0.50
query6	1.13	0.74	0.72
query7	0.02	0.02	0.02
query8	0.04	0.03	0.03
query9	0.57	0.51	0.51
query10	0.56	0.57	0.57
query11	0.14	0.11	0.10
query12	0.14	0.10	0.11
query13	0.61	0.59	0.59
query14	1.41	1.43	1.46
query15	0.87	0.88	0.85
query16	0.36	0.37	0.39
query17	1.02	1.05	1.08
query18	0.19	0.18	0.20
query19	1.95	1.82	1.78
query20	0.01	0.01	0.01
query21	15.38	0.55	0.55
query22	3.17	4.06	2.69
query23	17.59	1.03	0.93
query24	2.33	0.27	0.21
query25	0.24	0.05	0.06
query26	0.16	0.13	0.14
query27	0.04	0.03	0.03
query28	12.45	1.02	0.99
query29	12.59	3.27	3.28
query30	0.25	0.06	0.05
query31	2.88	0.39	0.37
query32	3.32	0.47	0.46
query33	3.01	3.03	3.05
query34	15.42	4.31	4.26
query35	4.28	4.33	4.29
query36	0.71	0.48	0.47
query37	0.08	0.06	0.06
query38	0.05	0.04	0.03
query39	0.03	0.02	0.02
query40	0.15	0.14	0.13
query41	0.07	0.02	0.02
query42	0.04	0.02	0.02
query43	0.03	0.03	0.03
Total cold run time: 105.78 s
Total hot run time: 31.11 s

gavinchou
gavinchou previously approved these changes Sep 19, 2024
View reviewed changes
@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Sep 19, 2024
@github-actions GitHub Actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

xinyiZzz
xinyiZzz previously approved these changes Sep 19, 2024
View reviewed changes
Copy link
Contributor

@xinyiZzz xinyiZzz left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@TangSiyang2001
Copy link
Collaborator Author

run buildall

@github-actions github-actions bot removed the approved Indicates a PR has been approved by one committer. label Sep 19, 2024
@doris-robot
Copy link

TPC-H: Total hot run time: 42086 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 7302a5c966cf23f047d874962a5e37e365b1a94c, data reload: false

------ Round 1 ----------------------------------
q1	18200	7437	7364	7364
q2	2815	161	170	161
q3	11360	1167	1272	1167
q4	10585	815	740	740
q5	8321	3165	3162	3162
q6	239	162	151	151
q7	1033	625	612	612
q8	9438	2076	2070	2070
q9	6952	6448	6435	6435
q10	7032	2283	2301	2283
q11	443	251	258	251
q12	417	213	212	212
q13	17785	3018	2991	2991
q14	261	212	219	212
q15	589	537	509	509
q16	666	601	613	601
q17	994	762	795	762
q18	7323	6815	6795	6795
q19	1407	1022	1004	1004
q20	610	299	288	288
q21	4161	3417	3319	3319
q22	1146	997	1022	997
Total cold run time: 111777 ms
Total hot run time: 42086 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7229	7270	7180	7180
q2	331	232	226	226
q3	2869	2764	2776	2764
q4	1922	1701	1668	1668
q5	5391	5411	5402	5402
q6	226	138	140	138
q7	2098	1701	1701	1701
q8	3211	3328	3363	3328
q9	8414	8390	8439	8390
q10	3407	3338	3357	3338
q11	576	469	478	469
q12	788	591	640	591
q13	5755	2984	2999	2984
q14	301	263	264	263
q15	575	526	517	517
q16	699	657	669	657
q17	1752	1543	1537	1537
q18	7771	7223	7425	7223
q19	1676	1700	1460	1460
q20	2052	1794	1798	1794
q21	5343	5268	5203	5203
q22	1090	1020	1028	1020
Total cold run time: 63476 ms
Total hot run time: 57853 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 194073 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 7302a5c966cf23f047d874962a5e37e365b1a94c, data reload: false

query1	974	368	360	360
query2	6507	2112	2097	2097
query3	6694	217	224	217
query4	34132	23282	23469	23282
query5	4269	478	470	470
query6	261	163	157	157
query7	4637	294	302	294
query8	268	214	219	214
query9	9624	2667	2680	2667
query10	449	288	300	288
query11	18175	15067	15169	15067
query12	158	102	98	98
query13	1617	429	395	395
query14	10120	7111	7472	7111
query15	280	174	174	174
query16	8079	465	492	465
query17	1669	575	546	546
query18	2114	313	307	307
query19	349	145	148	145
query20	113	106	106	106
query21	215	108	103	103
query22	4481	4344	4396	4344
query23	34752	34110	33939	33939
query24	11234	2982	2924	2924
query25	653	398	404	398
query26	1347	167	158	158
query27	2819	289	289	289
query28	8349	2487	2481	2481
query29	885	422	420	420
query30	323	157	156	156
query31	1055	798	812	798
query32	94	53	57	53
query33	762	295	289	289
query34	965	480	499	480
query35	872	709	719	709
query36	1110	955	969	955
query37	154	91	79	79
query38	3944	3910	3926	3910
query39	1470	1422	1399	1399
query40	277	94	94	94
query41	49	47	46	46
query42	115	93	94	93
query43	526	485	494	485
query44	1277	832	800	800
query45	192	160	163	160
query46	1143	768	757	757
query47	1937	1812	1831	1812
query48	453	361	364	361
query49	1135	400	395	395
query50	821	398	398	398
query51	7116	6850	6900	6850
query52	97	85	82	82
query53	257	186	178	178
query54	1288	457	458	457
query55	79	77	76	76
query56	285	256	255	255
query57	1194	1129	1096	1096
query58	239	224	221	221
query59	3118	3092	3059	3059
query60	296	271	269	269
query61	105	103	100	100
query62	870	658	675	658
query63	214	189	182	182
query64	5271	630	619	619
query65	3268	3149	3162	3149
query66	1435	308	318	308
query67	16243	15806	15586	15586
query68	3102	873	876	873
query69	448	336	340	336
query70	1205	1152	1184	1152
query71	329	351	326	326
query72	5914	3384	2967	2967
query73	594	576	569	569
query74	9567	8899	8958	8899
query75	3094	2824	2875	2824
query76	1934	852	846	846
query77	395	353	363	353
query78	9396	9301	9921	9301
query79	906	890	866	866
query80	596	572	568	568
query81	455	250	253	250
query82	231	229	229	229
query83	157	158	166	158
query84	231	105	99	99
query85	671	370	344	344
query86	330	307	311	307
query87	4359	4272	4378	4272
query88	4336	4016	4003	4003
query89	367	354	350	350
query90	1515	303	303	303
query91	163	162	165	162
query92	81	73	70	70
query93	915	902	890	890
query94	574	371	356	356
query95	420	413	399	399
query96	481	481	481	481
query97	3107	3145	3119	3119
query98	219	229	225	225
query99	1409	1291	1286	1286
Total cold run time: 292774 ms
Total hot run time: 194073 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 32.37 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 7302a5c966cf23f047d874962a5e37e365b1a94c, data reload: false

query1	0.05	0.04	0.04
query2	0.06	0.02	0.03
query3	0.23	0.06	0.06
query4	1.64	0.10	0.10
query5	0.49	0.52	0.52
query6	1.13	0.72	0.71
query7	0.02	0.02	0.02
query8	0.03	0.02	0.03
query9	0.56	0.52	0.50
query10	0.54	0.57	0.55
query11	0.14	0.10	0.10
query12	0.13	0.10	0.10
query13	0.59	0.60	0.59
query14	3.14	3.00	3.05
query15	0.88	0.82	0.81
query16	0.38	0.36	0.37
query17	1.00	1.00	0.98
query18	0.23	0.21	0.21
query19	1.97	1.89	1.86
query20	0.01	0.01	0.01
query21	15.35	0.58	0.57
query22	2.18	2.48	1.98
query23	17.30	0.88	0.72
query24	2.48	1.28	0.56
query25	0.16	0.18	0.10
query26	0.42	0.14	0.13
query27	0.04	0.05	0.03
query28	11.28	1.09	1.06
query29	12.58	3.20	3.19
query30	0.24	0.06	0.05
query31	2.88	0.39	0.38
query32	3.24	0.46	0.45
query33	2.99	2.99	3.04
query34	16.82	4.39	4.42
query35	4.46	4.45	4.44
query36	0.68	0.48	0.49
query37	0.08	0.06	0.06
query38	0.04	0.04	0.04
query39	0.04	0.02	0.02
query40	0.15	0.12	0.13
query41	0.08	0.02	0.02
query42	0.03	0.02	0.01
query43	0.03	0.03	0.02
Total cold run time: 106.77 s
Total hot run time: 32.37 s

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 37.33% (9584/25676)
Line Coverage: 28.71% (79202/275901)
Region Coverage: 28.18% (41013/145524)
Branch Coverage: 24.82% (20901/84224)
Coverage Report: http://coverage.selectdb-in.cc/coverage/7302a5c966cf23f047d874962a5e37e365b1a94c_7302a5c966cf23f047d874962a5e37e365b1a94c/report/index.html

@TangSiyang2001
Copy link
Collaborator Author

run external

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Sep 20, 2024
@github-actions GitHub Actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

xinyiZzz
xinyiZzz approved these changes Sep 26, 2024
View reviewed changes
Copy link
Contributor

@xinyiZzz xinyiZzz left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@xinyiZzz xinyiZzz merged commit 62a64d1 into apache:master Sep 26, 2024
25 of 30 checks passed
@kaijchen kaijchen mentioned this pull request Sep 26, 2024
Merged
yiguolei pushed a commit that referenced this pull request Sep 27, 2024
@kaijchen
Revert "[enhancement](mem-tracker) Use thread local mem tracker to tr…
fd2e20f 
…ack s3 file buffer memory usage" (#41360)

Reverts #40597

This PR cased memtable memtracker check in cloud_p0 to fail.

```
*** Check failure stack trace: ***
[17:31:12 ](http://43.132.222.7:8111/buildConfiguration/Doris_DorisRegression_CloudP0/536070?buildTab=log&linesState=43518&logView=flowAware&focusLine=43518)  F20240926 17:27:30.342573 15767 memtable.cpp:145] memtable flush success but cosumption is not 0, it is 5242880F20240926 17:27:30.357038 15763 memtable.cpp:145] memtable flush success but cosumption is not 0, it is 5242880F20240926 17:27:30.357043 15762
[17:31:12 ](http://43.132.222.7:8111/buildConfiguration/Doris_DorisRegression_CloudP0/536070?buildTab=log&linesState=43519&logView=flowAware&focusLine=43519)   mtable.cpp:145] memtable flush success but cosumption is not 0, it is 5242880
[17:31:12 ](http://43.132.222.7:8111/buildConfiguration/Doris_DorisRegression_CloudP0/536070?buildTab=log&linesState=43520&logView=flowAware&focusLine=43520)  *** Check failure stack trace: ***
[17:31:12 ](http://43.132.222.7:8111/buildConfiguration/Doris_DorisRegression_CloudP0/536070?buildTab=log&linesState=43521&logView=flowAware&focusLine=43521)  F20240926 17:27:30.342573 15767 memtable.cpp:145] memtable flush success but cosumption is not 0, it is 5242880F20240926 17:27:30.357038 15763 memtable.cpp:145] memtable flush success but cosumption is not 0, it is 5242880F20240926 17:27:30.357043 15762
[17:31:12 ](http://43.132.222.7:8111/buildConfiguration/Doris_DorisRegression_CloudP0/536070?buildTab=log&linesState=43522&logView=flowAware&focusLine=43522)   mtable.cpp:145] memtable flush success but cosumption is not 0, it is 5242880F20240926 17:27:30.358129 15765 memtable.cpp:145] memtable flush success but cosumption is not 0, it is 5242880
[17:31:12 ](http://43.132.222.7:8111/buildConfiguration/Doris_DorisRegression_CloudP0/536070?buildTab=log&linesState=43523&logView=flowAware&focusLine=43523)  *** Check failure stack trace: ***
[17:31:12 ](http://43.132.222.7:8111/buildConfiguration/Doris_DorisRegression_CloudP0/536070?buildTab=log&linesState=43524&logView=flowAware&focusLine=43524)  F20240926 17:27:30.342573 15767 memtable.cpp:145] memtable flush success but cosumption is not 0, it is 5242880F20240926 17:27:30.357038 15763 memtable.cpp:145] memtable flush success but cosumption is not 0, it is 5242880F20240926 17:27:30.357043 15762
[17:31:12 ](http://43.132.222.7:8111/buildConfiguration/Doris_DorisRegression_CloudP0/536070?buildTab=log&linesState=43525&logView=flowAware&focusLine=43525)   mtable.cpp:145] memtable flush success but cosumption is not 0, it is 5242880F20240926 17:27:30.358129 15765 memtable.cpp:145] memtable flush success but cosumption is not 0, it is 5242880F20240926 17:27:30.358175 15766 memtable.cpp:145] memtable flush success but cosumption is not 0, it is 5242880
[17:31:12 ](http://43.132.222.7:8111/buildConfiguration/Doris_DorisRegression_CloudP0/536070?buildTab=log&linesState=43526&logView=flowAware&focusLine=43526)  *** Check failure stack trace: ***
```


http://43.132.222.7:8111/buildConfiguration/Doris_DorisRegression_CloudP0/536070?buildTab=log&linesState=25290&logView=flowAware&focusLine=44023
dataroaring pushed a commit that referenced this pull request Oct 9, 2024
@TangSiyang2001 @dataroaring
[enhancement](mem-tracker) Use thread local mem tracker to track s3 f…
86a26d7 
…ile buffer memory usage (#40597)

Track s3 file buffer memory usage with thread local tracker, so that
memory usage will be specified to detail.
dataroaring pushed a commit that referenced this pull request Oct 9, 2024
@kaijchen @dataroaring
Revert "[enhancement](mem-tracker) Use thread local mem tracker to tr…
b0e20ac 
…ack s3 file buffer memory usage" (#41360)

Reverts #40597

This PR cased memtable memtracker check in cloud_p0 to fail.

```
*** Check failure stack trace: ***
[17:31:12 ](http://43.132.222.7:8111/buildConfiguration/Doris_DorisRegression_CloudP0/536070?buildTab=log&linesState=43518&logView=flowAware&focusLine=43518)  F20240926 17:27:30.342573 15767 memtable.cpp:145] memtable flush success but cosumption is not 0, it is 5242880F20240926 17:27:30.357038 15763 memtable.cpp:145] memtable flush success but cosumption is not 0, it is 5242880F20240926 17:27:30.357043 15762
[17:31:12 ](http://43.132.222.7:8111/buildConfiguration/Doris_DorisRegression_CloudP0/536070?buildTab=log&linesState=43519&logView=flowAware&focusLine=43519)   mtable.cpp:145] memtable flush success but cosumption is not 0, it is 5242880
[17:31:12 ](http://43.132.222.7:8111/buildConfiguration/Doris_DorisRegression_CloudP0/536070?buildTab=log&linesState=43520&logView=flowAware&focusLine=43520)  *** Check failure stack trace: ***
[17:31:12 ](http://43.132.222.7:8111/buildConfiguration/Doris_DorisRegression_CloudP0/536070?buildTab=log&linesState=43521&logView=flowAware&focusLine=43521)  F20240926 17:27:30.342573 15767 memtable.cpp:145] memtable flush success but cosumption is not 0, it is 5242880F20240926 17:27:30.357038 15763 memtable.cpp:145] memtable flush success but cosumption is not 0, it is 5242880F20240926 17:27:30.357043 15762
[17:31:12 ](http://43.132.222.7:8111/buildConfiguration/Doris_DorisRegression_CloudP0/536070?buildTab=log&linesState=43522&logView=flowAware&focusLine=43522)   mtable.cpp:145] memtable flush success but cosumption is not 0, it is 5242880F20240926 17:27:30.358129 15765 memtable.cpp:145] memtable flush success but cosumption is not 0, it is 5242880
[17:31:12 ](http://43.132.222.7:8111/buildConfiguration/Doris_DorisRegression_CloudP0/536070?buildTab=log&linesState=43523&logView=flowAware&focusLine=43523)  *** Check failure stack trace: ***
[17:31:12 ](http://43.132.222.7:8111/buildConfiguration/Doris_DorisRegression_CloudP0/536070?buildTab=log&linesState=43524&logView=flowAware&focusLine=43524)  F20240926 17:27:30.342573 15767 memtable.cpp:145] memtable flush success but cosumption is not 0, it is 5242880F20240926 17:27:30.357038 15763 memtable.cpp:145] memtable flush success but cosumption is not 0, it is 5242880F20240926 17:27:30.357043 15762
[17:31:12 ](http://43.132.222.7:8111/buildConfiguration/Doris_DorisRegression_CloudP0/536070?buildTab=log&linesState=43525&logView=flowAware&focusLine=43525)   mtable.cpp:145] memtable flush success but cosumption is not 0, it is 5242880F20240926 17:27:30.358129 15765 memtable.cpp:145] memtable flush success but cosumption is not 0, it is 5242880F20240926 17:27:30.358175 15766 memtable.cpp:145] memtable flush success but cosumption is not 0, it is 5242880
[17:31:12 ](http://43.132.222.7:8111/buildConfiguration/Doris_DorisRegression_CloudP0/536070?buildTab=log&linesState=43526&logView=flowAware&focusLine=43526)  *** Check failure stack trace: ***
```


http://43.132.222.7:8111/buildConfiguration/Doris_DorisRegression_CloudP0/536070?buildTab=log&linesState=25290&logView=flowAware&focusLine=44023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@gavinchou gavinchou gavinchou approved these changes

@xinyiZzz xinyiZzz xinyiZzz approved these changes

@yiguolei yiguolei yiguolei left review comments

Assignees
No one assigned
Labels
approved Indicates a PR has been approved by one committer. reviewed
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@TangSiyang2001 @doris-robot @xinyiZzz @gavinchou @yiguolei