Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[bugfix](loadmem) load memory is too small because flush memory is not right #40696

Merged
merged 1 commit into from
Sep 12, 2024
Merged

[bugfix](loadmem) load memory is too small because flush memory is not right #40696

merged 1 commit into from
Sep 12, 2024

Conversation

yiguolei
Copy link
Contributor

@yiguolei yiguolei commented Sep 11, 2024 •
edited
Loading

Proposed changes

image

to block will clear input block and will release some memory, so if scoped, then the result will be small

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@yiguolei
Copy link
Contributor Author

run buildall

@github-actions GitHub Actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

xinyiZzz
xinyiZzz approved these changes Sep 11, 2024
View reviewed changes
Copy link
Contributor

@xinyiZzz xinyiZzz left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Sep 11, 2024
@github-actions GitHub Actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions GitHub Actions
Copy link
Contributor

PR approved by anyone and no changes requested.

@doris-robot
Copy link

TPC-H: Total hot run time: 38606 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 2b216000a7122de151b1f3d4f4de128f25453b9d, data reload: false

------ Round 1 ----------------------------------
q1	17949	4552	4443	4443
q2	2525	195	186	186
q3	11686	1150	1180	1150
q4	10813	736	706	706
q5	8948	2881	2844	2844
q6	233	141	142	141
q7	970	624	590	590
q8	9336	2079	2093	2079
q9	7301	6591	6558	6558
q10	6996	2233	2280	2233
q11	460	244	255	244
q12	408	227	222	222
q13	17773	3124	3108	3108
q14	291	250	246	246
q15	534	484	483	483
q16	529	453	441	441
q17	988	618	668	618
q18	7367	6861	6995	6861
q19	1400	1094	1124	1094
q20	679	346	330	330
q21	4139	3144	3010	3010
q22	1123	1038	1019	1019
Total cold run time: 112448 ms
Total hot run time: 38606 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4458	4835	4386	4386
q2	378	273	289	273
q3	2868	2707	2645	2645
q4	1963	1627	1674	1627
q5	5440	5443	5460	5443
q6	227	136	132	132
q7	2145	1728	1780	1728
q8	3232	3388	3360	3360
q9	8501	8486	8509	8486
q10	3512	3212	3200	3200
q11	612	507	527	507
q12	815	617	633	617
q13	9730	3124	3073	3073
q14	312	278	289	278
q15	539	489	486	486
q16	517	468	481	468
q17	1811	1514	1501	1501
q18	7788	7550	7398	7398
q19	1670	1585	1636	1585
q20	2084	1822	1892	1822
q21	5490	5352	5225	5225
q22	1134	1045	1037	1037
Total cold run time: 65226 ms
Total hot run time: 55277 ms

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 36.90% (9456/25626)
Line Coverage: 28.24% (77751/275280)
Region Coverage: 27.64% (40127/145175)
Branch Coverage: 24.25% (20386/84068)
Coverage Report: http://coverage.selectdb-in.cc/coverage/2b216000a7122de151b1f3d4f4de128f25453b9d_2b216000a7122de151b1f3d4f4de128f25453b9d/report/index.html

@doris-robot
Copy link

TPC-DS: Total hot run time: 192773 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 2b216000a7122de151b1f3d4f4de128f25453b9d, data reload: false

query1	938	376	399	376
query2	6481	1854	1895	1854
query3	6655	211	227	211
query4	31312	23342	23196	23196
query5	4180	505	500	500
query6	261	174	165	165
query7	4595	302	298	298
query8	288	232	228	228
query9	8706	2518	2496	2496
query10	449	275	281	275
query11	17879	15031	15016	15016
query12	155	100	102	100
query13	1654	416	367	367
query14	9895	7473	7299	7299
query15	261	167	172	167
query16	8075	447	437	437
query17	1562	559	541	541
query18	2093	283	285	283
query19	260	157	145	145
query20	121	105	105	105
query21	206	103	106	103
query22	4467	4113	4180	4113
query23	34203	33823	33602	33602
query24	11183	2886	2834	2834
query25	606	376	392	376
query26	1209	153	154	153
query27	2346	276	277	276
query28	7285	2065	2047	2047
query29	831	430	420	420
query30	313	164	154	154
query31	1012	792	759	759
query32	100	58	61	58
query33	760	308	294	294
query34	955	474	483	474
query35	868	729	741	729
query36	1081	933	906	906
query37	162	100	85	85
query38	4004	3849	3907	3849
query39	1448	1468	1407	1407
query40	205	121	120	120
query41	50	51	49	49
query42	124	100	96	96
query43	508	472	472	472
query44	1258	765	744	744
query45	201	166	168	166
query46	1102	743	751	743
query47	1868	1783	1820	1783
query48	378	300	290	290
query49	1131	470	464	464
query50	837	406	417	406
query51	7032	6842	6793	6793
query52	100	89	87	87
query53	265	184	187	184
query54	1026	465	459	459
query55	79	77	79	77
query56	369	268	259	259
query57	1192	1056	1068	1056
query58	242	233	230	230
query59	3095	2787	2800	2787
query60	302	265	267	265
query61	102	97	100	97
query62	832	665	707	665
query63	230	190	181	181
query64	5322	670	647	647
query65	3241	3167	3185	3167
query66	1433	367	334	334
query67	15914	15457	15186	15186
query68	3140	863	846	846
query69	426	320	315	315
query70	1182	1128	1186	1128
query71	348	345	340	340
query72	6129	3528	3529	3528
query73	605	593	583	583
query74	9030	8886	8945	8886
query75	3052	2965	2943	2943
query76	1846	849	834	834
query77	492	404	401	401
query78	9342	10026	9232	9232
query79	924	897	868	868
query80	849	823	867	823
query81	444	266	263	263
query82	268	273	264	264
query83	194	191	196	191
query84	229	114	105	105
query85	641	400	378	378
query86	315	303	294	294
query87	4295	4320	4343	4320
query88	4409	4093	4118	4093
query89	380	370	374	370
query90	1529	326	320	320
query91	121	123	124	123
query92	81	74	72	72
query93	933	962	931	931
query94	590	376	350	350
query95	421	469	410	410
query96	475	474	478	474
query97	3077	3098	3111	3098
query98	234	226	225	225
query99	1365	1260	1292	1260
Total cold run time: 284663 ms
Total hot run time: 192773 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 31.78 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 2b216000a7122de151b1f3d4f4de128f25453b9d, data reload: false

query1	0.06	0.04	0.04
query2	0.08	0.04	0.04
query3	0.23	0.05	0.05
query4	1.67	0.08	0.07
query5	0.52	0.49	0.49
query6	1.13	0.73	0.73
query7	0.01	0.02	0.01
query8	0.06	0.04	0.05
query9	0.53	0.49	0.48
query10	0.56	0.56	0.55
query11	0.16	0.13	0.12
query12	0.15	0.13	0.12
query13	0.60	0.59	0.60
query14	1.43	1.42	1.42
query15	0.85	0.83	0.83
query16	0.38	0.37	0.36
query17	0.98	1.01	0.98
query18	0.22	0.20	0.21
query19	1.90	1.75	1.80
query20	0.01	0.01	0.00
query21	15.40	0.67	0.67
query22	4.51	6.36	2.17
query23	18.27	1.38	1.28
query24	2.11	0.22	0.23
query25	0.14	0.09	0.08
query26	0.27	0.19	0.17
query27	0.08	0.07	0.08
query28	13.23	1.03	1.01
query29	12.61	3.38	3.38
query30	0.24	0.05	0.06
query31	2.91	0.42	0.41
query32	3.23	0.48	0.48
query33	2.99	3.03	3.05
query34	16.78	4.39	4.45
query35	4.49	4.42	4.51
query36	0.68	0.48	0.51
query37	0.19	0.16	0.15
query38	0.16	0.15	0.14
query39	0.05	0.04	0.05
query40	0.15	0.13	0.13
query41	0.09	0.05	0.05
query42	0.06	0.05	0.05
query43	0.05	0.04	0.04
Total cold run time: 110.22 s
Total hot run time: 31.78 s

@yiguolei yiguolei merged commit b7516c9 into apache:master Sep 12, 2024
27 of 31 checks passed
yiguolei added a commit to yiguolei/incubator-doris that referenced this pull request Sep 16, 2024
@yiguolei @Doris-Extras
[bugfix](loadmem) load memory is too small because flush memory is no…
4541f32 
…t right (apache#40696)

## Proposed changes

<img width="946" alt="image"
src="https://github.com/user-attachments/assets/1a203137-57bd-4f54-8771-b1536ca5ddd3">


to block will clear input block and will release some memory, so if
scoped, then the result will be small

Co-authored-by: yiguolei <[email protected]>
yiguolei added a commit to yiguolei/incubator-doris that referenced this pull request Sep 16, 2024
@yiguolei @Doris-Extras
[bugfix](loadmem) load memory is too small because flush memory is no…
643d7d1 
…t right (apache#40696)

## Proposed changes

<img width="946" alt="image"
src="https://github.com/user-attachments/assets/1a203137-57bd-4f54-8771-b1536ca5ddd3">


to block will clear input block and will release some memory, so if
scoped, then the result will be small

Co-authored-by: yiguolei <[email protected]>
yiguolei added a commit that referenced this pull request Sep 16, 2024
@yiguolei
[bugfix](loadmem) load memory is too small because flush memory is no…
76c7bf0 
…t right (#40696) (#40884)
dataroaring pushed a commit that referenced this pull request Oct 9, 2024
@yiguolei @Doris-Extras @dataroaring
[bugfix](loadmem) load memory is too small because flush memory is no…
8838587 
…t right (#40696)

## Proposed changes

<img width="946" alt="image"
src="https://github.com/user-attachments/assets/1a203137-57bd-4f54-8771-b1536ca5ddd3">


to block will clear input block and will release some memory, so if
scoped, then the result will be small

Co-authored-by: yiguolei <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@kaijchen kaijchen kaijchen approved these changes

@xinyiZzz xinyiZzz xinyiZzz approved these changes

Assignees
No one assigned
Labels
approved Indicates a PR has been approved by one committer. dev/2.1.7-merged dev/3.0.x reviewed
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@yiguolei @doris-robot @kaijchen @xinyiZzz @Doris-Extras