Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Feature](inverted index) add inverted index reader memory size into searcher cache #35149

Merged
merged 1 commit into from
May 22, 2024
Merged

[Feature](inverted index) add inverted index reader memory size into searcher cache #35149

merged 1 commit into from
May 22, 2024

Conversation

airborne12
Copy link
Member

Proposed changes

Issue Number: close #xxx

Further comments

If this is a relatively large or complex change, kick off the discussion at [email protected] by explaining why you chose the solution you did and what alternatives you considered, etc...

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

qidaye
qidaye approved these changes May 21, 2024
View reviewed changes
Copy link
Contributor

@qidaye qidaye left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@qidaye
Copy link
Contributor

qidaye commented May 21, 2024

run buildall

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label May 21, 2024
@github-actions GitHub Actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions GitHub Actions
Copy link
Contributor

PR approved by anyone and no changes requested.

zzzxl1993
zzzxl1993 approved these changes May 21, 2024
View reviewed changes
Copy link
Contributor

@zzzxl1993 zzzxl1993 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions GitHub Actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

TPC-H: Total hot run time: 40432 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 96992bd64eb6e0e91a55d69b6b6b7af7b531f9a0, data reload: false

------ Round 1 ----------------------------------
q1	17603	5118	4215	4215
q2	2017	197	195	195
q3	10783	1158	1227	1158
q4	10615	763	715	715
q5	7622	2724	2608	2608
q6	217	133	134	133
q7	949	555	533	533
q8	9426	2086	2125	2086
q9	9872	7368	7227	7227
q10	9348	3669	3701	3669
q11	446	246	239	239
q12	511	212	217	212
q13	18552	2963	2952	2952
q14	249	208	223	208
q15	527	481	478	478
q16	492	383	379	379
q17	969	612	662	612
q18	8166	7481	7499	7481
q19	1622	1545	1532	1532
q20	644	301	304	301
q21	5074	3895	3229	3229
q22	343	270	274	270
Total cold run time: 116047 ms
Total hot run time: 40432 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4322	4240	4211	4211
q2	375	269	259	259
q3	2938	2764	2683	2683
q4	1839	1546	1557	1546
q5	5286	5244	5286	5244
q6	213	123	122	122
q7	1737	1363	1393	1363
q8	3184	3369	3312	3312
q9	8924	9010	8970	8970
q10	3837	3715	3642	3642
q11	578	482	493	482
q12	761	593	591	591
q13	15688	2973	3009	2973
q14	280	248	253	248
q15	518	462	480	462
q16	466	407	405	405
q17	1758	1483	1451	1451
q18	7678	7567	7455	7455
q19	1632	1599	1535	1535
q20	1996	1753	1801	1753
q21	4807	4774	4924	4774
q22	562	481	477	477
Total cold run time: 69379 ms
Total hot run time: 53958 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 180706 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 96992bd64eb6e0e91a55d69b6b6b7af7b531f9a0, data reload: false

query1	915	374	363	363
query2	6475	2514	2375	2375
query3	6660	218	223	218
query4	24086	21236	21326	21236
query5	4187	421	424	421
query6	270	178	168	168
query7	4576	301	287	287
query8	242	190	199	190
query9	8640	2398	2361	2361
query10	437	239	249	239
query11	14708	14195	14451	14195
query12	135	86	88	86
query13	1636	371	366	366
query14	10606	6821	7655	6821
query15	217	166	169	166
query16	7814	254	251	251
query17	1869	559	539	539
query18	1949	276	262	262
query19	213	174	149	149
query20	94	83	86	83
query21	199	128	134	128
query22	4982	4787	4779	4779
query23	33920	33447	33792	33447
query24	11574	2871	2877	2871
query25	693	377	361	361
query26	1794	153	150	150
query27	3043	318	315	315
query28	7534	2066	2061	2061
query29	1173	605	581	581
query30	298	175	177	175
query31	979	748	752	748
query32	84	52	56	52
query33	741	252	251	251
query34	1089	475	479	475
query35	817	697	671	671
query36	1114	888	905	888
query37	179	69	68	68
query38	2891	2763	2735	2735
query39	1623	1559	1584	1559
query40	273	123	127	123
query41	46	42	42	42
query42	103	95	97	95
query43	589	574	577	574
query44	1170	741	752	741
query45	268	254	249	249
query46	1072	727	736	727
query47	1963	1857	1884	1857
query48	373	295	293	293
query49	1182	384	393	384
query50	763	390	385	385
query51	6894	6763	6720	6720
query52	103	86	95	86
query53	350	285	294	285
query54	861	428	427	427
query55	76	75	72	72
query56	267	223	220	220
query57	1271	1138	1166	1138
query58	213	197	197	197
query59	3692	3384	3432	3384
query60	242	228	232	228
query61	89	90	86	86
query62	687	494	472	472
query63	305	288	279	279
query64	9764	2209	1732	1732
query65	3208	3116	3126	3116
query66	1362	376	334	334
query67	15345	15174	15025	15025
query68	4592	561	549	549
query69	475	305	298	298
query70	1132	1144	1106	1106
query71	368	264	258	258
query72	7130	2583	2334	2334
query73	718	321	331	321
query74	6659	6142	6068	6068
query75	3296	2607	2575	2575
query76	2619	861	1006	861
query77	428	265	265	265
query78	10594	10082	10096	10082
query79	1684	520	538	520
query80	1101	461	480	461
query81	520	243	239	239
query82	949	94	99	94
query83	241	163	163	163
query84	234	85	85	85
query85	1289	262	255	255
query86	397	312	296	296
query87	3284	3099	3103	3099
query88	3377	2364	2381	2364
query89	455	372	385	372
query90	1893	185	187	185
query91	122	108	99	99
query92	53	51	50	50
query93	1657	529	509	509
query94	1241	177	179	177
query95	396	301	292	292
query96	585	270	265	265
query97	3162	2999	3016	2999
query98	242	223	223	223
query99	1153	880	909	880
Total cold run time: 285033 ms
Total hot run time: 180706 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.19 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 96992bd64eb6e0e91a55d69b6b6b7af7b531f9a0, data reload: false

query1	0.04	0.04	0.04
query2	0.07	0.04	0.04
query3	0.23	0.04	0.06
query4	1.68	0.07	0.07
query5	0.50	0.49	0.49
query6	1.13	0.72	0.72
query7	0.02	0.01	0.02
query8	0.05	0.04	0.04
query9	0.55	0.49	0.49
query10	0.55	0.55	0.53
query11	0.15	0.12	0.11
query12	0.15	0.11	0.11
query13	0.59	0.59	0.60
query14	0.78	0.77	0.77
query15	0.83	0.81	0.81
query16	0.37	0.36	0.37
query17	0.95	0.94	0.98
query18	0.23	0.24	0.23
query19	1.74	1.66	1.71
query20	0.01	0.02	0.01
query21	15.54	0.67	0.67
query22	3.87	7.61	1.77
query23	18.23	1.37	1.29
query24	1.76	0.28	0.22
query25	0.14	0.08	0.08
query26	0.27	0.17	0.17
query27	0.08	0.07	0.07
query28	13.29	1.01	0.99
query29	12.67	3.25	3.28
query30	0.24	0.05	0.06
query31	2.88	0.38	0.37
query32	3.28	0.48	0.47
query33	2.84	2.92	2.87
query34	17.26	4.44	4.37
query35	4.50	4.51	4.47
query36	0.64	0.47	0.47
query37	0.17	0.14	0.16
query38	0.15	0.14	0.14
query39	0.05	0.03	0.03
query40	0.16	0.13	0.15
query41	0.08	0.05	0.04
query42	0.05	0.04	0.05
query43	0.04	0.03	0.04
Total cold run time: 108.81 s
Total hot run time: 30.19 s

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 35.70% (9018/25262)
Line Coverage: 27.34% (74554/272691)
Region Coverage: 26.56% (38545/145134)
Branch Coverage: 23.40% (19663/84018)
Coverage Report: http://coverage.selectdb-in.cc/coverage/96992bd64eb6e0e91a55d69b6b6b7af7b531f9a0_96992bd64eb6e0e91a55d69b6b6b7af7b531f9a0/report/index.html

@eldenmoon eldenmoon merged commit f77c53d into apache:master May 22, 2024
26 of 29 checks passed
dataroaring pushed a commit that referenced this pull request May 26, 2024
@airborne12 @dataroaring
[Feature](inverted index) add inverted index reader memory size into …
1d550f6 
…searcher cache (#35149)
airborne12 added a commit to airborne12/apache-doris that referenced this pull request Jun 12, 2024
@airborne12
[Feature](inverted index) add inverted index reader memory size into …
4b347f5 
…searcher cache (apache#35149)
@airborne12 airborne12 mentioned this pull request Jun 12, 2024
Merged
qidaye pushed a commit that referenced this pull request Jun 12, 2024
@airborne12
[Pick 2.1](inverted index) add inverted index reader memory size into…
14ece32 
… searcher cache (#36160)

Pick from #35149
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@qidaye qidaye qidaye approved these changes

@zzzxl1993 zzzxl1993 zzzxl1993 approved these changes

Assignees
No one assigned
Labels
approved Indicates a PR has been approved by one committer. dev/2.1.4-merged dev/3.0.0-merged reviewed
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
8 participants
@airborne12 @doris-robot @qidaye @zzzxl1993 @xiaokang @yiguolei @eldenmoon @dataroaring