Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Feature](inverted index) add no need read data optimize config #36686 #38652

Merged
merged 2 commits into from
Aug 1, 2024
Merged

[Feature](inverted index) add no need read data optimize config #36686 #38652

merged 2 commits into from
Aug 1, 2024

Conversation

airborne12
Copy link
Member

cherry pick from #36686

@airborne12
[Feature](inverted index) add no need read data optimize config (apac…
1a0ad08 
…he#36686)

Add an FE session variable for the no-read-data optimization in
segment_iterator to prevent incorrect optimizations.
@airborne12
Copy link
Member Author

run buildall

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@airborne12
Copy link
Member Author

run buildall

@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Aug 1, 2024

clang-tidy review says "All clean, LGTM! 👍"

1 similar comment
@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Aug 1, 2024

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

TPC-H: Total hot run time: 50311 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 3793f28fd01a2fea447d986a7b11ded084812edd, data reload: false

------ Round 1 ----------------------------------
q1	17864	4441	4393	4393
q2	2077	159	155	155
q3	10429	1909	1942	1909
q4	10349	1304	1354	1304
q5	8641	3894	3936	3894
q6	233	125	125	125
q7	2043	1634	1611	1611
q8	9349	2747	2725	2725
q9	10585	10350	10315	10315
q10	8893	3670	3647	3647
q11	427	243	251	243
q12	486	315	314	314
q13	18384	3977	4044	3977
q14	354	334	328	328
q15	511	457	457	457
q16	691	572	578	572
q17	1156	970	969	969
q18	7413	6884	7026	6884
q19	1812	1652	1614	1614
q20	522	323	312	312
q21	4395	4174	4118	4118
q22	533	445	459	445
Total cold run time: 117147 ms
Total hot run time: 50311 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4361	4318	4402	4318
q2	322	233	226	226
q3	4217	4170	4155	4155
q4	2772	2749	2753	2749
q5	7162	7123	7153	7123
q6	244	118	119	118
q7	3253	2843	2789	2789
q8	4381	4458	4516	4458
q9	16941	16878	16646	16646
q10	4271	4271	4267	4267
q11	848	713	684	684
q12	1022	877	873	873
q13	7161	3732	3749	3732
q14	456	426	427	426
q15	503	465	464	464
q16	767	684	683	683
q17	3820	3853	3809	3809
q18	8855	8716	8796	8716
q19	1740	1737	1680	1680
q20	2388	2131	2094	2094
q21	8497	8523	8505	8505
q22	1081	994	946	946
Total cold run time: 85062 ms
Total hot run time: 79461 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 204108 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 3793f28fd01a2fea447d986a7b11ded084812edd, data reload: false

query1	921	430	386	386
query2	6533	2785	2724	2724
query3	6919	214	208	208
query4	20176	18159	18041	18041
query5	19725	6565	6555	6555
query6	275	210	226	210
query7	4141	298	317	298
query8	435	431	417	417
query9	3082	2646	2564	2564
query10	410	306	318	306
query11	11312	10768	10810	10768
query12	129	75	74	74
query13	5615	705	695	695
query14	17726	13402	13400	13400
query15	359	244	249	244
query16	6476	299	288	288
query17	1656	1523	876	876
query18	2312	421	401	401
query19	210	150	148	148
query20	78	76	79	76
query21	194	101	97	97
query22	5138	5058	5075	5058
query23	32579	31976	32056	31976
query24	6973	6567	6536	6536
query25	514	414	440	414
query26	528	164	160	160
query27	1852	297	295	295
query28	6173	2362	2326	2326
query29	2942	2828	2883	2828
query30	249	169	163	163
query31	904	750	736	736
query32	75	62	73	62
query33	397	255	255	255
query34	863	480	461	461
query35	1151	895	916	895
query36	1204	1155	1216	1155
query37	89	59	60	59
query38	3039	2923	3005	2923
query39	1377	1309	1319	1309
query40	207	99	95	95
query41	46	44	44	44
query42	86	97	87	87
query43	735	636	663	636
query44	1155	717	715	715
query45	248	235	238	235
query46	1219	948	979	948
query47	1882	1836	1933	1836
query48	1018	711	715	711
query49	626	378	363	363
query50	852	594	631	594
query51	4702	4666	4640	4640
query52	93	97	79	79
query53	450	328	316	316
query54	2644	2461	2480	2461
query55	83	82	80	80
query56	246	217	218	217
query57	1162	1056	1081	1056
query58	223	216	189	189
query59	4239	4066	3798	3798
query60	213	214	212	212
query61	97	93	106	93
query62	836	448	494	448
query63	485	328	338	328
query64	2440	1531	1489	1489
query65	3627	3577	3580	3577
query66	807	381	380	380
query67	16505	17418	15542	15542
query68	8456	641	640	640
query69	571	332	343	332
query70	1721	1553	1498	1498
query71	399	328	320	320
query72	6503	3445	3493	3445
query73	730	317	311	311
query74	6358	5823	5853	5823
query75	4698	3658	3649	3649
query76	4724	1089	1134	1089
query77	632	248	252	248
query78	12588	11740	11568	11568
query79	8856	649	628	628
query80	1298	404	399	399
query81	503	235	236	235
query82	1689	104	99	99
query83	179	137	129	129
query84	257	72	74	72
query85	900	334	334	334
query86	327	323	285	285
query87	3295	3004	3071	3004
query88	5106	2295	2308	2295
query89	396	283	293	283
query90	1863	209	202	202
query91	176	138	140	138
query92	56	54	54	54
query93	5726	593	544	544
query94	681	210	202	202
query95	1109	1084	1060	1060
query96	638	325	325	325
query97	6423	6361	6511	6361
query98	193	180	171	171
query99	2928	809	862	809
Total cold run time: 311636 ms
Total hot run time: 204108 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.6 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 3793f28fd01a2fea447d986a7b11ded084812edd, data reload: false

query1	0.02	0.02	0.02
query2	0.08	0.02	0.02
query3	0.25	0.04	0.05
query4	1.79	0.06	0.06
query5	0.54	0.53	0.52
query6	1.23	0.62	0.62
query7	0.01	0.00	0.01
query8	0.03	0.03	0.02
query9	0.54	0.50	0.49
query10	0.53	0.54	0.55
query11	0.12	0.09	0.09
query12	0.11	0.09	0.10
query13	0.62	0.61	0.63
query14	0.79	0.78	0.80
query15	0.77	0.76	0.76
query16	0.37	0.39	0.37
query17	1.03	1.01	1.04
query18	0.22	0.27	0.24
query19	1.88	1.72	1.85
query20	0.01	0.00	0.00
query21	15.53	0.56	0.54
query22	2.53	2.21	1.55
query23	17.07	1.03	1.02
query24	6.38	0.92	0.82
query25	0.32	0.08	0.06
query26	0.76	0.15	0.16
query27	0.05	0.04	0.04
query28	7.04	0.81	0.72
query29	12.67	2.36	2.37
query30	0.57	0.53	0.51
query31	2.82	0.39	0.37
query32	3.39	0.50	0.49
query33	3.06	3.09	3.09
query34	15.29	4.81	4.78
query35	4.80	4.83	4.83
query36	1.04	1.02	1.02
query37	0.05	0.04	0.05
query38	0.03	0.03	0.02
query39	0.02	0.01	0.01
query40	0.16	0.14	0.14
query41	0.06	0.01	0.02
query42	0.02	0.02	0.01
query43	0.02	0.02	0.02
Total cold run time: 104.62 s
Total hot run time: 30.6 s

@doris-robot
Copy link

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit 3793f28fd01a2fea447d986a7b11ded084812edd with default session variables
Stream load json:         19 seconds loaded 2358488459 Bytes, about 118 MB/s
Stream load orc:          58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet:      31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select:       21.4 seconds inserted 10000000 Rows, about 467K ops/s

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 37.75% (8104/21466)
Line Coverage: 29.39% (66369/225805)
Region Coverage: 28.91% (34256/118487)
Branch Coverage: 24.80% (17604/70990)
Coverage Report: http://coverage.selectdb-in.cc/coverage/3793f28fd01a2fea447d986a7b11ded084812edd_3793f28fd01a2fea447d986a7b11ded084812edd/report/index.html

@xiaokang xiaokang merged commit 90a2d71 into apache:branch-2.0 Aug 1, 2024
21 of 24 checks passed
mongo360 pushed a commit to mongo360/doris that referenced this pull request Aug 16, 2024
@airborne12 @Baymine
[Feature](inverted index) add no need read data optimize config apach…
0298f09 
…e#36686 (apache#38652)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@airborne12 @doris-robot @xiaokang