Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Configurations](multi-catalog) Add enable_parquet_filter_by_min_max and enable_orc_filter_by_min_max Session variables. #35290

Merged
merged 3 commits into from
May 28, 2024
Merged

[Configurations](multi-catalog) Add enable_parquet_filter_by_min_max and enable_orc_filter_by_min_max Session variables. #35290

merged 3 commits into from
May 28, 2024

Conversation

kaka11chen
Copy link
Contributor

@kaka11chen kaka11chen commented May 23, 2024 •
edited
Loading

Proposed changes

backport #35012 #35320

Further comments

If this is a relatively large or complex change, kick off the discussion at [email protected] by explaining why you chose the solution you did and what alternatives you considered, etc...

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@kaka11chen kaka11chen force-pushed the cherry-pick-35012-to-2.0 branch 2 times, most recently from 4058057 to 320ccbc Compare May 23, 2024 10:09
@kaka11chen
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 50219 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 320ccbc356850a049d04e572961627f169e8b883, data reload: false

------ Round 1 ----------------------------------
q1	17672	4550	4357	4357
q2	2067	153	142	142
q3	10452	1975	1978	1975
q4	10367	1303	1366	1303
q5	8652	3967	3930	3930
q6	235	127	129	127
q7	2048	1594	1628	1594
q8	9552	2761	2739	2739
q9	14321	10699	10574	10574
q10	8660	3545	3525	3525
q11	414	237	246	237
q12	479	298	298	298
q13	18325	3925	4022	3925
q14	350	331	329	329
q15	524	473	458	458
q16	671	575	571	571
q17	1166	952	908	908
q18	7170	6858	6844	6844
q19	1707	1648	1561	1561
q20	537	312	310	310
q21	4451	4203	4130	4130
q22	497	394	382	382
Total cold run time: 120317 ms
Total hot run time: 50219 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4324	4282	4259	4259
q2	324	223	222	222
q3	4148	4164	4163	4163
q4	2764	2741	2744	2741
q5	7244	7202	7203	7202
q6	242	122	120	120
q7	3264	2871	2850	2850
q8	4370	4498	4481	4481
q9	17440	17128	16999	16999
q10	4231	4259	4281	4259
q11	752	693	736	693
q12	1031	856	858	856
q13	7038	3725	3722	3722
q14	455	412	422	412
q15	514	453	449	449
q16	732	676	674	674
q17	3802	3861	3781	3781
q18	8711	8679	8665	8665
q19	1760	1688	1698	1688
q20	2407	2126	2088	2088
q21	8467	8493	8487	8487
q22	1035	911	934	911
Total cold run time: 85055 ms
Total hot run time: 79722 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 204355 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 320ccbc356850a049d04e572961627f169e8b883, data reload: false

query1	941	387	372	372
query2	6543	2857	2645	2645
query3	6925	208	208	208
query4	20113	18079	18090	18079
query5	19715	6519	6533	6519
query6	279	218	226	218
query7	4150	289	302	289
query8	267	242	276	242
query9	3129	2678	2599	2599
query10	409	308	299	299
query11	11472	10804	10844	10804
query12	120	78	73	73
query13	5577	692	668	668
query14	17798	13148	13393	13148
query15	360	227	221	221
query16	6468	273	258	258
query17	1766	1477	863	863
query18	2310	403	402	402
query19	198	145	145	145
query20	78	77	81	77
query21	186	100	92	92
query22	5268	4873	5042	4873
query23	32467	31837	31794	31794
query24	6978	6584	6512	6512
query25	524	433	429	429
query26	624	157	157	157
query27	2052	294	296	294
query28	6157	2363	2319	2319
query29	2817	2690	2732	2690
query30	241	162	162	162
query31	898	737	725	725
query32	72	62	60	60
query33	403	256	255	255
query34	839	465	488	465
query35	1133	924	937	924
query36	1656	1157	1331	1157
query37	93	62	60	60
query38	3050	2944	2927	2927
query39	1388	1315	1315	1315
query40	249	94	86	86
query41	49	44	35	35
query42	85	80	81	80
query43	700	634	666	634
query44	1127	707	722	707
query45	244	234	226	226
query46	1249	968	993	968
query47	1900	1659	1692	1659
query48	1014	720	699	699
query49	623	359	380	359
query50	871	626	603	603
query51	4730	4664	4684	4664
query52	92	78	80	78
query53	448	314	318	314
query54	2641	2465	2495	2465
query55	89	83	84	83
query56	224	240	229	229
query57	1208	1113	1082	1082
query58	210	210	206	206
query59	4146	3992	3924	3924
query60	203	183	187	183
query61	91	84	83	83
query62	782	503	459	459
query63	479	342	333	333
query64	2625	1489	1534	1489
query65	3605	3546	3534	3534
query66	800	376	393	376
query67	16066	15484	15933	15484
query68	9311	670	660	660
query69	559	336	354	336
query70	1837	1449	1562	1449
query71	420	301	320	301
query72	6381	3439	3427	3427
query73	740	330	335	330
query74	6295	5934	5815	5815
query75	5411	3802	3764	3764
query76	5901	1145	1159	1145
query77	983	256	268	256
query78	12591	12838	12611	12611
query79	11980	668	673	668
query80	778	393	422	393
query81	489	235	233	233
query82	536	94	96	94
query83	165	137	133	133
query84	255	70	69	69
query85	750	309	297	297
query86	330	296	340	296
query87	3246	3020	3053	3020
query88	5120	2366	2396	2366
query89	399	296	296	296
query90	1934	216	206	206
query91	193	136	145	136
query92	58	53	54	53
query93	4234	538	585	538
query94	710	206	201	201
query95	1098	1058	1046	1046
query96	659	334	344	334
query97	6571	6464	6473	6464
query98	195	189	177	177
query99	2909	862	860	860
Total cold run time: 315052 ms
Total hot run time: 204355 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.46 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 320ccbc356850a049d04e572961627f169e8b883, data reload: false

query1	0.02	0.02	0.02
query2	0.07	0.02	0.02
query3	0.24	0.05	0.04
query4	1.80	0.06	0.06
query5	0.55	0.52	0.52
query6	1.27	0.62	0.67
query7	0.02	0.01	0.00
query8	0.03	0.02	0.02
query9	0.52	0.47	0.48
query10	0.54	0.53	0.55
query11	0.11	0.08	0.08
query12	0.11	0.09	0.09
query13	0.62	0.62	0.60
query14	0.78	0.79	0.77
query15	0.78	0.76	0.76
query16	0.37	0.39	0.37
query17	1.00	0.97	1.03
query18	0.22	0.26	0.23
query19	1.95	1.79	1.88
query20	0.01	0.01	0.01
query21	15.50	0.56	0.56
query22	2.03	2.07	1.54
query23	17.51	1.09	1.02
query24	3.41	0.89	1.62
query25	0.36	0.15	0.04
query26	0.44	0.15	0.15
query27	0.04	0.04	0.04
query28	9.09	0.71	0.72
query29	12.79	2.25	2.31
query30	0.56	0.58	0.52
query31	2.81	0.38	0.37
query32	3.38	0.49	0.50
query33	3.08	3.03	3.08
query34	15.22	4.81	4.79
query35	4.87	4.84	4.84
query36	1.06	1.01	1.00
query37	0.06	0.05	0.04
query38	0.03	0.02	0.02
query39	0.02	0.02	0.01
query40	0.16	0.14	0.14
query41	0.06	0.01	0.02
query42	0.02	0.02	0.01
query43	0.02	0.02	0.02
Total cold run time: 103.53 s
Total hot run time: 30.46 s

@doris-robot
Copy link

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit 320ccbc356850a049d04e572961627f169e8b883 with default session variables
Stream load json:         20 seconds loaded 2358488459 Bytes, about 112 MB/s
Stream load orc:          58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet:      31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select:       21.2 seconds inserted 10000000 Rows, about 471K ops/s

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 37.79% (8076/21369)
Line Coverage: 29.45% (65942/223928)
Region Coverage: 28.92% (33954/117409)
Branch Coverage: 24.77% (17424/70332)
Coverage Report: http://coverage.selectdb-in.cc/coverage/320ccbc356850a049d04e572961627f169e8b883_320ccbc356850a049d04e572961627f169e8b883/report/index.html

@kaka11chen
[Configurations](multi-catalog) Add `enable_parquet_filter_by_min_max…
b77d820 
…` and `enable_orc_filter_by_min_max` Session variables. (apache#35012)
@kaka11chen
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 37.79% (8078/21378)
Line Coverage: 29.44% (65956/224014)
Region Coverage: 28.93% (33995/117491)
Branch Coverage: 24.80% (17456/70394)
Coverage Report: http://coverage.selectdb-in.cc/coverage/b77d820fc52d922617a30038b9f8a2a55675ea84_b77d820fc52d922617a30038b9f8a2a55675ea84/report/index.html

@kaka11chen
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 37.80% (8080/21378)
Line Coverage: 29.45% (65975/224014)
Region Coverage: 28.95% (34010/117491)
Branch Coverage: 24.81% (17464/70394)
Coverage Report: http://coverage.selectdb-in.cc/coverage/9076f9d9b5640886f380179e12b59c6a6a5c040e_9076f9d9b5640886f380179e12b59c6a6a5c040e/report/index.html

@kaka11chen
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 37.78% (8078/21379)
Line Coverage: 29.44% (65953/224028)
Region Coverage: 28.93% (33994/117500)
Branch Coverage: 24.79% (17453/70398)
Coverage Report: http://coverage.selectdb-in.cc/coverage/11a859af14747f485712fea3780c26381b34d7a2_11a859af14747f485712fea3780c26381b34d7a2/report/index.html

@morningman morningman merged commit a096107 into apache:branch-2.0 May 28, 2024
19 of 24 checks passed
mongo360 pushed a commit to mongo360/doris that referenced this pull request Aug 16, 2024
@kaka11chen
[Configurations](multi-catalog) Add `enable_parquet_filter_by_min_max…
34fa53f 
…` and `enable_orc_filter_by_min_max` Session variables. (apache#35290)

backport apache#35012 apache#35320
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
kind/test
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@kaka11chen @doris-robot @morningman