Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[opt](nereids) push down runtime filter to all children of SetOperation #32982

Merged
merged 5 commits into from
Apr 2, 2024

Conversation

englefly
Copy link
Contributor

@englefly englefly commented Mar 28, 2024

Proposed changes

generate more rf for tpcds 14

Issue Number: close #xxx

Further comments

If this is a relatively large or complex change, kick off the discussion at [email protected] by explaining why you chose the solution you did and what alternatives you considered, etc...

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@englefly
Copy link
Contributor Author

run buildall

1 similar comment
@englefly
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 37514 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 4545a011b8cc6e80dc527a6904e9e8ddc3f08782, data reload: false

------ Round 1 ----------------------------------
q1	17621	4191	4133	4133
q2	2123	157	148	148
q3	10583	1095	1198	1095
q4	10228	763	744	744
q5	7450	2872	2871	2871
q6	200	122	123	122
q7	1030	588	566	566
q8	9342	1948	1963	1948
q9	6933	6320	6295	6295
q10	8444	3464	3596	3464
q11	443	240	212	212
q12	437	197	194	194
q13	17795	2863	2880	2863
q14	243	199	207	199
q15	504	470	460	460
q16	494	364	369	364
q17	950	631	554	554
q18	7029	6423	6416	6416
q19	1548	1426	1492	1426
q20	536	255	261	255
q21	3611	2908	2894	2894
q22	337	294	291	291
Total cold run time: 107881 ms
Total hot run time: 37514 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4132	4074	4069	4069
q2	326	232	230	230
q3	2970	2840	2831	2831
q4	1842	1520	1553	1520
q5	5201	5234	5268	5234
q6	195	116	117	116
q7	2189	1794	1810	1794
q8	3162	3310	3294	3294
q9	8437	8357	8402	8357
q10	3799	3777	3900	3777
q11	547	459	451	451
q12	779	576	595	576
q13	17029	3019	3081	3019
q14	303	280	272	272
q15	523	487	479	479
q16	494	457	455	455
q17	1763	1506	1531	1506
q18	8101	7670	7698	7670
q19	1690	1571	1569	1569
q20	2039	1895	1715	1715
q21	5015	4816	4833	4816
q22	541	440	445	440
Total cold run time: 71077 ms
Total hot run time: 54190 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 183048 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 4545a011b8cc6e80dc527a6904e9e8ddc3f08782, data reload: false

query1	920	353	357	353
query2	6344	1990	2079	1990
query3	6702	227	216	216
query4	31388	21385	21484	21385
query5	4254	412	388	388
query6	266	183	174	174
query7	4627	284	290	284
query8	229	173	160	160
query9	9337	2275	2279	2275
query10	432	258	251	251
query11	15652	14238	14205	14205
query12	136	97	94	94
query13	1627	433	427	427
query14	9781	7716	7485	7485
query15	249	187	190	187
query16	8124	275	269	269
query17	1934	579	552	552
query18	2018	306	294	294
query19	263	166	158	158
query20	96	90	88	88
query21	197	129	132	129
query22	5002	4838	4815	4815
query23	33637	33116	33315	33116
query24	10968	2951	2933	2933
query25	621	401	418	401
query26	1188	168	162	162
query27	2570	375	378	375
query28	7344	1925	1943	1925
query29	933	657	628	628
query30	308	159	154	154
query31	993	768	740	740
query32	91	65	59	59
query33	757	245	247	245
query34	1444	503	526	503
query35	908	736	735	735
query36	1038	914	891	891
query37	117	63	63	63
query38	3746	3707	3613	3613
query39	1651	1615	1595	1595
query40	175	112	106	106
query41	49	48	50	48
query42	111	100	94	94
query43	472	469	468	468
query44	1274	753	769	753
query45	285	267	252	252
query46	1144	740	717	717
query47	2062	1958	1917	1917
query48	468	378	368	368
query49	945	355	345	345
query50	778	390	370	370
query51	6827	6813	6734	6734
query52	113	97	86	86
query53	348	269	284	269
query54	303	249	245	245
query55	81	85	79	79
query56	245	246	244	244
query57	1279	1189	1216	1189
query58	235	220	212	212
query59	2973	2757	2520	2520
query60	267	271	246	246
query61	116	114	110	110
query62	607	447	437	437
query63	302	275	272	272
query64	5536	4044	4134	4044
query65	3045	3052	3021	3021
query66	799	367	386	367
query67	15600	14926	14913	14913
query68	8559	541	538	538
query69	636	396	383	383
query70	1162	1172	1153	1153
query71	506	271	267	267
query72	6296	2532	2359	2359
query73	743	325	323	323
query74	8267	6449	6419	6419
query75	3864	2202	2192	2192
query76	5228	847	889	847
query77	609	268	259	259
query78	11004	10246	10170	10170
query79	11155	534	529	529
query80	2000	382	370	370
query81	523	217	213	213
query82	802	87	82	82
query83	202	146	147	146
query84	288	84	76	76
query85	1397	308	302	302
query86	401	303	293	293
query87	3741	3569	3498	3498
query88	4947	2417	2410	2410
query89	493	357	357	357
query90	1945	178	179	178
query91	162	134	138	134
query92	59	47	45	45
query93	6945	490	493	490
query94	1077	181	172	172
query95	407	312	313	312
query96	627	286	271	271
query97	2715	2475	2483	2475
query98	227	208	205	205
query99	1160	855	837	837
Total cold run time: 312478 ms
Total hot run time: 183048 ms

@doris-robot
Copy link

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit 4545a011b8cc6e80dc527a6904e9e8ddc3f08782 with default session variables
Stream load json:         18 seconds loaded 2358488459 Bytes, about 124 MB/s
Stream load orc:          59 seconds loaded 1101869774 Bytes, about 17 MB/s
Stream load parquet:      31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select:       13.6 seconds inserted 10000000 Rows, about 735K ops/s

@englefly
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 38248 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 954fd48afbd8d6e8e6a737cbf16fda85884b6660, data reload: false

------ Round 1 ----------------------------------
q1	17610	4489	4236	4236
q2	2083	192	185	185
q3	10455	1124	1173	1124
q4	10190	794	797	794
q5	7463	2810	2701	2701
q6	214	134	134	134
q7	1021	610	581	581
q8	9211	2049	2026	2026
q9	7523	6575	6543	6543
q10	8479	3534	3539	3534
q11	456	254	241	241
q12	403	222	212	212
q13	17911	2937	2937	2937
q14	266	230	232	230
q15	518	464	461	461
q16	534	384	389	384
q17	968	548	561	548
q18	7367	6774	6563	6563
q19	1590	1437	1423	1423
q20	667	313	291	291
q21	3448	2798	2805	2798
q22	359	302	306	302
Total cold run time: 108736 ms
Total hot run time: 38248 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4207	4137	4167	4137
q2	378	257	269	257
q3	2993	2684	2628	2628
q4	1868	1492	1530	1492
q5	5239	5178	5199	5178
q6	210	124	125	124
q7	2223	1859	1860	1859
q8	3190	3295	3338	3295
q9	8524	8503	8542	8503
q10	3886	3693	3586	3586
q11	598	482	474	474
q12	722	583	586	583
q13	17775	2923	2973	2923
q14	306	263	283	263
q15	509	458	463	458
q16	488	423	414	414
q17	1794	1429	1408	1408
q18	7555	7273	7374	7273
q19	1607	1511	1513	1511
q20	1939	1766	1740	1740
q21	4790	4718	4790	4718
q22	531	455	449	449
Total cold run time: 71332 ms
Total hot run time: 53273 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 182445 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 954fd48afbd8d6e8e6a737cbf16fda85884b6660, data reload: false

query1	895	1123	1116	1116
query2	7321	1971	1899	1899
query3	6664	217	216	216
query4	25032	21563	21579	21563
query5	4236	403	408	403
query6	270	181	175	175
query7	4602	295	292	292
query8	224	175	180	175
query9	8430	2288	2279	2279
query10	537	240	250	240
query11	14770	14252	14285	14252
query12	137	92	89	89
query13	1642	371	381	371
query14	8796	6840	6887	6840
query15	208	180	185	180
query16	7051	265	268	265
query17	1370	604	559	559
query18	1471	289	283	283
query19	204	162	155	155
query20	93	94	94	94
query21	201	128	127	127
query22	5009	4807	4849	4807
query23	33522	32810	32829	32810
query24	12874	2870	2764	2764
query25	634	381	391	381
query26	1903	161	154	154
query27	3223	317	330	317
query28	7377	1846	1841	1841
query29	999	625	612	612
query30	303	152	150	150
query31	1004	732	714	714
query32	82	57	57	57
query33	705	256	259	256
query34	1135	484	479	479
query35	814	672	687	672
query36	1082	885	923	885
query37	274	67	69	67
query38	3570	3434	3404	3404
query39	1580	1608	1536	1536
query40	279	134	126	126
query41	47	44	43	43
query42	107	97	98	97
query43	482	461	452	452
query44	1214	749	705	705
query45	268	254	273	254
query46	1085	677	712	677
query47	1912	1845	1856	1845
query48	368	298	306	298
query49	1175	353	364	353
query50	783	385	380	380
query51	6795	6701	6669	6669
query52	105	90	91	90
query53	344	276	278	276
query54	281	231	228	228
query55	84	80	80	80
query56	244	219	220	219
query57	1228	1126	1121	1121
query58	222	195	206	195
query59	2968	2730	2541	2541
query60	266	239	233	233
query61	95	96	91	91
query62	663	431	428	428
query63	310	279	287	279
query64	6682	4070	4055	4055
query65	3160	3067	3017	3017
query66	1415	350	345	345
query67	15327	15240	15169	15169
query68	5570	520	536	520
query69	519	313	313	313
query70	1311	1187	1161	1161
query71	439	277	272	272
query72	6832	2742	2599	2599
query73	748	328	334	328
query74	6739	6410	6357	6357
query75	3100	2258	2253	2253
query76	3193	1051	1084	1051
query77	623	270	268	268
query78	11032	10192	10098	10098
query79	9120	525	540	525
query80	1952	447	459	447
query81	524	1085	226	226
query82	923	94	94	94
query83	345	170	176	170
query84	268	86	87	86
query85	1941	264	266	264
query86	477	275	285	275
query87	3721	3545	3521	3521
query88	5771	2395	2396	2395
query89	562	375	367	367
query90	1980	180	181	180
query91	128	100	96	96
query92	60	47	47	47
query93	6525	519	504	504
query94	1227	180	184	180
query95	400	318	306	306
query96	604	267	271	267
query97	2684	2479	2483	2479
query98	231	213	215	213
query99	1203	839	838	838
Total cold run time: 298905 ms
Total hot run time: 182445 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 29.54 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 954fd48afbd8d6e8e6a737cbf16fda85884b6660, data reload: false

query1	0.04	0.04	0.02
query2	0.08	0.04	0.04
query3	0.23	0.05	0.05
query4	1.67	0.07	0.07
query5	0.47	0.47	0.49
query6	1.14	0.66	0.66
query7	0.02	0.02	0.01
query8	0.05	0.04	0.04
query9	0.54	0.49	0.50
query10	0.55	0.55	0.55
query11	0.15	0.10	0.12
query12	0.14	0.12	0.11
query13	0.58	0.58	0.58
query14	0.76	0.76	0.76
query15	0.82	0.80	0.79
query16	0.37	0.37	0.37
query17	1.01	0.96	0.97
query18	0.21	0.21	0.26
query19	1.71	1.65	1.66
query20	0.01	0.01	0.01
query21	15.60	0.64	0.64
query22	2.91	3.09	1.80
query23	17.23	0.82	0.90
query24	1.16	0.25	0.23
query25	0.09	0.08	0.08
query26	0.23	0.18	0.17
query27	0.08	0.07	0.08
query28	14.02	0.93	0.93
query29	12.56	3.25	3.29
query30	0.28	0.09	0.07
query31	2.81	0.40	0.37
query32	3.27	0.45	0.44
query33	2.80	2.78	2.86
query34	16.85	4.43	4.42
query35	4.51	4.51	4.49
query36	0.61	0.46	0.46
query37	0.19	0.15	0.15
query38	0.15	0.14	0.13
query39	0.06	0.03	0.04
query40	0.18	0.14	0.15
query41	0.09	0.05	0.04
query42	0.06	0.05	0.06
query43	0.05	0.04	0.04
Total cold run time: 106.34 s
Total hot run time: 29.54 s

@doris-robot
Copy link

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit 954fd48afbd8d6e8e6a737cbf16fda85884b6660 with default session variables
Stream load json:         18 seconds loaded 2358488459 Bytes, about 124 MB/s
Stream load orc:          58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet:      31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select:       13.7 seconds inserted 10000000 Rows, about 729K ops/s

starocean999
starocean999 previously approved these changes Apr 1, 2024
@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Apr 1, 2024
Copy link
Contributor

github-actions bot commented Apr 1, 2024

PR approved by at least one committer and no changes requested.

Copy link
Contributor

github-actions bot commented Apr 1, 2024

PR approved by anyone and no changes requested.

@englefly
Copy link
Contributor Author

englefly commented Apr 1, 2024

run buildall

@github-actions github-actions bot removed the approved Indicates a PR has been approved by one committer. label Apr 1, 2024
@doris-robot
Copy link

TPC-H: Total hot run time: 38877 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 4f4afc655ca3810e0312a27261f918354c301c66, data reload: false

------ Round 1 ----------------------------------
q1	17607	4117	4111	4111
q2	2022	189	187	187
q3	10477	1213	1427	1213
q4	10207	869	1011	869
q5	7475	2960	2916	2916
q6	222	138	138	138
q7	1118	659	610	610
q8	9419	2080	2052	2052
q9	6678	6191	6138	6138
q10	8464	3510	3504	3504
q11	429	252	245	245
q12	382	228	217	217
q13	17790	2918	2936	2918
q14	274	253	247	247
q15	521	496	494	494
q16	523	381	383	381
q17	974	919	920	919
q18	7434	6509	6423	6423
q19	1608	1553	1550	1550
q20	585	326	322	322
q21	3635	3123	3130	3123
q22	371	300	313	300
Total cold run time: 108215 ms
Total hot run time: 38877 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4119	4043	4032	4032
q2	334	223	226	223
q3	2970	2965	2992	2965
q4	1899	1827	1842	1827
q5	5269	5229	5249	5229
q6	209	127	124	124
q7	2265	1803	1789	1789
q8	3208	3275	3282	3275
q9	8455	8499	8447	8447
q10	3750	4006	3997	3997
q11	580	471	465	465
q12	738	605	574	574
q13	16823	3090	3143	3090
q14	314	291	313	291
q15	546	509	500	500
q16	517	446	456	446
q17	1775	1753	1740	1740
q18	8264	7657	7663	7657
q19	1716	1714	1714	1714
q20	2030	1841	1845	1841
q21	5125	5010	4990	4990
q22	513	442	453	442
Total cold run time: 71419 ms
Total hot run time: 55658 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 181948 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 4f4afc655ca3810e0312a27261f918354c301c66, data reload: false

query1	1226	1111	1118	1111
query2	6293	2074	1924	1924
query3	6664	214	204	204
query4	24865	21484	21515	21484
query5	4163	398	387	387
query6	284	190	180	180
query7	4599	301	305	301
query8	229	183	182	182
query9	8448	2244	2228	2228
query10	460	281	275	275
query11	15106	14508	14561	14508
query12	143	103	96	96
query13	1653	393	396	393
query14	8436	7008	6969	6969
query15	209	180	180	180
query16	6905	280	283	280
query17	971	593	596	593
query18	1867	289	294	289
query19	218	168	180	168
query20	109	97	94	94
query21	198	131	131	131
query22	4979	4775	4710	4710
query23	33614	32544	32552	32544
query24	11640	3241	3214	3214
query25	697	437	469	437
query26	1573	178	172	172
query27	3324	375	405	375
query28	6832	1927	1863	1863
query29	1202	627	607	607
query30	308	162	164	162
query31	1077	771	764	764
query32	98	68	65	65
query33	713	266	247	247
query34	1090	507	516	507
query35	881	752	741	741
query36	1021	883	885	883
query37	154	74	84	74
query38	3604	3584	3595	3584
query39	1675	1626	1617	1617
query40	246	142	142	142
query41	50	44	50	44
query42	133	116	122	116
query43	462	416	416	416
query44	1165	721	731	721
query45	295	251	268	251
query46	1129	802	797	797
query47	1998	1862	1900	1862
query48	399	313	310	310
query49	925	376	376	376
query50	830	416	419	416
query51	7061	6869	6884	6869
query52	115	97	101	97
query53	380	308	300	300
query54	318	237	244	237
query55	95	80	75	75
query56	251	239	233	233
query57	1248	1191	1169	1169
query58	250	241	231	231
query59	2764	2554	2398	2398
query60	250	222	232	222
query61	90	86	85	85
query62	642	453	450	450
query63	308	282	285	282
query64	5792	2980	3024	2980
query65	3030	3017	3009	3009
query66	1313	326	314	314
query67	15616	15043	14639	14639
query68	9342	575	576	575
query69	606	329	332	329
query70	1397	1113	1094	1094
query71	497	274	270	270
query72	6635	2574	2424	2424
query73	1566	323	334	323
query74	6687	6295	6317	6295
query75	3587	2280	2268	2268
query76	5456	1096	1189	1096
query77	558	253	248	248
query78	10801	10196	10043	10043
query79	10636	535	524	524
query80	1613	427	428	427
query81	493	231	221	221
query82	375	110	103	103
query83	209	164	164	164
query84	266	92	90	90
query85	940	325	283	283
query86	358	274	278	274
query87	3727	3453	3538	3453
query88	3810	2273	2292	2273
query89	532	374	367	367
query90	1898	178	182	178
query91	134	106	109	106
query92	66	54	56	54
query93	6541	545	538	538
query94	1216	192	196	192
query95	440	339	342	339
query96	602	277	264	264
query97	2660	2483	2499	2483
query98	228	215	212	212
query99	1182	879	854	854
Total cold run time: 299741 ms
Total hot run time: 181948 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.82 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 4f4afc655ca3810e0312a27261f918354c301c66, data reload: false

query1	0.04	0.04	0.03
query2	0.08	0.04	0.04
query3	0.24	0.05	0.04
query4	1.68	0.06	0.06
query5	0.49	0.48	0.49
query6	1.15	0.65	0.65
query7	0.02	0.01	0.01
query8	0.05	0.05	0.05
query9	0.54	0.51	0.50
query10	0.57	0.57	0.57
query11	0.15	0.11	0.11
query12	0.13	0.11	0.12
query13	0.63	0.60	0.59
query14	0.76	0.79	0.78
query15	0.85	0.82	0.83
query16	0.36	0.34	0.36
query17	0.98	0.99	0.97
query18	0.25	0.25	0.26
query19	1.82	1.71	1.78
query20	0.02	0.01	0.01
query21	15.40	0.74	0.70
query22	3.59	6.31	2.49
query23	17.79	1.25	1.09
query24	1.38	0.23	0.38
query25	0.13	0.08	0.08
query26	0.28	0.17	0.18
query27	0.08	0.09	0.08
query28	13.46	0.97	0.93
query29	12.60	3.53	3.34
query30	0.25	0.06	0.06
query31	2.87	0.38	0.38
query32	3.28	0.49	0.46
query33	2.90	2.87	2.93
query34	15.55	4.32	4.32
query35	4.38	4.37	4.37
query36	0.67	0.48	0.47
query37	0.20	0.17	0.18
query38	0.17	0.17	0.16
query39	0.05	0.04	0.05
query40	0.20	0.15	0.15
query41	0.10	0.05	0.05
query42	0.07	0.06	0.05
query43	0.05	0.04	0.04
Total cold run time: 106.26 s
Total hot run time: 30.82 s

@doris-robot
Copy link

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit 4f4afc655ca3810e0312a27261f918354c301c66 with default session variables
Stream load json:         18 seconds loaded 2358488459 Bytes, about 124 MB/s
Stream load orc:          59 seconds loaded 1101869774 Bytes, about 17 MB/s
Stream load parquet:      31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select:       15.5 seconds inserted 10000000 Rows, about 645K ops/s

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Apr 2, 2024
Copy link
Contributor

github-actions bot commented Apr 2, 2024

PR approved by at least one committer and no changes requested.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by one committer. dev/2.1.3-merged reviewed
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants