[fix](nereids) runtime filter with probe expr should be pushed though set operator

[englefly]

Proposed changes

select count() from ((select l_linenumber from lineitem) except (select o_orderkey from orders)) T join region on abs(T.l_linenumber) = r_regionkey

because of ABS and set operator, the runtime filter is not generated.

Issue Number: close #xxx

Further comments

If this is a relatively large or complex change, kick off the discussion at [email protected] by explaining why you chose the solution you did and what alternatives you considered, etc...

[doris-robot]

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

[englefly]

run buildall

[doris-robot]

TPC-H: Total hot run time: 37306 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 3f6cecf956cf5c9344fa462ff8a19737630194b8, data reload: false

------ Round 1 ----------------------------------
q1	17595	4700	4098	4098
q2	2104	154	149	149
q3	10585	1117	1187	1117
q4	10228	816	826	816
q5	7469	2928	2891	2891
q6	200	125	125	125
q7	1044	584	567	567
q8	9336	1988	1986	1986
q9	7063	6287	6300	6287
q10	8420	3418	3590	3418
q11	437	239	226	226
q12	405	202	197	197
q13	17793	2873	2821	2821
q14	248	200	197	197
q15	508	463	459	459
q16	479	368	366	366
q17	952	523	592	523
q18	7100	6393	6293	6293
q19	1928	1459	1359	1359
q20	555	247	241	241
q21	3573	2885	2961	2885
q22	331	285	295	285
Total cold run time: 108353 ms
Total hot run time: 37306 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4100	4073	4077	4073
q2	331	232	231	231
q3	2953	2844	2838	2838
q4	1829	1554	1570	1554
q5	5232	5389	5217	5217
q6	197	121	121	121
q7	2183	1795	1802	1795
q8	3152	3274	3283	3274
q9	8403	8392	8408	8392
q10	3755	3736	4658	3736
q11	571	470	459	459
q12	750	582	555	555
q13	17044	3018	3068	3018
q14	294	261	285	261
q15	536	492	483	483
q16	494	498	476	476
q17	1807	1573	1542	1542
q18	7901	7690	7640	7640
q19	1665	1594	1570	1570
q20	1997	1827	1831	1827
q21	4806	4870	4898	4870
q22	543	442	437	437
Total cold run time: 70543 ms
Total hot run time: 54369 ms

[doris-robot]

TPC-DS: Total hot run time: 182194 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 3f6cecf956cf5c9344fa462ff8a19737630194b8, data reload: false

query1	931	351	339	339
query2	6335	2138	1888	1888
query3	6697	217	213	213
query4	31520	21321	21292	21292
query5	4290	408	400	400
query6	288	197	177	177
query7	4631	294	292	292
query8	229	168	171	168
query9	9466	2297	2271	2271
query10	445	257	260	257
query11	14783	14227	14245	14227
query12	134	90	90	90
query13	1612	419	417	417
query14	8917	8036	7936	7936
query15	253	181	177	177
query16	8251	264	265	264
query17	1966	574	558	558
query18	2110	290	300	290
query19	347	155	164	155
query20	95	92	85	85
query21	197	127	126	126
query22	4992	4866	4777	4777
query23	33470	32822	33137	32822
query24	10897	2978	3015	2978
query25	628	418	414	414
query26	1177	168	173	168
query27	2796	372	388	372
query28	7325	1958	1970	1958
query29	923	648	626	626
query30	317	157	160	157
query31	1004	761	767	761
query32	95	58	59	58
query33	790	268	265	265
query34	1232	496	497	496
query35	912	708	725	708
query36	1056	912	924	912
query37	123	69	66	66
query38	3782	3680	3643	3643
query39	1084	1041	1061	1041
query40	183	104	109	104
query41	49	52	48	48
query42	108	96	104	96
query43	505	471	450	450
query44	1173	775	769	769
query45	290	251	231	231
query46	1121	707	713	707
query47	2042	1971	1961	1961
query48	490	398	376	376
query49	890	357	341	341
query50	770	407	398	398
query51	6838	6787	6718	6718
query52	102	92	95	92
query53	364	281	272	272
query54	294	248	243	243
query55	91	84	86	84
query56	255	240	262	240
query57	1260	1199	1171	1171
query58	240	210	219	210
query59	2872	2567	2588	2567
query60	256	234	239	234
query61	93	89	88	88
query62	601	445	437	437
query63	300	270	272	270
query64	5272	4007	4009	4007
query65	3058	3018	3026	3018
query66	788	353	349	349
query67	15739	14755	14780	14755
query68	10252	530	531	530
query69	706	377	371	371
query70	1563	1100	1092	1092
query71	536	265	265	265
query72	7133	2526	2383	2383
query73	1692	325	321	321
query74	7099	6374	6445	6374
query75	4954	2244	2191	2191
query76	6706	930	854	854
query77	616	267	254	254
query78	11145	10166	10129	10129
query79	10809	522	538	522
query80	2182	380	371	371
query81	496	217	215	215
query82	166	82	83	82
query83	222	141	147	141
query84	282	79	77	77
query85	1217	322	303	303
query86	362	314	312	312
query87	3757	3506	3539	3506
query88	5084	2399	2389	2389
query89	479	363	357	357
query90	2356	170	171	170
query91	172	130	134	130
query92	62	45	47	45
query93	6325	486	485	485
query94	1509	172	175	172
query95	412	313	304	304
query96	616	273	269	269
query97	2619	2467	2497	2467
query98	224	215	209	209
query99	1042	824	830	824
Total cold run time: 314939 ms
Total hot run time: 182194 ms

[doris-robot]

ClickBench: Total hot run time: 31.2 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 3f6cecf956cf5c9344fa462ff8a19737630194b8, data reload: false

query1	0.05	0.04	0.03
query2	0.06	0.03	0.03
query3	0.22	0.06	0.06
query4	1.65	0.10	0.10
query5	0.50	0.50	0.52
query6	1.14	0.65	0.65
query7	0.02	0.01	0.02
query8	0.04	0.03	0.03
query9	0.57	0.50	0.49
query10	0.54	0.55	0.55
query11	0.13	0.09	0.10
query12	0.12	0.09	0.09
query13	0.59	0.58	0.57
query14	0.76	0.77	0.80
query15	0.82	0.81	0.82
query16	0.39	0.37	0.37
query17	0.99	0.97	0.97
query18	0.23	0.22	0.22
query19	1.78	1.73	1.70
query20	0.01	0.01	0.01
query21	15.52	0.58	0.57
query22	2.56	2.19	2.15
query23	17.13	0.80	0.73
query24	3.30	2.00	2.48
query25	0.18	0.22	0.15
query26	0.70	0.12	0.11
query27	0.03	0.03	0.06
query28	8.44	0.89	0.86
query29	12.57	3.17	3.22
query30	0.29	0.09	0.08
query31	2.82	0.36	0.36
query32	3.30	0.46	0.45
query33	2.88	2.85	2.85
query34	16.14	4.45	4.40
query35	4.48	4.50	4.45
query36	0.67	0.48	0.49
query37	0.08	0.05	0.06
query38	0.05	0.03	0.03
query39	0.04	0.02	0.02
query40	0.18	0.14	0.14
query41	0.08	0.02	0.02
query42	0.03	0.02	0.02
query43	0.03	0.03	0.03
Total cold run time: 102.11 s
Total hot run time: 31.2 s

[doris-robot]

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit 3f6cecf956cf5c9344fa462ff8a19737630194b8 with default session variables
Stream load json:         18 seconds loaded 2358488459 Bytes, about 124 MB/s
Stream load orc:          59 seconds loaded 1101869774 Bytes, about 17 MB/s
Stream load parquet:      31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select:       14.2 seconds inserted 10000000 Rows, about 704K ops/s

[englefly]

run buildall

[doris-robot]

TPC-H: Total hot run time: 38375 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit a85db1a364acff1fac0b396794a60aa2f9d4b808, data reload: false

------ Round 1 ----------------------------------
q1	17600	4379	4274	4274
q2	2089	191	186	186
q3	10465	1150	1189	1150
q4	10328	754	804	754
q5	7440	2796	2690	2690
q6	217	133	131	131
q7	1011	596	597	596
q8	9210	2053	2024	2024
q9	7592	6590	6516	6516
q10	8520	3616	3537	3537
q11	468	243	238	238
q12	396	219	220	219
q13	19093	2960	2968	2960
q14	279	233	246	233
q15	531	478	459	459
q16	537	391	390	390
q17	958	558	609	558
q18	7304	6719	6637	6637
q19	1571	1424	1473	1424
q20	683	314	295	295
q21	3469	2797	2837	2797
q22	361	312	307	307
Total cold run time: 110122 ms
Total hot run time: 38375 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4265	4227	4191	4191
q2	386	258	265	258
q3	2972	2750	2703	2703
q4	1836	1561	1513	1513
q5	5286	5167	5209	5167
q6	212	122	125	122
q7	2215	1875	1909	1875
q8	3196	3290	3296	3290
q9	8512	8587	8543	8543
q10	3881	3716	3622	3622
q11	609	466	484	466
q12	757	592	578	578
q13	17658	2995	2973	2973
q14	312	274	271	271
q15	512	467	477	467
q16	486	426	435	426
q17	1783	1435	1436	1435
q18	7634	7364	7322	7322
q19	1607	1493	1549	1493
q20	1953	1775	1746	1746
q21	4987	4869	4976	4869
q22	537	477	476	476
Total cold run time: 71596 ms
Total hot run time: 53806 ms

[doris-robot]

TPC-DS: Total hot run time: 182039 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit a85db1a364acff1fac0b396794a60aa2f9d4b808, data reload: false

query1	907	1137	1145	1137
query2	7048	2031	1845	1845
query3	6661	216	211	211
query4	23301	21492	21332	21332
query5	4263	404	385	385
query6	274	180	169	169
query7	4600	304	290	290
query8	231	175	183	175
query9	8499	2283	2259	2259
query10	535	245	255	245
query11	14798	14173	14328	14173
query12	147	93	90	90
query13	1639	372	365	365
query14	9902	6798	6785	6785
query15	250	180	176	176
query16	6997	272	270	270
query17	1458	581	564	564
query18	1807	294	284	284
query19	207	161	155	155
query20	93	93	89	89
query21	202	132	128	128
query22	4979	4821	4842	4821
query23	33377	32890	32991	32890
query24	12411	2818	2801	2801
query25	702	391	390	390
query26	1713	153	157	153
query27	3374	321	321	321
query28	7131	1834	1830	1830
query29	1018	622	605	605
query30	295	151	151	151
query31	994	714	729	714
query32	97	59	55	55
query33	727	261	246	246
query34	1191	479	478	478
query35	814	694	686	686
query36	1054	904	915	904
query37	215	73	68	68
query38	3610	3511	3449	3449
query39	1590	1576	1544	1544
query40	292	140	130	130
query41	50	47	47	47
query42	106	103	97	97
query43	505	460	459	459
query44	1223	734	709	709
query45	271	265	264	264
query46	1086	693	715	693
query47	1936	1832	1832	1832
query48	390	315	298	298
query49	1164	370	372	370
query50	776	385	384	384
query51	6775	6604	6677	6604
query52	106	88	99	88
query53	347	277	278	277
query54	324	233	239	233
query55	88	79	82	79
query56	265	228	226	226
query57	1233	1118	1130	1118
query58	263	211	216	211
query59	2904	2744	2574	2574
query60	270	243	265	243
query61	116	116	117	116
query62	641	446	453	446
query63	309	278	288	278
query64	6581	4144	4117	4117
query65	3133	3050	3020	3020
query66	1495	351	333	333
query67	15397	14994	15035	14994
query68	9412	538	538	538
query69	612	301	313	301
query70	1439	1090	1159	1090
query71	522	279	268	268
query72	7268	2657	2457	2457
query73	1636	324	331	324
query74	6968	6396	6461	6396
query75	4017	2312	2299	2299
query76	5592	923	1046	923
query77	601	262	251	251
query78	10947	10255	10262	10255
query79	9492	546	528	528
query80	1303	430	418	418
query81	502	226	222	222
query82	228	89	94	89
query83	220	172	170	170
query84	265	84	81	81
query85	971	276	266	266
query86	365	299	297	297
query87	3729	3501	3502	3501
query88	5577	2401	2394	2394
query89	529	390	386	386
query90	2047	178	183	178
query91	130	99	98	98
query92	58	50	47	47
query93	6442	534	504	504
query94	1259	183	186	183
query95	403	313	315	313
query96	619	278	265	265
query97	2675	2499	2489	2489
query98	237	225	212	212
query99	1098	847	823	823
Total cold run time: 304320 ms
Total hot run time: 182039 ms

[doris-robot]

ClickBench: Total hot run time: 29.4 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit a85db1a364acff1fac0b396794a60aa2f9d4b808, data reload: false

query1	0.04	0.04	0.03
query2	0.08	0.03	0.04
query3	0.23	0.05	0.06
query4	1.65	0.08	0.10
query5	0.50	0.47	0.50
query6	1.15	0.66	0.65
query7	0.02	0.02	0.01
query8	0.06	0.04	0.04
query9	0.55	0.49	0.50
query10	0.55	0.54	0.56
query11	0.16	0.10	0.12
query12	0.14	0.12	0.11
query13	0.58	0.58	0.59
query14	0.76	0.77	0.78
query15	0.83	0.81	0.81
query16	0.36	0.35	0.36
query17	1.01	0.97	1.01
query18	0.23	0.23	0.25
query19	1.92	1.68	1.72
query20	0.02	0.01	0.01
query21	15.55	0.64	0.64
query22	2.46	2.98	1.52
query23	17.06	1.01	0.84
query24	1.17	0.25	0.27
query25	0.10	0.08	0.08
query26	0.22	0.17	0.17
query27	0.08	0.08	0.07
query28	13.94	0.93	0.93
query29	12.52	3.23	3.27
query30	0.28	0.09	0.07
query31	2.82	0.40	0.38
query32	3.25	0.46	0.47
query33	2.84	2.86	2.85
query34	16.80	4.40	4.45
query35	4.48	4.47	4.47
query36	0.61	0.46	0.46
query37	0.18	0.16	0.15
query38	0.15	0.14	0.14
query39	0.05	0.03	0.04
query40	0.20	0.14	0.15
query41	0.09	0.05	0.05
query42	0.05	0.04	0.04
query43	0.04	0.03	0.04
Total cold run time: 105.78 s
Total hot run time: 29.4 s

[doris-robot]

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit a85db1a364acff1fac0b396794a60aa2f9d4b808 with default session variables
Stream load json:         18 seconds loaded 2358488459 Bytes, about 124 MB/s
Stream load orc:          58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet:      31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select:       14.3 seconds inserted 10000000 Rows, about 699K ops/s

[englefly]

run buildall

[doris-robot]

TPC-H: Total hot run time: 38491 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit e5847a9a2b42300639ae5285d1d3922a5e6b15cd, data reload: false

------ Round 1 ----------------------------------
q1	17629	4054	4053	4053
q2	2007	194	181	181
q3	10471	1219	1388	1219
q4	10210	826	990	826
q5	7454	2959	2912	2912
q6	218	135	131	131
q7	1120	627	605	605
q8	9412	2079	2026	2026
q9	6646	6194	6138	6138
q10	8443	3512	3485	3485
q11	432	244	246	244
q12	380	223	214	214
q13	17781	2865	2924	2865
q14	271	244	244	244
q15	534	491	486	486
q16	525	399	384	384
q17	944	912	875	875
q18	7105	6460	6367	6367
q19	1587	1525	1534	1525
q20	591	323	317	317
q21	3605	3098	3133	3098
q22	365	298	296	296
Total cold run time: 107730 ms
Total hot run time: 38491 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4043	4066	4047	4047
q2	327	221	220	220
q3	2948	2978	2929	2929
q4	1826	1898	1831	1831
q5	5183	5185	5206	5185
q6	207	124	120	120
q7	2211	1796	1839	1796
q8	3204	3251	3255	3251
q9	8471	8406	8456	8406
q10	3751	3901	3987	3901
q11	558	459	463	459
q12	759	555	599	555
q13	16831	3073	3090	3073
q14	305	263	281	263
q15	541	524	503	503
q16	496	434	432	432
q17	1762	1719	1680	1680
q18	8356	7951	7611	7611
q19	1704	1673	1667	1667
q20	2020	1886	1768	1768
q21	5176	4955	4977	4955
q22	516	432	423	423
Total cold run time: 71195 ms
Total hot run time: 55075 ms

[doris-robot]

TPC-DS: Total hot run time: 183038 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit e5847a9a2b42300639ae5285d1d3922a5e6b15cd, data reload: false

query1	880	1122	1124	1122
query2	6273	1899	1878	1878
query3	6665	213	212	212
query4	24911	21445	21531	21445
query5	4205	411	404	404
query6	284	199	194	194
query7	4609	310	293	293
query8	230	190	184	184
query9	8452	2261	2285	2261
query10	451	254	273	254
query11	14875	14500	14596	14500
query12	145	98	96	96
query13	1636	377	387	377
query14	8554	6924	7119	6924
query15	208	178	193	178
query16	6823	286	275	275
query17	968	602	586	586
query18	1836	290	291	290
query19	202	168	163	163
query20	97	95	99	95
query21	194	130	130	130
query22	5045	4886	4872	4872
query23	33386	32966	32640	32640
query24	11315	3254	3205	3205
query25	696	457	438	438
query26	1640	183	166	166
query27	3175	385	423	385
query28	7028	1898	1891	1891
query29	1063	592	617	592
query30	301	162	156	156
query31	1078	746	730	730
query32	106	58	62	58
query33	689	247	244	244
query34	1313	524	518	518
query35	846	755	733	733
query36	1027	855	861	855
query37	154	80	77	77
query38	3683	3577	3665	3577
query39	1651	1614	1594	1594
query40	224	141	149	141
query41	46	48	46	46
query42	116	117	113	113
query43	461	414	419	414
query44	1132	715	714	714
query45	287	264	268	264
query46	1142	858	796	796
query47	2012	1913	1953	1913
query48	408	323	309	309
query49	911	369	365	365
query50	836	409	417	409
query51	6987	6877	6913	6877
query52	118	106	121	106
query53	378	303	304	303
query54	320	256	268	256
query55	89	80	79	79
query56	250	263	235	235
query57	1267	1241	1188	1188
query58	251	233	238	233
query59	2727	2495	2321	2321
query60	266	245	252	245
query61	111	106	106	106
query62	663	463	457	457
query63	310	282	284	282
query64	5067	3474	3482	3474
query65	3037	3032	3036	3032
query66	960	338	322	322
query67	15383	14878	14845	14845
query68	9197	584	584	584
query69	573	332	332	332
query70	1330	1100	1046	1046
query71	525	267	269	267
query72	6380	2581	2402	2402
query73	1407	335	333	333
query74	6696	6393	6445	6393
query75	3739	2326	2327	2326
query76	5588	1130	1194	1130
query77	588	244	244	244
query78	10908	10065	10135	10065
query79	9907	540	532	532
query80	1879	428	413	413
query81	490	227	1067	227
query82	730	104	102	102
query83	199	159	160	159
query84	272	90	92	90
query85	1332	309	281	281
query86	399	276	290	276
query87	3678	3489	3514	3489
query88	3641	2373	2376	2373
query89	546	361	364	361
query90	1922	181	180	180
query91	133	105	104	104
query92	59	49	49	49
query93	6819	524	529	524
query94	1056	194	191	191
query95	438	334	327	327
query96	613	279	270	270
query97	2665	2480	2509	2480
query98	240	218	220	218
query99	1302	833	839	833
Total cold run time: 297704 ms
Total hot run time: 183038 ms

[doris-robot]

ClickBench: Total hot run time: 30.57 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit e5847a9a2b42300639ae5285d1d3922a5e6b15cd, data reload: false

query1	0.04	0.04	0.02
query2	0.08	0.04	0.04
query3	0.23	0.05	0.05
query4	1.68	0.07	0.07
query5	0.49	0.48	0.48
query6	1.14	0.66	0.65
query7	0.02	0.01	0.01
query8	0.05	0.04	0.04
query9	0.55	0.51	0.50
query10	0.58	0.57	0.55
query11	0.15	0.11	0.12
query12	0.14	0.11	0.11
query13	0.60	0.59	0.60
query14	0.76	0.78	0.79
query15	0.85	0.84	0.84
query16	0.36	0.36	0.35
query17	0.96	0.96	0.97
query18	0.25	0.25	0.25
query19	1.83	1.68	1.71
query20	0.02	0.01	0.01
query21	15.40	0.78	0.71
query22	2.88	6.35	2.34
query23	17.89	1.24	1.07
query24	1.82	0.23	0.23
query25	0.14	0.09	0.09
query26	0.31	0.18	0.19
query27	0.09	0.09	0.09
query28	13.44	0.93	0.94
query29	12.52	3.43	3.29
query30	0.27	0.06	0.06
query31	2.86	0.38	0.38
query32	3.27	0.48	0.47
query33	2.79	2.88	2.94
query34	15.50	4.34	4.32
query35	4.37	4.35	4.37
query36	0.68	0.47	0.48
query37	0.22	0.16	0.18
query38	0.16	0.16	0.15
query39	0.05	0.04	0.04
query40	0.18	0.14	0.14
query41	0.10	0.06	0.05
query42	0.06	0.06	0.05
query43	0.04	0.03	0.04
Total cold run time: 105.82 s
Total hot run time: 30.57 s

[doris-robot]

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit e5847a9a2b42300639ae5285d1d3922a5e6b15cd with default session variables
Stream load json:         18 seconds loaded 2358488459 Bytes, about 124 MB/s
Stream load orc:          58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet:      31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select:       16.2 seconds inserted 10000000 Rows, about 617K ops/s

[github-actions]

PR approved by at least one committer and no changes requested.

[github-actions]

PR approved by anyone and no changes requested.