Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[fix](nereids) null-safe-eq runtime filter denies outer join #32927

Merged
merged 1 commit into from
Mar 29, 2024
Merged

[fix](nereids) null-safe-eq runtime filter denies outer join #32927

merged 1 commit into from
Mar 29, 2024

Conversation

englefly
Copy link
Contributor

Proposed changes

Issue Number: close #xxx

Further comments

If this is a relatively large or complex change, kick off the discussion at [email protected] by explaining why you chose the solution you did and what alternatives you considered, etc...

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@englefly
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 37864 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit d6f2046c5d2c2421a7fcfb065dcb8b276c0637fc, data reload: false

------ Round 1 ----------------------------------
q1	18154	4184	4111	4111
q2	2116	164	157	157
q3	10586	1122	1188	1122
q4	10232	732	785	732
q5	7469	3031	2965	2965
q6	203	122	121	121
q7	1042	580	583	580
q8	9318	1976	1994	1976
q9	7327	6659	6498	6498
q10	8402	3455	3582	3455
q11	425	220	210	210
q12	371	209	191	191
q13	17786	2861	2860	2860
q14	246	217	206	206
q15	511	466	453	453
q16	472	375	374	374
q17	941	547	613	547
q18	7121	6441	6450	6441
q19	1549	1468	1410	1410
q20	560	267	260	260
q21	3668	2899	2918	2899
q22	333	296	298	296
Total cold run time: 108832 ms
Total hot run time: 37864 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4101	4059	4083	4059
q2	325	226	227	226
q3	2997	2789	2785	2785
q4	1842	1527	1529	1527
q5	5299	5327	5308	5308
q6	191	116	118	116
q7	2216	1853	1872	1853
q8	3180	3301	3279	3279
q9	8637	8601	8638	8601
q10	3753	3796	3739	3739
q11	543	441	439	439
q12	714	575	557	557
q13	16919	2854	2840	2840
q14	276	249	252	249
q15	489	457	462	457
q16	480	435	421	421
q17	1708	1487	1467	1467
q18	7466	7251	7169	7169
q19	1612	1510	1514	1510
q20	1905	1718	1686	1686
q21	4829	4599	4648	4599
q22	560	463	456	456
Total cold run time: 70042 ms
Total hot run time: 53343 ms

starocean999
starocean999 previously approved these changes Mar 27, 2024
View reviewed changes
@github-actions GitHub Actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions github-actions bot added approved Indicates a PR has been approved by one committer. reviewed labels Mar 27, 2024
@github-actions GitHub Actions
Copy link
Contributor

PR approved by anyone and no changes requested.

@englefly englefly mentioned this pull request Mar 27, 2024
Merged
@doris-robot
Copy link

TPC-DS: Total hot run time: 181276 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit d6f2046c5d2c2421a7fcfb065dcb8b276c0637fc, data reload: false

query1	955	373	353	353
query2	6542	1908	1923	1908
query3	6704	212	208	208
query4	32000	21318	21276	21276
query5	4275	402	391	391
query6	271	175	177	175
query7	4631	289	294	289
query8	231	181	184	181
query9	9451	2332	2278	2278
query10	561	248	276	248
query11	17019	14254	14240	14240
query12	135	91	83	83
query13	1618	411	416	411
query14	10038	7287	7589	7287
query15	258	193	193	193
query16	8154	258	264	258
query17	2163	564	557	557
query18	2077	290	288	288
query19	233	160	161	160
query20	93	91	87	87
query21	206	126	128	126
query22	5032	4823	4818	4818
query23	33533	32699	32700	32699
query24	11793	2844	2859	2844
query25	673	386	384	384
query26	1788	156	157	156
query27	3159	359	346	346
query28	7397	1908	1868	1868
query29	1052	633	614	614
query30	307	148	160	148
query31	993	727	730	727
query32	102	59	54	54
query33	771	252	258	252
query34	1133	499	490	490
query35	850	620	614	614
query36	1031	899	884	884
query37	277	68	65	65
query38	3633	3433	3475	3433
query39	1470	1554	1420	1420
query40	304	117	116	116
query41	52	48	48	48
query42	108	96	93	93
query43	496	451	430	430
query44	1167	722	720	720
query45	261	258	271	258
query46	1123	713	723	713
query47	1948	1835	1833	1833
query48	454	357	349	349
query49	1235	336	334	334
query50	758	366	364	364
query51	6786	6565	6557	6557
query52	112	86	93	86
query53	341	282	275	275
query54	327	258	236	236
query55	85	79	79	79
query56	249	229	224	224
query57	1237	1145	1145	1145
query58	233	198	205	198
query59	2769	2663	2609	2609
query60	260	234	233	233
query61	93	90	98	90
query62	656	445	452	445
query63	297	272	271	271
query64	6636	4056	4086	4056
query65	3122	3002	3020	3002
query66	1409	360	359	359
query67	15486	15011	14829	14829
query68	5445	499	520	499
query69	554	384	370	370
query70	1208	1181	1215	1181
query71	414	266	262	262
query72	6120	2727	2551	2551
query73	703	320	320	320
query74	6845	6501	6411	6411
query75	2944	2249	2232	2232
query76	3483	834	889	834
query77	468	255	242	242
query78	10785	10146	10217	10146
query79	6043	507	519	507
query80	1611	392	390	390
query81	551	225	219	219
query82	1078	84	89	84
query83	337	153	153	153
query84	291	87	83	83
query85	2018	386	360	360
query86	501	318	279	279
query87	3728	3518	3513	3513
query88	4911	2334	2286	2286
query89	492	361	358	358
query90	2042	178	178	178
query91	175	146	167	146
query92	62	45	48	45
query93	4946	492	477	477
query94	1217	174	176	174
query95	431	336	330	330
query96	602	275	270	270
query97	2645	2469	2517	2469
query98	228	220	217	217
query99	1237	925	941	925
Total cold run time: 304728 ms
Total hot run time: 181276 ms

@doris-robot
Copy link

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit d6f2046c5d2c2421a7fcfb065dcb8b276c0637fc with default session variables
Stream load json:         18 seconds loaded 2358488459 Bytes, about 124 MB/s
Stream load orc:          58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet:      32 seconds loaded 861443392 Bytes, about 25 MB/s
Insert into select:       13.8 seconds inserted 10000000 Rows, about 724K ops/s

@github-actions github-actions bot removed the approved Indicates a PR has been approved by one committer. label Mar 28, 2024
@englefly
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 37520 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit d6f2046c5d2c2421a7fcfb065dcb8b276c0637fc, data reload: false

------ Round 1 ----------------------------------
q1	17602	4372	4076	4076
q2	2117	174	159	159
q3	10674	1098	1132	1098
q4	10295	756	736	736
q5	7462	2883	2931	2883
q6	200	128	128	128
q7	1013	584	580	580
q8	9332	1942	1978	1942
q9	6833	6283	6264	6264
q10	8432	3485	3554	3485
q11	443	226	224	224
q12	376	203	199	199
q13	17805	2863	2851	2851
q14	234	205	200	200
q15	515	480	463	463
q16	488	374	372	372
q17	941	541	552	541
q18	7146	6468	6478	6468
q19	3348	1509	1392	1392
q20	548	266	264	264
q21	3501	2939	2905	2905
q22	336	293	290	290
Total cold run time: 109641 ms
Total hot run time: 37520 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4133	4063	4061	4061
q2	334	228	223	223
q3	2950	2881	2791	2791
q4	1853	1593	1566	1566
q5	5174	5167	5202	5167
q6	194	119	121	119
q7	2198	1813	1812	1812
q8	3167	3297	3261	3261
q9	8399	8374	8389	8374
q10	3816	3811	3815	3811
q11	558	452	447	447
q12	747	582	594	582
q13	17141	3011	3043	3011
q14	307	263	267	263
q15	512	476	481	476
q16	505	438	447	438
q17	1759	1513	1507	1507
q18	7799	7677	7547	7547
q19	1643	1552	1549	1549
q20	2024	1749	1717	1717
q21	4890	4877	4730	4730
q22	562	447	469	447
Total cold run time: 70665 ms
Total hot run time: 53899 ms

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Mar 29, 2024
@github-actions GitHub Actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@englefly englefly merged commit 360d5b2 into apache:master Mar 29, 2024
42 of 47 checks passed
@englefly englefly deleted the nullEqRfPushDown branch March 29, 2024 07:33
yiguolei pushed a commit that referenced this pull request Apr 1, 2024
@englefly @Doris-Extras
[fix](nereids) null-safe-eq runtime filter denies outer join #32927
e0927e7
yiguolei pushed a commit that referenced this pull request Apr 10, 2024
@englefly @Doris-Extras
[fix](nereids) null-safe-eq runtime filter denies outer join #32927
fdb9500
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@starocean999 starocean999 starocean999 approved these changes

@LiBinfeng-01 LiBinfeng-01 LiBinfeng-01 approved these changes

Assignees
No one assigned
Labels
approved Indicates a PR has been approved by one committer. dev/2.1.1-merged reviewed
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@englefly @doris-robot @starocean999 @LiBinfeng-01 @yiguolei