Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[fix](nereids) do not transpose semi join agg when mark join #32475

Merged
merged 5 commits into from
Apr 11, 2024
Merged

[fix](nereids) do not transpose semi join agg when mark join #32475

merged 5 commits into from
Apr 11, 2024

Conversation

iwanttobepowerful
Copy link
Contributor

Proposed changes

Issue Number: close #31308 #31300

Further comments

If this is a relatively large or complex change, kick off the discussion at [email protected] by explaining why you chose the solution you did and what alternatives you considered, etc...

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@iwanttobepowerful
Copy link
Contributor Author

run buildall

@iwanttobepowerful
Copy link
Contributor Author

run p0

1 similar comment
@iwanttobepowerful
Copy link
Contributor Author

run p0

@iwanttobepowerful
Copy link
Contributor Author

run buildall

@iwanttobepowerful
Copy link
Contributor Author

@morrySnow @jackwener Hi, can you help review? Thx

@iwanttobepowerful
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 37618 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit e43f4d4bb27c29cf10b2ad1bde020a5d8b6e620e, data reload: false

------ Round 1 ----------------------------------
q1	17635	4219	4103	4103
q2	2026	151	144	144
q3	10597	1214	1238	1214
q4	10233	729	749	729
q5	7460	3038	2920	2920
q6	208	124	123	123
q7	1041	583	579	579
q8	9327	2018	1998	1998
q9	7039	6492	6471	6471
q10	8404	3256	3469	3256
q11	436	223	216	216
q12	380	201	201	201
q13	17822	2877	2863	2863
q14	227	197	196	196
q15	500	462	455	455
q16	472	356	359	356
q17	962	524	595	524
q18	7278	6551	6425	6425
q19	1585	1432	1508	1432
q20	534	263	251	251
q21	3616	3002	2859	2859
q22	357	303	326	303
Total cold run time: 108139 ms
Total hot run time: 37618 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4108	4050	4028	4028
q2	328	221	229	221
q3	3008	2839	2794	2794
q4	1873	1579	1535	1535
q5	5232	5268	5259	5259
q6	198	116	116	116
q7	2253	1890	1827	1827
q8	3181	3283	3279	3279
q9	8532	8545	8536	8536
q10	3691	3674	3626	3626
q11	545	448	437	437
q12	749	532	554	532
q13	16906	2837	2860	2837
q14	294	249	256	249
q15	487	456	448	448
q16	461	414	420	414
q17	1743	1471	1430	1430
q18	7609	7230	7080	7080
q19	1619	1531	1534	1531
q20	1873	1714	1694	1694
q21	4701	4630	4897	4630
q22	547	447	427	427
Total cold run time: 69938 ms
Total hot run time: 52930 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 180768 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit e43f4d4bb27c29cf10b2ad1bde020a5d8b6e620e, data reload: false

query1	931	364	348	348
query2	7591	2092	1885	1885
query3	6738	220	211	211
query4	31253	20569	20638	20569
query5	4347	401	402	401
query6	268	175	189	175
query7	4624	283	282	282
query8	233	163	170	163
query9	9941	2254	2260	2254
query10	465	237	250	237
query11	17248	14092	14169	14092
query12	134	87	85	85
query13	1628	422	423	422
query14	12050	10695	11071	10695
query15	255	206	189	189
query16	8118	259	266	259
query17	1942	574	535	535
query18	2067	282	279	279
query19	245	162	153	153
query20	92	88	86	86
query21	201	134	128	128
query22	4664	4544	4456	4456
query23	31781	30993	31217	30993
query24	10800	2834	2776	2776
query25	640	371	384	371
query26	1539	155	153	153
query27	3066	359	355	355
query28	8137	1912	1888	1888
query29	953	676	632	632
query30	303	144	145	144
query31	959	736	750	736
query32	93	57	55	55
query33	854	254	331	254
query34	930	473	485	473
query35	831	616	591	591
query36	1017	875	874	874
query37	124	80	77	77
query38	3522	3436	3416	3416
query39	1414	1365	1374	1365
query40	244	117	113	113
query41	49	46	47	46
query42	105	99	96	96
query43	490	440	441	440
query44	1180	710	703	703
query45	272	265	261	261
query46	1093	681	668	668
query47	1674	1582	1610	1582
query48	449	370	350	350
query49	1171	351	333	333
query50	761	379	372	372
query51	6710	6598	6619	6598
query52	108	87	90	87
query53	340	279	282	279
query54	308	252	270	252
query55	80	90	77	77
query56	247	241	228	228
query57	1092	1009	1001	1001
query58	243	203	209	203
query59	2720	2476	2534	2476
query60	284	249	265	249
query61	114	111	111	111
query62	607	394	405	394
query63	311	281	283	281
query64	6619	3961	3812	3812
query65	3100	3010	3016	3010
query66	1423	344	333	333
query67	15137	14371	14475	14371
query68	8262	512	519	512
query69	635	376	378	376
query70	1230	1139	1146	1139
query71	509	282	281	281
query72	6340	2695	2504	2504
query73	726	318	312	312
query74	7810	6537	6531	6531
query75	4324	2816	2887	2816
query76	5027	933	947	933
query77	655	255	256	255
query78	10410	9632	9631	9631
query79	12483	517	512	512
query80	1826	375	381	375
query81	506	212	207	207
query82	884	202	201	201
query83	225	144	145	144
query84	280	76	76	76
query85	1480	320	316	316
query86	468	288	322	288
query87	3706	3543	3534	3534
query88	5490	2292	2274	2274
query89	501	361	367	361
query90	2000	180	171	171
query91	158	152	131	131
query92	64	47	47	47
query93	7586	492	479	479
query94	1196	172	171	171
query95	424	328	324	324
query96	594	271	266	266
query97	3025	2861	2867	2861
query98	254	212	205	205
query99	1256	759	752	752
Total cold run time: 319263 ms
Total hot run time: 180768 ms

@doris-robot
Copy link

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit e43f4d4bb27c29cf10b2ad1bde020a5d8b6e620e with default session variables
Stream load json:         19 seconds loaded 2358488459 Bytes, about 118 MB/s
Stream load orc:          59 seconds loaded 1101869774 Bytes, about 17 MB/s
Stream load parquet:      31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select:       20.3 seconds inserted 10000000 Rows, about 492K ops/s

@iwanttobepowerful
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 38372 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 6399c12bbadb0ef2220e87eeb8b2f4c23dcc55d6, data reload: false

------ Round 1 ----------------------------------
q1	17650	4533	4256	4256
q2	2428	173	147	147
q3	11312	1122	1202	1122
q4	10768	751	850	751
q5	7817	3119	3052	3052
q6	208	127	123	123
q7	1057	615	574	574
q8	9541	2058	2039	2039
q9	7188	6689	6630	6630
q10	8868	3490	3539	3490
q11	438	229	220	220
q12	382	191	193	191
q13	17797	2842	2866	2842
q14	239	213	204	204
q15	507	466	449	449
q16	446	355	346	346
q17	966	666	557	557
q18	7424	6665	6576	6576
q19	1595	1429	1456	1429
q20	546	264	241	241
q21	3593	2827	2962	2827
q22	341	316	306	306
Total cold run time: 111111 ms
Total hot run time: 38372 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4200	4054	4087	4054
q2	323	225	227	225
q3	3000	2871	2862	2862
q4	1850	1528	1543	1528
q5	5227	5256	5264	5256
q6	201	121	122	121
q7	2343	1906	1865	1865
q8	3170	3334	3323	3323
q9	8633	8584	8599	8584
q10	3650	3687	3660	3660
q11	551	439	433	433
q12	727	562	563	562
q13	16901	2890	2832	2832
q14	267	259	259	259
q15	491	441	440	440
q16	468	431	418	418
q17	1751	1490	1474	1474
q18	7565	7125	7071	7071
q19	1612	1540	1590	1540
q20	1914	1720	1677	1677
q21	4892	4748	4761	4748
q22	568	466	468	466
Total cold run time: 70304 ms
Total hot run time: 53398 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 181010 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 6399c12bbadb0ef2220e87eeb8b2f4c23dcc55d6, data reload: false

query1	937	364	336	336
query2	7414	2105	1951	1951
query3	6709	224	217	217
query4	30915	20736	20643	20643
query5	4386	420	665	420
query6	268	202	168	168
query7	4637	295	284	284
query8	227	165	172	165
query9	9669	2267	2259	2259
query10	467	233	242	233
query11	17355	14268	14379	14268
query12	142	97	88	88
query13	1638	410	410	410
query14	11704	10783	10177	10177
query15	258	209	196	196
query16	8174	258	256	256
query17	1947	570	536	536
query18	2109	292	277	277
query19	271	160	176	160
query20	97	97	86	86
query21	203	130	131	130
query22	4643	4462	4489	4462
query23	31844	31055	31155	31055
query24	10723	2848	2843	2843
query25	617	377	372	372
query26	1195	152	154	152
query27	2530	347	349	347
query28	7724	1959	1872	1872
query29	895	655	614	614
query30	302	147	149	147
query31	978	728	740	728
query32	100	58	57	57
query33	773	275	253	253
query34	1032	475	489	475
query35	846	616	610	610
query36	1013	884	872	872
query37	120	83	79	79
query38	3534	3411	3465	3411
query39	1419	1408	1359	1359
query40	218	113	108	108
query41	46	44	43	43
query42	105	93	92	92
query43	474	456	471	456
query44	1277	697	704	697
query45	289	221	253	221
query46	1106	681	682	681
query47	1647	1608	1578	1578
query48	459	347	345	345
query49	1118	325	325	325
query50	772	361	360	360
query51	6732	6602	6618	6602
query52	107	85	92	85
query53	336	267	269	267
query54	304	227	245	227
query55	80	76	75	75
query56	254	219	231	219
query57	1100	1025	1012	1012
query58	232	205	203	203
query59	2992	2831	2660	2660
query60	263	244	244	244
query61	93	91	92	91
query62	648	417	390	390
query63	302	278	280	278
query64	5453	3954	3866	3866
query65	3060	3008	3006	3006
query66	855	345	345	345
query67	14997	14466	14188	14188
query68	5317	518	512	512
query69	583	368	366	366
query70	1229	1161	1152	1152
query71	433	282	273	273
query72	6341	2840	2614	2614
query73	721	316	332	316
query74	7357	6534	6644	6534
query75	3623	2942	2904	2904
query76	3817	995	964	964
query77	634	262	271	262
query78	10500	9631	9618	9618
query79	8711	539	538	538
query80	1992	424	409	409
query81	561	212	215	212
query82	1351	201	207	201
query83	329	146	154	146
query84	293	83	84	83
query85	1719	326	313	313
query86	482	318	305	305
query87	3702	3539	3530	3530
query88	5230	2322	2320	2320
query89	525	364	380	364
query90	1955	183	176	176
query91	169	147	128	128
query92	67	49	46	46
query93	6570	502	492	492
query94	1258	181	175	175
query95	447	327	332	327
query96	623	278	276	276
query97	3099	2852	2868	2852
query98	232	224	212	212
query99	1133	758	758	758
Total cold run time: 306137 ms
Total hot run time: 181010 ms

@doris-robot
Copy link

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit 6399c12bbadb0ef2220e87eeb8b2f4c23dcc55d6 with default session variables
Stream load json:         18 seconds loaded 2358488459 Bytes, about 124 MB/s
Stream load orc:          58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet:      31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select:       22.2 seconds inserted 10000000 Rows, about 450K ops/s

@iwanttobepowerful
Copy link
Contributor Author

run p0

@iwanttobepowerful
Copy link
Contributor Author

run buildall

@iwanttobepowerful
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 38418 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 37cc32415655af2ca2e8f61b51c8cb762d810bb8, data reload: false

------ Round 1 ----------------------------------
q1	17598	5140	4454	4454
q2	1988	191	194	191
q3	10491	1117	1147	1117
q4	10197	761	699	699
q5	7444	2832	2718	2718
q6	212	128	133	128
q7	1017	604	606	604
q8	9214	2066	2091	2066
q9	7529	6637	6566	6566
q10	8548	3523	3541	3523
q11	454	238	229	229
q12	485	220	207	207
q13	18914	2936	2947	2936
q14	277	245	239	239
q15	533	476	482	476
q16	523	397	383	383
q17	972	560	634	560
q18	7438	6811	6694	6694
q19	4481	1478	1414	1414
q20	655	297	307	297
q21	3372	2623	2765	2623
q22	361	294	309	294
Total cold run time: 112703 ms
Total hot run time: 38418 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4344	4264	4210	4210
q2	374	279	284	279
q3	2965	2748	2714	2714
q4	1819	1512	1509	1509
q5	5285	5215	5265	5215
q6	208	122	122	122
q7	2243	1898	1872	1872
q8	3237	3322	3341	3322
q9	8561	8574	8591	8574
q10	3900	3677	3708	3677
q11	579	484	498	484
q12	776	580	649	580
q13	17670	2952	2943	2943
q14	307	282	254	254
q15	501	466	473	466
q16	496	443	441	441
q17	1774	1423	1456	1423
q18	7889	8018	7902	7902
q19	1725	1577	1532	1532
q20	2076	1834	1885	1834
q21	9721	4933	5082	4933
q22	560	505	493	493
Total cold run time: 77010 ms
Total hot run time: 54779 ms

@iwanttobepowerful iwanttobepowerful mentioned this pull request Apr 8, 2024
Closed
@iwanttobepowerful
Copy link
Contributor Author

run buildall

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Apr 11, 2024
@github-actions GitHub Actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions GitHub Actions
Copy link
Contributor

PR approved by anyone and no changes requested.

@starocean999 starocean999 merged commit f06765f into apache:master Apr 11, 2024
26 of 29 checks passed
@iwanttobepowerful iwanttobepowerful deleted the fix31308 branch April 11, 2024 11:09
yiguolei pushed a commit that referenced this pull request Apr 12, 2024
@iwanttobepowerful @Doris-Extras
[fix](nereids) do not transpose semi join agg when mark join (#32475)
49bb063
yiguolei pushed a commit that referenced this pull request Apr 13, 2024
@iwanttobepowerful @Doris-Extras
[fix](nereids) do not transpose semi join agg when mark join (#32475)
5e43fd6
yiguolei pushed a commit that referenced this pull request Apr 13, 2024
@iwanttobepowerful @Doris-Extras
[fix](nereids) do not transpose semi join agg when mark join (#32475)
0423419
yiguolei pushed a commit that referenced this pull request Apr 17, 2024
@iwanttobepowerful @Doris-Extras
[fix](nereids) do not transpose semi join agg when mark join (#32475)
82d2bde
@jgq2008303393
Copy link

@iwanttobepowerful 这个没有合入2.0，比较明显的查询问题

@starocean999 starocean999 mentioned this pull request Apr 22, 2024
Merged
dataroaring pushed a commit that referenced this pull request Apr 24, 2024
@iwanttobepowerful @dataroaring
[fix](nereids) do not transpose semi join agg when mark join (#32475)
8efb658
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@jackwener jackwener jackwener approved these changes

@starocean999 starocean999 starocean999 approved these changes

Assignees
No one assigned
Labels
approved Indicates a PR has been approved by one committer. dev/2.0.10-merged reviewed
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

[Bug] Only support subquery in binary predicate in case statement
7 participants
@iwanttobepowerful @doris-robot @jgq2008303393 @jackwener @starocean999 @xiaokang @wm1581066