Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Enhancement](Short Circuit) short circuit query supports IN #39468

Open
wants to merge 64 commits into
base: master
Choose a base branch
from
Open

[Enhancement](Short Circuit) short circuit query supports IN #39468

wants to merge 64 commits into from

Conversation

lxr599
Copy link
Contributor

@lxr599 lxr599 commented Aug 16, 2024

Proposed changes

short circuit query support IN

lxr599 and others added 30 commits July 2, 2024 16:44
@lxr599
change default value of row_column_page_size to 16KB
ec59d28
@lxr599
allow to set row_column_page_size for tables
76a78da
@lxr599
[feature](Row store)allow to set row_column_page_size for tables
288bf5a
@lxr599
format codes
b544145
@lxr599
change default value of row_column_page_size to 16KB
2339bd1
@lxr599
allow to set row_column_page_size for tables
9a211b1
@lxr599
[feature](Row store)allow to set row_column_page_size for tables
9552bd5
@lxr599
format codes
8c4c0f7
@lxr599
rebase master
a9e584a
@lxr599
Merge branch 'master' into page_size
eedaa21
@lxr599
format codes
3f271b0
@lxr599
format fe codes
843d26d
@lxr599
Merge remote-tracking branch 'upstream/master' into page_size
2936a27
@lxr599
Merge branch 'master' into page_size
82a3e84
@lxr599
Merge branch 'master' into page_size
50dc477
@lxr599
display row_column_page_size only when row store enabled
fe22103
@lxr599
Merge branch 'master' into page_size
9706f9a
@lxr599
add test case for property row_column_page_size
9860a7f
@lxr599
Merge branch 'master' into page_size
29f3fd7
@lxr599
Merge branch 'apache:master' into page_size
78ba814
@lxr599
add more test cases
05374ba
@lxr599
Merge branch 'master' into page_size
a607007
@lxr599
change the property name
c32b788
@lxr599
change names of vars related to row_store_page_size
484160a
@lxr599
Merge remote-tracking branch 'upstream/master' into page_size
5c543f7
@lxr599
Merge branch 'master' into page_size
617dafd
@lxr599
Merge branch 'apache:master' into page_size
e0ab2fd
@lxr599
Merge branch 'master' of github.com:lxr599/doris into short_IN
779e08b
@lxr599
IN for short circuit query
a18e510
@lxr599
Merge branch 'master' of github.com:lxr599/doris into short_IN
1d92fdf
@lxr599
Copy link
Contributor Author

lxr599 commented Sep 13, 2024

run buildall

eldenmoon
eldenmoon previously approved these changes Sep 13, 2024
View reviewed changes
Copy link
Member

@eldenmoon eldenmoon left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

@github-actions GitHub Actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions github-actions bot added approved Indicates a PR has been approved by one committer. reviewed labels Sep 13, 2024
@github-actions GitHub Actions
Copy link
Contributor

PR approved by anyone and no changes requested.

@doris-robot
Copy link

TPC-H: Total hot run time: 43197 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit e93d59afaf0a77595efde92d02a010d90a36d62f, data reload: false

------ Round 1 ----------------------------------
q1	18173	7550	7299	7299
q2	2573	196	185	185
q3	11054	1427	1397	1397
q4	10725	976	1103	976
q5	8102	3274	3203	3203
q6	246	153	160	153
q7	1057	655	655	655
q8	9638	2044	2070	2044
q9	6803	6284	6286	6284
q10	7013	2548	2543	2543
q11	428	259	260	259
q12	432	234	231	231
q13	17769	3047	3025	3025
q14	295	259	267	259
q15	582	542	523	523
q16	527	451	429	429
q17	981	953	941	941
q18	7526	6713	6876	6713
q19	1384	1241	1228	1228
q20	613	355	329	329
q21	3929	3579	3534	3534
q22	1134	987	1035	987
Total cold run time: 110984 ms
Total hot run time: 43197 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7228	7189	7176	7176
q2	340	229	234	229
q3	2923	2923	2908	2908
q4	1963	1984	1940	1940
q5	5474	5446	5459	5446
q6	235	146	145	145
q7	2074	1699	1677	1677
q8	3245	3327	3295	3295
q9	8464	8449	8422	8422
q10	3394	3457	3452	3452
q11	571	455	474	455
q12	778	567	624	567
q13	5752	3077	3067	3067
q14	292	265	265	265
q15	579	529	521	521
q16	477	454	451	451
q17	1753	1703	1705	1703
q18	8066	7648	7520	7520
q19	1714	1692	1696	1692
q20	2047	1779	1818	1779
q21	5623	5451	5427	5427
q22	1097	1037	1032	1032
Total cold run time: 64089 ms
Total hot run time: 59169 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 195339 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit e93d59afaf0a77595efde92d02a010d90a36d62f, data reload: false

query1	919	392	373	373
query2	6491	1741	1776	1741
query3	6658	213	229	213
query4	26273	24244	23629	23629
query5	5414	539	517	517
query6	275	169	170	169
query7	4594	312	304	304
query8	287	217	214	214
query9	8485	2592	2576	2576
query10	494	296	284	284
query11	16188	15498	15560	15498
query12	172	101	102	101
query13	1693	408	378	378
query14	11584	6832	7047	6832
query15	220	176	180	176
query16	7495	472	469	469
query17	1543	594	571	571
query18	1882	298	296	296
query19	203	156	162	156
query20	130	113	112	112
query21	222	107	105	105
query22	4618	4273	4451	4273
query23	34497	33825	33725	33725
query24	9411	3176	3089	3089
query25	689	419	430	419
query26	1296	160	163	160
query27	2789	285	285	285
query28	6827	2112	2070	2070
query29	931	439	435	435
query30	301	154	155	154
query31	996	815	809	809
query32	108	57	61	57
query33	747	308	316	308
query34	920	484	487	484
query35	892	739	733	733
query36	1074	915	894	894
query37	175	84	86	84
query38	3959	3916	3992	3916
query39	1459	1440	1454	1440
query40	246	120	121	120
query41	52	50	49	49
query42	121	99	101	99
query43	489	441	453	441
query44	1266	813	786	786
query45	198	167	176	167
query46	1121	825	824	824
query47	1906	1792	1791	1791
query48	372	294	293	293
query49	1135	479	466	466
query50	906	441	452	441
query51	7116	6949	6949	6949
query52	105	89	88	88
query53	258	192	188	188
query54	830	480	488	480
query55	81	79	78	78
query56	298	276	279	276
query57	1248	1081	1066	1066
query58	261	265	250	250
query59	2675	2544	2573	2544
query60	316	291	284	284
query61	126	242	110	110
query62	900	686	702	686
query63	228	189	199	189
query64	5346	686	684	684
query65	3243	3181	3188	3181
query66	1391	298	296	296
query67	15856	15528	15411	15411
query68	3180	573	578	573
query69	447	279	276	276
query70	1160	1092	1120	1092
query71	353	289	282	282
query72	5991	4066	4112	4066
query73	769	332	335	332
query74	9392	9128	9105	9105
query75	3399	2753	2732	2732
query76	1912	1344	1311	1311
query77	459	324	313	313
query78	9974	10016	9377	9377
query79	1399	883	875	875
query80	992	851	829	829
query81	521	267	276	267
query82	881	267	265	265
query83	211	193	194	193
query84	244	113	110	110
query85	757	413	401	401
query86	337	315	313	313
query87	4397	4326	4339	4326
query88	4212	4126	4124	4124
query89	383	372	369	369
query90	1374	326	316	316
query91	125	126	127	126
query92	78	78	77	77
query93	1129	1078	1066	1066
query94	542	375	398	375
query95	488	429	426	426
query96	474	474	477	474
query97	3156	3136	3122	3122
query98	240	233	233	233
query99	1563	1308	1307	1307
Total cold run time: 281540 ms
Total hot run time: 195339 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 32.16 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit e93d59afaf0a77595efde92d02a010d90a36d62f, data reload: false

query1	0.05	0.05	0.04
query2	0.07	0.04	0.04
query3	0.23	0.05	0.04
query4	1.69	0.08	0.07
query5	0.49	0.50	0.50
query6	1.14	0.73	0.72
query7	0.02	0.01	0.02
query8	0.05	0.04	0.04
query9	0.57	0.53	0.52
query10	0.60	0.59	0.57
query11	0.16	0.11	0.12
query12	0.15	0.13	0.12
query13	0.62	0.61	0.61
query14	1.47	1.50	1.44
query15	0.92	0.89	0.88
query16	0.37	0.37	0.37
query17	1.02	1.02	1.02
query18	0.22	0.21	0.20
query19	1.94	1.80	1.84
query20	0.02	0.01	0.01
query21	15.40	0.66	0.65
query22	4.23	7.07	2.37
query23	18.00	1.28	1.29
query24	2.31	0.22	0.22
query25	0.19	0.08	0.07
query26	0.29	0.18	0.19
query27	0.08	0.08	0.08
query28	13.17	1.15	1.11
query29	12.52	3.34	3.33
query30	0.25	0.06	0.05
query31	2.86	0.43	0.42
query32	3.22	0.50	0.49
query33	3.03	3.03	3.07
query34	15.46	4.33	4.31
query35	4.34	4.34	4.37
query36	0.69	0.50	0.51
query37	0.18	0.17	0.16
query38	0.16	0.14	0.16
query39	0.06	0.04	0.04
query40	0.17	0.13	0.13
query41	0.10	0.05	0.05
query42	0.06	0.06	0.06
query43	0.05	0.04	0.04
Total cold run time: 108.62 s
Total hot run time: 32.16 s

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 36.88% (9463/25656)
Line Coverage: 28.26% (77836/275470)
Region Coverage: 27.66% (40198/145305)
Branch Coverage: 24.27% (20425/84146)
Coverage Report: http://coverage.selectdb-in.cc/coverage/e93d59afaf0a77595efde92d02a010d90a36d62f_e93d59afaf0a77595efde92d02a010d90a36d62f/report/index.html

@github-actions github-actions bot removed the approved Indicates a PR has been approved by one committer. label Sep 13, 2024
@lxr599
Copy link
Contributor Author

lxr599 commented Sep 13, 2024

run buildall

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 36.89% (9465/25656)
Line Coverage: 28.25% (77810/275470)
Region Coverage: 27.66% (40193/145305)
Branch Coverage: 24.27% (20420/84146)
Coverage Report: http://coverage.selectdb-in.cc/coverage/13463e5d1e9de36c3246703ae42c70a65411ef5c_13463e5d1e9de36c3246703ae42c70a65411ef5c/report/index.html

eldenmoon
eldenmoon previously approved these changes Sep 18, 2024
View reviewed changes
Copy link
Member

@eldenmoon eldenmoon left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Sep 18, 2024
@github-actions GitHub Actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@doris-robot
Copy link

TPC-H: Total hot run time: 41554 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 13463e5d1e9de36c3246703ae42c70a65411ef5c, data reload: false

------ Round 1 ----------------------------------
q1	17859	7430	7265	7265
q2	2037	168	163	163
q3	10643	1086	1169	1086
q4	10493	799	699	699
q5	7751	3089	3061	3061
q6	240	151	153	151
q7	1000	618	596	596
q8	9438	2105	2075	2075
q9	6876	6440	6430	6430
q10	7007	2282	2318	2282
q11	433	245	247	245
q12	420	217	224	217
q13	17789	3001	2960	2960
q14	241	211	217	211
q15	587	530	527	527
q16	668	631	617	617
q17	982	806	800	800
q18	7467	6734	6820	6734
q19	1403	975	991	975
q20	608	295	284	284
q21	4070	3168	3310	3168
q22	1111	1027	1008	1008
Total cold run time: 109123 ms
Total hot run time: 41554 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7284	7210	7196	7196
q2	326	228	228	228
q3	3061	2991	2972	2972
q4	2084	1918	1850	1850
q5	5639	5587	5707	5587
q6	237	142	142	142
q7	2187	1802	1812	1802
q8	3325	3423	3432	3423
q9	8773	8922	8874	8874
q10	3477	3478	3535	3478
q11	588	486	483	483
q12	816	613	637	613
q13	11358	3150	3144	3144
q14	311	279	304	279
q15	572	526	532	526
q16	708	657	678	657
q17	1812	1629	1578	1578
q18	8290	7871	7865	7865
q19	1736	1577	1401	1401
q20	2111	1890	1881	1881
q21	5570	5231	5483	5231
q22	1131	1092	1082	1082
Total cold run time: 71396 ms
Total hot run time: 60292 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 199842 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 13463e5d1e9de36c3246703ae42c70a65411ef5c, data reload: false

query1	1304	878	917	878
query2	6383	1953	2008	1953
query3	10777	3982	3950	3950
query4	63229	29661	23408	23408
query5	5172	470	472	470
query6	430	166	163	163
query7	5457	294	292	292
query8	305	222	222	222
query9	8388	2702	2699	2699
query10	466	296	277	277
query11	17716	15219	15599	15219
query12	161	100	103	100
query13	1477	419	439	419
query14	10817	7205	7528	7205
query15	206	175	179	175
query16	6780	507	498	498
query17	1164	632	580	580
query18	1472	305	306	305
query19	214	155	153	153
query20	123	111	115	111
query21	213	105	105	105
query22	4994	4731	4760	4731
query23	34806	33812	34117	33812
query24	6002	2883	2900	2883
query25	508	415	418	415
query26	643	157	162	157
query27	1791	280	281	280
query28	4284	2490	2471	2471
query29	679	427	433	427
query30	231	165	155	155
query31	986	767	791	767
query32	70	56	56	56
query33	432	297	300	297
query34	914	485	500	485
query35	856	737	708	708
query36	1060	914	924	914
query37	145	88	88	88
query38	3996	3884	3943	3884
query39	1451	1395	1387	1387
query40	212	96	97	96
query41	46	48	46	46
query42	119	101	97	97
query43	503	481	474	474
query44	1165	834	800	800
query45	190	163	167	163
query46	1129	787	754	754
query47	1884	1787	1827	1787
query48	466	363	357	357
query49	724	410	479	410
query50	840	403	403	403
query51	7164	6963	7079	6963
query52	102	89	86	86
query53	243	178	179	178
query54	563	444	464	444
query55	75	72	79	72
query56	265	250	252	250
query57	1199	1085	1116	1085
query58	224	218	234	218
query59	3233	2979	3025	2979
query60	295	261	282	261
query61	107	106	102	102
query62	771	670	682	670
query63	213	183	186	183
query64	1395	646	650	646
query65	3232	3149	3157	3149
query66	671	328	288	288
query67	16509	15804	15711	15711
query68	1328	852	843	843
query69	475	348	341	341
query70	1141	1210	1202	1202
query71	347	326	337	326
query72	6384	3386	3408	3386
query73	592	575	583	575
query74	9250	9031	8993	8993
query75	3040	2888	2940	2888
query76	1166	868	918	868
query77	438	358	367	358
query78	9786	9303	9342	9303
query79	920	893	872	872
query80	604	598	576	576
query81	459	258	250	250
query82	232	232	232	232
query83	164	158	156	156
query84	268	107	132	107
query85	687	372	362	362
query86	317	317	315	315
query87	4354	4356	4366	4356
query88	4466	4080	4060	4060
query89	371	364	363	363
query90	1327	307	309	307
query91	160	158	164	158
query92	73	70	76	70
query93	907	907	894	894
query94	483	366	347	347
query95	423	409	405	405
query96	487	484	490	484
query97	3193	3158	3174	3158
query98	232	231	217	217
query99	1432	1299	1300	1299
Total cold run time: 305147 ms
Total hot run time: 199842 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 33.59 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 13463e5d1e9de36c3246703ae42c70a65411ef5c, data reload: false

query1	0.05	0.04	0.04
query2	0.08	0.04	0.04
query3	0.23	0.06	0.05
query4	1.65	0.08	0.09
query5	0.52	0.52	0.51
query6	1.12	0.75	0.74
query7	0.03	0.02	0.02
query8	0.06	0.05	0.05
query9	0.55	0.52	0.50
query10	0.54	0.57	0.54
query11	0.16	0.12	0.11
query12	0.16	0.12	0.12
query13	0.61	0.58	0.58
query14	3.10	2.96	2.94
query15	0.91	0.83	0.84
query16	0.39	0.38	0.37
query17	1.05	1.06	1.05
query18	0.17	0.17	0.17
query19	1.91	1.96	1.95
query20	0.02	0.01	0.02
query21	15.37	0.67	0.67
query22	3.92	6.83	1.85
query23	18.26	1.38	1.30
query24	2.08	0.23	0.22
query25	0.15	0.08	0.08
query26	0.26	0.18	0.18
query27	0.08	0.08	0.08
query28	13.24	1.16	1.14
query29	12.62	3.42	3.39
query30	0.24	0.06	0.05
query31	2.86	0.40	0.41
query32	3.24	0.49	0.49
query33	3.00	3.03	3.03
query34	16.93	4.50	4.42
query35	4.53	4.49	4.48
query36	0.67	0.49	0.52
query37	0.20	0.16	0.17
query38	0.16	0.15	0.15
query39	0.05	0.04	0.04
query40	0.15	0.13	0.14
query41	0.09	0.05	0.05
query42	0.07	0.05	0.05
query43	0.05	0.04	0.04
Total cold run time: 111.53 s
Total hot run time: 33.59 s

@github-actions github-actions bot removed the approved Indicates a PR has been approved by one committer. label Sep 29, 2024
@lxr599
Copy link
Contributor Author

lxr599 commented Sep 29, 2024

run buildall

@lxr599
Copy link
Contributor Author

lxr599 commented Sep 29, 2024

run buildall

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 37.28% (9625/25817)
Line Coverage: 28.67% (79652/277800)
Region Coverage: 28.12% (41201/146543)
Branch Coverage: 24.73% (20977/84822)
Coverage Report: http://coverage.selectdb-in.cc/coverage/461a11a4f9e0463454b622be246fb50dd973952f_461a11a4f9e0463454b622be246fb50dd973952f/report/index.html

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@eldenmoon eldenmoon eldenmoon left review comments

At least 1 approving review is required to merge this pull request.

Assignees
No one assigned
Labels
reviewed
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@lxr599 @doris-robot @eldenmoon @Liuyushiii