Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[exec](table_fun) opt numbers table func performance #33804

Merged
merged 2 commits into from
Apr 18, 2024

Conversation

HappenLee
Copy link
Contributor

@HappenLee HappenLee commented Apr 17, 2024

Proposed changes

before:

[tpch]>select count(e1) from (select 1 as k1) as t lateral view explode_numbers(1000000000) tmp1 as e1;
+------------+
| count(e1)  |
+------------+
| 1000000000 |
+------------+
1 row in set (5.60 sec)

after:

[tpch]>select count(e1) from (select 1 as k1) as t lateral view explode_numbers(1000000000) tmp1 as e1;
+------------+
| count(e1)  |
+------------+
| 1000000000 |
+------------+
1 row in set (0.60 sec)

Further comments

If this is a relatively large or complex change, kick off the discussion at [email protected] by explaining why you chose the solution you did and what alternatives you considered, etc...

@HappenLee
Copy link
Contributor Author

run buildall

Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 35.45% (8908/25127)
Line Coverage: 27.14% (73164/269569)
Region Coverage: 26.29% (37834/143936)
Branch Coverage: 23.08% (19279/83522)
Coverage Report: http://coverage.selectdb-in.cc/coverage/89c8c31fa81a9a96a43452b32ef8936bee1adae4_89c8c31fa81a9a96a43452b32ef8936bee1adae4/report/index.html

@HappenLee
Copy link
Contributor Author

run buildall

Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 35.45% (8908/25127)
Line Coverage: 27.14% (73154/269576)
Region Coverage: 26.28% (37828/143939)
Branch Coverage: 23.08% (19274/83524)
Coverage Report: http://coverage.selectdb-in.cc/coverage/ecc46e2a24e9b4be42a34cac0fd45ce3d47701cb_ecc46e2a24e9b4be42a34cac0fd45ce3d47701cb/report/index.html

@doris-robot
Copy link

TPC-H: Total hot run time: 38224 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit ecc46e2a24e9b4be42a34cac0fd45ce3d47701cb, data reload: false

------ Round 1 ----------------------------------
q1	17604	4303	4259	4259
q2	2006	189	187	187
q3	10452	1117	1160	1117
q4	10187	779	692	692
q5	7492	2679	2635	2635
q6	216	131	133	131
q7	1036	600	593	593
q8	9215	2082	2037	2037
q9	7351	6557	6533	6533
q10	8559	3510	3496	3496
q11	436	224	231	224
q12	501	213	206	206
q13	17775	2958	2985	2958
q14	276	229	227	227
q15	511	492	466	466
q16	527	376	371	371
q17	959	640	718	640
q18	7274	6696	6670	6670
q19	7125	1575	1465	1465
q20	647	313	300	300
q21	3451	2720	2899	2720
q22	361	297	301	297
Total cold run time: 113961 ms
Total hot run time: 38224 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4396	4275	4298	4275
q2	363	259	272	259
q3	2984	2704	2738	2704
q4	1862	1582	1527	1527
q5	5347	5332	5314	5314
q6	207	123	123	123
q7	2178	1879	1838	1838
q8	3216	3327	3351	3327
q9	8539	8590	8593	8590
q10	4056	3920	3977	3920
q11	584	511	502	502
q12	802	607	653	607
q13	17329	3286	3104	3104
q14	310	306	287	287
q15	518	478	482	478
q16	497	445	459	445
q17	1870	1536	1529	1529
q18	8078	8052	7971	7971
q19	1674	1566	1511	1511
q20	2047	1835	1847	1835
q21	9613	4937	5095	4937
q22	545	464	475	464
Total cold run time: 77015 ms
Total hot run time: 55547 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 185178 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit ecc46e2a24e9b4be42a34cac0fd45ce3d47701cb, data reload: false

query1	879	381	365	365
query2	6598	2796	2414	2414
query3	6652	206	198	198
query4	23136	21300	21224	21224
query5	4140	409	430	409
query6	279	173	176	173
query7	4590	288	290	288
query8	234	178	172	172
query9	8489	2357	2332	2332
query10	422	252	264	252
query11	14612	14275	14110	14110
query12	140	93	84	84
query13	1640	362	357	357
query14	9554	7101	7797	7101
query15	255	183	194	183
query16	8148	266	257	257
query17	1939	575	561	561
query18	2090	279	276	276
query19	197	153	158	153
query20	89	89	86	86
query21	197	127	122	122
query22	5098	4859	4891	4859
query23	33664	33221	33341	33221
query24	10997	3004	2940	2940
query25	598	388	374	374
query26	704	163	155	155
query27	2447	355	368	355
query28	6232	2086	2097	2086
query29	857	632	640	632
query30	314	175	171	171
query31	1014	755	751	751
query32	88	53	52	52
query33	647	246	251	246
query34	894	485	496	485
query35	792	701	717	701
query36	1074	941	942	941
query37	106	68	69	68
query38	3435	3295	3258	3258
query39	1638	1605	1581	1581
query40	170	142	138	138
query41	44	45	42	42
query42	104	97	103	97
query43	611	534	562	534
query44	1090	747	733	733
query45	295	263	285	263
query46	1081	743	721	721
query47	2011	1911	1936	1911
query48	378	308	321	308
query49	817	372	396	372
query50	789	401	415	401
query51	6837	6855	6733	6733
query52	99	91	92	91
query53	344	276	282	276
query54	296	224	219	219
query55	73	70	68	68
query56	231	219	230	219
query57	1263	1143	1130	1130
query58	215	195	200	195
query59	3632	3356	3334	3334
query60	242	227	227	227
query61	85	83	85	83
query62	590	434	435	434
query63	297	274	277	274
query64	4707	3951	4069	3951
query65	3112	3020	3006	3006
query66	749	321	324	321
query67	15513	15015	15114	15015
query68	5134	524	527	524
query69	487	292	300	292
query70	1207	1105	1120	1105
query71	1395	1263	1268	1263
query72	6382	2609	2425	2425
query73	704	322	363	322
query74	6824	6424	6411	6411
query75	3347	2589	2629	2589
query76	2892	966	959	959
query77	569	268	265	265
query78	10967	10285	10199	10199
query79	4936	522	545	522
query80	1763	447	440	440
query81	517	248	242	242
query82	1213	99	96	96
query83	326	177	167	167
query84	263	94	84	84
query85	1651	271	258	258
query86	472	294	261	261
query87	3474	3319	3308	3308
query88	4863	2397	2407	2397
query89	477	374	378	374
query90	1905	187	185	185
query91	122	97	96	96
query92	56	47	50	47
query93	5658	503	502	502
query94	1133	186	181	181
query95	445	299	305	299
query96	586	267	259	259
query97	3093	2938	2958	2938
query98	227	230	218	218
query99	1187	863	861	861
Total cold run time: 284829 ms
Total hot run time: 185178 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.5 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit ecc46e2a24e9b4be42a34cac0fd45ce3d47701cb, data reload: false

query1	0.04	0.04	0.04
query2	0.08	0.03	0.04
query3	0.23	0.04	0.04
query4	1.70	0.07	0.06
query5	0.50	0.50	0.50
query6	1.48	0.72	0.71
query7	0.02	0.01	0.02
query8	0.04	0.04	0.04
query9	0.55	0.49	0.50
query10	0.55	0.55	0.55
query11	0.15	0.12	0.12
query12	0.14	0.12	0.11
query13	0.61	0.59	0.58
query14	0.75	0.81	0.77
query15	0.84	0.81	0.81
query16	0.37	0.37	0.36
query17	0.94	1.02	1.03
query18	0.22	0.22	0.27
query19	1.87	1.79	1.80
query20	0.01	0.01	0.01
query21	15.40	0.65	0.65
query22	4.50	6.08	2.06
query23	18.29	1.39	1.25
query24	1.62	0.31	0.20
query25	0.15	0.09	0.08
query26	0.28	0.17	0.17
query27	0.08	0.07	0.08
query28	13.52	1.00	0.99
query29	12.57	3.25	3.23
query30	0.25	0.08	0.05
query31	2.86	0.39	0.37
query32	3.27	0.47	0.47
query33	2.80	2.78	2.82
query34	17.12	4.39	4.37
query35	4.46	4.46	4.45
query36	0.65	0.46	0.46
query37	0.18	0.15	0.15
query38	0.15	0.15	0.15
query39	0.04	0.04	0.04
query40	0.17	0.14	0.13
query41	0.10	0.05	0.05
query42	0.05	0.04	0.05
query43	0.04	0.03	0.04
Total cold run time: 109.64 s
Total hot run time: 30.5 s

@doris-robot
Copy link

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit ecc46e2a24e9b4be42a34cac0fd45ce3d47701cb with default session variables
Stream load json:         18 seconds loaded 2358488459 Bytes, about 124 MB/s
Stream load orc:          58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet:      33 seconds loaded 861443392 Bytes, about 24 MB/s
Insert into select:       13.3 seconds inserted 10000000 Rows, about 751K ops/s

Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions github-actions bot added approved Indicates a PR has been approved by one committer. reviewed labels Apr 18, 2024
Copy link
Contributor

PR approved by anyone and no changes requested.

@HappenLee HappenLee merged commit c860237 into apache:master Apr 18, 2024
26 of 29 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by one committer. reviewed
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants