Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[feature](java-udtf) support java-udtf #33595

Merged
merged 9 commits into from
Apr 19, 2024
Merged

[feature](java-udtf) support java-udtf #33595

merged 9 commits into from
Apr 19, 2024

Conversation

zhangstar333
Copy link
Contributor

@zhangstar333 zhangstar333 commented Apr 12, 2024 •
edited
Loading

Proposed changes

support java-udtf, the doc in: apache/doris-website#569

 public class UDTFStringTest {
     public ArrayList<String> evaluate(String value, String separator) {
         if (value == null || separator == null) {
             return null;
         } else {
             return new ArrayList<>(Arrays.asList(value.split(separator)));
         }
     }
 }
CREATE TABLES FUNCTION java-utdf(string, string) RETURNS array<string> PROPERTIES (
    "file"="file:///pathTo/java-udaf.jar",
    "symbol"="org.apache.doris.udf.demo.UDTFStringTest",
    "always_nullable"="true",
    "type"="JAVA_UDF"
);

Issue Number: close #xxx

Further comments

If this is a relatively large or complex change, kick off the discussion at [email protected] by explaining why you chose the solution you did and what alternatives you considered, etc...

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@zhangstar333
Copy link
Contributor Author

run buildall

@zhangstar333 zhangstar333 changed the title [test](test) fe [feature](java-udtf) support java-udtf Apr 17, 2024
@zhangstar333 zhangstar333 marked this pull request as ready for review April 17, 2024 03:58
github-actions[bot]
github-actions bot reviewed Apr 17, 2024
View reviewed changes
Copy link
Contributor

@github-actions github-actions bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

clang-tidy made some suggestions

be/src/pipeline/exec/table_function_operator.h Show resolved Hide resolved
be/src/vec/exprs/table_function/table_function_factory.cpp Show resolved Hide resolved
be/src/vec/exprs/table_function/table_function_factory.h Show resolved Hide resolved
be/src/vec/exprs/table_function/udf_table_function.h Outdated Show resolved Hide resolved
be/src/vec/exprs/table_function/udf_table_function.h Outdated Show resolved Hide resolved
be/src/vec/exprs/table_function/udf_table_function.h Outdated Show resolved Hide resolved
@zhangstar333
Copy link
Contributor Author

run buildall

github-actions[bot]
github-actions bot reviewed Apr 17, 2024
View reviewed changes
Copy link
Contributor

@github-actions github-actions bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

clang-tidy made some suggestions

be/src/pipeline/exec/table_function_operator.h Show resolved Hide resolved
@zhangstar333
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 35.46% (8905/25113)
Line Coverage: 27.19% (73131/268947)
Region Coverage: 26.31% (37806/143672)
Branch Coverage: 23.10% (19263/83374)
Coverage Report: http://coverage.selectdb-in.cc/coverage/c63347440a952984e8573a48560ca5545d196b4b_c63347440a952984e8573a48560ca5545d196b4b/report/index.html

@doris-robot
Copy link

TPC-H: Total hot run time: 38578 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit c63347440a952984e8573a48560ca5545d196b4b, data reload: false

------ Round 1 ----------------------------------
q1	17610	4261	4249	4249
q2	2011	194	202	194
q3	10439	1142	1201	1142
q4	10182	913	719	719
q5	7548	2686	2668	2668
q6	219	136	137	136
q7	1011	604	602	602
q8	9236	2062	2035	2035
q9	7293	6663	6578	6578
q10	8641	3543	3528	3528
q11	453	228	222	222
q12	422	227	220	220
q13	17768	2953	2985	2953
q14	264	237	228	228
q15	520	492	477	477
q16	515	391	385	385
q17	963	676	717	676
q18	7332	6723	6619	6619
q19	7034	1532	1545	1532
q20	651	316	303	303
q21	3514	2815	2849	2815
q22	368	297	298	297
Total cold run time: 113994 ms
Total hot run time: 38578 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4348	4258	4232	4232
q2	372	266	269	266
q3	2971	2692	2738	2692
q4	1876	1593	1641	1593
q5	5368	5354	5322	5322
q6	212	129	128	128
q7	2234	1858	1831	1831
q8	3227	3385	3364	3364
q9	8622	8565	8693	8565
q10	4088	3883	4015	3883
q11	609	494	492	492
q12	782	633	633	633
q13	17179	3261	3229	3229
q14	343	304	297	297
q15	526	492	488	488
q16	496	460	450	450
q17	1839	1549	1498	1498
q18	8052	7899	7828	7828
q19	1672	1620	1604	1604
q20	2034	1867	1827	1827
q21	5089	5003	4922	4922
q22	538	494	471	471
Total cold run time: 72477 ms
Total hot run time: 55615 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 186296 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit c63347440a952984e8573a48560ca5545d196b4b, data reload: false

query1	893	390	372	372
query2	6201	2552	2491	2491
query3	6653	202	198	198
query4	23784	21383	21303	21303
query5	4194	406	415	406
query6	270	177	168	168
query7	4586	281	290	281
query8	229	174	178	174
query9	8574	2399	2340	2340
query10	425	238	253	238
query11	14722	14158	14262	14158
query12	131	87	88	87
query13	1637	357	350	350
query14	9960	7510	7971	7510
query15	259	178	185	178
query16	8269	263	268	263
query17	1996	589	593	589
query18	2117	279	279	279
query19	333	153	157	153
query20	94	86	86	86
query21	195	125	123	123
query22	5000	4839	4857	4839
query23	33721	32980	33290	32980
query24	10555	3129	3129	3129
query25	608	411	400	400
query26	731	162	160	160
query27	2328	370	379	370
query28	5965	2084	2135	2084
query29	880	668	643	643
query30	266	180	182	180
query31	1010	770	746	746
query32	94	56	57	56
query33	655	256	262	256
query34	902	502	524	502
query35	851	717	720	717
query36	1069	923	907	907
query37	115	74	70	70
query38	3423	3290	3331	3290
query39	1634	1590	1582	1582
query40	177	132	132	132
query41	49	45	48	45
query42	101	99	98	98
query43	598	553	565	553
query44	1150	739	746	739
query45	316	292	283	283
query46	1105	779	744	744
query47	2053	1976	1982	1976
query48	392	306	301	301
query49	840	400	407	400
query50	778	398	389	389
query51	6894	6821	6829	6821
query52	104	88	88	88
query53	341	271	279	271
query54	305	224	217	217
query55	74	70	68	68
query56	236	216	228	216
query57	1242	1142	1113	1113
query58	222	204	216	204
query59	3423	3153	3069	3069
query60	248	233	232	232
query61	88	86	85	85
query62	591	453	434	434
query63	305	278	274	274
query64	4700	4034	4017	4017
query65	3069	3034	3028	3028
query66	750	332	338	332
query67	15723	15080	14744	14744
query68	9047	546	543	543
query69	604	313	302	302
query70	1327	1172	1126	1126
query71	1530	1269	1268	1268
query72	6861	2636	2431	2431
query73	1628	332	323	323
query74	6836	6430	6415	6415
query75	4223	2609	2625	2609
query76	5663	911	1000	911
query77	613	265	272	265
query78	10953	10189	10081	10081
query79	12781	519	516	516
query80	2492	442	436	436
query81	524	243	241	241
query82	378	97	91	91
query83	243	162	164	162
query84	266	85	90	85
query85	908	270	255	255
query86	338	305	307	305
query87	3491	3368	3252	3252
query88	5557	2454	2448	2448
query89	522	369	372	369
query90	2063	191	187	187
query91	135	99	96	96
query92	57	49	46	46
query93	7295	491	489	489
query94	1364	184	178	178
query95	1099	1086	1089	1086
query96	632	271	270	270
query97	3107	2901	2911	2901
query98	244	233	211	211
query99	1165	867	878	867
Total cold run time: 305149 ms
Total hot run time: 186296 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.34 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit c63347440a952984e8573a48560ca5545d196b4b, data reload: false

query1	0.04	0.03	0.03
query2	0.08	0.04	0.04
query3	0.23	0.06	0.06
query4	1.66	0.07	0.07
query5	0.51	0.49	0.50
query6	1.48	0.72	0.71
query7	0.02	0.01	0.01
query8	0.05	0.04	0.05
query9	0.54	0.47	0.50
query10	0.54	0.56	0.55
query11	0.16	0.12	0.11
query12	0.15	0.12	0.12
query13	0.61	0.60	0.58
query14	0.77	0.78	0.79
query15	0.85	0.81	0.81
query16	0.36	0.36	0.37
query17	1.03	0.94	0.93
query18	0.21	0.25	0.23
query19	1.88	1.68	1.82
query20	0.01	0.01	0.01
query21	15.40	0.65	0.65
query22	4.47	7.58	1.81
query23	18.66	1.38	1.25
query24	2.08	0.21	0.24
query25	0.15	0.09	0.09
query26	0.27	0.18	0.17
query27	0.08	0.08	0.07
query28	13.33	1.00	1.00
query29	12.59	3.30	3.27
query30	0.26	0.06	0.05
query31	2.88	0.38	0.39
query32	3.27	0.48	0.47
query33	2.83	2.87	2.88
query34	17.05	4.47	4.41
query35	4.49	4.51	4.51
query36	0.65	0.46	0.46
query37	0.18	0.15	0.16
query38	0.16	0.15	0.14
query39	0.05	0.03	0.04
query40	0.17	0.14	0.14
query41	0.10	0.05	0.05
query42	0.06	0.05	0.04
query43	0.04	0.04	0.04
Total cold run time: 110.4 s
Total hot run time: 30.34 s

@doris-robot
Copy link

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit c63347440a952984e8573a48560ca5545d196b4b with default session variables
Stream load json:         18 seconds loaded 2358488459 Bytes, about 124 MB/s
Stream load orc:          58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet:      33 seconds loaded 861443392 Bytes, about 24 MB/s
Insert into select:       12.9 seconds inserted 10000000 Rows, about 775K ops/s

@zhangstar333 zhangstar333 mentioned this pull request Apr 17, 2024
Merged
@zhangstar333
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 35.43% (8908/25140)
Line Coverage: 27.13% (73168/269736)
Region Coverage: 26.27% (37833/144026)
Branch Coverage: 23.07% (19275/83566)
Coverage Report: http://coverage.selectdb-in.cc/coverage/fab6e7dfabee5684c9c94dfec4bd5ac80b81ceac_fab6e7dfabee5684c9c94dfec4bd5ac80b81ceac/report/index.html

@zhangstar333
Copy link
Contributor Author

run buildall

@zhangstar333
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 35.41% (8911/25163)
Line Coverage: 27.11% (73210/270040)
Region Coverage: 26.24% (37825/144156)
Branch Coverage: 23.06% (19280/83626)
Coverage Report: http://coverage.selectdb-in.cc/coverage/b39b66ca5b83cb4f0ef5f6fdc55460e00d775977_b39b66ca5b83cb4f0ef5f6fdc55460e00d775977/report/index.html

@zhangstar333
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 35.41% (8913/25170)
Line Coverage: 27.11% (73241/270147)
Region Coverage: 26.25% (37844/144190)
Branch Coverage: 23.05% (19283/83650)
Coverage Report: http://coverage.selectdb-in.cc/coverage/8a58d458739f04f81e57136f497ff403755ff2b9_8a58d458739f04f81e57136f497ff403755ff2b9/report/index.html

@zhangstar333
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 35.41% (8913/25170)
Line Coverage: 27.11% (73240/270147)
Region Coverage: 26.24% (37832/144190)
Branch Coverage: 23.05% (19282/83650)
Coverage Report: http://coverage.selectdb-in.cc/coverage/8a58d458739f04f81e57136f497ff403755ff2b9_8a58d458739f04f81e57136f497ff403755ff2b9/report/index.html

@zhangstar333
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 35.44% (8921/25174)
Line Coverage: 27.14% (73319/270182)
Region Coverage: 26.26% (37863/144205)
Branch Coverage: 23.07% (19297/83658)
Coverage Report: http://coverage.selectdb-in.cc/coverage/bc87583a1de8b9d0923d1e253f04054128e851d7_bc87583a1de8b9d0923d1e253f04054128e851d7/report/index.html

@zhangstar333
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 35.40% (8920/25200)
Line Coverage: 27.12% (73282/270214)
Region Coverage: 26.24% (37839/144228)
Branch Coverage: 23.06% (19284/83630)
Coverage Report: http://coverage.selectdb-in.cc/coverage/43496cce48500510628cb02188df8d34de5b3c02_43496cce48500510628cb02188df8d34de5b3c02/report/index.html

HappenLee
HappenLee approved these changes Apr 19, 2024
View reviewed changes
Copy link
Contributor

@HappenLee HappenLee left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Apr 19, 2024
@github-actions GitHub Actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions GitHub Actions
Copy link
Contributor

PR approved by anyone and no changes requested.

@zhangstar333 zhangstar333 merged commit 3a4b7bb into apache:master Apr 19, 2024
25 of 29 checks passed
zhangstar333 added a commit to zhangstar333/incubator-doris that referenced this pull request May 11, 2024
@zhangstar333
[feature](java-udtf) support java-udtf (apache#33595)
72e9edd
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@github-actions github-actions[bot] github-actions[bot] left review comments

@BiteTheDDDDt BiteTheDDDDt BiteTheDDDDt approved these changes

@HappenLee HappenLee HappenLee approved these changes

Assignees
No one assigned
Labels
2.1.0-conflict approved Indicates a PR has been approved by one committer. meta-change reviewed
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@zhangstar333 @doris-robot @BiteTheDDDDt @HappenLee @yiguolei