Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[fix](glue)support access glue iceberg with credential list #32381

Merged
merged 1 commit into from
Mar 18, 2024
Merged

[fix](glue)support access glue iceberg with credential list #32381

merged 1 commit into from
Mar 18, 2024

Conversation

wsjz
Copy link
Contributor

@wsjz wsjz commented Mar 18, 2024

Proposed changes

from #30292

Further comments

If this is a relatively large or complex change, kick off the discussion at [email protected] by explaining why you chose the solution you did and what alternatives you considered, etc...

@wsjz
[fix](glue)support access glue iceberg with credential list
c7dbe62 
support access glue and s3 iceberg with credential list
support iceberg hadoop catalog on s3

(cherry picked from commit ef85d0d)
@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

@wsjz
Copy link
Contributor Author

wsjz commented Mar 18, 2024

run buildall

@github-actions github-actions bot added kind/docs Categorizes issue or PR as related to documentation. kind/test labels Mar 18, 2024
@github-actions GitHub Actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 37.82% (8042/21265)
Line Coverage: 29.49% (65638/222569)
Region Coverage: 28.96% (33777/116622)
Branch Coverage: 24.81% (17338/69890)
Coverage Report: http://coverage.selectdb-in.cc/coverage/c7dbe62da40b684f382d5e7b7f2b3f46ee01ed89_c7dbe62da40b684f382d5e7b7f2b3f46ee01ed89/report/index.html

@doris-robot
Copy link

TPC-H: Total hot run time: 49940 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit c7dbe62da40b684f382d5e7b7f2b3f46ee01ed89, data reload: false

------ Round 1 ----------------------------------
q1	17528	4311	4330	4311
q2	2021	150	140	140
q3	10278	1879	1899	1879
q4	10102	1231	1316	1231
q5	8441	3933	3943	3933
q6	224	121	123	121
q7	2007	1604	1585	1585
q8	9328	2739	2715	2715
q9	10952	11366	10659	10659
q10	8620	3519	3520	3519
q11	425	241	235	235
q12	475	290	300	290
q13	18350	3994	4039	3994
q14	342	324	319	319
q15	508	457	451	451
q16	696	589	599	589
q17	1137	943	979	943
q18	7341	6961	6764	6764
q19	1714	1548	1512	1512
q20	526	312	310	310
q21	4468	4042	4094	4042
q22	506	401	398	398
Total cold run time: 115989 ms
Total hot run time: 49940 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4302	4301	4302	4301
q2	317	221	215	215
q3	4162	4126	4150	4126
q4	2745	2746	2750	2746
q5	7307	7179	7194	7179
q6	234	119	118	118
q7	3263	2836	2838	2836
q8	4399	4482	4491	4482
q9	17083	16964	16964	16964
q10	4255	4282	4239	4239
q11	749	679	686	679
q12	1024	847	851	847
q13	4211	3757	3782	3757
q14	460	425	430	425
q15	499	465	452	452
q16	756	706	696	696
q17	3831	3845	3942	3845
q18	8910	8739	8795	8739
q19	1690	1709	1647	1647
q20	2402	2158	2129	2129
q21	8476	8569	8492	8492
q22	1029	939	915	915
Total cold run time: 82104 ms
Total hot run time: 79829 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 202965 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit c7dbe62da40b684f382d5e7b7f2b3f46ee01ed89, data reload: false

query1	917	389	380	380
query2	6516	2245	2265	2245
query3	6917	194	196	194
query4	21001	17925	17958	17925
query5	19731	6490	6510	6490
query6	298	218	233	218
query7	4160	315	298	298
query8	254	273	235	235
query9	3187	2756	2659	2659
query10	440	302	310	302
query11	11404	10664	10666	10664
query12	118	75	74	74
query13	5578	635	649	635
query14	17673	13474	13369	13369
query15	363	232	239	232
query16	6441	263	268	263
query17	1574	1442	873	873
query18	2275	407	398	398
query19	209	143	144	143
query20	79	74	79	74
query21	186	88	99	88
query22	5061	5054	4955	4955
query23	32646	31974	31962	31962
query24	6966	6534	6491	6491
query25	529	434	416	416
query26	526	158	158	158
query27	1921	293	295	293
query28	6106	2226	2201	2201
query29	2966	2810	2601	2601
query30	238	160	163	160
query31	888	709	765	709
query32	65	59	56	56
query33	387	262	254	254
query34	848	472	495	472
query35	1112	894	917	894
query36	1520	1381	1585	1381
query37	94	67	59	59
query38	3075	2920	2915	2915
query39	1368	1318	1315	1315
query40	210	97	93	93
query41	36	32	33	32
query42	86	89	87	87
query43	651	562	639	562
query44	1138	719	723	719
query45	243	224	230	224
query46	1221	950	971	950
query47	1906	1864	1809	1809
query48	984	686	665	665
query49	608	366	353	353
query50	873	629	627	627
query51	4756	4668	4600	4600
query52	77	85	78	78
query53	451	314	326	314
query54	2641	2476	2439	2439
query55	88	73	85	73
query56	225	214	205	205
query57	1152	1143	1097	1097
query58	201	206	206	206
query59	3475	3341	3320	3320
query60	208	193	217	193
query61	85	85	92	85
query62	869	437	458	437
query63	470	331	333	331
query64	2462	1506	1423	1423
query65	3592	3564	3571	3564
query66	818	370	363	363
query67	15628	15998	16978	15998
query68	9813	665	659	659
query69	562	331	363	331
query70	2070	1955	1934	1934
query71	408	292	311	292
query72	6505	3459	3425	3425
query73	733	320	317	317
query74	6330	5792	5862	5792
query75	5423	3712	3645	3645
query76	6234	1124	1166	1124
query77	1055	250	262	250
query78	13284	12051	11661	11661
query79	7199	645	636	636
query80	762	393	385	385
query81	491	232	227	227
query82	1289	96	92	92
query83	173	137	130	130
query84	251	69	69	69
query85	683	285	278	278
query86	397	410	384	384
query87	3212	3064	2994	2994
query88	4601	2341	2323	2323
query89	370	284	299	284
query90	1897	204	216	204
query91	149	114	115	114
query92	55	55	49	49
query93	5506	569	610	569
query94	660	208	209	208
query95	1093	1083	1067	1067
query96	627	326	327	326
query97	6540	6348	6259	6259
query98	188	174	160	160
query99	2837	879	890	879
Total cold run time: 312188 ms
Total hot run time: 202965 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.52 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit c7dbe62da40b684f382d5e7b7f2b3f46ee01ed89, data reload: false

query1	0.02	0.02	0.02
query2	0.06	0.02	0.02
query3	0.24	0.05	0.04
query4	1.82	0.06	0.06
query5	0.54	0.53	0.52
query6	1.23	0.61	0.62
query7	0.02	0.02	0.01
query8	0.04	0.02	0.02
query9	0.50	0.49	0.49
query10	0.54	0.53	0.54
query11	0.11	0.09	0.08
query12	0.11	0.08	0.09
query13	0.62	0.61	0.61
query14	0.79	0.79	0.78
query15	0.77	0.76	0.76
query16	0.37	0.36	0.35
query17	1.00	1.02	1.03
query18	0.23	0.26	0.25
query19	1.93	1.86	1.86
query20	0.02	0.01	0.01
query21	15.49	0.56	0.54
query22	1.87	2.00	1.09
query23	17.19	0.80	1.04
query24	7.25	2.92	1.39
query25	0.39	0.09	0.06
query26	1.01	0.16	0.16
query27	0.05	0.04	0.03
query28	3.79	0.78	0.73
query29	12.64	2.31	2.32
query30	0.57	0.52	0.52
query31	2.80	0.38	0.38
query32	3.42	0.51	0.49
query33	3.10	3.04	3.06
query34	15.25	4.80	4.78
query35	4.86	4.85	4.83
query36	1.07	1.01	1.03
query37	0.05	0.05	0.05
query38	0.04	0.02	0.03
query39	0.02	0.01	0.02
query40	0.16	0.13	0.14
query41	0.06	0.02	0.01
query42	0.02	0.01	0.01
query43	0.02	0.02	0.01
Total cold run time: 102.08 s
Total hot run time: 30.52 s

@doris-robot
Copy link

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit c7dbe62da40b684f382d5e7b7f2b3f46ee01ed89 with default session variables
Stream load json:         20 seconds loaded 2358488459 Bytes, about 112 MB/s
Stream load orc:          58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet:      31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select:       20.7 seconds inserted 10000000 Rows, about 483K ops/s

@morningman morningman merged commit ecc80dc into apache:branch-2.0 Mar 18, 2024
22 of 26 checks passed
wsjz added a commit to wsjz/incubator-doris that referenced this pull request May 28, 2024
@wsjz
Revert "[fix](glue)support access glue iceberg with credential list (a…
da9c8ff 
…pache#32381)"

This reverts commit ecc80dc.
mongo360 pushed a commit to mongo360/doris that referenced this pull request Aug 16, 2024
@wsjz
[fix](glue)support access glue iceberg with credential list (apache#3…
7f80dd0 
…2381)

from apache#30292
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
kind/docs Categorizes issue or PR as related to documentation. kind/test
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@wsjz @doris-robot @morningman