Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[improvement](ranger) minor optimization for ranger plugin #32976

Closed
wants to merge 2 commits into from

Conversation

xianjingfeng
Copy link
Member

Proposed changes

  1. Modify ranger.service.name to ranger.service.type
  2. Optimize RangerHiveAuditLogFlusher

Further comments

If this is a relatively large or complex change, kick off the discussion at [email protected] by explaining why you chose the solution you did and what alternatives you considered, etc...

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@xianjingfeng
Copy link
Member Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 37727 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit e02da1282a9dc06801d1b58f8742594a087e773b, data reload: false

------ Round 1 ----------------------------------
q1	17611	4286	4130	4130
q2	2116	160	171	160
q3	10563	1177	1202	1177
q4	10231	784	793	784
q5	7498	2958	2904	2904
q6	203	128	125	125
q7	1049	573	588	573
q8	9354	2040	2002	2002
q9	6823	6346	6299	6299
q10	8423	3551	3533	3533
q11	434	234	219	219
q12	371	199	197	197
q13	17808	2888	2838	2838
q14	251	204	214	204
q15	505	477	465	465
q16	481	373	361	361
q17	964	535	559	535
q18	7345	6677	6357	6357
q19	5098	1428	1495	1428
q20	562	264	250	250
q21	3560	2898	2888	2888
q22	345	315	298	298
Total cold run time: 111595 ms
Total hot run time: 37727 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4221	4099	4112	4099
q2	330	232	234	232
q3	2963	2877	2830	2830
q4	1869	1597	1533	1533
q5	5214	5230	5291	5230
q6	195	116	117	116
q7	2176	1811	1817	1811
q8	3168	3302	3278	3278
q9	8467	8425	8466	8425
q10	3814	3803	3815	3803
q11	562	460	454	454
q12	738	605	567	567
q13	17721	3002	3047	3002
q14	298	259	261	259
q15	519	469	468	468
q16	494	457	430	430
q17	1753	1561	1531	1531
q18	7962	7721	7471	7471
q19	1673	1564	1546	1546
q20	1921	1797	1798	1797
q21	4886	4844	4813	4813
q22	518	485	458	458
Total cold run time: 71462 ms
Total hot run time: 54153 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 181200 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit e02da1282a9dc06801d1b58f8742594a087e773b, data reload: false

query1	754	363	354	354
query2	6526	1937	1874	1874
query3	6444	214	206	206
query4	31177	21319	21290	21290
query5	4100	400	400	400
query6	258	178	175	175
query7	4439	283	290	283
query8	230	185	171	171
query9	9164	2308	2301	2301
query10	540	248	263	248
query11	17191	14236	14181	14181
query12	139	88	85	85
query13	1537	416	417	416
query14	9994	7785	7618	7618
query15	268	185	199	185
query16	8185	265	263	263
query17	1882	579	543	543
query18	2103	299	278	278
query19	323	156	153	153
query20	94	85	92	85
query21	205	123	125	123
query22	5084	4833	4852	4833
query23	33538	32721	32838	32721
query24	11587	2865	2852	2852
query25	652	389	384	384
query26	1768	157	157	157
query27	2939	309	326	309
query28	7588	1897	1843	1843
query29	1038	644	639	639
query30	304	153	150	150
query31	1008	740	720	720
query32	93	59	59	59
query33	777	264	261	261
query34	1028	482	499	482
query35	878	699	687	687
query36	1017	857	901	857
query37	276	64	67	64
query38	3542	3456	3425	3425
query39	1617	1564	1550	1550
query40	289	106	106	106
query41	50	48	49	48
query42	103	97	98	97
query43	486	466	460	460
query44	1329	707	724	707
query45	289	255	267	255
query46	1113	709	717	709
query47	1957	1827	1814	1814
query48	445	346	369	346
query49	1166	348	345	345
query50	767	374	374	374
query51	6645	6671	6568	6568
query52	118	89	90	89
query53	349	273	276	273
query54	316	260	248	248
query55	87	82	86	82
query56	259	242	230	230
query57	1205	1135	1095	1095
query58	243	220	210	210
query59	2793	2697	2696	2696
query60	258	252	249	249
query61	118	117	113	113
query62	620	451	476	451
query63	312	274	274	274
query64	6774	4156	3935	3935
query65	3123	3059	3038	3038
query66	1382	362	359	359
query67	15447	14900	15098	14900
query68	8666	520	526	520
query69	619	385	375	375
query70	1206	1192	1143	1143
query71	520	271	257	257
query72	6283	2523	2378	2378
query73	777	318	313	313
query74	8179	6443	6326	6326
query75	4061	2159	2217	2159
query76	5375	798	861	798
query77	663	262	256	256
query78	11141	10201	10238	10201
query79	10022	539	520	520
query80	1499	377	366	366
query81	505	214	217	214
query82	725	86	83	83
query83	206	141	141	141
query84	282	79	79	79
query85	1349	310	297	297
query86	420	270	304	270
query87	3735	3474	3529	3474
query88	4814	2299	2285	2285
query89	506	365	371	365
query90	2047	180	177	177
query91	175	135	133	133
query92	57	47	47	47
query93	6708	496	481	481
query94	1200	179	175	175
query95	412	305	304	304
query96	608	277	264	264
query97	2680	2488	2523	2488
query98	231	214	203	203
query99	1239	826	812	812
Total cold run time: 315240 ms
Total hot run time: 181200 ms

@doris-robot
Copy link

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit e02da1282a9dc06801d1b58f8742594a087e773b with default session variables
Stream load json:         18 seconds loaded 2358488459 Bytes, about 124 MB/s
Stream load orc:          58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet:      32 seconds loaded 861443392 Bytes, about 25 MB/s
Insert into select:       13.5 seconds inserted 10000000 Rows, about 740K ops/s

@xianjingfeng
Copy link
Member Author

@morningman @Yulei-Yang PTAL

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants