[improvement](jdbc catalog) Optimize connection pool parameter settings #30588

zy-kkk · 2024-01-30T10:03:09Z

Proposed changes

Issue Number: close #xxx

This PR makes the following changes to the connection pool of JDBC Catalog

Set the maximum connection survival time, the default is 30 minutes

Moreover, one-half of the maximum survival time is the recyclable time,
One-tenth is the check interval for recycling connections

Keepalive only takes effect on the connection pool on BE, and will be activated based on one-fifth of the maximum survival time.
The maximum number of existing connections is changed from 100 to 10
Add the connection cache recycling thread on BE, and add a parameter to control the recycling time, the default is 28800 (8 hours)
Add CatalogID to the key of the connection pool cache to achieve better isolation, requires refresh catalog to take effect
Upgrade druid connection pool to version 1.2.20
Added JdbcResource's setting of default parameters when upgrading the FE version to avoid errors due to unset parameters.

Further comments

If this is a relatively large or complex change, kick off the discussion at [email protected] by explaining why you chose the solution you did and what alternatives you considered, etc...

zy-kkk · 2024-01-30T10:03:29Z

run buildall

github-actions · 2024-01-30T10:12:42Z

clang-tidy review says "All clean, LGTM! 👍"

doris-robot · 2024-01-30T11:19:48Z

TPC-H: Total hot run time: 36790 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 4d8c1b88aeeb6472f11ec258ce5754601c7b8fce, data reload: false

------ Round 1 ----------------------------------
q1	17774	4688	4373	4373
q2	2038	136	137	136
q3	10684	891	895	891
q4	4662	800	719	719
q5	7675	2845	2770	2770
q6	179	117	117	117
q7	1136	720	718	718
q8	9283	1991	2004	1991
q9	7629	6346	6326	6326
q10	8102	2430	2428	2428
q11	414	209	216	209
q12	738	273	275	273
q13	18003	3305	3291	3291
q14	285	254	257	254
q15	532	499	485	485
q16	467	412	402	402
q17	947	555	500	500
q18	6858	5988	5897	5897
q19	1589	1377	1351	1351
q20	599	355	338	338
q21	6906	3045	3030	3030
q22	791	300	291	291
Total cold run time: 107291 ms
Total hot run time: 36790 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4440	4454	4364	4364
q2	330	234	230	230
q3	2965	2844	2787	2787
q4	1861	1680	1618	1618
q5	5213	5278	5219	5219
q6	188	112	114	112
q7	2152	1772	1789	1772
q8	3149	3256	3268	3256
q9	8335	8324	8268	8268
q10	5871	3573	3595	3573
q11	530	455	445	445
q12	745	589	544	544
q13	12772	3068	3082	3068
q14	272	248	262	248
q15	538	497	490	490
q16	510	479	462	462
q17	1882	1664	1721	1664
q18	8038	7702	7628	7628
q19	9183	1555	1559	1555
q20	2120	1929	1896	1896
q21	4653	4508	4473	4473
q22	523	449	450	449
Total cold run time: 76270 ms
Total hot run time: 54121 ms

doris-robot · 2024-01-30T11:30:39Z

TPC-DS: Total hot run time: 172819 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 4d8c1b88aeeb6472f11ec258ce5754601c7b8fce, data reload: false

query1	945	333	323	323
query2	6577	1866	1851	1851
query3	6710	202	198	198
query4	27751	22107	22131	22107
query5	4455	410	409	409
query6	245	151	147	147
query7	4609	257	259	257
query8	260	179	174	174
query9	9059	2314	2283	2283
query10	416	205	203	203
query11	16766	15340	15403	15340
query12	130	64	66	64
query13	1685	370	362	362
query14	9197	6481	6414	6414
query15	237	182	182	182
query16	6546	251	251	251
query17	1901	465	459	459
query18	1950	249	244	244
query19	179	129	124	124
query20	70	72	64	64
query21	200	128	125	125
query22	4971	4949	4750	4750
query23	31118	30323	30348	30323
query24	7392	2681	2725	2681
query25	541	304	308	304
query26	822	131	134	131
query27	2776	283	283	283
query28	5547	1859	1835	1835
query29	942	645	598	598
query30	280	138	139	138
query31	936	708	717	708
query32	90	49	47	47
query33	447	202	199	199
query34	826	454	461	454
query35	866	759	765	759
query36	1279	1175	1161	1161
query37	91	56	56	56
query38	3367	3193	3176	3176
query39	1309	1235	1256	1235
query40	197	84	79	79
query41	35	33	33	33
query42	82	87	81	81
query43	524	503	489	489
query44	1026	679	697	679
query45	194	172	175	172
query46	1034	643	642	642
query47	1542	1519	1534	1519
query48	383	302	290	290
query49	1109	277	276	276
query50	684	304	303	303
query51	5284	5140	5142	5140
query52	85	79	70	70
query53	321	259	254	254
query54	231	179	190	179
query55	76	79	72	72
query56	167	165	157	157
query57	961	922	918	918
query58	188	150	155	150
query59	2458	2192	2319	2192
query60	193	174	172	172
query61	80	80	89	80
query62	584	351	357	351
query63	285	263	267	263
query64	4020	1740	1681	1681
query65	3304	3202	3230	3202
query66	1137	315	307	307
query67	14599	14465	14519	14465
query68	4019	468	482	468
query69	448	291	304	291
query70	1489	1516	1521	1516
query71	274	205	206	205
query72	3761	2807	2809	2807
query73	676	312	313	312
query74	6600	6161	6311	6161
query75	2772	2321	2313	2313
query76	2977	980	973	973
query77	337	224	222	222
query78	9085	8635	8484	8484
query79	2722	492	488	488
query80	2066	311	301	301
query81	504	193	195	193
query82	757	81	80	80
query83	235	114	116	114
query84	299	72	75	72
query85	1968	334	308	308
query86	510	371	372	371
query87	3403	3303	3304	3303
query88	3946	2202	2199	2199
query89	422	344	366	344
query90	1917	183	182	182
query91	146	115	112	112
query92	57	43	42	42
query93	3228	422	433	422
query94	1237	155	158	155
query95	495	455	462	455
query96	608	317	311	311
query97	4251	4107	4115	4107
query98	211	188	185	185
query99	1087	690	688	688
Total cold run time: 266778 ms
Total hot run time: 172819 ms

doris-robot · 2024-01-30T11:32:53Z

TeamCity be ut coverage result:
Function Coverage: 36.18% (8625/23836)
Line Coverage: 28.25% (70539/249692)
Region Coverage: 27.25% (36395/133581)
Branch Coverage: 24.04% (18644/77562)
Coverage Report: http://coverage.selectdb-in.cc/coverage/4d8c1b88aeeb6472f11ec258ce5754601c7b8fce_4d8c1b88aeeb6472f11ec258ce5754601c7b8fce/report/index.html

doris-robot · 2024-01-30T11:36:05Z

ClickBench: Total hot run time: 30.36 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 4d8c1b88aeeb6472f11ec258ce5754601c7b8fce, data reload: false

query1	0.04	0.03	0.03
query2	0.06	0.02	0.02
query3	0.23	0.06	0.06
query4	1.68	0.10	0.10
query5	0.52	0.52	0.51
query6	1.20	0.62	0.63
query7	0.01	0.01	0.02
query8	0.03	0.02	0.02
query9	0.55	0.50	0.50
query10	0.56	0.56	0.55
query11	0.12	0.09	0.09
query12	0.12	0.09	0.10
query13	0.60	0.61	0.61
query14	0.78	0.79	0.81
query15	0.80	0.78	0.76
query16	0.40	0.38	0.36
query17	1.01	1.02	1.01
query18	0.22	0.28	0.23
query19	1.93	1.79	1.78
query20	0.01	0.02	0.02
query21	15.40	0.62	0.58
query22	2.82	3.16	1.89
query23	17.66	0.81	0.70
query24	2.52	0.97	0.21
query25	0.28	0.15	0.13
query26	0.51	0.13	0.14
query27	0.05	0.05	0.04
query28	11.99	0.83	0.88
query29	12.55	3.26	3.25
query30	0.63	0.56	0.54
query31	2.79	0.35	0.35
query32	3.37	0.47	0.48
query33	3.24	3.21	3.25
query34	15.79	4.33	4.27
query35	4.30	4.28	4.28
query36	1.09	1.06	1.05
query37	0.06	0.05	0.06
query38	0.04	0.03	0.03
query39	0.02	0.02	0.02
query40	0.16	0.13	0.14
query41	0.07	0.02	0.01
query42	0.03	0.01	0.01
query43	0.02	0.02	0.02
Total cold run time: 106.26 s
Total hot run time: 30.36 s

doris-robot · 2024-01-30T11:39:09Z

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit 4d8c1b88aeeb6472f11ec258ce5754601c7b8fce with default session variables
Stream load json:         18 seconds loaded 2358488459 Bytes, about 124 MB/s
Stream load orc:          58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet:      32 seconds loaded 861443392 Bytes, about 25 MB/s
Insert into select:       14.4 seconds inserted 10000000 Rows, about 694K ops/s

morningman · 2024-01-30T13:21:13Z

fe/fe-core/src/main/java/org/apache/doris/datasource/jdbc/JdbcExternalCatalog.java

+        if (connectionPoolMaxWaitTime > 30000) {
+            throw new DdlException("connection_pool_max_wait_time must be less than or equal to 30000");
+        }
+        if (connectionPoolMaxLifeTime < 150000) {


We need to define this value somewhere, there are too many magic number in code

zy-kkk · 2024-01-31T08:09:22Z

run buildall

github-actions · 2024-01-31T08:17:34Z

clang-tidy review says "All clean, LGTM! 👍"

doris-robot · 2024-01-31T09:10:21Z

TPC-H: Total hot run time: 36940 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 97b3996076b1099cbe4322e2ee8ce26cc4659473, data reload: false

------ Round 1 ----------------------------------
q1	17608	4551	4471	4471
q2	2041	140	130	130
q3	10614	906	883	883
q4	4664	815	715	715
q5	7839	2785	2736	2736
q6	184	119	119	119
q7	1115	714	727	714
q8	9269	1995	2020	1995
q9	7263	6363	6360	6360
q10	8076	2451	2427	2427
q11	413	207	200	200
q12	736	276	289	276
q13	17998	3315	3292	3292
q14	280	251	248	248
q15	526	486	491	486
q16	460	418	405	405
q17	958	547	507	507
q18	6684	5987	5994	5987
q19	1570	1345	1391	1345
q20	596	312	308	308
q21	6512	3151	3035	3035
q22	802	301	302	301
Total cold run time: 106208 ms
Total hot run time: 36940 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4514	4357	4437	4357
q2	331	230	230	230
q3	2972	2850	2792	2792
q4	1858	1635	1630	1630
q5	5183	5222	5231	5222
q6	189	111	113	111
q7	2146	1728	1723	1723
q8	3110	3252	3269	3252
q9	8373	8299	8317	8299
q10	5840	3517	3504	3504
q11	531	460	469	460
q12	729	570	568	568
q13	5233	3060	3081	3060
q14	275	253	278	253
q15	533	501	484	484
q16	502	462	493	462
q17	1849	1693	1741	1693
q18	7977	7753	7589	7589
q19	6967	1541	1514	1514
q20	2126	1918	1893	1893
q21	4786	4754	4437	4437
q22	560	494	480	480
Total cold run time: 66584 ms
Total hot run time: 54013 ms

doris-robot · 2024-01-31T09:21:32Z

TPC-DS: Total hot run time: 174711 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 97b3996076b1099cbe4322e2ee8ce26cc4659473, data reload: false

query1	926	334	326	326
query2	6550	1820	1932	1820
query3	6700	201	200	200
query4	32295	21997	22036	21997
query5	4449	416	389	389
query6	249	149	164	149
query7	4609	253	255	253
query8	246	176	174	174
query9	9133	2277	2258	2258
query10	411	208	202	202
query11	18138	15327	15217	15217
query12	120	69	73	69
query13	1637	370	365	365
query14	9278	7251	7541	7251
query15	241	177	181	177
query16	7343	263	247	247
query17	1829	493	469	469
query18	1918	254	250	250
query19	177	135	132	132
query20	75	70	66	66
query21	194	125	122	122
query22	5119	4982	4822	4822
query23	31286	30173	30281	30173
query24	10687	2738	2680	2680
query25	546	318	314	314
query26	1054	138	138	138
query27	2470	278	278	278
query28	7070	1853	1841	1841
query29	952	622	596	596
query30	277	130	134	130
query31	940	696	712	696
query32	89	49	46	46
query33	691	210	214	210
query34	1094	451	456	451
query35	872	757	741	741
query36	1243	1146	1170	1146
query37	91	55	54	54
query38	3214	3210	3171	3171
query39	1312	1250	1243	1243
query40	202	81	82	81
query41	37	34	34	34
query42	88	82	85	82
query43	515	457	467	457
query44	1156	679	693	679
query45	195	177	173	173
query46	1035	627	625	625
query47	1558	1423	1488	1423
query48	406	308	304	304
query49	1147	284	280	280
query50	690	300	301	300
query51	5217	5249	5158	5158
query52	89	74	81	74
query53	332	260	261	260
query54	274	188	189	188
query55	80	77	81	77
query56	182	164	167	164
query57	959	876	897	876
query58	185	146	155	146
query59	2403	2249	2221	2221
query60	199	170	185	170
query61	85	91	87	87
query62	630	360	344	344
query63	277	260	267	260
query64	5455	3683	3443	3443
query65	3228	3217	3216	3216
query66	941	310	313	310
query67	14485	14190	14496	14190
query68	4120	494	485	485
query69	430	307	303	303
query70	1575	1447	1470	1447
query71	284	212	202	202
query72	5442	3095	2845	2845
query73	694	311	317	311
query74	6683	6188	6217	6188
query75	3000	2312	2346	2312
query76	2519	958	928	928
query77	364	227	223	223
query78	9098	8782	8626	8626
query79	2736	480	492	480
query80	2118	315	315	315
query81	539	192	198	192
query82	781	77	84	77
query83	237	119	115	115
query84	296	76	78	76
query85	2318	331	333	331
query86	506	376	390	376
query87	3442	3267	3251	3251
query88	3887	2178	2164	2164
query89	436	346	353	346
query90	1934	183	180	180
query91	148	116	119	116
query92	63	41	43	41
query93	2575	420	436	420
query94	1370	156	158	156
query95	504	456	446	446
query96	627	321	322	321
query97	4267	4123	4130	4123
query98	216	185	180	180
query99	1216	686	650	650
Total cold run time: 282000 ms
Total hot run time: 174711 ms

doris-robot · 2024-01-31T09:26:58Z

ClickBench: Total hot run time: 30.97 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 97b3996076b1099cbe4322e2ee8ce26cc4659473, data reload: false

query1	0.04	0.03	0.03
query2	0.05	0.02	0.02
query3	0.23	0.06	0.06
query4	1.69	0.10	0.10
query5	0.52	0.52	0.52
query6	1.25	0.64	0.64
query7	0.02	0.01	0.02
query8	0.04	0.03	0.03
query9	0.57	0.50	0.50
query10	0.55	0.55	0.56
query11	0.11	0.08	0.08
query12	0.10	0.08	0.08
query13	0.60	0.61	0.60
query14	0.77	0.79	0.79
query15	0.78	0.79	0.77
query16	0.39	0.39	0.36
query17	1.01	1.00	1.01
query18	0.22	0.27	0.25
query19	1.89	1.79	1.75
query20	0.01	0.01	0.01
query21	15.44	0.55	0.55
query22	2.48	2.22	2.00
query23	17.37	0.87	0.82
query24	2.44	0.74	0.91
query25	0.30	0.36	0.12
query26	0.54	0.14	0.14
query27	0.07	0.04	0.04
query28	12.37	0.83	0.82
query29	12.60	3.25	3.18
query30	0.59	0.55	0.52
query31	2.83	0.34	0.36
query32	3.37	0.48	0.47
query33	3.23	3.22	3.18
query34	15.80	4.30	4.31
query35	4.35	4.38	4.30
query36	1.11	1.06	1.06
query37	0.06	0.04	0.04
query38	0.03	0.02	0.03
query39	0.02	0.01	0.02
query40	0.16	0.13	0.13
query41	0.07	0.02	0.01
query42	0.02	0.02	0.01
query43	0.02	0.02	0.02
Total cold run time: 106.11 s
Total hot run time: 30.97 s

doris-robot · 2024-01-31T09:30:01Z

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit 97b3996076b1099cbe4322e2ee8ce26cc4659473 with default session variables
Stream load json:         18 seconds loaded 2358488459 Bytes, about 124 MB/s
Stream load orc:          58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet:      32 seconds loaded 861443392 Bytes, about 25 MB/s
Insert into select:       13.6 seconds inserted 10000000 Rows, about 735K ops/s

doris-robot · 2024-01-31T09:40:04Z

TeamCity be ut coverage result:
Function Coverage: 36.18% (8626/23839)
Line Coverage: 28.25% (70549/249749)
Region Coverage: 27.25% (36408/133602)
Branch Coverage: 24.04% (18651/77572)
Coverage Report: http://coverage.selectdb-in.cc/coverage/97b3996076b1099cbe4322e2ee8ce26cc4659473_97b3996076b1099cbe4322e2ee8ce26cc4659473/report/index.html

morningman

LGTM

github-actions · 2024-01-31T15:40:19Z

PR approved by at least one committer and no changes requested.

github-actions · 2024-01-31T15:40:21Z

PR approved by anyone and no changes requested.

morningman

Please add some ut or test case to test these parameters

github-actions · 2024-01-31T16:59:22Z

clang-tidy review says "All clean, LGTM! 👍"

github-actions · 2024-02-01T08:41:08Z

clang-tidy review says "All clean, LGTM! 👍"

github-actions · 2024-02-01T08:54:17Z

clang-tidy review says "All clean, LGTM! 👍"

doris-robot · 2024-02-01T11:40:49Z

TPC-H: Total hot run time: 37003 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 005c627e3f08506332f647d4749f11cbe5f15364, data reload: false

------ Round 1 ----------------------------------
q1	17813	5226	4434	4434
q2	2057	139	138	138
q3	10691	937	949	937
q4	4667	785	743	743
q5	7694	2834	2884	2834
q6	184	120	118	118
q7	1148	721	715	715
q8	9308	2030	2040	2030
q9	7281	6390	6335	6335
q10	8132	2455	2421	2421
q11	408	201	196	196
q12	737	287	282	282
q13	18035	3281	3314	3281
q14	279	261	247	247
q15	527	497	500	497
q16	505	413	424	413
q17	945	536	555	536
q18	6740	5977	5896	5896
q19	1580	1403	1321	1321
q20	635	316	331	316
q21	7089	3014	3129	3014
q22	796	309	299	299
Total cold run time: 107251 ms
Total hot run time: 37003 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4349	4462	4357	4357
q2	339	240	231	231
q3	2998	2935	2845	2845
q4	1855	1634	1635	1634
q5	5157	5282	5266	5266
q6	189	111	112	111
q7	2176	1822	1729	1729
q8	3112	3246	3297	3246
q9	8439	8325	8269	8269
q10	5884	3601	3594	3594
q11	537	466	459	459
q12	749	604	574	574
q13	4371	3084	3075	3075
q14	282	237	274	237
q15	537	495	492	492
q16	504	474	484	474
q17	1808	1679	1712	1679
q18	8022	7607	7540	7540
q19	10453	1584	1528	1528
q20	2129	1901	1907	1901
q21	4825	4650	4727	4650
q22	555	476	471	471
Total cold run time: 69270 ms
Total hot run time: 54362 ms

doris-robot · 2024-02-01T11:52:07Z

TPC-DS: Total hot run time: 175247 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 005c627e3f08506332f647d4749f11cbe5f15364, data reload: false

query1	932	339	328	328
query2	6568	2134	1881	1881
query3	6695	214	202	202
query4	31940	22225	22120	22120
query5	4450	355	412	355
query6	252	160	155	155
query7	4604	266	259	259
query8	240	173	199	173
query9	9024	2231	2236	2231
query10	415	226	198	198
query11	18154	15312	15327	15312
query12	120	68	64	64
query13	1634	368	359	359
query14	9741	7429	7334	7334
query15	214	179	177	177
query16	7246	272	256	256
query17	1835	488	469	469
query18	1927	256	256	256
query19	187	132	131	131
query20	74	70	73	70
query21	195	133	139	133
query22	4982	4780	4738	4738
query23	31342	30333	30194	30194
query24	10997	2756	2712	2712
query25	543	315	307	307
query26	1134	138	136	136
query27	2629	283	290	283
query28	6870	1852	1814	1814
query29	979	639	610	610
query30	278	133	133	133
query31	933	696	719	696
query32	96	55	57	55
query33	688	212	210	210
query34	1054	458	474	458
query35	862	749	756	749
query36	1350	1260	1159	1159
query37	91	61	57	57
query38	3272	3201	3179	3179
query39	1289	1264	1251	1251
query40	199	87	84	84
query41	41	41	35	35
query42	88	81	84	81
query43	501	451	468	451
query44	1082	666	704	666
query45	190	180	173	173
query46	1061	637	610	610
query47	1597	1501	1502	1501
query48	375	304	309	304
query49	1144	283	282	282
query50	692	300	307	300
query51	5256	5158	5186	5158
query52	96	74	79	74
query53	317	266	265	265
query54	260	179	185	179
query55	86	77	75	75
query56	177	163	162	162
query57	968	928	912	912
query58	179	155	155	155
query59	2385	2237	2417	2237
query60	195	184	175	175
query61	82	81	81	81
query62	656	351	362	351
query63	276	267	258	258
query64	5575	3610	3456	3456
query65	3246	3230	3227	3227
query66	1056	315	314	314
query67	14299	14303	14102	14102
query68	4304	488	496	488
query69	431	309	298	298
query70	1583	1545	1531	1531
query71	301	209	212	209
query72	5575	3133	2838	2838
query73	676	320	327	320
query74	6681	6257	6205	6205
query75	2919	2270	2297	2270
query76	2535	984	986	984
query77	366	237	232	232
query78	9120	8918	8584	8584
query79	3265	498	497	497
query80	2157	324	303	303
query81	539	195	195	195
query82	1224	89	75	75
query83	233	119	118	118
query84	285	72	87	72
query85	2361	334	328	328
query86	526	428	382	382
query87	3458	3258	3273	3258
query88	4537	2226	2239	2226
query89	427	364	334	334
query90	1947	188	183	183
query91	145	118	129	118
query92	61	42	42	42
query93	5271	444	403	403
query94	1374	161	157	157
query95	489	448	454	448
query96	644	328	324	324
query97	4204	4092	4112	4092
query98	218	185	187	185
query99	1085	697	676	676
Total cold run time: 286856 ms
Total hot run time: 175247 ms

doris-robot · 2024-02-01T11:57:33Z

ClickBench: Total hot run time: 30.61 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 005c627e3f08506332f647d4749f11cbe5f15364, data reload: false

query1	0.04	0.03	0.03
query2	0.06	0.02	0.02
query3	0.23	0.05	0.06
query4	1.68	0.10	0.10
query5	0.52	0.52	0.52
query6	1.19	0.62	0.64
query7	0.02	0.01	0.02
query8	0.03	0.02	0.02
query9	0.57	0.50	0.48
query10	0.55	0.56	0.57
query11	0.11	0.08	0.08
query12	0.11	0.09	0.10
query13	0.60	0.60	0.61
query14	0.77	0.80	0.80
query15	0.80	0.77	0.76
query16	0.38	0.39	0.40
query17	1.00	0.99	1.01
query18	0.22	0.28	0.24
query19	1.92	1.79	1.75
query20	0.01	0.01	0.01
query21	15.41	0.56	0.55
query22	2.77	2.23	1.63
query23	17.14	0.79	0.81
query24	2.87	1.45	0.77
query25	0.38	0.16	0.12
query26	0.56	0.14	0.14
query27	0.05	0.05	0.05
query28	11.28	0.83	0.83
query29	12.57	3.18	3.20
query30	0.64	0.55	0.55
query31	2.79	0.35	0.35
query32	3.36	0.48	0.48
query33	3.21	3.22	3.27
query34	16.10	4.26	4.31
query35	4.36	4.28	4.30
query36	1.09	1.06	1.04
query37	0.07	0.05	0.05
query38	0.04	0.02	0.03
query39	0.02	0.02	0.01
query40	0.15	0.13	0.13
query41	0.07	0.02	0.01
query42	0.03	0.01	0.02
query43	0.03	0.02	0.02
Total cold run time: 105.8 s
Total hot run time: 30.61 s

doris-robot · 2024-02-01T12:00:36Z

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit 005c627e3f08506332f647d4749f11cbe5f15364 with default session variables
Stream load json:         18 seconds loaded 2358488459 Bytes, about 124 MB/s
Stream load orc:          58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet:      31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select:       13.4 seconds inserted 10000000 Rows, about 746K ops/s

doris-robot · 2024-02-01T12:05:08Z

TeamCity be ut coverage result:
Function Coverage: 36.18% (8627/23847)
Line Coverage: 28.23% (70496/249731)
Region Coverage: 27.24% (36394/133612)
Branch Coverage: 24.02% (18636/77584)
Coverage Report: http://coverage.selectdb-in.cc/coverage/005c627e3f08506332f647d4749f11cbe5f15364_005c627e3f08506332f647d4749f11cbe5f15364/report/index.html

fe/fe-core/src/main/java/org/apache/doris/catalog/JdbcTable.java

fe/fe-core/src/test/java/org/apache/doris/catalog/JdbcResourceTest.java

zy-kkk · 2024-02-02T02:42:49Z

run buildall

github-actions · 2024-02-02T02:51:48Z

clang-tidy review says "All clean, LGTM! 👍"

doris-robot · 2024-02-02T03:43:34Z

TPC-H: Total hot run time: 37521 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 2d4449a945f06107e5fc837afded7a8a1b858140, data reload: false

------ Round 1 ----------------------------------
q1	17741	4881	4551	4551
q2	2446	145	151	145
q3	11507	957	942	942
q4	4713	796	755	755
q5	8357	2936	2866	2866
q6	191	126	123	123
q7	1162	746	736	736
q8	10081	2064	2079	2064
q9	7433	6365	6315	6315
q10	8117	2439	2453	2439
q11	405	212	206	206
q12	778	270	282	270
q13	18022	3327	3312	3312
q14	278	269	252	252
q15	528	494	493	493
q16	476	410	412	410
q17	942	548	503	503
q18	6609	6089	6173	6089
q19	1591	1361	1377	1361
q20	636	342	320	320
q21	6551	3070	3092	3070
q22	794	311	299	299
Total cold run time: 109358 ms
Total hot run time: 37521 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4563	4436	4407	4407
q2	339	237	236	236
q3	3001	2868	2860	2860
q4	1843	1688	1679	1679
q5	5206	5199	5248	5199
q6	190	114	116	114
q7	2180	1737	1803	1737
q8	3146	3259	3253	3253
q9	8361	8347	8271	8271
q10	5979	3574	3590	3574
q11	551	470	469	469
q12	729	580	574	574
q13	3912	3186	3057	3057
q14	276	249	255	249
q15	526	483	485	483
q16	515	486	474	474
q17	1845	1696	1740	1696
q18	7945	7725	7666	7666
q19	7596	1532	1527	1527
q20	2149	1930	1907	1907
q21	4777	4786	4685	4685
q22	554	487	485	485
Total cold run time: 66183 ms
Total hot run time: 54602 ms

doris-robot · 2024-02-02T03:54:52Z

TPC-DS: Total hot run time: 174929 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 2d4449a945f06107e5fc837afded7a8a1b858140, data reload: false

query1	932	339	333	333
query2	6542	2107	1924	1924
query3	6722	210	204	204
query4	32021	22354	22082	22082
query5	4456	411	349	349
query6	253	151	161	151
query7	4603	260	260	260
query8	257	174	178	174
query9	8963	2232	2213	2213
query10	407	196	202	196
query11	18294	15709	15438	15438
query12	123	66	62	62
query13	1620	367	377	367
query14	9914	7541	7291	7291
query15	240	180	186	180
query16	7464	266	251	251
query17	1856	499	463	463
query18	1939	256	251	251
query19	350	136	137	136
query20	75	73	67	67
query21	193	136	132	132
query22	4808	4611	4620	4611
query23	31303	30432	30458	30432
query24	10813	2765	2797	2765
query25	520	319	300	300
query26	1310	134	149	134
query27	2332	288	282	282
query28	7055	1817	1817	1817
query29	1087	642	620	620
query30	285	132	139	132
query31	915	701	719	701
query32	92	54	58	54
query33	715	214	204	204
query34	1049	469	470	469
query35	878	786	772	772
query36	1081	921	952	921
query37	94	55	55	55
query38	3240	3160	3126	3126
query39	1308	1262	1248	1248
query40	324	89	83	83
query41	39	38	34	34
query42	85	80	80	80
query43	526	494	478	478
query44	1100	675	689	675
query45	199	180	177	177
query46	1053	647	625	625
query47	1556	1476	1523	1476
query48	403	303	309	303
query49	1211	278	277	277
query50	692	311	313	311
query51	5213	5110	5205	5110
query52	98	75	77	75
query53	346	265	254	254
query54	268	172	184	172
query55	78	81	80	80
query56	183	166	151	151
query57	1009	876	890	876
query58	194	150	157	150
query59	2627	2427	2210	2210
query60	204	175	178	175
query61	82	85	87	85
query62	631	350	336	336
query63	284	260	258	258
query64	6404	3773	3651	3651
query65	3235	3225	3219	3219
query66	1153	334	321	321
query67	14309	14367	14015	14015
query68	4380	500	502	500
query69	433	305	315	305
query70	1288	1251	1203	1203
query71	297	216	212	212
query72	5681	3114	2828	2828
query73	691	319	325	319
query74	6720	6198	6185	6185
query75	2960	2312	2357	2312
query76	2798	941	958	941
query77	379	229	228	228
query78	9079	8750	8665	8665
query79	3531	495	489	489
query80	2323	323	312	312
query81	530	197	200	197
query82	1556	85	81	81
query83	373	123	118	118
query84	277	78	81	78
query85	1877	332	313	313
query86	469	270	284	270
query87	3458	3272	3285	3272
query88	4375	2262	2262	2262
query89	436	354	345	345
query90	1939	189	190	189
query91	151	113	121	113
query92	58	44	45	44
query93	4393	456	438	438
query94	1341	160	155	155
query95	501	452	453	452
query96	640	335	321	321
query97	4215	4114	4103	4103
query98	216	198	186	186
query99	1170	669	672	669
Total cold run time: 288150 ms
Total hot run time: 174929 ms

doris-robot · 2024-02-02T04:00:19Z

ClickBench: Total hot run time: 31.71 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 2d4449a945f06107e5fc837afded7a8a1b858140, data reload: false

query1	0.03	0.04	0.03
query2	0.06	0.03	0.02
query3	0.23	0.06	0.06
query4	1.68	0.10	0.10
query5	0.53	0.51	0.52
query6	1.19	0.63	0.64
query7	0.02	0.02	0.01
query8	0.04	0.03	0.02
query9	0.54	0.50	0.50
query10	0.55	0.58	0.56
query11	0.12	0.08	0.08
query12	0.11	0.09	0.10
query13	0.61	0.60	0.60
query14	0.77	0.82	0.81
query15	0.80	0.78	0.80
query16	0.37	0.38	0.40
query17	1.00	1.03	1.01
query18	0.23	0.28	0.24
query19	1.86	1.77	1.81
query20	0.01	0.01	0.02
query21	15.40	0.59	0.57
query22	2.59	1.95	2.20
query23	17.22	0.78	0.80
query24	2.52	1.36	1.31
query25	0.30	0.21	0.28
query26	0.56	0.14	0.13
query27	0.06	0.04	0.05
query28	11.12	0.84	0.83
query29	12.50	3.28	3.28
query30	0.61	0.55	0.49
query31	2.79	0.35	0.35
query32	3.36	0.48	0.48
query33	3.22	3.26	3.25
query34	15.81	4.30	4.27
query35	4.36	4.30	4.27
query36	1.11	1.06	1.05
query37	0.06	0.04	0.04
query38	0.03	0.03	0.04
query39	0.02	0.01	0.02
query40	0.17	0.12	0.13
query41	0.07	0.02	0.01
query42	0.03	0.01	0.01
query43	0.02	0.02	0.02
Total cold run time: 104.68 s
Total hot run time: 31.71 s

doris-robot · 2024-02-02T04:03:22Z

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit 2d4449a945f06107e5fc837afded7a8a1b858140 with default session variables
Stream load json:         18 seconds loaded 2358488459 Bytes, about 124 MB/s
Stream load orc:          58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet:      32 seconds loaded 861443392 Bytes, about 25 MB/s
Insert into select:       13.3 seconds inserted 10000000 Rows, about 751K ops/s

doris-robot · 2024-02-02T04:14:18Z

TeamCity be ut coverage result:
Function Coverage: 36.17% (8625/23847)
Line Coverage: 28.22% (70530/249904)
Region Coverage: 27.24% (36398/133622)
Branch Coverage: 24.03% (18641/77582)
Coverage Report: http://coverage.selectdb-in.cc/coverage/2d4449a945f06107e5fc837afded7a8a1b858140_2d4449a945f06107e5fc837afded7a8a1b858140/report/index.html

morningman

LGTM

github-actions · 2024-02-02T13:24:35Z

PR approved by at least one committer and no changes requested.

…gs (#30588) This PR makes the following changes to the connection pool of JDBC Catalog 1. Set the maximum connection survival time, the default is 30 minutes - Moreover, one-half of the maximum survival time is the recyclable time, - One-tenth is the check interval for recycling connections 2. Keepalive only takes effect on the connection pool on BE, and will be activated based on one-fifth of the maximum survival time. 3. The maximum number of existing connections is changed from 100 to 10 4. Add the connection cache recycling thread on BE, and add a parameter to control the recycling time, the default is 28800 (8 hours) 5. Add CatalogID to the key of the connection pool cache to achieve better isolation, requires refresh catalog to take effect 6. Upgrade druid connection pool to version 1.2.20 7. Added JdbcResource's setting of default parameters when upgrading the FE version to avoid errors due to unset parameters.

…gs (apache#30588) This PR makes the following changes to the connection pool of JDBC Catalog 1. Set the maximum connection survival time, the default is 30 minutes - Moreover, one-half of the maximum survival time is the recyclable time, - One-tenth is the check interval for recycling connections 2. Keepalive only takes effect on the connection pool on BE, and will be activated based on one-fifth of the maximum survival time. 3. The maximum number of existing connections is changed from 100 to 10 4. Add the connection cache recycling thread on BE, and add a parameter to control the recycling time, the default is 28800 (8 hours) 5. Add CatalogID to the key of the connection pool cache to achieve better isolation, requires refresh catalog to take effect 6. Upgrade druid connection pool to version 1.2.20 7. Added JdbcResource's setting of default parameters when upgrading the FE version to avoid errors due to unset parameters.

…gs (#31177) pick from (#30588)

…gs (apache#31177) pick from (apache#30588)

github-actions bot added the meta-change label Jan 30, 2024

morningman reviewed Jan 30, 2024

View reviewed changes

zy-kkk mentioned this pull request Jan 31, 2024

[doc](jdbc catalog) add connection pool doc #30641

Merged

morningman previously approved these changes Jan 31, 2024

View reviewed changes

github-actions bot added the approved Indicates a PR has been approved by one committer. label Jan 31, 2024

github-actions bot added the reviewed label Jan 31, 2024

morningman reviewed Jan 31, 2024

View reviewed changes

zy-kkk dismissed morningman’s stale review via 9be7858 January 31, 2024 16:48

zy-kkk force-pushed the jdbc_pool_isolation branch from 97b3996 to 9be7858 Compare January 31, 2024 16:48

github-actions bot removed the approved Indicates a PR has been approved by one committer. label Jan 31, 2024

zy-kkk force-pushed the jdbc_pool_isolation branch from 4e0e10c to a325cde Compare February 1, 2024 08:44

morningman reviewed Feb 1, 2024

View reviewed changes

fe/fe-core/src/main/java/org/apache/doris/catalog/JdbcTable.java Outdated Show resolved Hide resolved

fe/fe-core/src/test/java/org/apache/doris/catalog/JdbcResourceTest.java Outdated Show resolved Hide resolved

fix

2d4449a

morningman approved these changes Feb 2, 2024

View reviewed changes

github-actions bot added the approved Indicates a PR has been approved by one committer. label Feb 2, 2024

BePPPower approved these changes Feb 2, 2024

View reviewed changes

zy-kkk merged commit 78238fc into apache:master Feb 2, 2024
31 of 32 checks passed

zy-kkk deleted the jdbc_pool_isolation branch February 2, 2024 13:39

zy-kkk mentioned this pull request Feb 20, 2024

[pick][improvement](jdbc catalog) Optimize connection pool parameter settings #31177

Merged

morningman pushed a commit that referenced this pull request Feb 21, 2024

[improvement](jdbc catalog) Optimize connection pool parameter settin…

7e7fcf1

…gs (#31177) pick from (#30588)

morningman added dev/2.0.5 dev/2.0.5-merged and removed dev/2.0.5 labels Feb 21, 2024

zy-kkk mentioned this pull request Mar 3, 2024

[fix](jdbc catalog) Change Druid Pool dependency to version 1.2.11 #31703

Merged

mongo360 pushed a commit to mongo360/doris that referenced this pull request Aug 16, 2024

[improvement](jdbc catalog) Optimize connection pool parameter settin…

34508ac

…gs (apache#31177) pick from (apache#30588)

[improvement](jdbc catalog) Optimize connection pool parameter settings #30588

[improvement](jdbc catalog) Optimize connection pool parameter settings #30588

Conversation

zy-kkk commented Jan 30, 2024 • edited Loading

Proposed changes

Further comments

zy-kkk commented Jan 30, 2024

github-actions bot commented Jan 30, 2024

doris-robot commented Jan 30, 2024

doris-robot commented Jan 30, 2024

doris-robot commented Jan 30, 2024

doris-robot commented Jan 30, 2024

doris-robot commented Jan 30, 2024

morningman Jan 30, 2024

Choose a reason for hiding this comment

zy-kkk commented Jan 31, 2024

github-actions bot commented Jan 31, 2024

doris-robot commented Jan 31, 2024

doris-robot commented Jan 31, 2024

doris-robot commented Jan 31, 2024

doris-robot commented Jan 31, 2024

doris-robot commented Jan 31, 2024

morningman left a comment

Choose a reason for hiding this comment

github-actions bot commented Jan 31, 2024

github-actions bot commented Jan 31, 2024

morningman left a comment

Choose a reason for hiding this comment

github-actions bot commented Jan 31, 2024

github-actions bot commented Feb 1, 2024

github-actions bot commented Feb 1, 2024

doris-robot commented Feb 1, 2024

doris-robot commented Feb 1, 2024

doris-robot commented Feb 1, 2024

doris-robot commented Feb 1, 2024

doris-robot commented Feb 1, 2024

zy-kkk commented Feb 2, 2024

github-actions bot commented Feb 2, 2024

doris-robot commented Feb 2, 2024

doris-robot commented Feb 2, 2024

doris-robot commented Feb 2, 2024

doris-robot commented Feb 2, 2024

doris-robot commented Feb 2, 2024

morningman left a comment

Choose a reason for hiding this comment

github-actions bot commented Feb 2, 2024

zy-kkk commented Jan 30, 2024 •

edited

Loading