Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[opt](catalog) cache the converted properties #30668

Merged
merged 4 commits into from
Feb 5, 2024
Merged

[opt](catalog) cache the converted properties #30668

merged 4 commits into from
Feb 5, 2024

Conversation

morningman
Copy link
Contributor

Proposed changes

convert properties may be a heavy operation, so we cache the result.

        at sun.misc.URLClassPath.getNextLoader(URLClassPath.java:480)
        - waiting to lock <0x00000005e99cca00> (a sun.misc.URLClassPath)
        at sun.misc.URLClassPath.findResource(URLClassPath.java:225)
        at java.net.URLClassLoader$2.run(URLClassLoader.java:577)
        at java.net.URLClassLoader$2.run(URLClassLoader.java:575)
        at java.security.AccessController.doPrivileged(Native Method)
        at java.net.URLClassLoader.findResource(URLClassLoader.java:574)
        at java.lang.ClassLoader.getResource(ClassLoader.java:1089)
        at org.apache.hadoop.conf.Configuration.getResource(Configuration.java:2839)
        at org.apache.hadoop.conf.Configuration.getStreamReader(Configuration.java:3113)
        at org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:3072)
        at org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:3045)
        at org.apache.hadoop.conf.Configuration.loadProps(Configuration.java:2923)
        - locked <0x000000065cfe78a8> (a org.apache.hadoop.hive.conf.HiveConf)
        at org.apache.hadoop.conf.Configuration.getProps(Configuration.java:2905)
        - locked <0x000000065cfe78a8> (a org.apache.hadoop.hive.conf.HiveConf)
        at org.apache.hadoop.conf.Configuration.iterator(Configuration.java:2968)
        at org.apache.hadoop.hive.conf.HiveConf.getProperties(HiveConf.java:5134)
        at org.apache.hadoop.hive.conf.HiveConf.getAllProperties(HiveConf.java:5130)
        at org.apache.hadoop.hive.conf.HiveConf.initialize(HiveConf.java:5147)
        at org.apache.hadoop.hive.conf.HiveConf.<init>(HiveConf.java:5102)
        at org.apache.doris.datasource.property.PropertyConverter.getPropertiesFromDLFConf(PropertyConverter.java:377)
        at org.apache.doris.datasource.property.PropertyConverter.convertToDLFProperties(PropertyConverter.java:365)
        at org.apache.doris.datasource.property.PropertyConverter.convertToMetaProperties(PropertyConverter.java:89)
        at org.apache.doris.datasource.ExternalCatalog.getProperties(ExternalCatalog.java:407)
        at org.apache.doris.datasource.hive.HiveMetaStoreCache.getFileCache(HiveMetaStoreCache.java:378)

Further comments

If this is a relatively large or complex change, kick off the discussion at [email protected] by explaining why you chose the solution you did and what alternatives you considered, etc...

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

@morningman
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 36599 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 57362bd6a837720388ac1bf14bff6b6dc1d9c51d, data reload: false

------ Round 1 ----------------------------------
q1	17645	4401	4307	4307
q2	2035	140	131	131
q3	10598	898	891	891
q4	4685	765	751	751
q5	7660	2888	2718	2718
q6	184	117	117	117
q7	1127	712	698	698
q8	9253	1998	1985	1985
q9	7163	6340	6276	6276
q10	8115	2405	2419	2405
q11	415	192	208	192
q12	735	269	286	269
q13	18013	3291	3266	3266
q14	275	244	246	244
q15	537	488	483	483
q16	458	401	425	401
q17	921	556	500	500
q18	6756	6092	6003	6003
q19	1548	1379	1348	1348
q20	570	328	338	328
q21	6743	2985	2990	2985
q22	788	303	301	301
Total cold run time: 106224 ms
Total hot run time: 36599 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4359	4359	4396	4359
q2	325	237	234	234
q3	3034	2829	2817	2817
q4	1853	1610	1665	1610
q5	5211	5224	5226	5224
q6	187	111	115	111
q7	2090	1796	1821	1796
q8	3141	3223	3246	3223
q9	8304	8291	8260	8260
q10	5845	3570	3554	3554
q11	561	451	450	450
q12	725	572	583	572
q13	12557	3064	3070	3064
q14	274	243	265	243
q15	523	486	489	486
q16	513	480	459	459
q17	1821	1675	1702	1675
q18	8034	7722	7564	7564
q19	9263	1542	1556	1542
q20	2140	1898	1890	1890
q21	4825	4654	4531	4531
q22	540	463	459	459
Total cold run time: 76125 ms
Total hot run time: 54123 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 175700 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 57362bd6a837720388ac1bf14bff6b6dc1d9c51d, data reload: false

query1	925	336	329	329
query2	6574	2026	1901	1901
query3	6700	202	200	200
query4	31945	22209	22190	22190
query5	4452	432	352	352
query6	250	161	155	155
query7	4599	262	271	262
query8	245	174	175	174
query9	9270	2270	2270	2270
query10	423	210	217	210
query11	18706	15470	15380	15380
query12	121	70	67	67
query13	1628	403	385	385
query14	10221	7551	7350	7350
query15	247	177	181	177
query16	7528	267	251	251
query17	1984	501	459	459
query18	1968	258	249	249
query19	328	138	135	135
query20	75	75	68	68
query21	206	138	125	125
query22	5013	4830	4771	4771
query23	31739	30342	30379	30342
query24	10217	2755	2781	2755
query25	503	314	310	310
query26	709	138	139	138
query27	2095	286	300	286
query28	5538	1853	1835	1835
query29	901	624	611	611
query30	282	134	139	134
query31	900	702	710	702
query32	88	52	48	48
query33	606	206	212	206
query34	815	448	461	448
query35	861	765	738	738
query36	1360	1178	1212	1178
query37	90	58	57	57
query38	3320	3183	3146	3146
query39	1310	1265	1240	1240
query40	191	90	84	84
query41	38	35	36	35
query42	88	81	83	81
query43	513	493	497	493
query44	1032	683	689	683
query45	194	184	177	177
query46	1059	640	622	622
query47	1555	1406	1507	1406
query48	419	301	307	301
query49	1097	289	272	272
query50	683	317	315	315
query51	5273	5200	5175	5175
query52	102	73	79	73
query53	320	264	266	264
query54	276	187	182	182
query55	82	73	78	73
query56	179	162	159	159
query57	965	909	904	904
query58	182	163	158	158
query59	2553	2277	2380	2277
query60	202	176	187	176
query61	83	83	83	83
query62	602	361	361	361
query63	276	260	268	260
query64	4617	3711	3468	3468
query65	3240	3236	3217	3217
query66	940	324	318	318
query67	14353	14199	14236	14199
query68	4383	498	499	498
query69	458	310	315	310
query70	1591	1568	1539	1539
query71	282	214	212	212
query72	5867	3134	2845	2845
query73	677	317	318	317
query74	6639	6179	6270	6179
query75	3050	2252	2313	2252
query76	3201	982	958	958
query77	377	236	229	229
query78	9219	8676	8553	8553
query79	2802	502	502	502
query80	2122	325	319	319
query81	541	194	204	194
query82	709	85	86	85
query83	334	114	119	114
query84	275	82	76	76
query85	1692	330	339	330
query86	517	403	376	376
query87	3418	3223	3273	3223
query88	3773	2156	2174	2156
query89	420	366	359	359
query90	1945	186	180	180
query91	148	114	120	114
query92	52	45	42	42
query93	2462	429	418	418
query94	1420	159	161	159
query95	503	465	455	455
query96	618	315	308	308
query97	4235	4114	4116	4114
query98	220	198	176	176
query99	1142	679	658	658
Total cold run time: 280948 ms
Total hot run time: 175700 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.77 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 57362bd6a837720388ac1bf14bff6b6dc1d9c51d, data reload: false

query1	0.04	0.03	0.03
query2	0.05	0.02	0.02
query3	0.24	0.06	0.06
query4	1.69	0.10	0.10
query5	0.52	0.52	0.51
query6	1.20	0.67	0.64
query7	0.02	0.02	0.01
query8	0.03	0.02	0.02
query9	0.57	0.52	0.50
query10	0.56	0.55	0.56
query11	0.12	0.08	0.08
query12	0.12	0.10	0.09
query13	0.61	0.61	0.61
query14	0.78	0.80	0.81
query15	0.80	0.78	0.77
query16	0.37	0.39	0.39
query17	0.99	1.01	1.01
query18	0.24	0.25	0.26
query19	1.87	1.70	1.77
query20	0.01	0.01	0.01
query21	15.40	0.58	0.57
query22	2.67	2.32	1.81
query23	17.11	0.88	0.85
query24	2.51	0.85	1.11
query25	0.26	0.26	0.07
query26	0.51	0.15	0.15
query27	0.06	0.04	0.04
query28	11.40	0.86	0.84
query29	12.55	3.08	3.12
query30	0.63	0.47	0.51
query31	2.78	0.35	0.34
query32	3.37	0.48	0.49
query33	3.22	3.23	3.22
query34	15.74	4.27	4.29
query35	4.29	4.33	4.27
query36	1.11	1.06	1.07
query37	0.07	0.05	0.05
query38	0.04	0.03	0.02
query39	0.02	0.02	0.02
query40	0.16	0.12	0.12
query41	0.06	0.02	0.01
query42	0.03	0.01	0.02
query43	0.02	0.02	0.02
Total cold run time: 104.84 s
Total hot run time: 30.77 s

@doris-robot
Copy link

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit 57362bd6a837720388ac1bf14bff6b6dc1d9c51d with default session variables
Stream load json:         18 seconds loaded 2358488459 Bytes, about 124 MB/s
Stream load orc:          58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet:      31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select:       13.6 seconds inserted 10000000 Rows, about 735K ops/s

@morningman
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 36977 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit ccf3fa1fa18bef86e69e3568943333c33a5a7539, data reload: false

------ Round 1 ----------------------------------
q1	17702	4665	4421	4421
q2	2042	135	131	131
q3	10678	920	896	896
q4	4676	711	715	711
q5	7672	2847	2843	2843
q6	181	117	117	117
q7	1134	725	717	717
q8	9310	2014	2026	2014
q9	7227	6327	6304	6304
q10	8084	2448	2435	2435
q11	409	201	209	201
q12	773	269	265	265
q13	18018	3276	3292	3276
q14	271	249	242	242
q15	530	480	495	480
q16	465	413	394	394
q17	934	576	569	569
q18	6849	5899	5830	5830
q19	1576	1373	1373	1373
q20	577	326	334	326
q21	6876	3177	3127	3127
q22	803	306	305	305
Total cold run time: 106787 ms
Total hot run time: 36977 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4428	4489	4372	4372
q2	328	239	245	239
q3	2943	2817	2807	2807
q4	1821	1740	1617	1617
q5	5238	5192	5251	5192
q6	188	113	113	113
q7	2170	1785	1750	1750
q8	3139	3221	3276	3221
q9	8274	8264	8364	8264
q10	5754	3581	3523	3523
q11	537	453	455	453
q12	750	574	590	574
q13	13522	3086	3071	3071
q14	272	246	257	246
q15	533	494	492	492
q16	497	471	471	471
q17	1804	1700	1690	1690
q18	8027	7566	7585	7566
q19	9281	1532	1482	1482
q20	2160	1903	1894	1894
q21	6312	4544	4537	4537
q22	541	468	430	430
Total cold run time: 78519 ms
Total hot run time: 54004 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 173261 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit ccf3fa1fa18bef86e69e3568943333c33a5a7539, data reload: false

query1	933	333	323	323
query2	6580	1927	1899	1899
query3	6696	201	195	195
query4	31961	21927	21955	21927
query5	4456	356	412	356
query6	249	158	153	153
query7	4597	265	261	261
query8	237	180	170	170
query9	9171	2300	2267	2267
query10	409	195	215	195
query11	18706	15471	15355	15355
query12	123	69	64	64
query13	1621	367	377	367
query14	9955	6914	7461	6914
query15	233	182	184	182
query16	7324	264	251	251
query17	1865	485	461	461
query18	1933	256	246	246
query19	199	127	134	127
query20	68	68	71	68
query21	196	147	124	124
query22	4962	4637	4566	4566
query23	31639	30184	30223	30184
query24	7315	2744	2726	2726
query25	526	309	316	309
query26	731	138	138	138
query27	2259	279	277	277
query28	5286	1848	1836	1836
query29	929	617	606	606
query30	284	133	138	133
query31	924	734	722	722
query32	95	58	56	56
query33	457	213	201	201
query34	814	442	466	442
query35	824	755	742	742
query36	950	920	921	920
query37	94	55	57	55
query38	3267	3131	3159	3131
query39	1310	1264	1235	1235
query40	193	87	88	87
query41	39	36	40	36
query42	91	82	79	79
query43	528	476	507	476
query44	1039	705	704	704
query45	192	178	174	174
query46	1028	619	612	612
query47	1531	1557	1489	1489
query48	385	305	311	305
query49	1122	285	284	284
query50	694	304	299	299
query51	5287	5152	5124	5124
query52	93	76	80	76
query53	319	258	263	258
query54	225	181	185	181
query55	79	74	78	74
query56	175	165	154	154
query57	975	894	911	894
query58	179	158	162	158
query59	2442	2298	2428	2298
query60	204	171	175	171
query61	84	87	84	84
query62	581	357	340	340
query63	281	255	260	255
query64	4681	3691	3335	3335
query65	3250	3228	3220	3220
query66	959	318	318	318
query67	14608	13974	13841	13841
query68	4025	497	502	497
query69	425	303	309	303
query70	1285	1218	1183	1183
query71	299	209	212	209
query72	5540	3074	2846	2846
query73	689	310	315	310
query74	6672	6223	6247	6223
query75	2942	2274	2264	2264
query76	2561	936	1017	936
query77	357	230	220	220
query78	9315	8596	8511	8511
query79	3068	481	482	481
query80	2034	325	309	309
query81	495	202	207	202
query82	763	80	81	80
query83	244	122	116	116
query84	280	72	73	72
query85	2006	345	336	336
query86	478	322	285	285
query87	3462	3271	3211	3211
query88	4127	2169	2152	2152
query89	445	360	352	352
query90	1912	185	184	184
query91	161	116	116	116
query92	53	44	40	40
query93	5362	452	443	443
query94	1210	157	164	157
query95	504	470	444	444
query96	630	310	309	309
query97	4229	4103	4115	4103
query98	207	189	184	184
query99	1177	673	673	673
Total cold run time: 278399 ms
Total hot run time: 173261 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.57 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit ccf3fa1fa18bef86e69e3568943333c33a5a7539, data reload: false

query1	0.04	0.03	0.03
query2	0.06	0.02	0.02
query3	0.23	0.06	0.06
query4	1.68	0.10	0.10
query5	0.53	0.51	0.51
query6	1.17	0.63	0.63
query7	0.02	0.01	0.01
query8	0.04	0.02	0.03
query9	0.55	0.50	0.50
query10	0.54	0.54	0.56
query11	0.12	0.08	0.08
query12	0.11	0.08	0.09
query13	0.60	0.61	0.61
query14	0.79	0.79	0.78
query15	0.78	0.77	0.77
query16	0.38	0.37	0.40
query17	0.99	0.98	1.01
query18	0.22	0.26	0.25
query19	1.79	1.74	1.76
query20	0.02	0.01	0.01
query21	15.41	0.58	0.57
query22	2.57	2.54	1.66
query23	16.87	0.89	0.80
query24	2.69	1.59	0.84
query25	0.26	0.31	0.08
query26	0.60	0.13	0.14
query27	0.05	0.05	0.05
query28	12.38	0.84	0.83
query29	12.66	3.10	3.22
query30	0.62	0.54	0.53
query31	2.79	0.33	0.36
query32	3.37	0.47	0.49
query33	3.23	3.22	3.22
query34	15.86	4.32	4.28
query35	4.29	4.27	4.30
query36	1.12	1.06	1.07
query37	0.07	0.05	0.05
query38	0.03	0.03	0.03
query39	0.02	0.01	0.02
query40	0.16	0.14	0.12
query41	0.07	0.01	0.01
query42	0.03	0.02	0.03
query43	0.03	0.02	0.02
Total cold run time: 105.84 s
Total hot run time: 30.57 s

@doris-robot
Copy link

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit ccf3fa1fa18bef86e69e3568943333c33a5a7539 with default session variables
Stream load json:         19 seconds loaded 2358488459 Bytes, about 118 MB/s
Stream load orc:          58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet:      31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select:       13.4 seconds inserted 10000000 Rows, about 746K ops/s

@morningman
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 37589 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit a6c3b7920a2a9b742a1b2d85377ac1a34b26f1a7, data reload: false

------ Round 1 ----------------------------------
q1	17886	4502	4443	4443
q2	2044	145	136	136
q3	10590	935	952	935
q4	5148	756	746	746
q5	7739	3000	2918	2918
q6	183	121	122	121
q7	1117	750	733	733
q8	9312	2025	2010	2010
q9	7382	6511	6569	6511
q10	8299	2615	2696	2615
q11	415	222	207	207
q12	763	300	288	288
q13	18704	3355	3383	3355
q14	287	256	269	256
q15	547	509	500	500
q16	1297	417	418	417
q17	1734	561	540	540
q18	6840	5988	5895	5895
q19	1584	1499	1375	1375
q20	629	356	318	318
q21	6800	2972	3056	2972
q22	800	304	298	298
Total cold run time: 110100 ms
Total hot run time: 37589 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4538	4345	4494	4345
q2	330	240	237	237
q3	2982	2867	2801	2801
q4	1821	1629	1665	1629
q5	5171	5262	5231	5231
q6	194	115	117	115
q7	2145	1721	1762	1721
q8	3087	3221	3275	3221
q9	8320	8260	8282	8260
q10	5774	3507	3573	3507
q11	539	467	456	456
q12	767	579	559	559
q13	14665	3072	3089	3072
q14	274	251	262	251
q15	528	491	492	491
q16	537	471	470	470
q17	1864	1677	1691	1677
q18	7987	7816	7479	7479
q19	8611	1512	1544	1512
q20	2146	1906	1898	1898
q21	5085	4617	4543	4543
q22	565	464	442	442
Total cold run time: 77930 ms
Total hot run time: 53917 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 174449 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit a6c3b7920a2a9b742a1b2d85377ac1a34b26f1a7, data reload: false

query1	935	349	333	333
query2	6544	2071	1909	1909
query3	6700	206	200	200
query4	31690	22036	22022	22022
query5	4270	356	361	356
query6	258	172	177	172
query7	4613	285	277	277
query8	252	175	194	175
query9	8863	2313	2296	2296
query10	409	213	214	213
query11	18649	15467	15326	15326
query12	124	76	78	76
query13	1612	407	406	406
query14	9412	6886	6939	6886
query15	289	182	182	182
query16	8160	253	253	253
query17	2043	532	490	490
query18	2104	271	282	271
query19	368	142	137	137
query20	81	73	79	73
query21	199	120	123	120
query22	4901	4645	4680	4645
query23	31056	29983	29946	29946
query24	6430	2754	2753	2753
query25	545	347	343	343
query26	702	147	142	142
query27	2127	303	304	303
query28	4811	1884	1854	1854
query29	882	614	609	609
query30	282	130	142	130
query31	931	711	696	696
query32	89	59	49	49
query33	464	220	217	217
query34	815	451	468	451
query35	846	780	745	745
query36	992	914	946	914
query37	98	59	62	59
query38	3216	3120	3108	3108
query39	1290	1258	1220	1220
query40	181	93	91	91
query41	37	36	34	34
query42	95	87	90	87
query43	512	472	463	463
query44	1022	699	699	699
query45	187	179	176	176
query46	1033	649	647	647
query47	1515	1531	1498	1498
query48	428	353	361	353
query49	1044	289	286	286
query50	762	369	369	369
query51	5272	5177	5094	5094
query52	93	87	81	81
query53	337	275	259	259
query54	258	220	220	220
query55	79	77	86	77
query56	214	195	196	195
query57	949	892	902	892
query58	202	177	179	177
query59	2464	2449	2375	2375
query60	237	214	220	214
query61	87	86	84	84
query62	627	356	346	346
query63	286	275	274	274
query64	4648	3537	3715	3537
query65	3273	3229	3201	3201
query66	830	312	303	303
query67	14209	14266	14129	14129
query68	4398	534	525	525
query69	471	337	345	337
query70	1214	1229	1224	1224
query71	315	250	252	250
query72	6090	2868	2685	2685
query73	698	316	317	316
query74	6571	6342	6234	6234
query75	3060	2321	2279	2279
query76	2593	986	998	986
query77	343	230	223	223
query78	9257	8786	8589	8589
query79	2270	495	490	490
query80	1529	351	347	347
query81	534	194	198	194
query82	645	79	81	79
query83	240	120	121	120
query84	237	80	78	78
query85	2152	364	337	337
query86	494	277	308	277
query87	3392	3214	3166	3166
query88	3705	2307	2320	2307
query89	437	356	367	356
query90	1944	168	163	163
query91	168	146	124	124
query92	57	42	44	42
query93	2644	505	458	458
query94	1302	176	169	169
query95	468	360	354	354
query96	596	264	267	264
query97	4240	4104	4114	4104
query98	219	197	197	197
query99	1192	696	657	657
Total cold run time: 272408 ms
Total hot run time: 174449 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.53 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit a6c3b7920a2a9b742a1b2d85377ac1a34b26f1a7, data reload: false

query1	0.04	0.02	0.03
query2	0.06	0.02	0.02
query3	0.24	0.06	0.06
query4	1.64	0.10	0.09
query5	0.52	0.51	0.53
query6	1.21	0.64	0.63
query7	0.02	0.01	0.01
query8	0.04	0.02	0.02
query9	0.57	0.52	0.48
query10	0.57	0.57	0.55
query11	0.12	0.08	0.08
query12	0.11	0.09	0.09
query13	0.60	0.60	0.61
query14	0.79	0.81	0.80
query15	0.79	0.78	0.78
query16	0.38	0.38	0.38
query17	1.02	1.03	1.03
query18	0.25	0.23	0.24
query19	1.81	1.81	1.78
query20	0.01	0.01	0.01
query21	15.40	0.57	0.58
query22	2.55	3.00	2.05
query23	17.33	0.77	0.72
query24	2.69	0.28	0.68
query25	0.40	0.19	0.08
query26	0.36	0.15	0.14
query27	0.04	0.05	0.05
query28	12.22	0.82	0.85
query29	12.55	3.20	3.20
query30	0.55	0.57	0.53
query31	2.78	0.35	0.36
query32	3.36	0.49	0.48
query33	3.21	3.26	3.21
query34	15.84	4.31	4.32
query35	4.40	4.36	4.27
query36	1.13	1.06	1.06
query37	0.06	0.05	0.04
query38	0.04	0.02	0.02
query39	0.03	0.01	0.01
query40	0.16	0.13	0.15
query41	0.07	0.02	0.01
query42	0.02	0.01	0.01
query43	0.03	0.02	0.02
Total cold run time: 106.01 s
Total hot run time: 30.53 s

@doris-robot
Copy link

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit a6c3b7920a2a9b742a1b2d85377ac1a34b26f1a7 with default session variables
Stream load json:         18 seconds loaded 2358488459 Bytes, about 124 MB/s
Stream load orc:          60 seconds loaded 1101869774 Bytes, about 17 MB/s
Stream load parquet:      32 seconds loaded 861443392 Bytes, about 25 MB/s
Insert into select:       13.7 seconds inserted 10000000 Rows, about 729K ops/s

@morningman
Copy link
Contributor Author

run buildall

@morningman morningman added dev/2.0.5 usercase Important user case type label labels Feb 5, 2024
@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Feb 5, 2024

PR approved by anyone and no changes requested.

zy-kkk
zy-kkk approved these changes Feb 5, 2024
View reviewed changes
Copy link
Member

@zy-kkk zy-kkk left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Feb 5, 2024
@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Feb 5, 2024

PR approved by at least one committer and no changes requested.

@morningman morningman merged commit 72caa40 into apache:master Feb 5, 2024
29 of 31 checks passed
morningman added a commit to morningman/doris that referenced this pull request Feb 5, 2024
@morningman
[opt](catalog) cache the converted properties (apache#30668)
b7b6f79 
convert properties may be a heavy operation, so we cache the result.
@morningman morningman mentioned this pull request Feb 5, 2024
Merged
morningman added a commit that referenced this pull request Feb 5, 2024
@morningman
[opt](catalog) cache the converted properties (#30668) (#30878)
ae356c9 
bp #30668
yiguolei pushed a commit that referenced this pull request Feb 6, 2024
@morningman @Doris-Extras
[opt](catalog) cache the converted properties (#30668)
06ed578 
convert properties may be a heavy operation, so we cache the result.
starocean999 pushed a commit to starocean999/incubator-doris that referenced this pull request Feb 6, 2024
@morningman @starocean999
[opt](catalog) cache the converted properties (apache#30668)
4a68062 
convert properties may be a heavy operation, so we cache the result.
mongo360 pushed a commit to mongo360/doris that referenced this pull request Aug 16, 2024
@morningman
[opt](catalog) cache the converted properties (apache#30668) (apache#…
53b7028 
…30878)

bp apache#30668
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@zddr zddr zddr approved these changes

@wuwenchi wuwenchi wuwenchi approved these changes

@zy-kkk zy-kkk zy-kkk approved these changes

Assignees
No one assigned
Labels
approved Indicates a PR has been approved by one committer. dev/2.0.5-merged reviewed usercase Important user case type label
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@morningman @doris-robot @zddr @wuwenchi @zy-kkk