Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[opt](catalog) cache the converted properties (#30668) #30878

Merged
merged 1 commit into from
Feb 5, 2024

Conversation

morningman
Copy link
Contributor

bp #30668

convert properties may be a heavy operation, so we cache the result.
@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

@morningman
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 49796 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit b7b6f796858333aa12139e54d313fca68fe44d29, data reload: false

------ Round 1 ----------------------------------
q1	18043	4429	4374	4374
q2	2103	159	140	140
q3	10264	1894	1923	1894
q4	10097	1259	1340	1259
q5	8404	3951	3988	3951
q6	230	123	119	119
q7	2038	1647	1593	1593
q8	9284	2683	2717	2683
q9	10978	10521	10445	10445
q10	8667	3508	3520	3508
q11	414	231	244	231
q12	466	297	299	297
q13	18367	3939	3974	3939
q14	360	325	318	318
q15	500	459	453	453
q16	697	597	583	583
q17	1107	945	979	945
q18	7218	7052	6859	6859
q19	1682	1526	1437	1437
q20	566	300	312	300
q21	4414	4098	4074	4074
q22	488	409	394	394
Total cold run time: 116387 ms
Total hot run time: 49796 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4324	4267	4320	4267
q2	324	217	217	217
q3	4168	4133	4131	4131
q4	2760	2759	2753	2753
q5	7364	7338	7185	7185
q6	234	117	115	115
q7	3233	2910	2853	2853
q8	4383	4448	4504	4448
q9	17130	17020	17151	17020
q10	4250	4261	4261	4261
q11	753	676	694	676
q12	1015	862	861	861
q13	7388	3722	3753	3722
q14	449	418	419	418
q15	497	465	447	447
q16	764	698	698	698
q17	3776	3831	3828	3828
q18	8800	8846	8834	8834
q19	1701	1691	1648	1648
q20	2376	2177	2102	2102
q21	8617	8426	8537	8426
q22	1033	952	934	934
Total cold run time: 85339 ms
Total hot run time: 79844 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 238545 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit b7b6f796858333aa12139e54d313fca68fe44d29, data reload: false

query1	897	398	384	384
query2	6526	2150	2226	2150
query3	6911	203	200	200
query4	21320	18062	18012	18012
query5	19724	6248	6228	6228
query6	285	223	221	221
query7	4148	289	288	288
query8	236	244	222	222
query9	3146	2745	2667	2667
query10	401	292	316	292
query11	11408	10715	10807	10715
query12	120	75	70	70
query13	5651	626	607	607
query14	17510	13155	13123	13123
query15	387	227	235	227
query16	6441	260	258	258
query17	3182	884	856	856
query18	2260	395	388	388
query19	199	145	142	142
query20	77	74	73	73
query21	186	89	90	89
query22	5329	5212	5206	5206
query23	32424	31795	31724	31724
query24	6988	6498	6521	6498
query25	508	429	396	396
query26	532	158	156	156
query27	1886	292	293	292
query28	6084	2243	2225	2225
query29	2966	2913	2608	2608
query30	244	161	160	160
query31	906	731	732	731
query32	59	58	55	55
query33	400	257	233	233
query34	838	455	471	455
query35	1122	879	878	878
query36	1491	1554	1515	1515
query37	87	57	57	57
query38	3082	2935	2955	2935
query39	1384	1318	1324	1318
query40	201	92	97	92
query41	34	36	31	31
query42	86	87	81	81
query43	613	639	648	639
query44	1149	726	717	717
query45	242	229	224	224
query46	1234	987	973	973
query47	1692	1672	1696	1672
query48	975	681	635	635
query49	620	365	361	361
query50	858	595	586	586
query51	5473	5496	5392	5392
query52	88	73	78	73
query53	438	318	314	314
query54	2423	2242	2227	2227
query55	79	81	79	79
query56	190	194	206	194
query57	1128	1038	1074	1038
query58	221	193	207	193
query59	3464	3276	3223	3223
query60	197	190	209	190
query61	81	87	79	79
query62	826	494	490	490
query63	471	334	328	328
query64	1963	1318	1307	1307
query65	3682	3649	3646	3646
query66	834	369	364	364
query67	15867	15337	16121	15337
query68	8822	625	662	625
query69	554	350	354	350
query70	1808	2001	1735	1735
query71	386	293	305	293
query72	4446	3288	3302	3288
query73	725	319	326	319
query74	6238	5959	5815	5815
query75	4804	3512	3549	3512
query76	5609	1195	1227	1195
query77	854	244	250	244
query78	32305	52038	48349	48349
query79	15972	665	627	627
query80	4894	373	376	373
query81	580	233	227	227
query82	1207	100	94	94
query83	452	138	132	132
query84	260	70	71	70
query85	2458	269	273	269
query86	458	371	381	371
query87	3265	3040	3052	3040
query88	6854	2370	2322	2322
query89	469	276	299	276
query90	2541	213	205	205
query91	151	117	116	116
query92	62	54	49	49
query93	6531	585	574	574
query94	1695	203	204	203
query95	1099	1062	1081	1062
query96	645	323	328	323
query97	6433	6319	6436	6319
query98	183	188	173	173
query99	3791	881	856	856
Total cold run time: 349025 ms
Total hot run time: 238545 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.42 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit b7b6f796858333aa12139e54d313fca68fe44d29, data reload: false

query1	0.03	0.02	0.02
query2	0.06	0.03	0.02
query3	0.25	0.05	0.05
query4	1.81	0.08	0.07
query5	0.53	0.53	0.52
query6	1.24	0.62	0.66
query7	0.02	0.01	0.01
query8	0.03	0.02	0.02
query9	0.52	0.48	0.47
query10	0.54	0.53	0.53
query11	0.11	0.08	0.08
query12	0.11	0.09	0.09
query13	0.61	0.62	0.62
query14	0.78	0.80	0.80
query15	0.80	0.76	0.75
query16	0.37	0.38	0.37
query17	1.02	1.02	1.02
query18	0.24	0.22	0.25
query19	1.93	1.81	1.84
query20	0.01	0.01	0.02
query21	15.47	0.55	0.55
query22	1.82	1.90	1.26
query23	17.15	0.96	0.94
query24	6.73	0.84	1.29
query25	1.63	0.11	0.11
query26	0.37	0.14	0.14
query27	0.11	0.11	0.12
query28	7.78	0.74	0.72
query29	12.66	2.33	2.30
query30	0.60	0.51	0.50
query31	2.83	0.38	0.40
query32	3.42	0.50	0.50
query33	3.08	3.06	3.11
query34	15.24	4.79	4.83
query35	4.83	4.86	4.88
query36	1.06	1.01	1.01
query37	0.06	0.05	0.05
query38	0.04	0.02	0.02
query39	0.01	0.02	0.01
query40	0.15	0.14	0.14
query41	0.07	0.01	0.01
query42	0.02	0.01	0.01
query43	0.03	0.02	0.01
Total cold run time: 106.17 s
Total hot run time: 30.42 s

@doris-robot
Copy link

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit b7b6f796858333aa12139e54d313fca68fe44d29 with default session variables
Stream load json:         20 seconds loaded 2358488459 Bytes, about 112 MB/s
Stream load orc:          59 seconds loaded 1101869774 Bytes, about 17 MB/s
Stream load parquet:      32 seconds loaded 861443392 Bytes, about 25 MB/s
Insert into select:       21.8 seconds inserted 10000000 Rows, about 458K ops/s

@morningman morningman merged commit ae356c9 into apache:branch-2.0 Feb 5, 2024
25 of 28 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants