Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Refactor](SchemaCache) remove redundant Schema cache #40350

Merged
merged 1 commit into from
Sep 5, 2024

Conversation

eldenmoon
Copy link
Member

We already introduced TabletSchema cache and it is enough at present,
and Schema cache may introduce inconsistency bettween Schema and
TabletSchema
@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@eldenmoon
Copy link
Member Author

run buildall

Copy link
Contributor

github-actions bot commented Sep 4, 2024

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

TPC-H: Total hot run time: 49141 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 4b975504aa578ac3692f9b069246730232f29887, data reload: false

------ Round 1 ----------------------------------
q1	17784	4473	4330	4330
q2	2065	161	143	143
q3	10465	1886	1946	1886
q4	10295	1265	1339	1265
q5	8544	3928	3919	3919
q6	230	123	123	123
q7	2052	1653	1619	1619
q8	9307	2746	2719	2719
q9	9983	9891	9869	9869
q10	8664	3552	3499	3499
q11	421	252	249	249
q12	464	302	300	300
q13	18325	3954	4026	3954
q14	357	324	335	324
q15	507	461	460	460
q16	547	466	454	454
q17	1138	969	941	941
q18	7200	6994	6814	6814
q19	1695	1568	1537	1537
q20	544	303	295	295
q21	4426	4118	4061	4061
q22	499	391	380	380
Total cold run time: 115512 ms
Total hot run time: 49141 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4310	4344	4326	4326
q2	330	237	222	222
q3	4153	4145	4175	4145
q4	2753	2748	2754	2748
q5	7236	7123	7079	7079
q6	241	118	120	118
q7	3283	2846	2810	2810
q8	4397	4484	4502	4484
q9	13652	13477	13586	13477
q10	4217	4259	4272	4259
q11	734	720	665	665
q12	1026	857	864	857
q13	6995	3741	3761	3741
q14	455	421	432	421
q15	509	456	461	456
q16	623	597	604	597
q17	3772	3852	3791	3791
q18	8814	8699	8731	8699
q19	1738	1664	1627	1627
q20	2385	2109	2108	2108
q21	8564	8486	8458	8458
q22	1033	914	935	914
Total cold run time: 81220 ms
Total hot run time: 76002 ms

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 37.83% (8133/21497)
Line Coverage: 29.58% (66949/226306)
Region Coverage: 29.07% (34542/118820)
Branch Coverage: 24.98% (17801/71258)
Coverage Report: http://coverage.selectdb-in.cc/coverage/4b975504aa578ac3692f9b069246730232f29887_4b975504aa578ac3692f9b069246730232f29887/report/index.html

@doris-robot
Copy link

TPC-DS: Total hot run time: 211713 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 4b975504aa578ac3692f9b069246730232f29887, data reload: false

query1	937	382	415	382
query2	6566	2253	2040	2040
query3	6920	212	199	199
query4	22869	21738	21821	21738
query5	19730	6480	6442	6442
query6	281	229	221	221
query7	4336	299	318	299
query8	266	276	281	276
query9	3053	2616	2571	2571
query10	471	307	302	302
query11	15831	15224	15175	15175
query12	132	80	75	75
query13	1050	457	429	429
query14	17172	13316	13063	13063
query15	362	224	220	220
query16	6471	284	263	263
query17	1740	954	879	879
query18	896	316	318	316
query19	214	150	145	145
query20	72	78	81	78
query21	190	93	93	93
query22	5271	4987	4966	4966
query23	34116	33374	33272	33272
query24	7914	6277	6308	6277
query25	515	432	441	432
query26	1270	160	160	160
query27	2426	294	291	291
query28	6153	2267	2239	2239
query29	2828	2738	2657	2657
query30	242	169	169	169
query31	963	770	745	745
query32	65	66	63	63
query33	452	270	261	261
query34	856	456	481	456
query35	1126	879	940	879
query36	1430	1169	1217	1169
query37	169	60	59	59
query38	3072	2936	2926	2926
query39	1381	1318	1310	1310
query40	314	94	97	94
query41	40	36	38	36
query42	91	90	88	88
query43	594	609	614	609
query44	1171	707	724	707
query45	247	232	227	227
query46	1234	946	951	946
query47	1873	1725	1829	1725
query48	511	420	410	410
query49	642	385	380	380
query50	856	567	619	567
query51	4824	4649	4622	4622
query52	94	89	91	89
query53	240	185	177	177
query54	2662	2435	2422	2422
query55	95	81	84	81
query56	233	216	224	216
query57	1271	1272	1064	1064
query58	215	215	198	198
query59	3564	3212	3213	3212
query60	233	207	198	198
query61	95	92	93	92
query62	853	498	500	498
query63	199	179	174	174
query64	3544	1606	1495	1495
query65	3629	3553	3559	3553
query66	769	399	403	399
query67	16069	15797	15012	15012
query68	9992	666	644	644
query69	481	270	258	258
query70	1840	1446	1540	1446
query71	416	311	324	311
query72	6855	4772	4693	4693
query73	756	311	320	311
query74	6309	5914	5924	5914
query75	5335	3758	3794	3758
query76	5819	1123	1202	1123
query77	993	251	249	249
query78	12614	11735	12916	11735
query79	12653	641	640	640
query80	1305	381	372	372
query81	488	243	233	233
query82	1018	98	97	97
query83	183	135	134	134
query84	262	71	71	71
query85	891	317	320	317
query86	338	298	318	298
query87	3272	2997	3023	2997
query88	4937	2290	2298	2290
query89	426	273	304	273
query90	2185	217	208	208
query91	169	123	135	123
query92	57	51	52	51
query93	5172	564	554	554
query94	968	212	202	202
query95	1919	1930	1994	1930
query96	646	329	339	329
query97	6385	6452	6295	6295
query98	220	206	205	205
query99	2820	863	824	824
Total cold run time: 323694 ms
Total hot run time: 211713 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.96 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 4b975504aa578ac3692f9b069246730232f29887, data reload: false

query1	0.02	0.02	0.02
query2	0.07	0.02	0.03
query3	0.26	0.05	0.05
query4	1.79	0.06	0.07
query5	0.54	0.53	0.52
query6	1.23	0.62	0.62
query7	0.01	0.01	0.01
query8	0.03	0.02	0.03
query9	0.51	0.48	0.48
query10	0.52	0.54	0.53
query11	0.12	0.09	0.09
query12	0.12	0.10	0.09
query13	0.62	0.62	0.61
query14	0.79	0.78	0.79
query15	0.79	0.77	0.77
query16	0.39	0.37	0.37
query17	1.03	1.02	1.02
query18	0.23	0.25	0.24
query19	1.93	1.85	1.85
query20	0.02	0.01	0.01
query21	15.44	0.57	0.55
query22	2.07	2.19	1.46
query23	17.03	1.16	0.95
query24	6.50	1.24	1.29
query25	0.43	0.06	0.08
query26	0.71	0.16	0.16
query27	0.04	0.03	0.04
query28	6.10	0.76	0.72
query29	12.63	2.32	2.38
query30	0.61	0.53	0.53
query31	2.81	0.39	0.38
query32	3.39	0.49	0.48
query33	3.13	3.06	3.09
query34	15.23	4.77	4.79
query35	4.85	4.83	4.81
query36	1.06	1.02	1.04
query37	0.05	0.05	0.05
query38	0.04	0.03	0.02
query39	0.02	0.02	0.01
query40	0.16	0.14	0.14
query41	0.06	0.02	0.01
query42	0.02	0.02	0.01
query43	0.03	0.02	0.02
Total cold run time: 103.43 s
Total hot run time: 30.96 s

@doris-robot
Copy link

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit 4b975504aa578ac3692f9b069246730232f29887 with default session variables
Stream load json:         20 seconds loaded 2358488459 Bytes, about 112 MB/s
Stream load orc:          58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet:      31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select:       21.1 seconds inserted 10000000 Rows, about 473K ops/s

@eldenmoon eldenmoon merged commit ef967c2 into apache:branch-2.0 Sep 5, 2024
20 of 24 checks passed
@eldenmoon eldenmoon deleted the branch-2.0 branch September 5, 2024 03:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants