Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[opt](partition) delete idToStoragePolicy from PartitionInfo.java #40690

Open
wants to merge 3 commits into
base: master
Choose a base branch
from

Conversation

DarvenDuan
Copy link
Contributor

Proposed changes

Issue Number: close #xxx

The partitions' storage policy is saved in both idToStoragePolicy and idToDataProperty, but idToStoragePolicy will not be written to editLog, this PR deletes the redundant info idToStoragePolicy to simplify partition info.

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@doris-robot
Copy link

TPC-H: Total hot run time: 38281 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit fa41c5b7c07bf545faa1b07f53dd51fd6721e502, data reload: false

------ Round 1 ----------------------------------
q1	17619	4556	4289	4289
q2	2025	194	195	194
q3	11204	977	1110	977
q4	10540	781	743	743
q5	7778	2866	2846	2846
q6	220	139	136	136
q7	953	622	602	602
q8	9342	2079	2096	2079
q9	7157	6574	6509	6509
q10	7048	2212	2197	2197
q11	473	251	251	251
q12	412	222	223	222
q13	19006	3080	3116	3080
q14	281	248	251	248
q15	551	483	488	483
q16	558	433	421	421
q17	1010	625	711	625
q18	7428	7001	6948	6948
q19	1385	1106	1084	1084
q20	695	342	331	331
q21	3980	2999	3033	2999
q22	1112	1025	1017	1017
Total cold run time: 110777 ms
Total hot run time: 38281 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4356	4306	4307	4306
q2	391	275	268	268
q3	2870	2680	2639	2639
q4	1920	1660	1653	1653
q5	5686	5708	5732	5708
q6	225	142	136	136
q7	2207	1812	1784	1784
q8	3336	3464	3472	3464
q9	8897	8873	8795	8795
q10	3594	3437	3370	3370
q11	634	534	514	514
q12	860	656	673	656
q13	14047	3266	3314	3266
q14	315	285	293	285
q15	540	483	496	483
q16	540	487	516	487
q17	1826	1545	1555	1545
q18	8244	7940	7837	7837
q19	1769	1438	1538	1438
q20	2174	1928	1871	1871
q21	5855	5575	5636	5575
q22	1126	1062	1018	1018
Total cold run time: 71412 ms
Total hot run time: 57098 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 196972 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit fa41c5b7c07bf545faa1b07f53dd51fd6721e502, data reload: false

query1	1262	895	850	850
query2	6360	1899	1845	1845
query3	10595	3946	3848	3848
query4	59818	26977	23386	23386
query5	5049	490	486	486
query6	392	176	172	172
query7	5635	307	308	307
query8	331	237	220	220
query9	7824	2514	2493	2493
query10	421	274	264	264
query11	17382	15123	15346	15123
query12	158	104	109	104
query13	1475	404	395	395
query14	10462	7003	7812	7003
query15	214	181	168	168
query16	6819	468	466	466
query17	1091	574	579	574
query18	1433	331	303	303
query19	206	148	149	148
query20	121	113	116	113
query21	216	107	103	103
query22	4488	4478	4433	4433
query23	34506	33543	33836	33543
query24	5986	2874	2831	2831
query25	514	415	411	411
query26	622	158	161	158
query27	1599	274	276	274
query28	3699	2043	2028	2028
query29	666	400	398	398
query30	245	155	148	148
query31	916	752	778	752
query32	72	53	53	53
query33	423	299	293	293
query34	877	465	455	455
query35	832	694	710	694
query36	1055	928	931	928
query37	146	84	78	78
query38	3957	3882	3985	3882
query39	1441	1381	1396	1381
query40	205	113	111	111
query41	46	47	46	46
query42	119	97	99	97
query43	504	474	468	468
query44	1076	744	739	739
query45	191	165	168	165
query46	1078	747	729	729
query47	1879	1787	1809	1787
query48	377	291	296	291
query49	765	446	432	432
query50	820	412	416	412
query51	7006	6890	6865	6865
query52	99	87	90	87
query53	249	182	172	172
query54	545	456	457	456
query55	73	110	72	72
query56	283	258	256	256
query57	1198	1095	1125	1095
query58	241	226	239	226
query59	3024	2726	2629	2629
query60	289	259	257	257
query61	103	103	103	103
query62	746	671	665	665
query63	219	186	189	186
query64	1371	670	728	670
query65	3236	3211	3184	3184
query66	628	344	343	343
query67	15778	15237	15331	15237
query68	1377	857	834	834
query69	442	329	321	321
query70	1088	1164	1191	1164
query71	353	346	337	337
query72	4585	3586	3525	3525
query73	577	569	579	569
query74	9018	8928	8895	8895
query75	3045	3003	2979	2979
query76	949	905	835	835
query77	418	406	402	402
query78	9292	9232	9280	9232
query79	892	871	848	848
query80	820	835	816	816
query81	451	256	260	256
query82	271	268	266	266
query83	191	185	187	185
query84	196	103	103	103
query85	597	466	387	387
query86	317	324	302	302
query87	4337	4400	4262	4262
query88	4117	4038	4018	4018
query89	375	364	369	364
query90	778	305	303	303
query91	123	123	124	123
query92	76	80	74	74
query93	924	910	902	902
query94	393	364	342	342
query95	422	400	406	400
query96	467	466	467	466
query97	3164	3097	3096	3096
query98	234	236	224	224
query99	1326	1281	1251	1251
Total cold run time: 293071 ms
Total hot run time: 196972 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 31.02 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit fa41c5b7c07bf545faa1b07f53dd51fd6721e502, data reload: false

query1	0.05	0.05	0.04
query2	0.09	0.04	0.04
query3	0.23	0.05	0.04
query4	1.67	0.08	0.08
query5	0.51	0.48	0.52
query6	1.13	0.73	0.73
query7	0.02	0.01	0.02
query8	0.05	0.04	0.04
query9	0.55	0.50	0.50
query10	0.55	0.55	0.53
query11	0.16	0.11	0.12
query12	0.15	0.12	0.12
query13	0.60	0.58	0.59
query14	1.38	1.40	1.41
query15	0.84	0.82	0.82
query16	0.37	0.37	0.36
query17	1.00	1.00	1.00
query18	0.18	0.18	0.18
query19	1.83	1.84	1.84
query20	0.01	0.01	0.01
query21	15.39	0.68	0.66
query22	4.60	6.69	1.41
query23	18.23	1.42	1.31
query24	2.06	0.22	0.22
query25	0.15	0.08	0.08
query26	0.25	0.18	0.19
query27	0.08	0.08	0.07
query28	13.34	1.01	1.00
query29	12.58	3.35	3.31
query30	0.24	0.06	0.05
query31	2.88	0.41	0.39
query32	3.24	0.48	0.48
query33	3.01	3.03	3.00
query34	17.18	4.43	4.53
query35	4.50	4.48	4.43
query36	0.66	0.49	0.49
query37	0.18	0.16	0.16
query38	0.16	0.14	0.14
query39	0.05	0.04	0.05
query40	0.16	0.12	0.13
query41	0.10	0.05	0.05
query42	0.06	0.05	0.05
query43	0.04	0.04	0.04
Total cold run time: 110.51 s
Total hot run time: 31.02 s

@DarvenDuan
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 43409 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 0720ade702de88db8372be296c11b5ab93957414, data reload: false

------ Round 1 ----------------------------------
q1	18053	7570	7306	7306
q2	2336	197	221	197
q3	11473	1408	1415	1408
q4	10275	987	1039	987
q5	7764	3220	3215	3215
q6	249	157	152	152
q7	1065	666	639	639
q8	9952	2044	2022	2022
q9	6708	6317	6286	6286
q10	7019	2534	2534	2534
q11	442	260	253	253
q12	417	238	239	238
q13	17758	3054	3053	3053
q14	284	247	258	247
q15	601	538	522	522
q16	522	448	430	430
q17	999	960	953	953
q18	7412	6951	6884	6884
q19	1370	1237	1228	1228
q20	609	337	339	337
q21	3951	3556	3569	3556
q22	1051	993	962	962
Total cold run time: 110310 ms
Total hot run time: 43409 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7167	7277	7160	7160
q2	344	234	237	234
q3	2942	2895	2925	2895
q4	1928	1971	1933	1933
q5	5457	5436	5442	5436
q6	237	141	143	141
q7	2093	1691	1734	1691
q8	3248	3400	3320	3320
q9	8434	8387	8487	8387
q10	3407	3473	3455	3455
q11	586	485	471	471
q12	825	605	619	605
q13	6903	3029	3065	3029
q14	298	270	275	270
q15	578	527	523	523
q16	501	463	465	463
q17	1753	1689	1708	1689
q18	8082	7540	7555	7540
q19	1745	1713	1710	1710
q20	2036	1782	1793	1782
q21	5813	5488	5433	5433
q22	1126	990	977	977
Total cold run time: 65503 ms
Total hot run time: 59144 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 194232 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 0720ade702de88db8372be296c11b5ab93957414, data reload: false

query1	930	388	393	388
query2	6532	1828	1800	1800
query3	6657	218	229	218
query4	26128	24236	24149	24149
query5	5176	560	527	527
query6	270	176	161	161
query7	4588	304	309	304
query8	280	216	213	213
query9	8512	2585	2582	2582
query10	485	279	284	279
query11	16178	15533	15501	15501
query12	160	104	109	104
query13	1694	406	390	390
query14	11083	6595	6793	6595
query15	226	172	176	172
query16	7607	472	458	458
query17	1430	563	561	561
query18	1976	292	285	285
query19	187	149	168	149
query20	123	114	111	111
query21	210	109	104	104
query22	4632	4419	4395	4395
query23	34363	33808	33666	33666
query24	10236	3088	3044	3044
query25	658	382	396	382
query26	1370	156	153	153
query27	2957	284	291	284
query28	6916	2089	2066	2066
query29	943	407	412	407
query30	314	165	152	152
query31	983	741	817	741
query32	108	57	64	57
query33	731	300	297	297
query34	915	476	481	476
query35	863	755	735	735
query36	1061	891	905	891
query37	166	84	76	76
query38	4062	3934	3895	3895
query39	1468	1406	1439	1406
query40	268	118	115	115
query41	48	46	45	45
query42	116	98	96	96
query43	496	450	464	450
query44	1241	808	773	773
query45	199	168	172	168
query46	1120	841	819	819
query47	1883	1770	1810	1770
query48	373	292	294	292
query49	1113	459	450	450
query50	907	427	456	427
query51	7032	6808	6812	6808
query52	101	89	114	89
query53	258	183	183	183
query54	759	470	457	457
query55	79	77	78	77
query56	273	266	264	264
query57	1212	1081	1100	1081
query58	252	244	241	241
query59	2899	2449	2693	2449
query60	295	284	287	284
query61	101	98	97	97
query62	933	664	677	664
query63	227	189	194	189
query64	5361	720	667	667
query65	3256	3207	3138	3138
query66	1410	302	302	302
query67	16008	15472	15475	15472
query68	3183	874	852	852
query69	431	327	320	320
query70	1188	1167	1145	1145
query71	361	340	339	339
query72	5980	3448	2632	2632
query73	594	589	577	577
query74	9105	9109	9229	9109
query75	3118	2992	3026	2992
query76	1900	857	846	846
query77	442	397	412	397
query78	9310	9293	9324	9293
query79	920	881	872	872
query80	860	855	844	844
query81	446	274	267	267
query82	268	263	268	263
query83	201	202	202	202
query84	242	116	111	111
query85	718	470	453	453
query86	310	302	324	302
query87	4320	4415	4478	4415
query88	4187	4133	4112	4112
query89	376	371	373	371
query90	1294	332	327	327
query91	138	154	140	140
query92	82	78	80	78
query93	1054	1090	1037	1037
query94	615	367	397	367
query95	474	434	437	434
query96	475	476	477	476
query97	3148	3120	3090	3090
query98	236	234	238	234
query99	1558	1321	1294	1294
Total cold run time: 278932 ms
Total hot run time: 194232 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 31.08 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 0720ade702de88db8372be296c11b5ab93957414, data reload: false

query1	0.05	0.05	0.04
query2	0.07	0.04	0.04
query3	0.23	0.04	0.05
query4	1.67	0.07	0.07
query5	0.50	0.51	0.50
query6	1.13	0.72	0.72
query7	0.02	0.01	0.02
query8	0.06	0.05	0.04
query9	0.58	0.51	0.51
query10	0.58	0.58	0.56
query11	0.16	0.12	0.12
query12	0.14	0.12	0.12
query13	0.62	0.62	0.61
query14	1.44	1.46	1.51
query15	0.90	0.88	0.89
query16	0.36	0.36	0.35
query17	1.03	1.03	1.04
query18	0.18	0.16	0.16
query19	1.93	1.84	1.84
query20	0.01	0.02	0.01
query21	15.48	0.69	0.68
query22	4.08	7.53	1.20
query23	17.82	1.31	1.35
query24	2.22	0.22	0.23
query25	0.19	0.09	0.08
query26	0.29	0.19	0.19
query27	0.07	0.08	0.07
query28	13.18	1.13	1.11
query29	12.54	3.39	3.36
query30	0.24	0.05	0.06
query31	2.89	0.43	0.41
query32	3.22	0.50	0.49
query33	3.05	3.06	3.07
query34	15.42	4.36	4.30
query35	4.33	4.33	4.33
query36	0.68	0.49	0.50
query37	0.19	0.16	0.16
query38	0.17	0.15	0.16
query39	0.05	0.04	0.04
query40	0.17	0.14	0.13
query41	0.09	0.06	0.05
query42	0.05	0.05	0.05
query43	0.05	0.04	0.05
Total cold run time: 108.13 s
Total hot run time: 31.08 s

@DarvenDuan
Copy link
Contributor Author

run feut

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants