Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[fix](cluster key) Cluster key skip non vertical compact #41413

Open
wants to merge 2 commits into
base: master
Choose a base branch
from

Conversation

mymeiyi
Copy link
Contributor

@mymeiyi mymeiyi commented Sep 27, 2024

now the cluster key does not support non vertical compaction.
#41208 try to implement it, but there are some problems, so we skip non vertical compaction for cluster key

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@mymeiyi
Copy link
Contributor Author

mymeiyi commented Sep 27, 2024

run buildall

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 37.30% (9627/25811)
Line Coverage: 28.70% (79668/277598)
Region Coverage: 28.13% (41192/146446)
Branch Coverage: 24.74% (20974/84762)
Coverage Report: http://coverage.selectdb-in.cc/coverage/a8c8b5df02569837e91c184417dbfc5276f6d9d4_a8c8b5df02569837e91c184417dbfc5276f6d9d4/report/index.html

@doris-robot
Copy link

TPC-H: Total hot run time: 41004 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit a8c8b5df02569837e91c184417dbfc5276f6d9d4, data reload: false

------ Round 1 ----------------------------------
q1	17586	7565	7328	7328
q2	2023	272	275	272
q3	12158	1101	1174	1101
q4	10562	722	782	722
q5	7755	2885	2809	2809
q6	237	156	154	154
q7	982	628	617	617
q8	9387	1941	1976	1941
q9	7564	6437	6415	6415
q10	6942	2276	2319	2276
q11	434	242	245	242
q12	429	219	221	219
q13	17786	2968	2965	2965
q14	246	209	212	209
q15	566	520	524	520
q16	658	588	594	588
q17	965	538	554	538
q18	7246	6675	6691	6675
q19	1386	1013	1124	1013
q20	466	196	204	196
q21	4005	3245	3214	3214
q22	1110	990	992	990
Total cold run time: 110493 ms
Total hot run time: 41004 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7312	7277	7312	7277
q2	334	226	234	226
q3	3070	2917	2881	2881
q4	2086	1825	1813	1813
q5	5810	5764	5747	5747
q6	245	142	145	142
q7	2206	1863	1772	1772
q8	3369	3562	3441	3441
q9	8931	8971	8854	8854
q10	3593	3588	3536	3536
q11	579	473	490	473
q12	825	586	668	586
q13	9996	3209	3194	3194
q14	296	281	283	281
q15	572	529	526	526
q16	693	659	648	648
q17	1856	1611	1600	1600
q18	8140	7764	7603	7603
q19	1696	1497	1548	1497
q20	2086	1870	1902	1870
q21	5584	5388	5510	5388
q22	1150	1039	1065	1039
Total cold run time: 70429 ms
Total hot run time: 60394 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 191675 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit a8c8b5df02569837e91c184417dbfc5276f6d9d4, data reload: false

query1	916	396	396	396
query2	6257	2074	2076	2074
query3	8677	195	205	195
query4	33577	23577	23486	23486
query5	3535	471	454	454
query6	266	165	168	165
query7	4201	302	312	302
query8	288	221	233	221
query9	9261	2670	2670	2670
query10	475	279	285	279
query11	17921	15289	15224	15224
query12	152	100	96	96
query13	1543	418	404	404
query14	9635	7544	7493	7493
query15	242	170	175	170
query16	7924	481	479	479
query17	1683	628	611	611
query18	1900	313	323	313
query19	388	165	159	159
query20	120	116	119	116
query21	223	102	112	102
query22	4905	4536	4503	4503
query23	35007	34130	33879	33879
query24	11011	2847	2815	2815
query25	639	410	404	404
query26	1135	158	156	156
query27	2273	298	298	298
query28	7502	2409	2390	2390
query29	796	434	423	423
query30	313	153	155	153
query31	1076	746	825	746
query32	99	57	58	57
query33	771	306	302	302
query34	925	503	518	503
query35	855	769	734	734
query36	1115	956	938	938
query37	150	86	85	85
query38	4004	3925	3891	3891
query39	1491	1432	1408	1408
query40	210	97	97	97
query41	52	47	49	47
query42	124	94	97	94
query43	547	465	473	465
query44	1217	799	800	799
query45	194	166	171	166
query46	1162	706	709	706
query47	1924	1777	1804	1777
query48	464	371	375	371
query49	969	413	407	407
query50	826	419	418	418
query51	7142	6937	6860	6860
query52	102	89	86	86
query53	247	181	184	181
query54	1214	471	468	468
query55	82	79	79	79
query56	299	247	248	247
query57	1187	1090	1080	1080
query58	234	217	227	217
query59	3172	2910	2963	2910
query60	289	268	276	268
query61	108	100	108	100
query62	865	661	670	661
query63	216	187	184	184
query64	4002	633	620	620
query65	3242	3160	3197	3160
query66	837	305	307	305
query67	15837	15509	15485	15485
query68	4596	562	554	554
query69	540	294	295	294
query70	1181	1118	1120	1118
query71	356	267	269	267
query72	7254	4114	4013	4013
query73	764	338	342	338
query74	10236	8950	8989	8950
query75	3469	2691	2642	2642
query76	2803	911	932	911
query77	617	299	294	294
query78	10619	9606	9513	9513
query79	2572	596	620	596
query80	2477	449	457	449
query81	585	239	239	239
query82	676	134	141	134
query83	325	133	139	133
query84	284	78	80	78
query85	1619	301	297	297
query86	456	304	297	297
query87	4425	4182	4306	4182
query88	4102	2390	2459	2390
query89	409	289	288	288
query90	2115	187	184	184
query91	179	141	163	141
query92	69	48	48	48
query93	2250	536	541	536
query94	1185	290	279	279
query95	352	254	254	254
query96	633	278	279	278
query97	3239	3126	3160	3126
query98	221	197	196	196
query99	1599	1304	1304	1304
Total cold run time: 302222 ms
Total hot run time: 191675 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 32.22 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit a8c8b5df02569837e91c184417dbfc5276f6d9d4, data reload: false

query1	0.05	0.05	0.04
query2	0.06	0.02	0.02
query3	0.23	0.06	0.06
query4	1.64	0.10	0.09
query5	0.51	0.52	0.53
query6	1.14	0.72	0.73
query7	0.02	0.01	0.01
query8	0.04	0.04	0.03
query9	0.59	0.52	0.51
query10	0.58	0.58	0.56
query11	0.14	0.11	0.11
query12	0.14	0.11	0.12
query13	0.61	0.60	0.60
query14	2.75	2.74	2.72
query15	0.90	0.83	0.82
query16	0.40	0.39	0.38
query17	1.04	1.04	1.00
query18	0.20	0.20	0.19
query19	1.87	1.82	2.03
query20	0.01	0.01	0.01
query21	15.35	0.55	0.55
query22	3.03	1.69	1.72
query23	17.21	0.97	0.78
query24	2.46	0.73	1.00
query25	0.22	0.08	0.17
query26	0.32	0.15	0.14
query27	0.05	0.05	0.04
query28	11.33	1.10	1.07
query29	12.52	3.24	3.23
query30	0.25	0.06	0.07
query31	2.88	0.39	0.38
query32	3.27	0.48	0.47
query33	3.04	3.02	3.07
query34	17.01	4.48	4.45
query35	4.52	4.44	4.50
query36	0.65	0.48	0.49
query37	0.08	0.06	0.06
query38	0.05	0.04	0.03
query39	0.02	0.02	0.02
query40	0.16	0.13	0.12
query41	0.07	0.02	0.02
query42	0.04	0.02	0.02
query43	0.03	0.04	0.02
Total cold run time: 107.48 s
Total hot run time: 32.22 s

@mymeiyi
Copy link
Contributor Author

mymeiyi commented Sep 29, 2024

run p0

@mymeiyi mymeiyi force-pushed the cluster-key-skip-non-vertical-compact branch from a8c8b5d to 2a99934 Compare September 29, 2024 03:00
@mymeiyi
Copy link
Contributor Author

mymeiyi commented Sep 29, 2024

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 40439 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 2a99934605fc0b28325fb33db5004575af9c6579, data reload: false

------ Round 1 ----------------------------------
q1	17588	7346	7230	7230
q2	2031	272	266	266
q3	12150	1058	1129	1058
q4	10574	729	688	688
q5	7772	2861	2764	2764
q6	239	152	148	148
q7	962	624	610	610
q8	9341	1913	1980	1913
q9	6546	6483	6363	6363
q10	6948	2315	2308	2308
q11	441	238	247	238
q12	425	223	216	216
q13	17783	2968	2985	2968
q14	245	206	204	204
q15	571	514	520	514
q16	658	587	584	584
q17	960	552	551	551
q18	7094	6727	6591	6591
q19	1401	960	1135	960
q20	501	202	194	194
q21	4022	3216	3082	3082
q22	1109	997	989	989
Total cold run time: 109361 ms
Total hot run time: 40439 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7243	7255	7258	7255
q2	326	230	232	230
q3	2903	2943	2959	2943
q4	2077	1860	1787	1787
q5	5804	5725	5741	5725
q6	228	141	139	139
q7	2256	1849	1846	1846
q8	3329	3563	3445	3445
q9	8936	8843	8835	8835
q10	3584	3579	3464	3464
q11	583	484	510	484
q12	824	671	686	671
q13	10046	3232	3153	3153
q14	319	289	267	267
q15	574	520	524	520
q16	686	642	642	642
q17	1867	1645	1596	1596
q18	8273	7665	7560	7560
q19	1704	1395	1458	1395
q20	2102	1882	1913	1882
q21	5523	5336	5449	5336
q22	1141	1073	1073	1073
Total cold run time: 70328 ms
Total hot run time: 60248 ms

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 37.30% (9627/25811)
Line Coverage: 28.69% (79667/277643)
Region Coverage: 28.13% (41205/146481)
Branch Coverage: 24.75% (20982/84792)
Coverage Report: http://coverage.selectdb-in.cc/coverage/2a99934605fc0b28325fb33db5004575af9c6579_2a99934605fc0b28325fb33db5004575af9c6579/report/index.html

@doris-robot
Copy link

TPC-DS: Total hot run time: 191392 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 2a99934605fc0b28325fb33db5004575af9c6579, data reload: false

query1	967	389	400	389
query2	6240	2180	2038	2038
query3	8680	200	203	200
query4	33778	23863	23338	23338
query5	3970	471	457	457
query6	264	165	160	160
query7	4184	297	313	297
query8	275	251	220	220
query9	9470	2673	2670	2670
query10	490	285	289	285
query11	17837	15129	15174	15129
query12	143	102	98	98
query13	1529	410	415	410
query14	10180	7359	7504	7359
query15	260	168	184	168
query16	7693	444	508	444
query17	1654	620	592	592
query18	1497	312	309	309
query19	392	161	154	154
query20	124	111	115	111
query21	209	104	103	103
query22	4882	4461	4609	4461
query23	34963	33805	33754	33754
query24	11047	2832	2853	2832
query25	590	418	417	417
query26	986	162	164	162
query27	2228	308	300	300
query28	7107	2437	2425	2425
query29	712	444	425	425
query30	293	149	146	146
query31	1056	803	802	802
query32	97	59	58	58
query33	770	297	303	297
query34	925	510	481	481
query35	885	714	745	714
query36	1133	961	952	952
query37	150	84	90	84
query38	4114	3976	3926	3926
query39	1509	1461	1421	1421
query40	205	94	95	94
query41	51	49	48	48
query42	117	95	95	95
query43	539	501	524	501
query44	1252	825	803	803
query45	194	162	171	162
query46	1148	700	715	700
query47	1922	1892	1855	1855
query48	482	374	386	374
query49	978	402	415	402
query50	817	408	406	406
query51	6921	6902	6873	6873
query52	96	85	87	85
query53	247	180	176	176
query54	1158	466	463	463
query55	81	81	76	76
query56	299	264	273	264
query57	1212	1096	1123	1096
query58	240	228	233	228
query59	3230	2952	2960	2952
query60	293	269	258	258
query61	98	98	111	98
query62	861	653	666	653
query63	211	182	183	182
query64	4028	637	619	619
query65	3216	3199	3168	3168
query66	791	289	296	289
query67	15951	15645	15537	15537
query68	4403	575	569	569
query69	508	292	290	290
query70	1201	1144	1117	1117
query71	362	272	269	269
query72	6994	4076	3954	3954
query73	765	338	348	338
query74	10323	9093	8997	8997
query75	3433	2683	2605	2605
query76	2947	889	946	889
query77	631	295	305	295
query78	10472	9678	9553	9553
query79	2140	600	597	597
query80	2420	431	456	431
query81	598	238	235	235
query82	697	136	136	136
query83	297	131	136	131
query84	280	72	79	72
query85	1648	288	291	288
query86	460	294	302	294
query87	4458	4387	4366	4366
query88	3693	2397	2355	2355
query89	400	288	285	285
query90	2140	181	183	181
query91	180	142	143	142
query92	69	49	47	47
query93	2151	546	558	546
query94	1229	298	290	290
query95	357	259	252	252
query96	613	282	280	280
query97	3243	3078	3079	3078
query98	218	195	186	186
query99	1675	1298	1279	1279
Total cold run time: 300926 ms
Total hot run time: 191392 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 31.64 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 2a99934605fc0b28325fb33db5004575af9c6579, data reload: false

query1	0.05	0.05	0.04
query2	0.06	0.02	0.03
query3	0.23	0.06	0.06
query4	1.64	0.11	0.10
query5	0.54	0.51	0.51
query6	1.12	0.72	0.72
query7	0.02	0.02	0.01
query8	0.04	0.03	0.03
query9	0.56	0.50	0.50
query10	0.56	0.58	0.55
query11	0.13	0.11	0.10
query12	0.14	0.11	0.10
query13	0.62	0.59	0.59
query14	2.74	2.74	2.73
query15	0.88	0.81	0.82
query16	0.37	0.39	0.39
query17	1.01	1.05	1.04
query18	0.20	0.20	0.19
query19	1.95	1.81	1.99
query20	0.02	0.01	0.01
query21	15.40	0.59	0.59
query22	2.64	2.82	1.76
query23	17.15	0.93	0.76
query24	2.99	1.28	0.13
query25	0.15	0.13	0.03
query26	0.33	0.14	0.14
query27	0.05	0.05	0.05
query28	11.61	1.09	1.07
query29	12.60	3.25	3.20
query30	0.24	0.06	0.06
query31	2.87	0.38	0.38
query32	3.29	0.47	0.46
query33	3.01	3.02	2.99
query34	16.83	4.48	4.44
query35	4.54	4.49	4.54
query36	0.69	0.47	0.47
query37	0.08	0.06	0.06
query38	0.05	0.04	0.04
query39	0.04	0.02	0.02
query40	0.15	0.13	0.12
query41	0.08	0.02	0.02
query42	0.03	0.02	0.02
query43	0.03	0.03	0.04
Total cold run time: 107.73 s
Total hot run time: 31.64 s

@mymeiyi mymeiyi force-pushed the cluster-key-skip-non-vertical-compact branch from 2a99934 to c964ec2 Compare September 29, 2024 08:31
@mymeiyi
Copy link
Contributor Author

mymeiyi commented Sep 29, 2024

run buildall

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 37.29% (9627/25815)
Line Coverage: 28.68% (79668/277756)
Region Coverage: 28.12% (41204/146533)
Branch Coverage: 24.73% (20979/84818)
Coverage Report: http://coverage.selectdb-in.cc/coverage/c964ec27f5bfd0b2f89c8a6fbbbb7478b956a68e_c964ec27f5bfd0b2f89c8a6fbbbb7478b956a68e/report/index.html

@mymeiyi mymeiyi force-pushed the cluster-key-skip-non-vertical-compact branch from c964ec2 to 3d39d0e Compare October 10, 2024 03:44
@mymeiyi
Copy link
Contributor Author

mymeiyi commented Oct 10, 2024

run buildall

Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 37.25% (9646/25894)
Line Coverage: 28.55% (79965/280118)
Region Coverage: 27.99% (41346/147742)
Branch Coverage: 24.60% (21051/85590)
Coverage Report: http://coverage.selectdb-in.cc/coverage/3d39d0ea471cdcc46e139ff0472e8ca1d980e6e8_3d39d0ea471cdcc46e139ff0472e8ca1d980e6e8/report/index.html

@mymeiyi
Copy link
Contributor Author

mymeiyi commented Oct 10, 2024

run performance

Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions github-actions bot added approved Indicates a PR has been approved by one committer. reviewed labels Oct 10, 2024
Copy link
Contributor

PR approved by anyone and no changes requested.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by one committer. reviewed
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants