Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[fix](mtmv)partition limit #32978

Merged
merged 1 commit into from
Mar 29, 2024
Merged

[fix](mtmv)partition limit #32978

merged 1 commit into from
Mar 29, 2024

Conversation

zddr
Copy link
Contributor

@zddr zddr commented Mar 28, 2024

Proposed changes

Issue Number: close #xxx

when base table has three partitions

[("2024-03-26"),("2024-03-27")),
[("2024-03-27"),("2024-03-28")),
[("2024-03-28"),("2024-03-29"))

current time is 2024-03-28 xx:xx:xx
when mtmv set 'partition_sync_limit'='1'
mtmv should have only one partition

[("2024-03-28"),("2024-03-29"))

but current have two

[("2024-03-27"),("2024-03-28")),
[("2024-03-28"),("2024-03-29"))

Further comments

If this is a relatively large or complex change, kick off the discussion at [email protected] by explaining why you chose the solution you did and what alternatives you considered, etc...

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@zddr zddr marked this pull request as draft March 28, 2024 10:26
@zddr zddr marked this pull request as ready for review March 28, 2024 10:35
@zddr
Copy link
Contributor Author

zddr commented Mar 28, 2024

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 37451 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit b55598a9431969c0722fa7137c842a80c355ff6a, data reload: false

------ Round 1 ----------------------------------
q1	17602	4162	4070	4070
q2	2105	152	161	152
q3	10578	1133	1182	1133
q4	10240	780	795	780
q5	7449	2860	2881	2860
q6	199	122	125	122
q7	1048	580	563	563
q8	9320	1978	1967	1967
q9	6907	6345	6319	6319
q10	8447	3460	3643	3460
q11	423	224	215	215
q12	420	199	202	199
q13	17801	2871	2868	2868
q14	239	203	204	203
q15	507	477	458	458
q16	484	366	364	364
q17	942	615	558	558
q18	6990	6493	6384	6384
q19	3539	1378	1357	1357
q20	556	251	260	251
q21	3462	3007	2885	2885
q22	351	283	309	283
Total cold run time: 109609 ms
Total hot run time: 37451 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4146	4072	4112	4072
q2	327	226	231	226
q3	2940	2843	2822	2822
q4	1810	1564	1524	1524
q5	5176	5197	5251	5197
q6	206	114	116	114
q7	2158	1831	1815	1815
q8	3157	3254	3253	3253
q9	8465	8326	8340	8326
q10	3750	3793	4184	3793
q11	558	455	445	445
q12	758	556	592	556
q13	17023	3051	3021	3021
q14	308	258	279	258
q15	511	496	479	479
q16	509	470	454	454
q17	1785	1534	1490	1490
q18	8031	7733	7681	7681
q19	1627	1579	1543	1543
q20	2032	1871	1771	1771
q21	5066	4865	4847	4847
q22	533	474	459	459
Total cold run time: 70876 ms
Total hot run time: 54146 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 181923 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit b55598a9431969c0722fa7137c842a80c355ff6a, data reload: false

query1	919	354	344	344
query2	6742	2006	1927	1927
query3	6696	207	211	207
query4	31622	21311	21258	21258
query5	4358	396	400	396
query6	278	179	178	178
query7	4629	291	288	288
query8	237	173	183	173
query9	9362	2311	2253	2253
query10	441	252	264	252
query11	17104	14269	14213	14213
query12	135	87	86	86
query13	1621	421	422	421
query14	9353	8263	7437	7437
query15	254	187	180	180
query16	8256	247	252	247
query17	2032	559	540	540
query18	2097	278	273	273
query19	369	149	157	149
query20	92	89	89	89
query21	205	127	129	127
query22	4973	4772	4796	4772
query23	33484	32606	32938	32606
query24	10763	3051	2950	2950
query25	634	409	394	394
query26	1136	180	159	159
query27	2804	360	363	360
query28	7630	1972	1877	1877
query29	912	645	634	634
query30	320	154	165	154
query31	1043	758	764	758
query32	94	62	68	62
query33	762	255	249	249
query34	1161	510	507	507
query35	945	727	711	711
query36	1037	889	902	889
query37	123	65	68	65
query38	3749	3665	3716	3665
query39	1617	1625	1654	1625
query40	191	105	115	105
query41	49	46	48	46
query42	109	97	100	97
query43	489	467	469	467
query44	1188	747	750	747
query45	284	259	267	259
query46	1114	725	714	714
query47	2046	1953	1934	1934
query48	457	359	353	353
query49	933	353	341	341
query50	789	386	390	386
query51	6955	6794	6802	6794
query52	116	93	89	89
query53	354	282	274	274
query54	314	269	238	238
query55	85	80	77	77
query56	259	241	230	230
query57	1280	1173	1174	1173
query58	237	227	212	212
query59	3118	2833	2608	2608
query60	262	237	256	237
query61	105	102	91	91
query62	601	437	431	431
query63	299	276	265	265
query64	5311	3935	4036	3935
query65	3052	3001	3010	3001
query66	790	344	353	344
query67	15455	15130	15181	15130
query68	8767	517	530	517
query69	622	377	365	365
query70	1267	1183	1117	1117
query71	498	258	253	253
query72	6287	2532	2351	2351
query73	799	313	320	313
query74	6900	6410	6318	6318
query75	3777	2196	2186	2186
query76	5245	839	833	833
query77	597	263	249	249
query78	10833	10117	10198	10117
query79	11218	514	529	514
query80	2135	376	368	368
query81	518	209	216	209
query82	695	85	83	83
query83	203	146	140	140
query84	285	77	74	74
query85	1327	307	308	307
query86	398	313	295	295
query87	3675	3494	3531	3494
query88	4861	2325	2315	2315
query89	498	359	364	359
query90	1905	177	174	174
query91	167	136	134	134
query92	61	46	47	46
query93	7221	498	476	476
query94	1074	176	174	174
query95	399	305	305	305
query96	603	267	268	267
query97	2673	2478	2466	2466
query98	233	209	205	205
query99	1139	846	879	846
Total cold run time: 313111 ms
Total hot run time: 181923 ms

@doris-robot
Copy link

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit b55598a9431969c0722fa7137c842a80c355ff6a with default session variables
Stream load json:         18 seconds loaded 2358488459 Bytes, about 124 MB/s
Stream load orc:          59 seconds loaded 1101869774 Bytes, about 17 MB/s
Stream load parquet:      31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select:       13.7 seconds inserted 10000000 Rows, about 729K ops/s

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Mar 29, 2024
Copy link
Contributor

PR approved by at least one committer and no changes requested.

Copy link
Contributor

PR approved by anyone and no changes requested.

@zddr zddr changed the title [fix]mtmv partition limit [fix](mtmv)partition limit Mar 29, 2024
@CalvinKirs CalvinKirs merged commit 97123cf into apache:master Mar 29, 2024
28 of 34 checks passed
yiguolei pushed a commit that referenced this pull request Apr 1, 2024
yiguolei pushed a commit that referenced this pull request Apr 10, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by one committer. dev/2.1.3-merged reviewed
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants