Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[opt](partial update) Forbid partial update on merge-on-write table with sync mv #40190

Merged

Conversation

bobhan1
Copy link
Contributor

@bobhan1 bobhan1 commented Aug 30, 2024

Proposed changes

doc: apache/doris-website#1103

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@bobhan1 bobhan1 force-pushed the forbid-partial-update-mow-if-exist-sync-mv branch from 043ee14 to 4a70b17 Compare August 30, 2024 08:44
@bobhan1
Copy link
Contributor Author

bobhan1 commented Aug 30, 2024

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 38103 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 4a70b17b4a69d5f287e2604906a70b73f2d85313, data reload: false

------ Round 1 ----------------------------------
q1	17606	4508	4295	4295
q2	2017	184	168	168
q3	11854	927	1065	927
q4	10497	763	754	754
q5	7758	2822	2806	2806
q6	227	137	139	137
q7	980	612	589	589
q8	9331	2053	2095	2053
q9	6972	6545	6539	6539
q10	7056	2201	2257	2201
q11	468	244	246	244
q12	403	233	234	233
q13	18393	3040	3055	3040
q14	270	233	234	233
q15	515	476	481	476
q16	603	517	499	499
q17	984	667	681	667
q18	7275	6894	6968	6894
q19	1385	1046	998	998
q20	674	335	328	328
q21	3874	3066	3028	3028
q22	1108	1036	994	994
Total cold run time: 110250 ms
Total hot run time: 38103 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4405	4301	4260	4260
q2	375	275	278	275
q3	2870	2649	2658	2649
q4	1952	1667	1633	1633
q5	5585	5702	5713	5702
q6	236	139	137	137
q7	2243	1830	1808	1808
q8	3274	3407	3413	3407
q9	8869	8851	8787	8787
q10	3594	3397	3369	3369
q11	637	516	507	507
q12	817	676	690	676
q13	13788	3151	3241	3151
q14	318	291	292	291
q15	526	501	489	489
q16	609	575	562	562
q17	1844	1557	1554	1554
q18	8096	7879	7749	7749
q19	1747	1618	1611	1611
q20	2172	1952	1905	1905
q21	5792	5389	5561	5389
q22	1121	1050	1035	1035
Total cold run time: 70870 ms
Total hot run time: 56946 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 191755 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 4a70b17b4a69d5f287e2604906a70b73f2d85313, data reload: false

query1	1270	871	856	856
query2	6338	1857	1865	1857
query3	10700	3997	4121	3997
query4	59951	26560	23167	23167
query5	5531	477	472	472
query6	415	173	174	173
query7	5896	299	286	286
query8	284	205	206	205
query9	8949	2461	2459	2459
query10	492	269	256	256
query11	18063	15049	15422	15049
query12	157	104	100	100
query13	1576	405	371	371
query14	11186	6662	7164	6662
query15	233	177	185	177
query16	7592	457	487	457
query17	1174	588	576	576
query18	1973	318	306	306
query19	306	153	154	153
query20	118	120	111	111
query21	210	103	108	103
query22	4704	4454	4477	4454
query23	34318	33189	33140	33140
query24	5916	2822	2834	2822
query25	525	372	379	372
query26	682	158	156	156
query27	1789	273	278	273
query28	3917	2073	2061	2061
query29	676	397	410	397
query30	232	153	154	153
query31	937	766	742	742
query32	81	54	56	54
query33	447	285	291	285
query34	861	480	476	476
query35	835	716	722	716
query36	1105	936	958	936
query37	183	92	92	92
query38	3961	3934	3846	3846
query39	1444	1401	1381	1381
query40	206	113	115	113
query41	47	46	44	44
query42	115	98	98	98
query43	499	458	450	450
query44	1090	737	738	737
query45	197	175	171	171
query46	1094	726	714	714
query47	1883	1809	1793	1793
query48	360	287	304	287
query49	767	420	436	420
query50	811	420	416	416
query51	7173	7013	6945	6945
query52	99	86	86	86
query53	246	179	179	179
query54	574	448	449	448
query55	75	76	75	75
query56	274	261	248	248
query57	1183	1103	1088	1088
query58	220	226	235	226
query59	2960	2817	2738	2738
query60	294	272	278	272
query61	107	98	98	98
query62	761	643	654	643
query63	216	184	187	184
query64	2833	676	653	653
query65	3236	3140	3132	3132
query66	674	344	335	335
query67	15640	15267	15262	15262
query68	2972	563	577	563
query69	409	288	297	288
query70	1171	1150	1170	1150
query71	339	275	278	275
query72	5152	4068	4020	4020
query73	739	325	328	325
query74	9207	8831	9044	8831
query75	3368	2644	2660	2644
query76	1465	1005	976	976
query77	537	325	329	325
query78	9773	9632	9111	9111
query79	1023	548	528	528
query80	681	495	547	495
query81	477	234	231	231
query82	253	146	151	146
query83	171	146	145	145
query84	252	79	73	73
query85	687	292	338	292
query86	306	294	271	271
query87	4411	4262	4244	4244
query88	3278	2335	2332	2332
query89	384	282	285	282
query90	1881	196	196	196
query91	130	102	98	98
query92	60	52	52	52
query93	1050	535	523	523
query94	652	292	298	292
query95	331	271	255	255
query96	591	286	270	270
query97	3180	3085	3047	3047
query98	229	201	209	201
query99	1519	1270	1287	1270
Total cold run time: 305339 ms
Total hot run time: 191755 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 32.11 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 4a70b17b4a69d5f287e2604906a70b73f2d85313, data reload: false

query1	0.05	0.05	0.04
query2	0.08	0.04	0.04
query3	0.23	0.05	0.05
query4	1.67	0.08	0.07
query5	0.52	0.50	0.50
query6	1.13	0.73	0.72
query7	0.02	0.01	0.02
query8	0.05	0.05	0.04
query9	0.55	0.46	0.50
query10	0.55	0.55	0.54
query11	0.16	0.11	0.11
query12	0.15	0.12	0.12
query13	0.61	0.59	0.59
query14	2.04	2.15	2.13
query15	0.89	0.81	0.82
query16	0.37	0.38	0.38
query17	1.01	1.01	1.00
query18	0.20	0.21	0.19
query19	1.93	1.76	1.73
query20	0.01	0.00	0.01
query21	15.41	0.67	0.66
query22	4.66	6.98	1.87
query23	18.31	1.42	1.35
query24	2.18	0.23	0.21
query25	0.16	0.08	0.09
query26	0.26	0.18	0.18
query27	0.07	0.08	0.08
query28	13.22	1.01	1.00
query29	12.63	3.30	3.29
query30	0.24	0.06	0.05
query31	2.89	0.40	0.40
query32	3.23	0.48	0.48
query33	2.96	2.98	3.06
query34	16.84	4.48	4.49
query35	4.49	4.43	4.45
query36	0.66	0.46	0.47
query37	0.19	0.15	0.15
query38	0.14	0.14	0.14
query39	0.04	0.04	0.04
query40	0.17	0.12	0.13
query41	0.10	0.05	0.04
query42	0.06	0.04	0.05
query43	0.04	0.05	0.05
Total cold run time: 111.17 s
Total hot run time: 32.11 s

@bobhan1 bobhan1 force-pushed the forbid-partial-update-mow-if-exist-sync-mv branch from 4a70b17 to c7868fc Compare September 2, 2024 02:06
@bobhan1
Copy link
Contributor Author

bobhan1 commented Sep 2, 2024

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 37973 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit c7868fc8c2948c6561cdbbff8886915a3890ee80, data reload: false

------ Round 1 ----------------------------------
q1	17621	5030	4264	4264
q2	2027	183	177	177
q3	11879	993	1253	993
q4	10527	683	674	674
q5	7766	2860	2820	2820
q6	224	136	138	136
q7	967	613	596	596
q8	9326	2073	2075	2073
q9	7142	6539	6547	6539
q10	7004	2246	2253	2246
q11	451	233	233	233
q12	394	226	228	226
q13	18796	3078	3060	3060
q14	280	227	234	227
q15	525	470	489	470
q16	576	492	509	492
q17	987	675	741	675
q18	7393	6832	6759	6759
q19	1394	1002	997	997
q20	677	326	341	326
q21	4181	3155	2938	2938
q22	1138	1053	1052	1052
Total cold run time: 111275 ms
Total hot run time: 37973 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4356	4290	4305	4290
q2	382	275	264	264
q3	2888	2665	2671	2665
q4	1948	1670	1712	1670
q5	5690	5687	5827	5687
q6	233	131	139	131
q7	2205	1846	1838	1838
q8	3266	3473	3472	3472
q9	8893	8788	8741	8741
q10	3566	3410	3352	3352
q11	600	496	504	496
q12	856	679	646	646
q13	14062	3185	3234	3185
q14	321	293	288	288
q15	534	482	476	476
q16	601	579	544	544
q17	1842	1568	1569	1568
q18	8057	7878	7895	7878
q19	1708	1607	1631	1607
q20	2154	1881	1891	1881
q21	5681	5515	5568	5515
q22	1162	1072	1037	1037
Total cold run time: 71005 ms
Total hot run time: 57231 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 193430 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit c7868fc8c2948c6561cdbbff8886915a3890ee80, data reload: false

query1	1255	893	905	893
query2	6291	1997	1840	1840
query3	10755	4114	4029	4029
query4	60014	24160	23224	23224
query5	5438	504	498	498
query6	422	168	160	160
query7	5790	297	294	294
query8	291	210	210	210
query9	8665	2507	2540	2507
query10	478	272	260	260
query11	17062	15005	15243	15005
query12	173	100	102	100
query13	1563	386	387	386
query14	10764	7053	7607	7053
query15	228	179	182	179
query16	7215	471	464	464
query17	1153	586	579	579
query18	1929	344	309	309
query19	290	155	153	153
query20	122	110	112	110
query21	206	103	106	103
query22	4545	4390	4382	4382
query23	34491	33625	33799	33625
query24	6009	2889	2960	2889
query25	556	396	396	396
query26	687	162	164	162
query27	1773	290	287	287
query28	3845	2083	2073	2073
query29	680	431	434	431
query30	238	154	163	154
query31	949	768	767	767
query32	90	55	59	55
query33	446	290	291	290
query34	858	487	488	487
query35	852	706	719	706
query36	1054	955	933	933
query37	147	97	94	94
query38	4048	3972	3861	3861
query39	1465	1398	1413	1398
query40	203	116	118	116
query41	48	47	46	46
query42	115	96	97	96
query43	511	480	455	455
query44	1094	761	761	761
query45	204	169	169	169
query46	1107	746	745	745
query47	1933	1780	1824	1780
query48	385	296	302	296
query49	764	440	456	440
query50	825	425	434	425
query51	7181	7000	7022	7000
query52	100	90	91	90
query53	251	186	186	186
query54	591	464	475	464
query55	82	76	74	74
query56	285	263	279	263
query57	1218	1095	1058	1058
query58	235	240	239	239
query59	3009	2808	2689	2689
query60	300	295	287	287
query61	123	122	122	122
query62	748	668	662	662
query63	213	190	190	190
query64	2843	765	664	664
query65	3210	3136	3155	3136
query66	626	340	344	340
query67	15557	15314	15479	15314
query68	4577	567	558	558
query69	419	280	280	280
query70	1157	1144	1150	1144
query71	348	275	289	275
query72	6544	4276	4071	4071
query73	758	333	337	333
query74	9212	8832	8877	8832
query75	3393	2713	2705	2705
query76	2037	1042	1034	1034
query77	605	320	328	320
query78	9873	9144	9131	9131
query79	1072	524	528	524
query80	778	509	502	502
query81	533	234	234	234
query82	288	149	147	147
query83	193	149	151	149
query84	257	73	80	73
query85	797	329	283	283
query86	304	309	298	298
query87	4651	4210	4309	4210
query88	3281	2356	2363	2356
query89	386	291	283	283
query90	1908	194	193	193
query91	124	101	100	100
query92	60	53	53	53
query93	1058	542	543	542
query94	699	306	306	306
query95	358	266	260	260
query96	592	268	267	267
query97	3205	3030	3074	3030
query98	225	216	197	197
query99	1543	1287	1296	1287
Total cold run time: 307833 ms
Total hot run time: 193430 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 32.36 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit c7868fc8c2948c6561cdbbff8886915a3890ee80, data reload: false

query1	0.04	0.04	0.04
query2	0.07	0.04	0.03
query3	0.22	0.05	0.05
query4	1.68	0.07	0.07
query5	0.50	0.50	0.50
query6	1.12	0.72	0.73
query7	0.02	0.01	0.01
query8	0.05	0.05	0.05
query9	0.54	0.50	0.48
query10	0.56	0.55	0.54
query11	0.15	0.12	0.11
query12	0.15	0.12	0.13
query13	0.62	0.59	0.59
query14	2.10	2.03	2.06
query15	0.84	0.81	0.82
query16	0.36	0.39	0.36
query17	1.05	0.98	1.07
query18	0.21	0.19	0.20
query19	1.90	1.71	1.78
query20	0.01	0.01	0.01
query21	15.42	0.67	0.67
query22	4.52	6.30	2.50
query23	18.26	1.38	1.23
query24	2.18	0.22	0.21
query25	0.15	0.09	0.07
query26	0.26	0.17	0.18
query27	0.08	0.08	0.08
query28	13.22	1.02	0.99
query29	12.65	3.33	3.29
query30	0.24	0.05	0.05
query31	2.91	0.41	0.39
query32	3.22	0.49	0.47
query33	2.92	2.97	3.03
query34	17.12	4.37	4.48
query35	4.47	4.44	4.47
query36	0.66	0.49	0.48
query37	0.19	0.16	0.15
query38	0.16	0.15	0.15
query39	0.05	0.04	0.04
query40	0.15	0.12	0.13
query41	0.10	0.04	0.05
query42	0.06	0.04	0.05
query43	0.04	0.04	0.05
Total cold run time: 111.22 s
Total hot run time: 32.36 s

@bobhan1
Copy link
Contributor Author

bobhan1 commented Sep 2, 2024

run feut

Copy link
Contributor

@zhannngchen zhannngchen left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Sep 11, 2024
Copy link
Contributor

PR approved by at least one committer and no changes requested.

Copy link
Contributor

PR approved by anyone and no changes requested.

@dataroaring dataroaring merged commit dc2579f into apache:master Sep 19, 2024
27 of 29 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by one committer. dev/2.0.x dev/2.1.x reviewed
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants