Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Fix](partial update) Fix partial update failed when merge_type=MERGE #40730

Merged
merged 2 commits into from
Sep 18, 2024

Conversation

bobhan1
Copy link
Contributor

@bobhan1 bobhan1 commented Sep 12, 2024

branch-2.1-pick: #40951
branch-2.0-pick: #40955

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@bobhan1
Copy link
Contributor Author

bobhan1 commented Sep 12, 2024

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 39819 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 8a08a5f47a07dd35bd61a580ddda739f7a4bd7a0, data reload: false

------ Round 1 ----------------------------------
q1	17633	4374	4182	4182
q2	2027	191	191	191
q3	10565	1256	1446	1256
q4	10525	983	975	975
q5	7717	3170	3155	3155
q6	231	138	138	138
q7	1058	636	620	620
q8	9482	2044	2031	2031
q9	6803	6343	6365	6343
q10	7029	2480	2490	2480
q11	434	245	251	245
q12	416	227	230	227
q13	17770	3033	3019	3019
q14	280	252	246	246
q15	538	494	491	491
q16	535	427	434	427
q17	1000	959	941	941
q18	7409	6752	6880	6752
q19	1397	1269	1249	1249
q20	601	338	331	331
q21	3923	3555	3538	3538
q22	1123	1013	982	982
Total cold run time: 108496 ms
Total hot run time: 39819 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4185	4182	4205	4182
q2	343	235	232	232
q3	2941	2917	2874	2874
q4	1967	1955	1987	1955
q5	5766	5717	5824	5717
q6	224	139	133	133
q7	2163	1780	1773	1773
q8	3376	3449	3474	3449
q9	8836	8817	8785	8785
q10	3536	3663	3613	3613
q11	589	471	464	464
q12	807	628	630	628
q13	7785	3212	3183	3183
q14	317	313	287	287
q15	537	511	507	507
q16	494	489	468	468
q17	1806	1729	1751	1729
q18	8452	8187	8057	8057
q19	1797	1765	1748	1748
q20	2137	1896	1886	1886
q21	5952	5789	5739	5739
q22	1126	982	1000	982
Total cold run time: 65136 ms
Total hot run time: 58391 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 199247 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 8a08a5f47a07dd35bd61a580ddda739f7a4bd7a0, data reload: false

query1	1286	877	905	877
query2	6201	1684	1708	1684
query3	10752	4007	3946	3946
query4	58080	26179	23592	23592
query5	5102	537	528	528
query6	320	161	171	161
query7	5623	304	288	288
query8	278	222	225	222
query9	6113	2455	2441	2441
query10	416	304	274	274
query11	15887	15372	15500	15372
query12	149	102	103	102
query13	1468	396	427	396
query14	10386	7297	7366	7297
query15	232	179	184	179
query16	6881	521	523	521
query17	1154	616	605	605
query18	1644	314	305	305
query19	208	160	164	160
query20	130	120	114	114
query21	208	106	106	106
query22	4814	4662	4828	4662
query23	34496	33985	33726	33726
query24	6156	3129	3029	3029
query25	528	413	411	411
query26	612	157	156	156
query27	1577	284	282	282
query28	2790	2042	2005	2005
query29	668	432	432	432
query30	222	159	162	159
query31	962	801	803	801
query32	76	56	56	56
query33	444	307	311	307
query34	893	484	483	483
query35	867	724	729	724
query36	1049	896	874	874
query37	142	88	83	83
query38	4102	3918	3924	3918
query39	1462	1436	1415	1415
query40	208	118	117	117
query41	49	48	48	48
query42	126	106	99	99
query43	479	434	434	434
query44	1145	771	755	755
query45	202	169	174	169
query46	1100	843	834	834
query47	1902	1822	1798	1798
query48	360	288	290	288
query49	743	456	458	456
query50	939	457	449	449
query51	6947	6852	6856	6852
query52	96	89	88	88
query53	253	180	176	176
query54	566	453	464	453
query55	76	73	78	73
query56	288	267	276	267
query57	1215	1081	1114	1081
query58	230	242	250	242
query59	2624	2420	2507	2420
query60	285	280	274	274
query61	102	115	102	102
query62	801	665	668	665
query63	220	194	193	193
query64	1606	668	710	668
query65	3257	3166	3171	3166
query66	676	345	367	345
query67	16033	15545	15541	15541
query68	2022	875	861	861
query69	424	327	327	327
query70	1142	1040	1145	1040
query71	361	373	341	341
query72	5961	3394	3406	3394
query73	588	570	582	570
query74	9261	9267	9018	9018
query75	3108	2994	3022	2994
query76	956	850	849	849
query77	520	403	411	403
query78	9607	9742	9296	9296
query79	929	900	886	886
query80	835	864	802	802
query81	459	258	266	258
query82	266	262	268	262
query83	185	188	184	184
query84	264	108	106	106
query85	657	402	394	394
query86	322	316	325	316
query87	4350	4447	4387	4387
query88	4094	4032	4016	4016
query89	371	359	366	359
query90	1793	313	306	306
query91	118	124	120	120
query92	79	73	76	73
query93	1050	1043	1043	1043
query94	641	390	379	379
query95	448	419	425	419
query96	468	469	464	464
query97	3122	3090	3138	3090
query98	235	232	229	229
query99	1681	1315	1323	1315
Total cold run time: 293223 ms
Total hot run time: 199247 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.85 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 8a08a5f47a07dd35bd61a580ddda739f7a4bd7a0, data reload: false

query1	0.05	0.04	0.04
query2	0.08	0.04	0.04
query3	0.22	0.05	0.04
query4	1.67	0.07	0.06
query5	0.50	0.50	0.50
query6	1.13	0.74	0.72
query7	0.02	0.02	0.02
query8	0.06	0.05	0.05
query9	0.58	0.51	0.53
query10	0.56	0.58	0.56
query11	0.16	0.12	0.12
query12	0.16	0.12	0.12
query13	0.62	0.61	0.61
query14	1.50	1.48	1.42
query15	0.90	0.87	0.88
query16	0.37	0.36	0.36
query17	1.04	1.05	1.03
query18	0.16	0.16	0.17
query19	1.89	1.83	1.83
query20	0.01	0.01	0.01
query21	15.39	0.68	0.68
query22	4.15	8.80	1.03
query23	17.84	1.32	1.34
query24	2.27	0.23	0.22
query25	0.18	0.08	0.09
query26	0.29	0.18	0.18
query27	0.07	0.08	0.08
query28	13.17	1.01	0.98
query29	12.58	3.35	3.35
query30	0.25	0.05	0.07
query31	2.86	0.43	0.41
query32	3.23	0.50	0.50
query33	3.06	3.03	3.06
query34	15.47	4.36	4.36
query35	4.37	4.39	4.40
query36	0.68	0.50	0.49
query37	0.19	0.18	0.17
query38	0.17	0.15	0.14
query39	0.04	0.04	0.04
query40	0.16	0.14	0.13
query41	0.10	0.05	0.05
query42	0.06	0.06	0.06
query43	0.05	0.04	0.04
Total cold run time: 108.31 s
Total hot run time: 30.85 s

@bobhan1 bobhan1 force-pushed the fix-partial-update-merge-type branch from 8a08a5f to b170f33 Compare September 12, 2024 11:40
@bobhan1
Copy link
Contributor Author

bobhan1 commented Sep 12, 2024

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 42687 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit b170f3380f70d7ab63f42a905f007e61df7064ff, data reload: false

------ Round 1 ----------------------------------
q1	17603	7253	7238	7238
q2	2020	186	182	182
q3	10769	1250	1384	1250
q4	10460	891	975	891
q5	7722	3131	3129	3129
q6	238	150	148	148
q7	1031	625	605	605
q8	9453	2021	2010	2010
q9	6769	6277	6311	6277
q10	7012	2549	2508	2508
q11	436	254	252	252
q12	398	229	229	229
q13	17749	3018	3028	3018
q14	294	258	260	258
q15	590	526	519	519
q16	526	429	427	427
q17	987	963	955	955
q18	7464	6763	6813	6763
q19	1383	1250	1231	1231
q20	614	337	333	333
q21	3892	3504	3481	3481
q22	1049	992	983	983
Total cold run time: 108459 ms
Total hot run time: 42687 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7184	7147	7112	7112
q2	340	235	244	235
q3	3035	3059	3028	3028
q4	2066	2142	2094	2094
q5	5634	5599	5648	5599
q6	240	147	155	147
q7	2163	1776	1773	1773
q8	3377	3423	3417	3417
q9	8812	8975	8783	8783
q10	3533	3557	3594	3557
q11	585	487	484	484
q12	824	594	651	594
q13	9059	3220	3161	3161
q14	308	291	287	287
q15	571	537	543	537
q16	520	455	462	455
q17	1776	1765	1709	1709
q18	8540	7939	8106	7939
q19	1774	1762	1749	1749
q20	2095	1877	1875	1875
q21	5911	5576	5613	5576
q22	1161	1041	999	999
Total cold run time: 69508 ms
Total hot run time: 61110 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 199267 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit b170f3380f70d7ab63f42a905f007e61df7064ff, data reload: false

query1	1275	867	887	867
query2	6228	1831	1861	1831
query3	10693	3804	3914	3804
query4	55060	26613	23605	23605
query5	5007	550	515	515
query6	303	165	164	164
query7	5576	312	305	305
query8	279	223	214	214
query9	5944	2598	2595	2595
query10	397	285	272	272
query11	15972	15592	15580	15580
query12	162	101	99	99
query13	1388	403	388	388
query14	10810	7184	7253	7184
query15	202	179	192	179
query16	7043	482	478	478
query17	1169	616	606	606
query18	1912	313	296	296
query19	209	158	164	158
query20	132	114	114	114
query21	217	112	115	112
query22	4617	4331	4452	4331
query23	34721	33838	33684	33684
query24	6111	3082	3037	3037
query25	519	416	426	416
query26	607	152	158	152
query27	1584	282	294	282
query28	2802	2110	2084	2084
query29	696	431	439	431
query30	217	156	151	151
query31	969	767	819	767
query32	77	56	58	56
query33	437	295	306	295
query34	891	492	472	472
query35	850	753	746	746
query36	1041	907	897	897
query37	144	84	81	81
query38	3996	4018	3949	3949
query39	1446	1412	1406	1406
query40	207	114	120	114
query41	50	50	47	47
query42	121	96	96	96
query43	491	452	448	448
query44	1180	815	783	783
query45	202	166	172	166
query46	1104	827	806	806
query47	1904	1789	1825	1789
query48	373	306	297	297
query49	748	458	446	446
query50	932	455	458	455
query51	6835	6946	6969	6946
query52	100	86	90	86
query53	249	182	181	181
query54	559	456	466	456
query55	77	73	76	73
query56	286	265	267	265
query57	1211	1091	1095	1091
query58	235	252	238	238
query59	2947	2617	2585	2585
query60	298	273	271	271
query61	98	101	111	101
query62	812	670	669	669
query63	221	182	182	182
query64	1460	659	686	659
query65	3234	3146	3176	3146
query66	637	297	295	295
query67	15762	15399	15326	15326
query68	1967	872	856	856
query69	437	323	326	323
query70	1119	1141	1172	1141
query71	349	346	349	346
query72	5990	3345	3337	3337
query73	600	583	578	578
query74	9196	9144	9005	9005
query75	3068	2992	2999	2992
query76	961	844	843	843
query77	512	399	401	399
query78	9509	9373	9386	9373
query79	908	886	873	873
query80	829	804	809	804
query81	458	305	263	263
query82	264	263	262	262
query83	188	189	186	186
query84	265	104	104	104
query85	646	410	397	397
query86	313	319	313	313
query87	4381	4420	4324	4324
query88	4215	4132	4124	4124
query89	373	362	370	362
query90	1790	311	307	307
query91	123	122	121	121
query92	75	72	76	72
query93	1042	1038	1034	1034
query94	640	362	394	362
query95	452	419	418	418
query96	472	472	471	471
query97	3077	3118	3148	3118
query98	245	271	232	232
query99	1653	1315	1307	1307
Total cold run time: 290153 ms
Total hot run time: 199267 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.96 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit b170f3380f70d7ab63f42a905f007e61df7064ff, data reload: false

query1	0.05	0.05	0.04
query2	0.07	0.04	0.04
query3	0.23	0.05	0.04
query4	1.67	0.07	0.08
query5	0.50	0.49	0.51
query6	1.15	0.74	0.72
query7	0.02	0.02	0.01
query8	0.05	0.04	0.04
query9	0.57	0.51	0.50
query10	0.56	0.57	0.56
query11	0.16	0.12	0.13
query12	0.16	0.13	0.12
query13	0.62	0.60	0.63
query14	1.45	1.48	1.48
query15	0.90	0.87	0.85
query16	0.37	0.38	0.36
query17	1.03	1.05	1.03
query18	0.16	0.16	0.17
query19	1.94	1.76	1.77
query20	0.02	0.01	0.01
query21	15.41	0.67	0.65
query22	4.04	7.60	1.17
query23	17.72	1.43	1.44
query24	2.24	0.22	0.21
query25	0.17	0.08	0.08
query26	0.29	0.17	0.17
query27	0.08	0.08	0.08
query28	13.18	1.13	1.10
query29	12.54	3.41	3.36
query30	0.24	0.05	0.05
query31	2.88	0.42	0.40
query32	3.23	0.49	0.49
query33	3.02	3.10	3.02
query34	15.44	4.33	4.31
query35	4.35	4.33	4.36
query36	0.70	0.49	0.49
query37	0.20	0.16	0.16
query38	0.16	0.15	0.15
query39	0.04	0.04	0.04
query40	0.17	0.15	0.13
query41	0.10	0.05	0.05
query42	0.06	0.05	0.06
query43	0.05	0.04	0.05
Total cold run time: 107.99 s
Total hot run time: 30.96 s

@bobhan1
Copy link
Contributor Author

bobhan1 commented Sep 13, 2024

run p0

Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions github-actions bot added approved Indicates a PR has been approved by one committer. reviewed labels Sep 18, 2024
Copy link
Contributor

PR approved by anyone and no changes requested.

Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by one committer. dev/2.0.15-merged dev/2.1.7-merged reviewed usercase Important user case type label
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants