Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[fix](transaction) commit txn check txn status #40064

Merged
merged 2 commits into from
Sep 24, 2024
Merged

[fix](transaction) commit txn check txn status #40064

merged 2 commits into from
Sep 24, 2024

Conversation

mymeiyi
Copy link
Contributor

@mymeiyi mymeiyi commented Aug 28, 2024 •
edited
Loading

  1. In checkBeforeUnprotectedCommitTransaction of DatabaseTransactionMgr, if the txn status is not PREPARE, should skip calculate partition version, but the return logic does not work.
  2. the preCommitTransaction2PC in DatabaseTransactionMgr does not hold DatabaseTransactionMgr write lock
  3. the abortTxnWhenCoordinateBeRestart and abortTxnWhenCoordinateBeDown in GlobalTransactionMgr does not hold table write lock
  4. the public void commitTransaction(long dbId, List<Table> tableList, long transactionId, List<TabletCommitInfo> tabletCommitInfos) is only used for UT, it requires the caller hold table write lock, and may wrongly use , mark it as Deprecated.

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@mymeiyi mymeiyi changed the title [fix](transaction) commit txn do more check [fix](transaction) commit txn check txn status Aug 28, 2024
@mymeiyi
Copy link
Contributor Author

mymeiyi commented Aug 29, 2024

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 38130 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 60a22df8fd637aab47d8ca40ae659ba3e61dae2e, data reload: false

------ Round 1 ----------------------------------
q1	17682	4461	4334	4334
q2	2008	186	179	179
q3	11760	955	1007	955
q4	10508	717	802	717
q5	7759	2855	2815	2815
q6	234	138	140	138
q7	960	611	597	597
q8	9305	2053	2082	2053
q9	7047	6527	6531	6527
q10	6996	2207	2288	2207
q11	456	235	237	235
q12	405	226	230	226
q13	17917	3017	3013	3013
q14	283	240	245	240
q15	517	492	494	492
q16	610	498	513	498
q17	996	711	747	711
q18	7170	6976	6795	6795
q19	1390	1067	1074	1067
q20	665	342	339	339
q21	3936	2994	2984	2984
q22	1120	1008	1048	1008
Total cold run time: 109724 ms
Total hot run time: 38130 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4358	4310	4279	4279
q2	384	280	274	274
q3	2885	2631	2649	2631
q4	1923	1672	1640	1640
q5	5549	5698	5639	5639
q6	229	134	137	134
q7	2282	1803	1834	1803
q8	3274	3482	3423	3423
q9	8856	8821	8780	8780
q10	3609	3313	3413	3313
q11	604	519	531	519
q12	845	642	674	642
q13	14805	3275	3165	3165
q14	316	289	289	289
q15	532	489	516	489
q16	645	587	580	580
q17	1828	1546	1519	1519
q18	8124	7727	7836	7727
q19	1714	1633	1554	1554
q20	2131	1935	1973	1935
q21	5777	5406	5506	5406
q22	1140	1056	1078	1056
Total cold run time: 71810 ms
Total hot run time: 56797 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 192049 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 60a22df8fd637aab47d8ca40ae659ba3e61dae2e, data reload: false

query1	1261	899	871	871
query2	6277	1984	1974	1974
query3	10707	4133	4229	4133
query4	60086	23925	23198	23198
query5	5482	511	509	509
query6	419	157	159	157
query7	5770	306	292	292
query8	299	215	219	215
query9	8927	2511	2506	2506
query10	484	290	264	264
query11	17668	15125	15309	15125
query12	152	104	107	104
query13	1565	401	381	381
query14	10435	7349	7350	7349
query15	225	181	187	181
query16	7635	481	508	481
query17	1111	578	588	578
query18	2059	299	303	299
query19	292	156	180	156
query20	122	115	114	114
query21	210	100	103	100
query22	4538	4449	4428	4428
query23	34196	33792	33595	33595
query24	6019	2907	2816	2816
query25	528	377	375	375
query26	689	156	152	152
query27	1804	283	272	272
query28	3936	2120	2105	2105
query29	697	399	400	399
query30	236	155	165	155
query31	906	763	765	763
query32	83	54	53	53
query33	458	283	283	283
query34	859	471	478	471
query35	843	752	720	720
query36	1068	953	940	940
query37	137	95	91	91
query38	3989	3843	3857	3843
query39	1444	1407	1393	1393
query40	204	122	118	118
query41	50	53	53	53
query42	116	99	98	98
query43	539	497	479	479
query44	1081	744	759	744
query45	213	176	174	174
query46	1093	740	724	724
query47	1884	1803	1828	1803
query48	375	293	301	293
query49	775	429	458	429
query50	820	433	409	409
query51	7235	7137	7056	7056
query52	103	89	89	89
query53	253	191	184	184
query54	562	475	463	463
query55	78	76	77	76
query56	278	270	265	265
query57	1218	1062	1048	1048
query58	235	238	219	219
query59	2986	2887	2796	2796
query60	307	265	269	265
query61	101	99	100	99
query62	725	641	653	641
query63	214	184	183	183
query64	2802	704	641	641
query65	3247	3161	3150	3150
query66	675	336	361	336
query67	15308	15151	15220	15151
query68	2961	588	581	581
query69	405	283	282	282
query70	1147	1111	1121	1111
query71	351	286	275	275
query72	6214	2574	3983	2574
query73	758	341	343	341
query74	9140	8791	8792	8791
query75	3395	2644	2667	2644
query76	1409	939	1042	939
query77	588	323	334	323
query78	9570	9158	8962	8962
query79	1035	547	525	525
query80	709	506	512	506
query81	462	238	236	236
query82	239	150	146	146
query83	172	152	149	149
query84	261	74	77	74
query85	672	286	282	282
query86	303	302	301	301
query87	4312	4308	4255	4255
query88	3550	2372	2344	2344
query89	383	285	356	285
query90	1895	204	199	199
query91	127	99	102	99
query92	61	53	53	53
query93	1051	538	548	538
query94	666	301	291	291
query95	336	264	267	264
query96	593	273	272	272
query97	3205	3122	3088	3088
query98	213	210	209	209
query99	1479	1293	1283	1283
Total cold run time: 304755 ms
Total hot run time: 192049 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 32.92 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 60a22df8fd637aab47d8ca40ae659ba3e61dae2e, data reload: false

query1	0.05	0.05	0.04
query2	0.08	0.04	0.05
query3	0.23	0.05	0.04
query4	1.68	0.08	0.08
query5	0.49	0.50	0.48
query6	1.13	0.72	0.73
query7	0.01	0.01	0.02
query8	0.05	0.04	0.05
query9	0.55	0.50	0.50
query10	0.54	0.54	0.54
query11	0.16	0.12	0.12
query12	0.16	0.12	0.12
query13	0.60	0.58	0.58
query14	2.11	2.14	2.12
query15	0.89	0.83	0.82
query16	0.38	0.38	0.37
query17	1.05	0.98	1.06
query18	0.22	0.20	0.20
query19	1.81	1.75	1.75
query20	0.01	0.02	0.01
query21	15.39	0.69	0.67
query22	3.92	6.74	2.55
query23	18.19	1.41	1.32
query24	2.22	0.23	0.22
query25	0.16	0.08	0.08
query26	0.26	0.19	0.18
query27	0.07	0.08	0.07
query28	13.14	1.04	1.00
query29	12.57	3.35	3.38
query30	0.24	0.06	0.06
query31	2.88	0.41	0.39
query32	3.25	0.48	0.48
query33	2.93	3.03	3.02
query34	16.98	4.42	4.45
query35	4.49	4.44	4.44
query36	0.66	0.50	0.49
query37	0.19	0.16	0.17
query38	0.16	0.16	0.16
query39	0.05	0.04	0.04
query40	0.15	0.13	0.13
query41	0.10	0.05	0.06
query42	0.06	0.04	0.04
query43	0.04	0.05	0.04
Total cold run time: 110.3 s
Total hot run time: 32.92 s

@mymeiyi mymeiyi force-pushed the fix-commit-3 branch 2 times, most recently from 75443d4 to 4fb82f4 Compare August 29, 2024 07:06
@mymeiyi
Copy link
Contributor Author

mymeiyi commented Aug 29, 2024

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 38456 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 4fb82f426b74f6a2fa85bc889a1b3a7bbbf279f6, data reload: false

------ Round 1 ----------------------------------
q1	17833	4586	4545	4545
q2	2556	195	174	174
q3	11715	1112	1170	1112
q4	10216	733	716	716
q5	7931	2883	2810	2810
q6	234	139	148	139
q7	976	619	612	612
q8	9532	2050	2030	2030
q9	7206	6519	6512	6512
q10	7016	2219	2218	2218
q11	443	250	245	245
q12	407	222	221	221
q13	17765	3042	3021	3021
q14	277	245	241	241
q15	535	494	475	475
q16	603	510	521	510
q17	971	643	753	643
q18	7512	6812	6904	6812
q19	1399	1061	1134	1061
q20	696	341	339	339
q21	3864	3111	2997	2997
q22	1116	1060	1023	1023
Total cold run time: 110803 ms
Total hot run time: 38456 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4388	4305	4332	4305
q2	364	280	281	280
q3	2891	2593	2605	2593
q4	1947	1690	1610	1610
q5	5410	5367	5407	5367
q6	224	129	130	129
q7	2076	1725	1784	1725
q8	3195	3332	3349	3332
q9	8429	8445	8457	8445
q10	3453	3228	3162	3162
q11	590	480	502	480
q12	801	604	608	604
q13	10048	3030	3046	3030
q14	312	282	276	276
q15	514	467	482	467
q16	618	567	542	542
q17	1780	1497	1478	1478
q18	7820	7525	7746	7525
q19	1671	1325	1527	1325
q20	2087	1854	1835	1835
q21	5529	5243	5301	5243
q22	1143	1064	1053	1053
Total cold run time: 65290 ms
Total hot run time: 54806 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 188511 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 4fb82f426b74f6a2fa85bc889a1b3a7bbbf279f6, data reload: false

query1	911	372	367	367
query2	6477	1995	1860	1860
query3	6656	214	227	214
query4	33741	23120	23136	23120
query5	4146	510	513	510
query6	247	170	179	170
query7	4581	300	295	295
query8	266	208	204	204
query9	8481	2488	2487	2487
query10	435	271	270	270
query11	17819	14953	15063	14953
query12	153	105	99	99
query13	1650	384	365	365
query14	9433	7506	7159	7159
query15	272	172	181	172
query16	8012	473	475	473
query17	1559	572	557	557
query18	2120	302	298	298
query19	240	154	177	154
query20	119	113	107	107
query21	204	105	104	104
query22	4333	4093	4134	4093
query23	34108	33510	33395	33395
query24	11085	3004	2982	2982
query25	634	387	383	383
query26	1334	168	164	164
query27	2867	291	291	291
query28	7687	2114	2118	2114
query29	863	425	417	417
query30	303	161	154	154
query31	981	752	802	752
query32	102	57	58	57
query33	753	294	299	294
query34	969	508	517	508
query35	908	744	725	725
query36	1114	935	983	935
query37	173	96	97	96
query38	3986	3898	3944	3898
query39	1435	1415	1385	1385
query40	275	121	121	121
query41	50	46	48	46
query42	120	98	103	98
query43	523	480	489	480
query44	1220	771	770	770
query45	200	166	169	166
query46	1120	761	760	760
query47	1893	1785	1817	1785
query48	386	307	300	300
query49	1084	475	432	432
query50	817	436	433	433
query51	7269	7128	7110	7110
query52	100	90	93	90
query53	263	195	236	195
query54	896	456	462	456
query55	81	79	78	78
query56	292	264	267	264
query57	1197	1090	1061	1061
query58	243	248	237	237
query59	2990	2721	2703	2703
query60	300	270	272	270
query61	99	102	104	102
query62	864	651	662	651
query63	221	190	190	190
query64	5129	774	752	752
query65	3288	3180	3198	3180
query66	1437	336	342	336
query67	15981	15055	15175	15055
query68	3249	588	575	575
query69	399	281	285	281
query70	1155	1056	1069	1056
query71	347	281	282	281
query72	6366	4002	4009	4002
query73	763	339	339	339
query74	9180	8852	8823	8823
query75	3441	2717	2727	2717
query76	1880	1030	982	982
query77	486	343	333	333
query78	9683	8991	8964	8964
query79	1074	547	561	547
query80	721	518	534	518
query81	453	236	232	232
query82	246	151	150	150
query83	178	159	154	154
query84	230	82	78	78
query85	684	298	287	287
query86	323	308	306	306
query87	4441	4318	4243	4243
query88	2987	2376	2369	2369
query89	391	292	294	292
query90	1801	206	211	206
query91	131	166	109	109
query92	67	55	52	52
query93	1060	554	558	554
query94	802	308	303	303
query95	366	275	271	271
query96	593	273	273	273
query97	3222	3044	3055	3044
query98	217	211	198	198
query99	1482	1292	1270	1270
Total cold run time: 287913 ms
Total hot run time: 188511 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 32.38 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 4fb82f426b74f6a2fa85bc889a1b3a7bbbf279f6, data reload: false

query1	0.05	0.04	0.03
query2	0.07	0.04	0.03
query3	0.23	0.05	0.05
query4	1.68	0.08	0.08
query5	0.51	0.50	0.49
query6	1.13	0.73	0.72
query7	0.02	0.02	0.01
query8	0.05	0.05	0.05
query9	0.56	0.49	0.49
query10	0.54	0.54	0.54
query11	0.16	0.12	0.12
query12	0.15	0.12	0.13
query13	0.61	0.59	0.58
query14	2.11	2.10	2.04
query15	0.84	0.81	0.82
query16	0.36	0.38	0.37
query17	1.00	1.00	1.01
query18	0.21	0.20	0.20
query19	1.84	1.80	1.71
query20	0.01	0.01	0.01
query21	15.42	0.68	0.67
query22	4.19	6.67	2.19
query23	18.27	1.43	1.29
query24	2.09	0.23	0.22
query25	0.15	0.09	0.08
query26	0.26	0.18	0.17
query27	0.07	0.08	0.07
query28	13.29	1.01	1.00
query29	12.71	3.36	3.37
query30	0.24	0.05	0.05
query31	2.89	0.40	0.38
query32	3.25	0.49	0.48
query33	2.98	3.02	2.99
query34	17.03	4.54	4.45
query35	4.55	4.52	4.44
query36	0.65	0.48	0.48
query37	0.19	0.17	0.16
query38	0.16	0.15	0.16
query39	0.05	0.03	0.03
query40	0.16	0.13	0.13
query41	0.10	0.05	0.05
query42	0.06	0.05	0.05
query43	0.04	0.04	0.04
Total cold run time: 110.93 s
Total hot run time: 32.38 s

dataroaring
dataroaring previously approved these changes Sep 3, 2024
View reviewed changes
Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Sep 3, 2024
@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Sep 3, 2024

PR approved by at least one committer and no changes requested.

@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Sep 3, 2024

PR approved by anyone and no changes requested.

@github-actions github-actions bot removed the approved Indicates a PR has been approved by one committer. label Sep 9, 2024
@mymeiyi
Copy link
Contributor Author

mymeiyi commented Sep 9, 2024

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 38450 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 607dd3184b7f31dc1f5a0c0a44a859aa84f591f8, data reload: false

------ Round 1 ----------------------------------
q1	17643	4507	4366	4366
q2	2015	187	185	185
q3	10471	1176	1134	1134
q4	10143	763	736	736
q5	7755	2919	2889	2889
q6	231	140	148	140
q7	1000	630	608	608
q8	9810	2069	2086	2069
q9	7380	6570	6549	6549
q10	7020	2232	2266	2232
q11	457	246	250	246
q12	404	229	224	224
q13	17763	3105	3080	3080
q14	287	242	251	242
q15	530	503	487	487
q16	510	421	450	421
q17	982	709	723	709
q18	7577	6999	6824	6824
q19	1386	1074	1092	1074
q20	683	331	331	331
q21	3984	2941	2869	2869
q22	1135	1035	1039	1035
Total cold run time: 109166 ms
Total hot run time: 38450 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4403	4441	4363	4363
q2	384	280	266	266
q3	2910	2687	2668	2668
q4	2003	1669	1707	1669
q5	5422	5417	5456	5417
q6	227	131	134	131
q7	2141	1809	1776	1776
q8	3210	3404	3392	3392
q9	8472	8513	8469	8469
q10	3480	3229	3238	3229
q11	600	525	514	514
q12	782	633	617	617
q13	11943	3111	3101	3101
q14	306	275	284	275
q15	533	490	492	490
q16	520	466	461	461
q17	1817	1515	1493	1493
q18	7765	7577	7414	7414
q19	1695	1583	1550	1550
q20	2035	1846	1845	1845
q21	5506	5224	5282	5224
q22	1120	1062	1030	1030
Total cold run time: 67274 ms
Total hot run time: 55394 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 194477 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 607dd3184b7f31dc1f5a0c0a44a859aa84f591f8, data reload: false

query1	909	373	377	373
query2	6500	1927	1888	1888
query3	6646	209	216	209
query4	27863	23402	23349	23349
query5	4146	507	480	480
query6	268	177	170	170
query7	4587	312	296	296
query8	283	238	229	229
query9	8844	2480	2487	2480
query10	448	277	276	276
query11	18009	15195	15051	15051
query12	160	99	96	96
query13	1629	383	373	373
query14	9822	7256	7373	7256
query15	296	173	175	173
query16	8068	474	475	474
query17	1573	577	565	565
query18	2077	301	297	297
query19	317	157	152	152
query20	131	114	112	112
query21	215	108	107	107
query22	4442	4283	4110	4110
query23	34295	33532	33523	33523
query24	11229	2980	2885	2885
query25	662	408	399	399
query26	1348	156	157	156
query27	2808	285	283	283
query28	7727	2050	2039	2039
query29	917	429	421	421
query30	315	165	155	155
query31	989	795	756	756
query32	105	56	58	56
query33	776	304	298	298
query34	966	493	485	485
query35	907	743	728	728
query36	1133	941	978	941
query37	172	91	93	91
query38	4065	3879	3856	3856
query39	1456	1405	1386	1386
query40	271	121	118	118
query41	49	45	45	45
query42	117	97	96	96
query43	526	469	462	462
query44	1228	778	756	756
query45	208	169	178	169
query46	1180	740	763	740
query47	1895	1795	1816	1795
query48	394	301	301	301
query49	1125	471	466	466
query50	878	421	411	411
query51	7132	6986	6944	6944
query52	101	92	91	91
query53	271	190	183	183
query54	1006	483	471	471
query55	78	78	79	78
query56	305	275	279	275
query57	1227	1074	1065	1065
query58	243	263	240	240
query59	2948	2802	2741	2741
query60	323	295	291	291
query61	131	124	206	124
query62	850	665	650	650
query63	230	186	178	178
query64	5296	666	672	666
query65	3240	3173	3154	3154
query66	1417	337	349	337
query67	15982	15424	15402	15402
query68	3332	892	887	887
query69	441	323	328	323
query70	1167	1250	1213	1213
query71	367	388	339	339
query72	6200	3646	3697	3646
query73	597	604	597	597
query74	9061	9121	8896	8896
query75	3221	3051	3156	3051
query76	1932	914	911	911
query77	482	425	417	417
query78	9386	9324	10380	9324
query79	990	949	930	930
query80	868	841	871	841
query81	456	267	270	267
query82	277	273	280	273
query83	195	201	198	198
query84	224	113	109	109
query85	643	453	396	396
query86	337	343	325	325
query87	4341	4368	4501	4368
query88	4576	4138	4137	4137
query89	404	381	385	381
query90	1506	333	326	326
query91	124	126	128	126
query92	81	77	119	77
query93	949	997	941	941
query94	602	381	399	381
query95	432	423	423	423
query96	479	479	479	479
query97	3138	3152	3140	3140
query98	249	228	226	226
query99	1454	1321	1306	1306
Total cold run time: 284265 ms
Total hot run time: 194477 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.91 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 607dd3184b7f31dc1f5a0c0a44a859aa84f591f8, data reload: false

query1	0.05	0.04	0.04
query2	0.07	0.04	0.04
query3	0.22	0.05	0.06
query4	1.67	0.09	0.10
query5	0.51	0.49	0.48
query6	1.13	0.73	0.73
query7	0.02	0.02	0.02
query8	0.05	0.04	0.05
query9	0.55	0.49	0.49
query10	0.55	0.57	0.54
query11	0.17	0.12	0.12
query12	0.16	0.12	0.12
query13	0.60	0.58	0.58
query14	1.43	1.43	1.43
query15	0.84	0.82	0.82
query16	0.37	0.37	0.37
query17	0.97	1.05	1.02
query18	0.21	0.19	0.20
query19	1.86	1.75	1.80
query20	0.02	0.01	0.01
query21	15.39	0.68	0.67
query22	4.37	8.37	1.27
query23	18.24	1.41	1.36
query24	1.97	0.25	0.22
query25	0.16	0.08	0.08
query26	0.27	0.19	0.18
query27	0.08	0.08	0.08
query28	13.22	1.02	1.00
query29	12.58	3.35	3.30
query30	0.24	0.05	0.05
query31	2.87	0.40	0.39
query32	3.24	0.48	0.49
query33	3.00	2.95	3.00
query34	17.11	4.43	4.42
query35	4.51	4.46	4.48
query36	0.66	0.47	0.48
query37	0.19	0.16	0.15
query38	0.15	0.14	0.15
query39	0.05	0.04	0.04
query40	0.16	0.13	0.13
query41	0.11	0.05	0.05
query42	0.05	0.05	0.05
query43	0.04	0.04	0.04
Total cold run time: 110.11 s
Total hot run time: 30.91 s

@mymeiyi
Copy link
Contributor Author

mymeiyi commented Sep 9, 2024

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 38980 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 607dd3184b7f31dc1f5a0c0a44a859aa84f591f8, data reload: false

------ Round 1 ----------------------------------
q1	17815	4353	4277	4277
q2	2012	192	181	181
q3	12235	932	1141	932
q4	10519	753	796	753
q5	7767	2876	2866	2866
q6	232	141	139	139
q7	964	630	625	625
q8	10564	2084	2126	2084
q9	7188	6631	6617	6617
q10	7611	2251	2181	2181
q11	449	248	253	248
q12	404	227	234	227
q13	18838	3112	3148	3112
q14	278	249	251	249
q15	566	518	493	493
q16	581	443	453	443
q17	1019	707	710	707
q18	8347	7293	7503	7293
q19	1406	1153	1037	1037
q20	696	342	331	331
q21	3901	3182	3158	3158
q22	1132	1027	1037	1027
Total cold run time: 114524 ms
Total hot run time: 38980 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4426	4369	4323	4323
q2	392	273	266	266
q3	2876	2714	2673	2673
q4	1945	1784	1795	1784
q5	5737	5713	5672	5672
q6	231	135	141	135
q7	2198	1835	1777	1777
q8	3286	3520	3453	3453
q9	8874	8805	8925	8805
q10	3635	3352	3274	3274
q11	615	513	518	513
q12	872	687	622	622
q13	17025	3265	3265	3265
q14	337	293	303	293
q15	524	504	508	504
q16	533	486	496	486
q17	1861	1548	1529	1529
q18	8109	7962	7857	7857
q19	1816	1698	1544	1544
q20	2168	1954	1885	1885
q21	5818	5475	5203	5203
q22	1190	1092	1084	1084
Total cold run time: 74468 ms
Total hot run time: 56947 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 192894 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 607dd3184b7f31dc1f5a0c0a44a859aa84f591f8, data reload: false

query1	1259	879	882	879
query2	6457	1966	1960	1960
query3	10747	4000	4096	4000
query4	59575	23836	23206	23206
query5	5429	514	528	514
query6	419	157	162	157
query7	5784	309	297	297
query8	331	223	229	223
query9	9201	2509	2494	2494
query10	502	277	260	260
query11	17115	15064	15270	15064
query12	165	106	108	106
query13	1605	375	387	375
query14	10828	7440	7508	7440
query15	226	187	186	186
query16	7598	471	487	471
query17	1115	607	595	595
query18	2078	305	319	305
query19	296	164	158	158
query20	131	116	115	115
query21	210	113	115	113
query22	4685	4319	4406	4319
query23	34678	33540	33331	33331
query24	6027	2895	2901	2895
query25	554	405	402	402
query26	693	159	163	159
query27	1786	285	282	282
query28	3724	2036	2008	2008
query29	720	430	423	423
query30	235	158	161	158
query31	948	764	774	764
query32	90	56	61	56
query33	437	300	297	297
query34	878	477	471	471
query35	840	708	748	708
query36	1079	947	923	923
query37	147	89	91	89
query38	3960	3835	3932	3835
query39	1445	1393	1383	1383
query40	198	117	118	117
query41	50	52	48	48
query42	117	99	96	96
query43	500	490	476	476
query44	1122	753	743	743
query45	199	170	170	170
query46	1099	740	739	739
query47	1887	1771	1847	1771
query48	384	296	299	296
query49	794	463	457	457
query50	835	430	425	425
query51	7095	6805	6796	6796
query52	102	88	89	88
query53	255	188	186	186
query54	585	470	463	463
query55	74	76	79	76
query56	290	276	281	276
query57	1242	1097	1081	1081
query58	251	267	237	237
query59	3170	2919	2904	2904
query60	300	275	268	268
query61	99	97	106	97
query62	748	653	658	653
query63	221	186	185	185
query64	2849	675	655	655
query65	3215	3128	3185	3128
query66	663	355	339	339
query67	15513	15086	15304	15086
query68	4397	572	557	557
query69	432	282	278	278
query70	1142	1043	1127	1043
query71	370	278	272	272
query72	6588	4147	4035	4035
query73	776	322	325	322
query74	9158	8747	8901	8747
query75	3322	2744	2700	2700
query76	1866	1108	1032	1032
query77	569	332	317	317
query78	10451	9063	9738	9063
query79	2202	536	554	536
query80	2116	516	514	514
query81	582	234	231	231
query82	362	143	139	139
query83	263	151	147	147
query84	278	78	73	73
query85	924	287	277	277
query86	335	277	300	277
query87	4507	4412	4228	4228
query88	3396	2442	2281	2281
query89	400	296	285	285
query90	1728	187	187	187
query91	120	101	98	98
query92	66	51	49	49
query93	2234	538	543	538
query94	705	299	299	299
query95	350	254	261	254
query96	603	261	260	260
query97	3176	3111	3112	3111
query98	213	197	197	197
query99	1688	1277	1270	1270
Total cold run time: 313173 ms
Total hot run time: 192894 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 31.75 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 607dd3184b7f31dc1f5a0c0a44a859aa84f591f8, data reload: false

query1	0.04	0.04	0.04
query2	0.08	0.05	0.04
query3	0.23	0.05	0.06
query4	1.66	0.08	0.07
query5	0.51	0.53	0.50
query6	1.13	0.74	0.72
query7	0.01	0.02	0.01
query8	0.05	0.04	0.04
query9	0.53	0.49	0.50
query10	0.54	0.57	0.53
query11	0.16	0.11	0.11
query12	0.15	0.12	0.11
query13	0.59	0.58	0.58
query14	1.40	1.42	1.45
query15	0.84	0.84	0.82
query16	0.38	0.38	0.39
query17	1.01	1.03	1.05
query18	0.22	0.21	0.20
query19	1.90	1.82	1.85
query20	0.01	0.01	0.01
query21	15.39	0.69	0.67
query22	3.93	7.14	2.14
query23	18.28	1.39	1.25
query24	2.16	0.21	0.23
query25	0.15	0.07	0.08
query26	0.26	0.18	0.18
query27	0.08	0.07	0.08
query28	13.21	1.02	0.99
query29	12.60	3.35	3.35
query30	0.25	0.06	0.06
query31	2.86	0.39	0.39
query32	3.27	0.47	0.48
query33	2.88	2.97	3.02
query34	17.08	4.39	4.40
query35	4.41	4.50	4.50
query36	0.67	0.49	0.48
query37	0.18	0.16	0.15
query38	0.16	0.15	0.15
query39	0.05	0.04	0.04
query40	0.15	0.12	0.13
query41	0.10	0.04	0.05
query42	0.05	0.05	0.05
query43	0.05	0.04	0.04
Total cold run time: 109.66 s
Total hot run time: 31.75 s

@dataroaring
Copy link
Contributor

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 38093 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit c4ecdbb9116e803fa04227dc2d92b5ee9aed2044, data reload: false

------ Round 1 ----------------------------------
q1	17600	4348	4339	4339
q2	2028	190	189	189
q3	11793	966	1180	966
q4	10520	657	751	657
q5	7730	2876	2832	2832
q6	227	143	143	143
q7	969	618	595	595
q8	9332	2084	2071	2071
q9	7350	6558	6576	6558
q10	7024	2182	2261	2182
q11	467	246	246	246
q12	390	223	229	223
q13	19025	3082	3104	3082
q14	280	239	242	239
q15	534	491	495	491
q16	533	443	434	434
q17	987	733	780	733
q18	7226	6858	6856	6856
q19	1383	951	951	951
q20	671	332	343	332
q21	3990	3023	2962	2962
q22	1120	1023	1012	1012
Total cold run time: 111179 ms
Total hot run time: 38093 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4338	4307	4434	4307
q2	388	274	270	270
q3	2869	2667	2639	2639
q4	1950	1661	1675	1661
q5	5714	5689	5838	5689
q6	224	131	137	131
q7	2281	1863	1821	1821
q8	3296	3495	3532	3495
q9	8936	8832	8915	8832
q10	3618	3383	3399	3383
q11	621	522	512	512
q12	837	667	659	659
q13	15662	3279	3281	3279
q14	331	284	307	284
q15	542	500	499	499
q16	559	488	517	488
q17	1864	1557	1566	1557
q18	8140	7832	7890	7832
q19	1734	1636	1558	1558
q20	2190	1912	1944	1912
q21	5832	5559	5558	5558
q22	1126	1039	1064	1039
Total cold run time: 73052 ms
Total hot run time: 57405 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 197776 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit c4ecdbb9116e803fa04227dc2d92b5ee9aed2044, data reload: false

query1	1272	908	927	908
query2	6441	1993	1922	1922
query3	10627	4081	3920	3920
query4	59565	24557	23226	23226
query5	5033	497	491	491
query6	399	172	159	159
query7	5619	311	290	290
query8	300	214	207	207
query9	7656	2478	2478	2478
query10	412	276	272	272
query11	15610	15240	15159	15159
query12	148	95	96	95
query13	1433	400	394	394
query14	10703	7735	7340	7340
query15	217	180	176	176
query16	6760	451	487	451
query17	1130	575	570	570
query18	1443	296	299	296
query19	209	154	144	144
query20	118	112	108	108
query21	218	105	104	104
query22	4669	4411	4369	4369
query23	34503	33832	33544	33544
query24	6038	2885	2874	2874
query25	514	399	399	399
query26	615	163	159	159
query27	1581	274	283	274
query28	3694	2058	2012	2012
query29	635	407	407	407
query30	247	153	149	149
query31	932	781	762	762
query32	74	56	53	53
query33	438	282	286	282
query34	891	484	471	471
query35	836	721	728	721
query36	1066	973	972	972
query37	146	88	83	83
query38	4025	3909	3899	3899
query39	1404	1385	1381	1381
query40	196	116	111	111
query41	47	44	44	44
query42	116	97	97	97
query43	512	465	482	465
query44	1097	747	743	743
query45	199	170	170	170
query46	1099	712	717	712
query47	1874	1811	1811	1811
query48	374	301	299	299
query49	767	469	471	469
query50	831	419	432	419
query51	7067	6947	6856	6856
query52	100	87	93	87
query53	252	180	186	180
query54	573	467	464	464
query55	78	76	75	75
query56	282	272	267	267
query57	1192	1087	1069	1069
query58	241	232	238	232
query59	3074	3014	2799	2799
query60	315	282	285	282
query61	127	121	122	121
query62	736	662	659	659
query63	222	190	185	185
query64	1414	719	744	719
query65	3259	3157	3169	3157
query66	675	347	354	347
query67	15766	15272	15288	15272
query68	1964	852	852	852
query69	443	326	343	326
query70	1145	1205	1196	1196
query71	346	334	334	334
query72	4758	3508	3484	3484
query73	585	575	581	575
query74	9044	8871	8947	8871
query75	3026	2896	2926	2896
query76	988	838	824	824
query77	516	389	395	389
query78	9312	9290	9243	9243
query79	890	865	862	862
query80	790	787	812	787
query81	447	261	256	256
query82	267	259	270	259
query83	189	187	186	186
query84	202	105	103	103
query85	603	400	377	377
query86	312	323	323	323
query87	4331	4308	4447	4308
query88	4360	4143	4125	4125
query89	364	363	359	359
query90	1382	314	303	303
query91	124	119	118	118
query92	77	74	73	73
query93	888	883	897	883
query94	521	364	351	351
query95	415	451	414	414
query96	475	474	469	469
query97	3029	3093	3113	3093
query98	221	236	230	230
query99	1320	1310	1275	1275
Total cold run time: 293440 ms
Total hot run time: 197776 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 31.37 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit c4ecdbb9116e803fa04227dc2d92b5ee9aed2044, data reload: false

query1	0.05	0.04	0.05
query2	0.07	0.04	0.04
query3	0.23	0.06	0.06
query4	1.65	0.08	0.11
query5	0.54	0.49	0.51
query6	1.12	0.73	0.72
query7	0.03	0.01	0.01
query8	0.04	0.04	0.04
query9	0.54	0.49	0.48
query10	0.53	0.57	0.53
query11	0.15	0.12	0.11
query12	0.14	0.12	0.12
query13	0.61	0.60	0.59
query14	1.36	1.41	1.41
query15	0.84	0.85	0.86
query16	0.37	0.37	0.37
query17	1.07	0.98	1.06
query18	0.21	0.20	0.20
query19	1.82	1.81	1.87
query20	0.01	0.01	0.01
query21	15.40	0.67	0.67
query22	4.03	8.14	1.72
query23	18.31	1.37	1.30
query24	2.15	0.22	0.21
query25	0.14	0.08	0.08
query26	0.28	0.18	0.18
query27	0.08	0.07	0.07
query28	13.25	1.04	1.04
query29	12.64	3.34	3.32
query30	0.24	0.06	0.05
query31	2.86	0.42	0.40
query32	3.23	0.48	0.49
query33	2.98	3.02	3.02
query34	17.01	4.40	4.39
query35	4.45	4.43	4.46
query36	0.66	0.49	0.50
query37	0.18	0.15	0.15
query38	0.15	0.14	0.15
query39	0.04	0.03	0.04
query40	0.15	0.14	0.13
query41	0.09	0.04	0.05
query42	0.06	0.05	0.05
query43	0.05	0.04	0.04
Total cold run time: 109.81 s
Total hot run time: 31.37 s

yujun777
yujun777 approved these changes Sep 19, 2024
View reviewed changes
Copy link
Collaborator

@yujun777 yujun777 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

dataroaring
dataroaring approved these changes Sep 24, 2024
View reviewed changes
Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Sep 24, 2024
@github-actions GitHub Actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@dataroaring dataroaring merged commit 008ddb2 into apache:master Sep 24, 2024
25 of 28 checks passed
mymeiyi added a commit to mymeiyi/doris that referenced this pull request Sep 24, 2024
@mymeiyi
[fix](transaction) commit txn check txn status (apache#40064)
80558dc
@mymeiyi mymeiyi mentioned this pull request Sep 24, 2024
Merged
mymeiyi added a commit to mymeiyi/doris that referenced this pull request Sep 24, 2024
@mymeiyi
[fix](transaction) commit txn check txn status (apache#40064)
baf78cb
mymeiyi added a commit to mymeiyi/doris that referenced this pull request Sep 24, 2024
@mymeiyi
[fix](transaction) commit txn check txn status (apache#40064)
895859b
@mymeiyi mymeiyi mentioned this pull request Sep 24, 2024
Merged
dataroaring pushed a commit that referenced this pull request Sep 25, 2024
@mymeiyi
[fix](transaction) commit txn check txn status (#40064) (#41227)
386c55f 
pick #40064
dataroaring pushed a commit that referenced this pull request Sep 25, 2024
@mymeiyi
[fix](transaction) commit txn check txn status (#40064) (#41229)
f8acd81 
pick #40064
dataroaring pushed a commit that referenced this pull request Oct 9, 2024
@mymeiyi @dataroaring
[fix](transaction) commit txn check txn status (#40064)
52d48a5 
1. In `checkBeforeUnprotectedCommitTransaction` of
`DatabaseTransactionMgr`, if the txn status is not `PREPARE`, should
skip calculate partition version, but the return logic does not work.
2. the `preCommitTransaction2PC` in `DatabaseTransactionMgr` does not
hold `DatabaseTransactionMgr` write lock
3. ~~the `abortTxnWhenCoordinateBeRestart` and
`abortTxnWhenCoordinateBeDown` in `GlobalTransactionMgr` does not hold
table write lock~~
4. the `public void commitTransaction(long dbId, List<Table> tableList,
long transactionId, List<TabletCommitInfo> tabletCommitInfos)` is only
used for UT, it requires the caller hold table write lock, and may
wrongly use , mark it as Deprecated.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@yujun777 yujun777 yujun777 approved these changes

@dataroaring dataroaring dataroaring approved these changes

Assignees
No one assigned
Labels
approved Indicates a PR has been approved by one committer. dev/2.0.16-merged dev/2.1.7-merged reviewed
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@mymeiyi @doris-robot @dataroaring @yujun777