Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[fix](bdb) Write OP_TIMESTAMP operation until it successed #34061

Merged
merged 2 commits into from
Apr 25, 2024
Merged

[fix](bdb) Write OP_TIMESTAMP operation until it successed #34061

merged 2 commits into from
Apr 25, 2024

Conversation

w41ter
Copy link
Contributor

@w41ter w41ter commented Apr 24, 2024

Cherry-pick #33967

@w41ter
[fix](bdb) Write OP_TIMESTAMP operation until it successed (apache#33967
66d6b71 
)

For now, it will reset the next journal id and return if the OP_TIMESTAMP
operation writes failed. Because BDBJE will replicate the committed txns (only
persisted in BDB log, but not replicated to other members) to FOLLOWERs after
the connection resumed, directly resetting the next journal id and returning
will cause subsequent txn written to the same journal ID not to be replayed by
the FOLLOWERS. So for OP_TIMESTAMP operation, try to write until it succeeds.
@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@w41ter
Copy link
Contributor Author

w41ter commented Apr 24, 2024

run buildall

dataroaring
dataroaring approved these changes Apr 24, 2024
View reviewed changes
Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions GitHub Actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Apr 24, 2024
@github-actions GitHub Actions
Copy link
Contributor

PR approved by anyone and no changes requested.

@doris-robot
Copy link

TPC-H: Total hot run time: 50098 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 66d6b716800420833a9e9df3e73f1415e39674d3, data reload: false

------ Round 1 ----------------------------------
q1	17845	4353	4305	4305
q2	2026	147	147	147
q3	10425	1867	1934	1867
q4	10330	1235	1328	1235
q5	8814	3944	3988	3944
q6	226	124	122	122
q7	2021	1597	1611	1597
q8	9306	2681	2703	2681
q9	11064	10798	10789	10789
q10	8667	3462	3462	3462
q11	437	246	257	246
q12	455	292	294	292
q13	18359	3984	3998	3984
q14	352	319	330	319
q15	512	459	459	459
q16	697	596	591	591
q17	1123	959	984	959
q18	7314	6947	6791	6791
q19	1694	1620	1533	1533
q20	515	315	317	315
q21	4380	4067	4081	4067
q22	522	393	408	393
Total cold run time: 117084 ms
Total hot run time: 50098 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4302	4300	4263	4263
q2	338	223	217	217
q3	4171	4140	4171	4140
q4	2733	2726	2716	2716
q5	7270	7130	7245	7130
q6	234	124	118	118
q7	3210	2860	2825	2825
q8	4334	4449	4482	4449
q9	17545	17301	17193	17193
q10	4204	4240	4229	4229
q11	769	693	691	691
q12	1037	836	842	836
q13	7310	3711	3744	3711
q14	443	430	418	418
q15	503	458	444	444
q16	744	699	687	687
q17	3847	3891	3871	3871
q18	8685	8628	8717	8628
q19	1724	1709	1620	1620
q20	2359	2137	2165	2137
q21	8549	8343	8484	8343
q22	1027	946	956	946
Total cold run time: 85338 ms
Total hot run time: 79612 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 203052 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 66d6b716800420833a9e9df3e73f1415e39674d3, data reload: false

query1	910	386	381	381
query2	6548	2739	2708	2708
query3	6922	205	204	204
query4	20018	17924	17813	17813
query5	19716	6519	6488	6488
query6	291	214	228	214
query7	4164	306	304	304
query8	289	212	219	212
query9	3116	2689	2609	2609
query10	421	294	310	294
query11	11322	10697	10788	10697
query12	115	78	71	71
query13	5574	681	666	666
query14	18111	13252	13123	13123
query15	356	233	236	233
query16	6442	281	262	262
query17	1715	1447	893	893
query18	2300	413	402	402
query19	204	150	147	147
query20	74	76	74	74
query21	186	96	90	90
query22	5109	5020	5154	5020
query23	32402	31884	31938	31884
query24	6930	6518	6471	6471
query25	527	440	403	403
query26	512	163	160	160
query27	1812	292	295	292
query28	6130	2355	2329	2329
query29	2885	2724	2924	2724
query30	235	162	160	160
query31	900	742	756	742
query32	73	60	56	56
query33	401	246	249	246
query34	836	460	496	460
query35	1167	901	901	901
query36	1295	1055	1133	1055
query37	91	61	62	61
query38	3069	2889	2911	2889
query39	1371	1335	1326	1326
query40	205	96	95	95
query41	37	39	37	37
query42	81	86	82	82
query43	849	654	654	654
query44	1119	734	731	731
query45	242	230	224	224
query46	1251	990	971	971
query47	1841	1670	1801	1670
query48	1000	697	698	697
query49	621	369	367	367
query50	859	619	624	619
query51	4872	4728	4678	4678
query52	86	78	79	78
query53	446	319	320	319
query54	2619	2424	2445	2424
query55	97	82	82	82
query56	221	204	210	204
query57	1097	1105	1029	1029
query58	214	208	203	203
query59	4315	4305	3603	3603
query60	218	192	196	192
query61	94	90	88	88
query62	741	504	513	504
query63	478	334	333	333
query64	2462	1606	1482	1482
query65	3621	3536	3552	3536
query66	768	374	376	374
query67	17697	15898	15841	15841
query68	8112	651	674	651
query69	575	335	344	335
query70	1609	1381	1537	1381
query71	404	304	320	304
query72	6465	3535	3510	3510
query73	733	328	324	324
query74	6296	5840	5897	5840
query75	4510	3661	3675	3661
query76	4596	1152	1244	1152
query77	540	245	240	240
query78	12699	11699	12641	11699
query79	10955	647	654	647
query80	1343	403	399	399
query81	490	231	231	231
query82	1319	97	97	97
query83	168	137	132	132
query84	266	70	68	68
query85	980	309	301	301
query86	339	311	291	291
query87	3253	3116	3005	3005
query88	5030	2340	2326	2326
query89	451	269	287	269
query90	1882	187	206	187
query91	180	135	138	135
query92	56	49	52	49
query93	5451	588	603	588
query94	783	208	204	204
query95	1107	1065	1053	1053
query96	644	328	322	322
query97	6429	6468	6444	6444
query98	187	178	171	171
query99	2768	932	855	855
Total cold run time: 313310 ms
Total hot run time: 203052 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.93 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 66d6b716800420833a9e9df3e73f1415e39674d3, data reload: false

query1	0.02	0.02	0.02
query2	0.06	0.02	0.02
query3	0.25	0.04	0.05
query4	1.80	0.08	0.06
query5	0.53	0.52	0.52
query6	1.24	0.62	0.62
query7	0.02	0.01	0.01
query8	0.03	0.02	0.03
query9	0.52	0.49	0.48
query10	0.54	0.54	0.52
query11	0.11	0.09	0.09
query12	0.12	0.09	0.08
query13	0.60	0.62	0.61
query14	0.77	0.78	0.77
query15	0.77	0.74	0.75
query16	0.36	0.37	0.38
query17	1.00	1.00	1.00
query18	0.23	0.27	0.22
query19	1.91	1.88	1.86
query20	0.02	0.00	0.01
query21	15.49	0.59	0.56
query22	1.90	2.48	1.45
query23	17.22	1.10	0.86
query24	5.48	2.17	1.54
query25	0.38	0.09	0.06
query26	0.75	0.16	0.16
query27	0.04	0.04	0.04
query28	6.41	0.75	0.78
query29	12.65	2.11	2.29
query30	0.54	0.48	0.51
query31	2.82	0.39	0.39
query32	3.36	0.50	0.49
query33	3.07	3.08	3.05
query34	15.28	4.83	4.82
query35	4.86	4.89	4.85
query36	1.06	1.02	1.02
query37	0.06	0.04	0.04
query38	0.04	0.02	0.02
query39	0.02	0.01	0.02
query40	0.16	0.14	0.15
query41	0.07	0.02	0.01
query42	0.02	0.01	0.01
query43	0.02	0.02	0.02
Total cold run time: 102.6 s
Total hot run time: 30.93 s

@doris-robot
Copy link

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit 66d6b716800420833a9e9df3e73f1415e39674d3 with default session variables
Stream load json:         20 seconds loaded 2358488459 Bytes, about 112 MB/s
Stream load orc:          58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet:      32 seconds loaded 861443392 Bytes, about 25 MB/s
Insert into select:       21.6 seconds inserted 10000000 Rows, about 462K ops/s

@w41ter
Copy link
Contributor Author

w41ter commented Apr 25, 2024

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 50101 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit be4ad206db74d0f2669dd9b0505c20d65729b361, data reload: false

------ Round 1 ----------------------------------
q1	17639	4432	4337	4337
q2	2062	160	148	148
q3	10492	1956	1876	1876
q4	10380	1241	1305	1241
q5	8532	3991	4007	3991
q6	230	125	123	123
q7	2038	1581	1624	1581
q8	9546	2744	2731	2731
q9	14344	10695	10684	10684
q10	8635	3485	3504	3485
q11	408	240	248	240
q12	465	297	304	297
q13	18313	3982	4032	3982
q14	360	323	341	323
q15	503	464	462	462
q16	678	596	595	595
q17	1128	927	904	904
q18	7276	6851	6734	6734
q19	1706	1576	1550	1550
q20	530	308	286	286
q21	4463	4136	4125	4125
q22	512	406	412	406
Total cold run time: 120240 ms
Total hot run time: 50101 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4304	4269	4341	4269
q2	316	225	218	218
q3	4131	4123	4121	4121
q4	2756	2753	2748	2748
q5	7289	7204	7167	7167
q6	236	120	118	118
q7	3245	2872	2792	2792
q8	4346	4433	4462	4433
q9	17445	17447	17337	17337
q10	4269	4245	4294	4245
q11	763	678	703	678
q12	1014	855	865	855
q13	7186	3729	3769	3729
q14	462	418	415	415
q15	514	467	466	466
q16	777	702	711	702
q17	3970	3820	3833	3820
q18	8756	8763	8779	8763
q19	1740	1685	1643	1643
q20	2413	2161	2098	2098
q21	8444	8541	8515	8515
q22	1060	941	919	919
Total cold run time: 85436 ms
Total hot run time: 80051 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 202642 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit be4ad206db74d0f2669dd9b0505c20d65729b361, data reload: false

query1	929	390	381	381
query2	6522	2663	2635	2635
query3	6917	200	200	200
query4	20097	18011	17886	17886
query5	19726	6582	6624	6582
query6	280	217	216	216
query7	4151	294	299	294
query8	247	243	240	240
query9	3156	2731	2635	2635
query10	405	310	308	308
query11	11221	10707	10685	10685
query12	123	77	71	71
query13	5576	685	691	685
query14	18078	13466	13180	13180
query15	376	234	238	234
query16	6464	271	253	253
query17	1731	1451	858	858
query18	2327	411	420	411
query19	203	139	162	139
query20	75	75	73	73
query21	192	95	100	95
query22	5219	4994	4926	4926
query23	32685	32029	32067	32029
query24	6962	6584	6495	6495
query25	529	438	415	415
query26	528	156	163	156
query27	1842	298	290	290
query28	6288	2385	2359	2359
query29	3026	2860	2793	2793
query30	245	159	160	159
query31	917	724	742	724
query32	67	60	60	60
query33	395	254	247	247
query34	853	467	483	467
query35	1131	969	940	940
query36	1371	1187	1233	1187
query37	86	65	60	60
query38	3024	2904	2934	2904
query39	1352	1345	1309	1309
query40	206	96	96	96
query41	36	38	34	34
query42	82	82	80	80
query43	760	708	681	681
query44	1108	728	724	724
query45	235	232	230	230
query46	1243	960	977	960
query47	1846	1831	1720	1720
query48	1012	722	709	709
query49	614	363	374	363
query50	881	612	610	610
query51	4737	4597	4692	4597
query52	89	77	82	77
query53	442	319	321	319
query54	2637	2527	2480	2480
query55	76	84	78	78
query56	213	215	199	199
query57	1139	1127	1065	1065
query58	216	208	203	203
query59	4290	4112	3771	3771
query60	198	202	214	202
query61	90	84	85	84
query62	809	522	498	498
query63	469	332	338	332
query64	2471	1594	1500	1500
query65	3624	3567	3582	3567
query66	802	367	388	367
query67	15475	14853	15060	14853
query68	8589	655	666	655
query69	563	319	331	319
query70	1720	1480	1349	1349
query71	404	299	305	299
query72	6529	3445	3452	3445
query73	737	324	322	322
query74	6248	5858	5877	5858
query75	5180	3650	3733	3650
query76	5267	1139	1213	1139
query77	877	246	256	246
query78	12980	11675	11811	11675
query79	9099	624	618	618
query80	1210	392	389	389
query81	495	229	231	229
query82	1675	93	100	93
query83	167	131	133	131
query84	262	69	69	69
query85	897	299	294	294
query86	329	290	281	281
query87	3279	3008	3034	3008
query88	4760	2338	2345	2338
query89	488	284	289	284
query90	1969	209	213	209
query91	170	134	141	134
query92	58	51	52	51
query93	6565	578	582	578
query94	704	202	200	200
query95	1102	1063	1054	1054
query96	642	328	326	326
query97	6558	6428	6414	6414
query98	187	182	160	160
query99	2881	871	845	845
Total cold run time: 313707 ms
Total hot run time: 202642 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.57 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit be4ad206db74d0f2669dd9b0505c20d65729b361, data reload: false

query1	0.03	0.02	0.02
query2	0.07	0.03	0.02
query3	0.25	0.04	0.05
query4	1.79	0.07	0.08
query5	0.54	0.51	0.52
query6	1.25	0.62	0.62
query7	0.01	0.01	0.01
query8	0.04	0.02	0.02
query9	0.52	0.50	0.48
query10	0.53	0.55	0.52
query11	0.12	0.09	0.09
query12	0.11	0.09	0.09
query13	0.62	0.61	0.61
query14	0.80	0.78	0.77
query15	0.77	0.75	0.75
query16	0.38	0.37	0.36
query17	1.02	1.01	1.01
query18	0.23	0.25	0.26
query19	1.90	1.83	1.79
query20	0.01	0.01	0.01
query21	15.45	0.56	0.56
query22	2.29	2.28	1.62
query23	17.11	1.00	1.05
query24	5.91	0.64	1.33
query25	0.36	0.08	0.07
query26	0.61	0.18	0.15
query27	0.05	0.04	0.05
query28	7.23	0.75	0.72
query29	12.76	2.34	2.41
query30	0.60	0.53	0.53
query31	2.82	0.38	0.38
query32	3.37	0.50	0.49
query33	3.07	3.08	3.09
query34	15.26	4.80	4.81
query35	4.86	4.86	4.88
query36	1.06	1.03	1.01
query37	0.05	0.05	0.04
query38	0.03	0.03	0.02
query39	0.02	0.02	0.01
query40	0.16	0.13	0.15
query41	0.07	0.02	0.01
query42	0.02	0.01	0.01
query43	0.02	0.02	0.02
Total cold run time: 104.17 s
Total hot run time: 30.57 s

@doris-robot
Copy link

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit be4ad206db74d0f2669dd9b0505c20d65729b361 with default session variables
Stream load json:         20 seconds loaded 2358488459 Bytes, about 112 MB/s
Stream load orc:          58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet:      31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select:       21.3 seconds inserted 10000000 Rows, about 469K ops/s

@dataroaring dataroaring merged commit ae9be51 into apache:branch-2.0 Apr 25, 2024
24 of 25 checks passed
@w41ter w41ter deleted the pick_33967 branch April 26, 2024 01:51
mongo360 pushed a commit to mongo360/doris that referenced this pull request Aug 16, 2024
@w41ter
[fix](bdb) Write OP_TIMESTAMP operation until it successed (apache#33967
8809e2c 
) (apache#34061)

For now, it will reset the next journal id and return if the OP_TIMESTAMP
operation writes failed. Because BDBJE will replicate the committed txns (only
persisted in BDB log, but not replicated to other members) to FOLLOWERs after
the connection resumed, directly resetting the next journal id and returning
will cause subsequent txn written to the same journal ID not to be replayed by
the FOLLOWERS. So for OP_TIMESTAMP operation, try to write until it succeeds.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@dataroaring dataroaring dataroaring approved these changes

Assignees
No one assigned
Labels
approved Indicates a PR has been approved by one committer. reviewed
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@w41ter @doris-robot @dataroaring