Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[fix](clone) Fix clone and alter tablet use same tablet path #34889 #36791

Merged
merged 3 commits into from
Jun 26, 2024
Merged

[fix](clone) Fix clone and alter tablet use same tablet path #34889 #36791

merged 3 commits into from
Jun 26, 2024

Conversation

deardeng
Copy link
Contributor

cherry pick from #34889

@deardeng
[fix](clone) Fix clone and alter tablet use same tablet path (apache#…
83210e2 
…34889)

The entire process is as follows:
1. Drop the tablet.
2. Successfully clone the tablet in full.
3. Start the incremental clone.
4. Start to move the tablet to the trash (the process of actually
cleaning the data begins from step 1, where the tablet was dropped).
5. The incremental clone fails.
6. The incremental clone is successfully retried.

Step 4 moved the data that was just pulled from the full clone to the
trash, leading to data loss.
The failure in step 5 of the incremental clone was also due to the
deletion of the just-pulled snapshot data.

Fix：
When cloning, check the tablet status and determine if the tablet
directory has already been moved to the trash directory. If it has not
been moved to the trash, the clone thread should help move it to the
trash directory.
@deardeng
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@deardeng
Copy link
Contributor Author

run buildall

@github-actions GitHub Actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

1 similar comment
@github-actions GitHub Actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@yujun777
Copy link
Collaborator

LGTM

yujun777
yujun777 approved these changes Jun 25, 2024
View reviewed changes
Copy link
Collaborator

@yujun777 yujun777 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@yujun777
Copy link
Collaborator

run buildall

@github-actions GitHub Actions
Copy link
Contributor

PR approved by anyone and no changes requested.

@deardeng
Copy link
Contributor Author

run buildall

@github-actions GitHub Actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

TPC-H: Total hot run time: 50236 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 731b122cb6753f9310b09859c97384c76df17d9c, data reload: false

------ Round 1 ----------------------------------
q1	17764	4833	4398	4398
q2	2102	170	154	154
q3	10542	1926	1968	1926
q4	10261	1303	1317	1303
q5	8504	3982	3961	3961
q6	265	148	130	130
q7	2094	1611	1615	1611
q8	9332	2757	2719	2719
q9	10725	10275	10267	10267
q10	8652	3598	3539	3539
q11	426	251	257	251
q12	478	308	309	308
q13	18344	3973	4010	3973
q14	355	327	329	327
q15	516	467	469	467
q16	693	587	571	571
q17	1160	964	968	964
q18	7404	6926	6924	6924
q19	1767	1664	1604	1604
q20	527	326	307	307
q21	4444	4171	4087	4087
q22	558	464	445	445
Total cold run time: 116913 ms
Total hot run time: 50236 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4374	4312	4334	4312
q2	324	227	220	220
q3	4181	4194	4186	4186
q4	2755	2765	2761	2761
q5	7250	7131	7086	7086
q6	241	121	122	121
q7	3223	2853	2791	2791
q8	4388	4535	4499	4499
q9	16906	16684	16812	16684
q10	4297	4279	4320	4279
q11	790	710	683	683
q12	1042	865	842	842
q13	7446	3765	3733	3733
q14	453	443	425	425
q15	501	462	461	461
q16	726	679	681	679
q17	3816	3851	3915	3851
q18	8799	8642	8793	8642
q19	1736	1694	1637	1637
q20	2395	2130	2108	2108
q21	8499	8585	8523	8523
q22	1015	971	991	971
Total cold run time: 85157 ms
Total hot run time: 79494 ms

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 37.87% (8112/21418)
Line Coverage: 29.54% (66430/224872)
Region Coverage: 29.00% (34227/118012)
Branch Coverage: 24.86% (17564/70654)
Coverage Report: http://coverage.selectdb-in.cc/coverage/731b122cb6753f9310b09859c97384c76df17d9c_731b122cb6753f9310b09859c97384c76df17d9c/report/index.html

@doris-robot
Copy link

TPC-DS: Total hot run time: 204525 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 731b122cb6753f9310b09859c97384c76df17d9c, data reload: false

query1	925	418	378	378
query2	6550	3001	2924	2924
query3	6923	214	204	204
query4	20236	18015	17963	17963
query5	19718	6541	6514	6514
query6	290	219	239	219
query7	4158	304	307	304
query8	439	443	468	443
query9	3118	2685	2615	2615
query10	409	306	301	301
query11	11313	10620	10805	10620
query12	128	83	76	76
query13	5597	690	681	681
query14	17763	13538	13669	13538
query15	366	242	253	242
query16	6499	298	268	268
query17	1681	1458	873	873
query18	2317	412	415	412
query19	216	154	148	148
query20	80	76	83	76
query21	189	98	96	96
query22	5226	5061	5080	5061
query23	32541	31925	32087	31925
query24	6912	6522	6535	6522
query25	528	446	413	413
query26	535	162	159	159
query27	1816	293	300	293
query28	6163	2385	2339	2339
query29	2873	2819	2823	2819
query30	252	168	166	166
query31	891	724	755	724
query32	76	68	61	61
query33	398	267	253	253
query34	858	475	469	469
query35	1127	929	926	926
query36	1295	1133	1036	1036
query37	89	63	61	61
query38	3093	2969	2928	2928
query39	1377	1333	1314	1314
query40	212	96	93	93
query41	46	44	44	44
query42	88	86	81	81
query43	819	816	660	660
query44	1109	726	721	721
query45	248	240	235	235
query46	1228	962	965	962
query47	2026	1710	1682	1682
query48	1011	717	702	702
query49	626	377	379	377
query50	875	643	583	583
query51	4788	4616	4692	4616
query52	90	77	86	77
query53	444	328	313	313
query54	2651	2488	2484	2484
query55	100	87	85	85
query56	231	208	224	208
query57	1167	1088	1112	1088
query58	201	210	196	196
query59	4363	3994	4122	3994
query60	240	203	222	203
query61	100	95	97	95
query62	873	478	482	478
query63	492	341	336	336
query64	2518	1555	1458	1458
query65	3648	3582	3561	3561
query66	830	384	387	384
query67	15934	15277	16041	15277
query68	11129	654	668	654
query69	586	364	358	358
query70	1999	1555	1392	1392
query71	421	308	322	308
query72	6554	3531	3519	3519
query73	746	315	325	315
query74	6377	5898	5893	5893
query75	5502	3693	3773	3693
query76	6953	1192	1216	1192
query77	1186	265	247	247
query78	12438	11835	11832	11832
query79	8904	629	638	629
query80	918	410	410	410
query81	458	238	235	235
query82	823	94	97	94
query83	165	138	134	134
query84	256	72	72	72
query85	886	319	314	314
query86	338	312	294	294
query87	3231	3060	3057	3057
query88	4377	2305	2331	2305
query89	482	298	279	279
query90	1913	207	206	206
query91	175	154	157	154
query92	59	55	52	52
query93	7097	604	561	561
query94	698	204	219	204
query95	1104	1054	1070	1054
query96	645	331	319	319
query97	6458	6357	6456	6357
query98	191	186	168	168
query99	3174	877	867	867
Total cold run time: 318166 ms
Total hot run time: 204525 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 31.15 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 731b122cb6753f9310b09859c97384c76df17d9c, data reload: false

query1	0.02	0.02	0.02
query2	0.07	0.02	0.03
query3	0.25	0.04	0.05
query4	1.79	0.07	0.06
query5	0.54	0.53	0.52
query6	1.26	0.63	0.62
query7	0.01	0.02	0.01
query8	0.04	0.03	0.02
query9	0.52	0.48	0.47
query10	0.55	0.54	0.53
query11	0.12	0.09	0.09
query12	0.11	0.08	0.10
query13	0.62	0.62	0.61
query14	0.79	0.77	0.79
query15	0.79	0.77	0.77
query16	0.39	0.38	0.39
query17	1.02	0.95	1.02
query18	0.24	0.25	0.20
query19	1.94	1.86	1.78
query20	0.01	0.01	0.01
query21	15.46	0.56	0.56
query22	1.98	2.56	2.11
query23	16.97	1.15	0.91
query24	7.57	1.12	0.98
query25	0.37	0.10	0.06
query26	0.78	0.17	0.16
query27	0.04	0.04	0.04
query28	5.49	0.81	0.76
query29	12.78	2.32	2.32
query30	0.57	0.52	0.52
query31	2.81	0.39	0.36
query32	3.37	0.49	0.49
query33	3.07	3.05	3.06
query34	15.25	4.82	4.80
query35	4.88	4.85	4.83
query36	1.04	1.02	1.01
query37	0.06	0.04	0.05
query38	0.04	0.03	0.02
query39	0.02	0.01	0.01
query40	0.17	0.14	0.14
query41	0.07	0.02	0.01
query42	0.02	0.01	0.02
query43	0.02	0.02	0.01
Total cold run time: 103.91 s
Total hot run time: 31.15 s

@doris-robot
Copy link

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit 731b122cb6753f9310b09859c97384c76df17d9c with default session variables
Stream load json:         20 seconds loaded 2358488459 Bytes, about 112 MB/s
Stream load orc:          58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet:      32 seconds loaded 861443392 Bytes, about 25 MB/s
Insert into select:       21.2 seconds inserted 10000000 Rows, about 471K ops/s

@dataroaring dataroaring merged commit fb72f2a into apache:branch-2.0 Jun 26, 2024
22 of 24 checks passed
mongo360 pushed a commit to mongo360/doris that referenced this pull request Aug 16, 2024
@deardeng
[fix](clone) Fix clone and alter tablet use same tablet path apache#3…
7c5d180 
…4889 (apache#36791)

cherry pick from apache#34889
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@yujun777 yujun777 yujun777 approved these changes

Assignees
No one assigned
Labels
kind/test reviewed
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@deardeng @doris-robot @yujun777 @dataroaring