Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[chore](backup) log backup/restore job during replay #35234

Merged
merged 1 commit into from
May 23, 2024
Merged

[chore](backup) log backup/restore job during replay #35234

merged 1 commit into from
May 23, 2024

Conversation

w41ter
Copy link
Contributor

@w41ter w41ter commented May 22, 2024

Proposed changes

Issue Number: close #xxx

Further comments

If this is a relatively large or complex change, kick off the discussion at [email protected] by explaining why you chose the solution you did and what alternatives you considered, etc...

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@w41ter
Copy link
Contributor Author

w41ter commented May 22, 2024

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 40574 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit b88ac255f8a8fb6e897e9f62491dd66ff3712a04, data reload: false

------ Round 1 ----------------------------------
q1	17604	4649	4269	4269
q2	2028	196	197	196
q3	10474	1183	1163	1163
q4	10155	776	756	756
q5	7506	2692	2647	2647
q6	221	138	138	138
q7	956	622	604	604
q8	9571	2081	2122	2081
q9	9232	6510	6521	6510
q10	8914	3684	3694	3684
q11	449	248	232	232
q12	455	223	219	219
q13	18725	2994	3009	2994
q14	250	231	226	226
q15	519	479	492	479
q16	521	379	383	379
q17	974	664	791	664
q18	8021	7560	7412	7412
q19	3450	1545	1490	1490
q20	679	308	299	299
q21	5006	3973	3859	3859
q22	345	273	276	273
Total cold run time: 116055 ms
Total hot run time: 40574 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4286	4270	4352	4270
q2	356	274	260	260
q3	3020	2757	2767	2757
q4	1880	1606	1600	1600
q5	5226	5265	5266	5265
q6	209	126	125	125
q7	2155	1714	1712	1712
q8	3218	3276	3287	3276
q9	8331	8348	8363	8348
q10	3911	3702	3667	3667
q11	572	482	486	482
q12	747	600	586	586
q13	17003	2945	2971	2945
q14	300	270	267	267
q15	508	464	457	457
q16	476	412	416	412
q17	1773	1517	1471	1471
q18	7629	7659	7385	7385
q19	1649	1548	1556	1548
q20	1991	1791	1793	1791
q21	4893	4703	4753	4703
q22	562	489	486	486
Total cold run time: 70695 ms
Total hot run time: 53813 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 169066 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit b88ac255f8a8fb6e897e9f62491dd66ff3712a04, data reload: false

query1	932	385	361	361
query2	6468	2340	2274	2274
query3	6646	206	210	206
query4	19314	17369	17259	17259
query5	4158	420	425	420
query6	260	179	171	171
query7	4590	290	284	284
query8	234	188	198	188
query9	8758	2398	2369	2369
query10	444	277	259	259
query11	10688	10320	9973	9973
query12	137	86	89	86
query13	1640	370	359	359
query14	9429	7496	6981	6981
query15	231	173	175	173
query16	7833	259	253	253
query17	1845	515	529	515
query18	1966	277	275	275
query19	221	161	160	160
query20	98	88	109	88
query21	198	136	130	130
query22	4258	3919	3872	3872
query23	33656	33182	33039	33039
query24	11964	2874	2800	2800
query25	681	351	355	351
query26	1807	159	158	158
query27	3000	311	316	311
query28	7549	2020	2033	2020
query29	1147	602	595	595
query30	313	171	177	171
query31	969	745	734	734
query32	94	52	54	52
query33	763	274	253	253
query34	1023	500	477	477
query35	744	589	607	589
query36	1086	915	895	895
query37	193	70	72	70
query38	2874	2747	2737	2737
query39	852	789	818	789
query40	269	121	129	121
query41	47	46	43	43
query42	104	98	98	98
query43	572	540	554	540
query44	1206	738	743	738
query45	184	163	164	163
query46	1068	691	768	691
query47	1830	1758	1756	1756
query48	375	291	289	289
query49	1206	383	385	383
query50	786	388	384	384
query51	6957	6756	6753	6753
query52	103	91	88	88
query53	353	289	302	289
query54	957	429	435	429
query55	75	71	73	71
query56	254	244	246	244
query57	1132	1047	1049	1047
query58	243	216	212	212
query59	3342	3036	2994	2994
query60	275	252	252	252
query61	88	108	111	108
query62	674	499	438	438
query63	310	292	286	286
query64	9853	2163	1692	1692
query65	3394	3111	3116	3111
query66	1373	341	323	323
query67	15419	14868	14816	14816
query68	4547	529	541	529
query69	450	281	274	274
query70	1224	1047	1107	1047
query71	412	279	275	275
query72	7660	2709	2587	2587
query73	708	321	321	321
query74	6043	5574	5691	5574
query75	3381	2636	2653	2636
query76	2580	1013	938	938
query77	425	270	273	270
query78	10211	10013	9986	9986
query79	2168	517	519	517
query80	905	448	440	440
query81	532	250	246	246
query82	652	96	99	96
query83	255	172	171	171
query84	240	87	89	87
query85	1821	286	338	286
query86	490	305	316	305
query87	3348	3111	3084	3084
query88	4183	2322	2302	2302
query89	512	394	399	394
query90	1996	187	186	186
query91	125	113	96	96
query92	67	51	50	50
query93	2320	523	493	493
query94	1265	198	186	186
query95	408	310	316	310
query96	594	276	260	260
query97	3138	2985	3001	2985
query98	233	230	223	223
query99	1094	863	856	856
Total cold run time: 275499 ms
Total hot run time: 169066 ms

dataroaring
dataroaring approved these changes May 22, 2024
View reviewed changes
Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@doris-robot
Copy link

ClickBench: Total hot run time: 30.12 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit b88ac255f8a8fb6e897e9f62491dd66ff3712a04, data reload: false

query1	0.03	0.03	0.03
query2	0.08	0.04	0.04
query3	0.24	0.05	0.05
query4	1.67	0.08	0.08
query5	0.49	0.49	0.52
query6	1.13	0.72	0.72
query7	0.02	0.02	0.01
query8	0.05	0.04	0.04
query9	0.56	0.49	0.49
query10	0.54	0.55	0.54
query11	0.16	0.11	0.12
query12	0.15	0.12	0.12
query13	0.60	0.58	0.59
query14	0.78	0.77	0.76
query15	0.85	0.83	0.81
query16	0.36	0.36	0.36
query17	0.95	1.02	1.00
query18	0.20	0.26	0.24
query19	1.88	1.65	1.65
query20	0.02	0.01	0.02
query21	15.56	0.68	0.65
query22	4.56	7.90	1.50
query23	18.25	1.38	1.32
query24	1.93	0.27	0.20
query25	0.14	0.09	0.08
query26	0.27	0.17	0.17
query27	0.09	0.08	0.08
query28	13.26	1.03	0.99
query29	13.18	3.32	3.28
query30	0.26	0.06	0.06
query31	2.86	0.38	0.40
query32	3.27	0.47	0.46
query33	2.90	2.88	2.92
query34	17.18	4.40	4.47
query35	4.54	4.51	4.63
query36	0.66	0.45	0.46
query37	0.18	0.15	0.15
query38	0.15	0.14	0.15
query39	0.04	0.04	0.03
query40	0.16	0.14	0.14
query41	0.09	0.05	0.04
query42	0.05	0.05	0.05
query43	0.04	0.03	0.04
Total cold run time: 110.38 s
Total hot run time: 30.12 s

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label May 22, 2024
@github-actions GitHub Actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions GitHub Actions
Copy link
Contributor

PR approved by anyone and no changes requested.

@dataroaring dataroaring merged commit 176e719 into apache:master May 23, 2024
27 of 29 checks passed
@w41ter w41ter deleted the chore/log_backup_restore_job branch May 23, 2024 02:12
yiguolei pushed a commit that referenced this pull request May 24, 2024
@w41ter @Doris-Extras
[chore](backup) log backup/restore job during replay (#35234)
473e14c
dataroaring pushed a commit that referenced this pull request May 26, 2024
@w41ter @dataroaring
[chore](backup) log backup/restore job during replay (#35234)
45564f7
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@SWJTU-ZhangLei SWJTU-ZhangLei SWJTU-ZhangLei approved these changes

@dataroaring dataroaring dataroaring approved these changes

Assignees
No one assigned
Labels
approved Indicates a PR has been approved by one committer. dev/3.0.0-merged reviewed
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@w41ter @doris-robot @SWJTU-ZhangLei @dataroaring