Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[fix](path-gc) Fix pending rowset guard check failure when ordered data compaction failed #33026

Merged
merged 1 commit into from
Mar 29, 2024
Merged

[fix](path-gc) Fix pending rowset guard check failure when ordered data compaction failed #33026

merged 1 commit into from
Mar 29, 2024

Conversation

platoneko
Copy link
Contributor

Proposed changes

Fix pending rowset guard check failure when ordered data compaction failed

Further comments

If this is a relatively large or complex change, kick off the discussion at [email protected] by explaining why you chose the solution you did and what alternatives you considered, etc...

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@platoneko
Copy link
Contributor Author

run buildall

@platoneko platoneko requested a review from qidaye March 29, 2024 07:59
@github-actions GitHub Actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

qidaye
qidaye approved these changes Mar 29, 2024
View reviewed changes
Copy link
Contributor

@qidaye qidaye left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Mar 29, 2024
@github-actions GitHub Actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions GitHub Actions
Copy link
Contributor

PR approved by anyone and no changes requested.

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 35.58% (8839/24843)
Line Coverage: 27.31% (72442/265293)
Region Coverage: 26.51% (37497/141449)
Branch Coverage: 23.32% (19118/81976)
Coverage Report: http://coverage.selectdb-in.cc/coverage/40ef65e7b3d0bc61aab5501051651d40eee3245d_40ef65e7b3d0bc61aab5501051651d40eee3245d/report/index.html

@doris-robot
Copy link

TPC-H: Total hot run time: 38529 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 40ef65e7b3d0bc61aab5501051651d40eee3245d, data reload: false

------ Round 1 ----------------------------------
q1	17608	4604	4317	4317
q2	2077	193	189	189
q3	10450	1162	1217	1162
q4	10193	729	774	729
q5	7489	2793	2777	2777
q6	217	130	132	130
q7	1016	614	597	597
q8	9213	2035	1999	1999
q9	7584	6610	6589	6589
q10	8642	3541	3522	3522
q11	450	238	242	238
q12	474	214	214	214
q13	18852	2951	3010	2951
q14	264	233	242	233
q15	529	483	467	467
q16	524	392	388	388
q17	998	650	568	568
q18	7554	6785	6658	6658
q19	6129	1433	1435	1433
q20	683	312	298	298
q21	3461	2764	2901	2764
q22	361	306	311	306
Total cold run time: 114768 ms
Total hot run time: 38529 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4348	4156	4173	4156
q2	369	262	267	262
q3	2941	2788	2717	2717
q4	1861	1536	1520	1520
q5	5260	5285	5225	5225
q6	215	126	127	126
q7	2238	1843	1856	1843
q8	3238	3293	3323	3293
q9	8653	8628	8704	8628
q10	4135	3888	3948	3888
q11	640	499	516	499
q12	772	645	634	634
q13	17778	3190	3167	3167
q14	314	328	291	291
q15	534	486	486	486
q16	500	461	467	461
q17	1865	1499	1515	1499
q18	8334	8011	7917	7917
q19	1644	1530	1544	1530
q20	2071	1857	1844	1844
q21	5380	5175	5001	5001
q22	567	487	478	478
Total cold run time: 73657 ms
Total hot run time: 55465 ms

@qidaye qidaye mentioned this pull request Mar 29, 2024
Merged
@doris-robot
Copy link

TPC-DS: Total hot run time: 182444 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 40ef65e7b3d0bc61aab5501051651d40eee3245d, data reload: false

query1	1214	1117	350	350
query2	6168	2079	1897	1897
query3	6652	217	210	210
query4	24935	21353	21431	21353
query5	4125	413	414	413
query6	280	173	179	173
query7	4597	293	291	291
query8	220	181	179	179
query9	8418	2261	2235	2235
query10	407	242	242	242
query11	14898	14540	14297	14297
query12	133	93	86	86
query13	1642	369	373	369
query14	9416	6839	6930	6839
query15	231	183	184	183
query16	6904	271	267	267
query17	1439	603	573	573
query18	1809	293	295	293
query19	205	159	157	157
query20	93	87	90	87
query21	201	128	127	127
query22	4958	4738	4829	4738
query23	33267	32896	32843	32843
query24	11067	2999	2971	2971
query25	621	407	424	407
query26	732	163	164	163
query27	3157	374	385	374
query28	6194	1931	1896	1896
query29	906	683	648	648
query30	301	163	1068	163
query31	1040	770	763	763
query32	103	63	56	56
query33	626	272	280	272
query34	972	495	515	495
query35	866	743	736	736
query36	1072	932	915	915
query37	113	70	73	70
query38	3842	3669	3603	3603
query39	1641	1597	1580	1580
query40	185	134	149	134
query41	52	52	49	49
query42	110	100	109	100
query43	501	472	471	471
query44	1139	747	742	742
query45	331	274	279	274
query46	1117	734	719	719
query47	1993	1927	1918	1918
query48	402	310	320	310
query49	817	392	400	392
query50	814	427	423	423
query51	6884	6932	6880	6880
query52	127	98	98	98
query53	357	290	286	286
query54	292	231	232	231
query55	79	79	75	75
query56	249	231	231	231
query57	1176	1124	1126	1124
query58	219	205	203	203
query59	2728	2583	2564	2564
query60	271	236	232	232
query61	99	93	92	92
query62	606	445	421	421
query63	304	291	287	287
query64	5284	4061	4191	4061
query65	3081	3048	3033	3033
query66	882	338	339	338
query67	15559	14988	14821	14821
query68	9082	545	554	545
query69	594	321	325	321
query70	1411	1210	1198	1198
query71	497	287	276	276
query72	7091	2626	2444	2444
query73	1689	325	324	324
query74	7508	6361	6391	6361
query75	3695	2344	2370	2344
query76	5081	936	1036	936
query77	579	249	256	249
query78	10807	10230	10135	10135
query79	12003	539	533	533
query80	1812	436	430	430
query81	511	217	219	217
query82	258	91	91	91
query83	208	185	171	171
query84	261	82	81	81
query85	838	284	266	266
query86	334	318	299	299
query87	3792	3526	3538	3526
query88	6031	2449	2411	2411
query89	495	386	375	375
query90	2015	181	181	181
query91	131	95	95	95
query92	66	47	46	46
query93	6878	513	503	503
query94	1318	185	184	184
query95	411	325	313	313
query96	613	270	264	264
query97	2653	2470	2496	2470
query98	237	219	219	219
query99	1113	847	813	813
Total cold run time: 301135 ms
Total hot run time: 182444 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 29.88 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 40ef65e7b3d0bc61aab5501051651d40eee3245d, data reload: false

query1	0.03	0.03	0.03
query2	0.07	0.04	0.04
query3	0.24	0.06	0.05
query4	1.67	0.08	0.08
query5	0.49	0.49	0.49
query6	1.13	0.65	0.65
query7	0.02	0.02	0.01
query8	0.04	0.03	0.04
query9	0.54	0.50	0.49
query10	0.54	0.54	0.54
query11	0.16	0.09	0.12
query12	0.14	0.12	0.11
query13	0.58	0.58	0.58
query14	0.76	0.79	0.80
query15	0.83	0.81	0.81
query16	0.40	0.35	0.37
query17	1.00	0.96	0.97
query18	0.22	0.21	0.24
query19	1.88	1.72	1.80
query20	0.01	0.01	0.02
query21	15.54	0.64	0.64
query22	2.42	2.87	2.08
query23	17.10	0.98	0.88
query24	1.13	0.23	0.20
query25	0.10	0.08	0.09
query26	0.23	0.18	0.17
query27	0.08	0.07	0.07
query28	14.06	0.93	0.93
query29	12.58	3.20	3.21
query30	0.28	0.10	0.10
query31	2.83	0.37	0.37
query32	3.27	0.45	0.45
query33	2.76	2.77	2.90
query34	16.56	4.46	4.41
query35	4.44	4.46	4.48
query36	0.60	0.46	0.46
query37	0.17	0.15	0.14
query38	0.15	0.14	0.14
query39	0.05	0.04	0.04
query40	0.17	0.14	0.15
query41	0.08	0.04	0.05
query42	0.05	0.05	0.04
query43	0.04	0.04	0.03
Total cold run time: 105.44 s
Total hot run time: 29.88 s

@doris-robot
Copy link

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit 40ef65e7b3d0bc61aab5501051651d40eee3245d with default session variables
Stream load json:         18 seconds loaded 2358488459 Bytes, about 124 MB/s
Stream load orc:          58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet:      32 seconds loaded 861443392 Bytes, about 25 MB/s
Insert into select:       13.6 seconds inserted 10000000 Rows, about 735K ops/s

dataroaring
dataroaring approved these changes Mar 29, 2024
View reviewed changes
Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@dataroaring dataroaring merged commit cb6287e into apache:master Mar 29, 2024
27 of 31 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@qidaye qidaye qidaye approved these changes

@dataroaring dataroaring dataroaring approved these changes

Assignees
No one assigned
Labels
approved Indicates a PR has been approved by one committer. dev/2.1.1-merged p0_c reviewed
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
7 participants
@platoneko @doris-robot @qidaye @dataroaring @xiaokang @morningman @yiguolei