Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[fix](inverted index) the rowset may be deleted and become nullptr. #33878

Merged
merged 1 commit into from
Apr 25, 2024
Merged

[fix](inverted index) the rowset may be deleted and become nullptr. #33878

merged 1 commit into from
Apr 25, 2024

Conversation

zzzxl1993
Copy link
Contributor

@zzzxl1993 zzzxl1993 commented Apr 19, 2024 •
edited
Loading

Proposed changes

  1. between compaction prepare and execute, a rowset may be deleted by cold down, leading to a nullptr exception
  2. intermittent bug that cannot be replicated with a test case

Further comments

If this is a relatively large or complex change, kick off the discussion at [email protected] by explaining why you chose the solution you did and what alternatives you considered, etc...

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@zzzxl1993
Copy link
Contributor Author

run buildall

@github-actions GitHub Actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 35.41% (8918/25184)
Line Coverage: 27.14% (73275/269981)
Region Coverage: 26.25% (37827/144112)
Branch Coverage: 23.07% (19279/83574)
Coverage Report: http://coverage.selectdb-in.cc/coverage/8ddb6edd4fff6eaba690cfd99618182d59d581a2_8ddb6edd4fff6eaba690cfd99618182d59d581a2/report/index.html

@github-actions GitHub Actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@zzzxl1993
Copy link
Contributor Author

run buildall

@github-actions GitHub Actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

1 similar comment
@github-actions GitHub Actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

TPC-H: Total hot run time: 38688 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 45bc42b952fc02b54443a11a3380386cc124047a, data reload: false

------ Round 1 ----------------------------------
q1	18114	7897	4372	4372
q2	2685	189	188	188
q3	11109	1207	1193	1193
q4	10505	786	725	725
q5	7646	2799	2675	2675
q6	225	136	137	136
q7	1025	639	611	611
q8	9466	2074	2074	2074
q9	7505	6696	6565	6565
q10	8601	3505	3497	3497
q11	465	234	230	230
q12	403	238	215	215
q13	17771	2947	2954	2947
q14	264	230	229	229
q15	520	466	471	466
q16	528	389	373	373
q17	953	673	716	673
q18	7381	6780	6704	6704
q19	6957	1507	1517	1507
q20	657	316	299	299
q21	3558	2701	2826	2701
q22	387	308	309	308
Total cold run time: 116725 ms
Total hot run time: 38688 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4365	4247	4226	4226
q2	364	274	283	274
q3	3040	2876	2717	2717
q4	1892	1598	1635	1598
q5	5346	5386	5323	5323
q6	219	121	122	121
q7	2246	1862	1890	1862
q8	3234	3378	3353	3353
q9	8565	8551	8545	8545
q10	3881	3721	3717	3717
q11	584	471	485	471
q12	730	617	581	581
q13	16343	2930	2922	2922
q14	303	259	271	259
q15	522	469	481	469
q16	479	428	428	428
q17	1762	1477	1460	1460
q18	7698	7492	7267	7267
q19	1631	1520	1576	1520
q20	1976	1782	1751	1751
q21	5963	4870	4740	4740
q22	532	468	465	465
Total cold run time: 71675 ms
Total hot run time: 54069 ms

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 35.37% (8916/25209)
Line Coverage: 27.10% (73328/270589)
Region Coverage: 26.24% (37881/144363)
Branch Coverage: 23.05% (19291/83696)
Coverage Report: http://coverage.selectdb-in.cc/coverage/45bc42b952fc02b54443a11a3380386cc124047a_45bc42b952fc02b54443a11a3380386cc124047a/report/index.html

@doris-robot
Copy link

TPC-DS: Total hot run time: 183917 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 45bc42b952fc02b54443a11a3380386cc124047a, data reload: false

query1	908	387	368	368
query2	6919	2598	2359	2359
query3	6651	218	210	210
query4	24066	21440	21273	21273
query5	4178	415	419	415
query6	261	175	169	169
query7	4571	293	281	281
query8	237	187	194	187
query9	8532	2292	2272	2272
query10	600	258	254	254
query11	14740	14162	14203	14162
query12	143	96	87	87
query13	1665	369	378	369
query14	10019	7018	7807	7018
query15	271	189	191	189
query16	8231	268	269	268
query17	1965	600	575	575
query18	2108	283	303	283
query19	329	153	154	153
query20	94	82	92	82
query21	199	130	125	125
query22	5011	4816	4816	4816
query23	33638	33241	33246	33241
query24	11764	2985	2911	2911
query25	647	380	384	380
query26	1829	150	147	147
query27	2942	314	305	305
query28	7281	1987	1966	1966
query29	1040	597	583	583
query30	269	171	169	169
query31	929	720	719	719
query32	98	54	55	54
query33	743	245	264	245
query34	1021	473	472	472
query35	831	685	689	685
query36	1043	878	930	878
query37	219	73	67	67
query38	3333	3150	3189	3150
query39	1567	1515	1521	1515
query40	275	126	124	124
query41	46	46	45	45
query42	104	97	92	92
query43	583	518	547	518
query44	1147	717	739	717
query45	283	267	266	266
query46	1073	760	765	760
query47	1905	1841	1861	1841
query48	368	300	291	291
query49	1136	393	386	386
query50	755	378	380	378
query51	6814	6604	6660	6604
query52	108	87	92	87
query53	349	276	276	276
query54	306	230	223	223
query55	76	71	74	71
query56	237	231	224	224
query57	1196	1096	1139	1096
query58	217	195	194	194
query59	3334	3239	3167	3167
query60	248	241	239	239
query61	102	86	90	86
query62	634	439	444	439
query63	300	269	271	269
query64	6140	3819	3995	3819
query65	3129	3115	3031	3031
query66	1369	345	330	330
query67	15557	14934	14791	14791
query68	5201	525	533	525
query69	474	350	302	302
query70	1228	1172	1184	1172
query71	1385	1265	1262	1262
query72	6312	2636	2420	2420
query73	703	319	319	319
query74	6781	6465	6429	6429
query75	3375	2673	2665	2665
query76	3343	1011	925	925
query77	399	264	262	262
query78	11137	10283	10226	10226
query79	2691	519	507	507
query80	1968	419	422	419
query81	563	243	244	243
query82	740	96	95	95
query83	279	178	176	176
query84	265	94	81	81
query85	2079	269	262	262
query86	508	298	288	288
query87	3479	3361	3260	3260
query88	4479	2398	2391	2391
query89	465	371	377	371
query90	1994	185	180	180
query91	125	96	100	96
query92	65	46	46	46
query93	2554	497	493	493
query94	1252	179	175	175
query95	386	295	299	295
query96	587	274	277	274
query97	3154	2923	2958	2923
query98	234	217	213	213
query99	1243	865	856	856
Total cold run time: 288131 ms
Total hot run time: 183917 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.12 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 45bc42b952fc02b54443a11a3380386cc124047a, data reload: false

query1	0.04	0.03	0.03
query2	0.08	0.04	0.04
query3	0.23	0.05	0.05
query4	1.68	0.08	0.08
query5	0.50	0.50	0.51
query6	1.50	0.73	0.72
query7	0.02	0.01	0.01
query8	0.05	0.05	0.04
query9	0.53	0.49	0.48
query10	0.55	0.55	0.55
query11	0.15	0.12	0.12
query12	0.14	0.12	0.13
query13	0.60	0.58	0.58
query14	0.77	0.76	0.77
query15	0.84	0.80	0.81
query16	0.35	0.37	0.37
query17	0.92	0.99	1.00
query18	0.22	0.24	0.23
query19	1.76	1.67	1.75
query20	0.01	0.01	0.02
query21	15.40	0.64	0.63
query22	4.80	6.75	1.83
query23	18.30	1.35	1.28
query24	2.00	0.22	0.21
query25	0.14	0.09	0.08
query26	0.25	0.16	0.16
query27	0.07	0.07	0.08
query28	13.23	1.01	0.98
query29	12.58	3.22	3.22
query30	0.26	0.07	0.05
query31	2.86	0.38	0.37
query32	3.27	0.46	0.46
query33	2.83	2.85	2.81
query34	16.97	4.35	4.42
query35	4.53	4.44	4.48
query36	0.65	0.46	0.45
query37	0.17	0.16	0.15
query38	0.15	0.15	0.15
query39	0.04	0.03	0.04
query40	0.18	0.13	0.14
query41	0.10	0.05	0.05
query42	0.06	0.04	0.05
query43	0.04	0.04	0.03
Total cold run time: 109.82 s
Total hot run time: 30.12 s

airborne12
airborne12 approved these changes Apr 23, 2024
View reviewed changes
Copy link
Member

@airborne12 airborne12 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@airborne12
Copy link
Member

need some testcases

@github-actions GitHub Actions
Copy link
Contributor

PR approved by anyone and no changes requested.

@zzzxl1993
Copy link
Contributor Author

run p0

xiaokang
xiaokang approved these changes Apr 23, 2024
View reviewed changes
Copy link
Contributor

@xiaokang xiaokang left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Apr 23, 2024
@github-actions GitHub Actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

qidaye
qidaye approved these changes Apr 23, 2024
View reviewed changes
Copy link
Contributor

@qidaye qidaye left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@yiguolei
Copy link
Contributor

run buildall

@github-actions GitHub Actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 35.20% (8918/25333)
Line Coverage: 26.97% (73321/271824)
Region Coverage: 26.15% (37886/144871)
Branch Coverage: 22.97% (19287/83972)
Coverage Report: http://coverage.selectdb-in.cc/coverage/7a74134891cddd4d4a3159dbee9120013033407e_7a74134891cddd4d4a3159dbee9120013033407e/report/index.html

@qidaye qidaye merged commit 3165b5e into apache:master Apr 25, 2024
23 of 26 checks passed
@zzzxl1993 zzzxl1993 mentioned this pull request Apr 26, 2024
Merged
@xiaokang xiaokang added the p0_b label Apr 26, 2024
dataroaring pushed a commit that referenced this pull request Apr 27, 2024
@zzzxl1993 @dataroaring
[fix](inverted index) the rowset may be deleted and become nullptr.(#…
bf5f5d3 
…33878)

1. between compaction prepare and execute, a rowset may be deleted by cold down, leading to a nullptr exception
2. intermittent bug that cannot be replicated with a test case
xiaokang pushed a commit that referenced this pull request Apr 29, 2024
@zzzxl1993
[fix](inverted index) the rowset may be deleted and become nullptr. #…
42681eb 
…33878 (#34183)
@zzzxl1993 zzzxl1993 mentioned this pull request May 7, 2024
Merged
xiaokang pushed a commit that referenced this pull request May 8, 2024
@zzzxl1993
[fix](inverted index) the rowset may be deleted and become nullptr. #…
8b6d624 
…33878 (#34464)
mongo360 pushed a commit to mongo360/doris that referenced this pull request Aug 16, 2024
@zzzxl1993
[fix](inverted index) the rowset may be deleted and become nullptr. a…
72d331a 
…pache#33878 (apache#34183)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@xiaokang xiaokang xiaokang approved these changes

@airborne12 airborne12 airborne12 approved these changes

@qidaye qidaye qidaye approved these changes

@zhangstar333 zhangstar333 zhangstar333 approved these changes

Assignees
No one assigned
Labels
approved Indicates a PR has been approved by one committer. dev/2.0.10-merged p0_b reviewed
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
7 participants
@zzzxl1993 @doris-robot @airborne12 @yiguolei @xiaokang @qidaye @zhangstar333