Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[fix](statistics)Fix drop stats log editlog bug. Catch drop stats exception while truncate table. (#40738) #40979

Merged
merged 1 commit into from
Sep 26, 2024

Conversation

Jibing-Li
Copy link
Contributor

backport: #40738

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@github-actions github-actions bot added area/planner Issues or PRs related to the query planner kind/test labels Sep 19, 2024
@Jibing-Li
Copy link
Contributor Author

run buildall

Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 37.84% (8141/21513)
Line Coverage: 29.60% (67011/226413)
Region Coverage: 29.08% (34570/118880)
Branch Coverage: 24.99% (17812/71288)
Coverage Report: http://coverage.selectdb-in.cc/coverage/79a282a801aaec84c9630e072e7d43dacc037575_79a282a801aaec84c9630e072e7d43dacc037575/report/index.html

@Jibing-Li
Copy link
Contributor Author

run buildall

Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

TPC-H: Total hot run time: 49080 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 817766f7a6c6eb7d30e12946e435b1ff0cb5272b, data reload: false

------ Round 1 ----------------------------------
q1	18003	4357	4319	4319
q2	2079	155	151	151
q3	10261	1920	1884	1884
q4	10099	1243	1331	1243
q5	8624	3986	3906	3906
q6	237	123	122	122
q7	2090	1587	1580	1580
q8	9500	2727	2702	2702
q9	13179	9796	9908	9796
q10	8614	3478	3541	3478
q11	418	236	253	236
q12	473	302	298	298
q13	18382	3996	4021	3996
q14	352	339	326	326
q15	508	468	461	461
q16	541	466	453	453
q17	1127	954	972	954
q18	7277	6865	6862	6862
q19	1712	1527	1492	1492
q20	548	311	319	311
q21	4440	4132	4127	4127
q22	491	406	383	383
Total cold run time: 118955 ms
Total hot run time: 49080 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4304	4290	4344	4290
q2	313	230	229	229
q3	4141	4131	4171	4131
q4	2747	2763	2746	2746
q5	7150	7094	7072	7072
q6	237	116	117	116
q7	3260	2878	2866	2866
q8	4337	4402	4442	4402
q9	13738	13553	13538	13538
q10	4239	4300	4229	4229
q11	777	688	695	688
q12	1025	880	840	840
q13	6766	3761	3722	3722
q14	470	437	421	421
q15	496	458	457	457
q16	633	619	609	609
q17	3867	3864	3874	3864
q18	8734	8729	8812	8729
q19	1729	1713	1665	1665
q20	2386	2199	2124	2124
q21	8523	8566	8543	8543
q22	1023	914	953	914
Total cold run time: 80895 ms
Total hot run time: 76195 ms

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 37.85% (8143/21513)
Line Coverage: 29.60% (67020/226413)
Region Coverage: 29.09% (34581/118880)
Branch Coverage: 24.99% (17817/71288)
Coverage Report: http://coverage.selectdb-in.cc/coverage/817766f7a6c6eb7d30e12946e435b1ff0cb5272b_817766f7a6c6eb7d30e12946e435b1ff0cb5272b/report/index.html

@doris-robot
Copy link

TPC-DS: Total hot run time: 212245 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 817766f7a6c6eb7d30e12946e435b1ff0cb5272b, data reload: false

query1	927	392	420	392
query2	6533	2290	2046	2046
query3	6915	208	206	206
query4	23582	21776	21835	21776
query5	19731	6474	6505	6474
query6	289	216	229	216
query7	4322	291	314	291
query8	294	245	258	245
query9	3058	2646	2557	2557
query10	470	310	312	310
query11	15823	14816	15073	14816
query12	126	78	74	74
query13	1022	436	434	434
query14	16941	13252	13281	13252
query15	394	219	229	219
query16	6470	287	273	273
query17	1773	932	913	913
query18	896	314	315	314
query19	214	160	149	149
query20	83	76	80	76
query21	187	101	92	92
query22	5214	5020	4938	4938
query23	34166	33551	33445	33445
query24	7845	6272	6267	6267
query25	511	430	422	422
query26	1271	160	166	160
query27	2412	294	292	292
query28	6071	2247	2216	2216
query29	2980	2762	2874	2762
query30	246	169	168	168
query31	981	718	720	718
query32	64	61	63	61
query33	461	281	258	258
query34	869	463	462	462
query35	1118	906	931	906
query36	1289	1266	1095	1095
query37	172	60	58	58
query38	3093	2953	2935	2935
query39	1389	1319	1310	1310
query40	307	96	98	96
query41	41	41	37	37
query42	90	95	82	82
query43	632	560	593	560
query44	1130	720	724	720
query45	243	234	230	230
query46	1235	951	960	951
query47	1834	1645	1700	1645
query48	512	415	404	404
query49	643	362	377	362
query50	860	588	600	588
query51	4755	4690	4685	4685
query52	92	86	88	86
query53	235	187	186	186
query54	2701	2467	2447	2447
query55	85	85	82	82
query56	226	215	212	212
query57	1296	1089	1161	1089
query58	223	211	225	211
query59	3420	3302	3303	3302
query60	207	201	208	201
query61	96	93	98	93
query62	805	458	485	458
query63	200	181	174	174
query64	3513	1602	1522	1522
query65	3633	3558	3572	3558
query66	768	395	397	395
query67	16343	15402	15281	15281
query68	9259	645	649	645
query69	490	268	274	268
query70	1613	1413	1377	1377
query71	402	309	304	304
query72	6869	4825	4736	4736
query73	764	330	324	324
query74	6464	5963	5997	5963
query75	5221	3753	3625	3625
query76	5250	1148	1217	1148
query77	893	257	262	257
query78	12407	11718	11700	11700
query79	7981	645	648	645
query80	1701	376	376	376
query81	496	240	239	239
query82	1705	98	96	96
query83	174	134	135	134
query84	256	71	71	71
query85	914	323	320	320
query86	338	295	313	295
query87	3238	3051	2997	2997
query88	4864	2410	2385	2385
query89	407	316	280	280
query90	1924	219	208	208
query91	158	126	123	123
query92	58	50	52	50
query93	6432	582	566	566
query94	726	210	210	210
query95	1942	1970	1978	1970
query96	661	349	335	335
query97	6497	6370	6361	6361
query98	231	211	211	211
query99	2890	771	861	771
Total cold run time: 319552 ms
Total hot run time: 212245 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.22 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 817766f7a6c6eb7d30e12946e435b1ff0cb5272b, data reload: false

query1	0.02	0.02	0.02
query2	0.07	0.03	0.02
query3	0.24	0.05	0.05
query4	1.79	0.08	0.06
query5	0.54	0.52	0.52
query6	1.27	0.62	0.61
query7	0.02	0.01	0.01
query8	0.04	0.02	0.02
query9	0.53	0.48	0.47
query10	0.54	0.54	0.52
query11	0.12	0.08	0.08
query12	0.11	0.09	0.09
query13	0.63	0.61	0.61
query14	0.77	0.78	0.80
query15	0.79	0.76	0.77
query16	0.37	0.39	0.36
query17	1.01	0.99	1.01
query18	0.22	0.26	0.24
query19	1.94	1.85	1.85
query20	0.01	0.01	0.01
query21	15.47	0.56	0.56
query22	2.41	2.22	1.48
query23	16.85	0.98	0.96
query24	6.89	0.68	1.10
query25	0.38	0.15	0.05
query26	0.69	0.17	0.15
query27	0.03	0.04	0.03
query28	6.49	0.75	0.75
query29	12.66	2.36	2.20
query30	0.56	0.51	0.53
query31	2.82	0.40	0.38
query32	3.35	0.50	0.49
query33	3.08	3.05	3.10
query34	15.25	4.78	4.79
query35	4.86	4.86	4.83
query36	1.06	1.01	1.02
query37	0.06	0.04	0.04
query38	0.04	0.02	0.02
query39	0.02	0.01	0.01
query40	0.16	0.14	0.15
query41	0.06	0.01	0.01
query42	0.02	0.02	0.01
query43	0.03	0.01	0.01
Total cold run time: 104.27 s
Total hot run time: 30.22 s

@doris-robot
Copy link

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit 817766f7a6c6eb7d30e12946e435b1ff0cb5272b with default session variables
Stream load json:         20 seconds loaded 2358488459 Bytes, about 112 MB/s
Stream load orc:          58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet:      31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select:       21.7 seconds inserted 10000000 Rows, about 460K ops/s

@Jibing-Li Jibing-Li changed the title Fix2.0 [fix](statistics)Fix drop stats log editlog bug. Catch drop stats exception while truncate table. (#40738) Sep 26, 2024
@Jibing-Li Jibing-Li marked this pull request as ready for review September 26, 2024 02:35
@Jibing-Li
Copy link
Contributor Author

run buildall

…eption while truncate table. (apache#40738)

Bug fix.
1. Fix drop stats log editlog NPE bug.
2. Catch drop stats exception while truncate table.
@Jibing-Li
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 49733 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 32d4321d73d462358cee535f4bf7576b11f3c35e, data reload: false

------ Round 1 ----------------------------------
q1	17686	4340	4298	4298
q2	2058	158	144	144
q3	10385	1947	1923	1923
q4	10367	1232	1347	1232
q5	8597	3969	3917	3917
q6	232	125	124	124
q7	2063	1610	1597	1597
q8	9285	2749	2722	2722
q9	10357	9969	9786	9786
q10	8661	3560	3606	3560
q11	420	251	257	251
q12	466	300	297	297
q13	18348	3977	4040	3977
q14	349	335	347	335
q15	520	489	457	457
q16	529	474	457	457
q17	1124	989	939	939
q18	7724	7271	7250	7250
q19	1706	1598	1589	1589
q20	530	336	290	290
q21	4510	4179	4175	4175
q22	510	415	413	413
Total cold run time: 116427 ms
Total hot run time: 49733 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4357	4352	4402	4352
q2	326	228	229	228
q3	4167	4124	4131	4124
q4	2730	2730	2734	2730
q5	7205	7105	7097	7097
q6	234	117	117	117
q7	3268	2872	2863	2863
q8	4376	4500	4485	4485
q9	13700	13569	13612	13569
q10	4222	4297	4280	4280
q11	770	677	675	675
q12	1032	844	852	844
q13	6874	3766	3756	3756
q14	458	421	416	416
q15	495	455	448	448
q16	639	593	595	593
q17	3878	3937	3906	3906
q18	8689	8643	8685	8643
q19	1713	1662	1663	1662
q20	2383	2109	2115	2109
q21	8600	8444	8495	8444
q22	1011	960	899	899
Total cold run time: 81127 ms
Total hot run time: 76240 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 211546 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 32d4321d73d462358cee535f4bf7576b11f3c35e, data reload: false

query1	939	421	378	378
query2	6552	2049	2106	2049
query3	6920	200	196	196
query4	22913	22235	21630	21630
query5	19728	6507	6585	6507
query6	277	212	237	212
query7	4320	312	313	312
query8	281	265	246	246
query9	3033	2646	2572	2572
query10	462	315	296	296
query11	15877	14996	15066	14996
query12	124	80	75	75
query13	1035	435	432	432
query14	17508	13552	13108	13108
query15	368	217	232	217
query16	6132	293	261	261
query17	1670	938	909	909
query18	896	302	311	302
query19	228	147	144	144
query20	101	98	102	98
query21	196	106	97	97
query22	5286	5143	4970	4970
query23	34016	33467	33403	33403
query24	7917	6292	6299	6292
query25	540	449	429	429
query26	1313	163	157	157
query27	2467	301	300	300
query28	6153	2243	2229	2229
query29	2885	2684	2700	2684
query30	245	167	163	163
query31	913	776	735	735
query32	68	62	59	59
query33	453	260	264	260
query34	851	480	466	466
query35	1129	942	948	942
query36	1481	1152	1179	1152
query37	166	62	60	60
query38	3125	2954	2959	2954
query39	1356	1319	1320	1319
query40	302	95	97	95
query41	41	36	37	36
query42	88	84	93	84
query43	738	582	620	582
query44	1137	704	716	704
query45	251	230	232	230
query46	1234	957	960	957
query47	1873	1639	1719	1639
query48	504	416	418	416
query49	644	393	393	393
query50	872	651	617	617
query51	4740	4621	4670	4621
query52	92	79	91	79
query53	229	191	192	191
query54	2658	2484	2471	2471
query55	94	77	81	77
query56	217	202	209	202
query57	1354	1177	1118	1118
query58	216	207	211	207
query59	3416	3193	3300	3193
query60	217	217	191	191
query61	98	98	95	95
query62	842	496	433	433
query63	205	171	171	171
query64	3569	1602	1443	1443
query65	3624	3517	3592	3517
query66	780	400	390	390
query67	16055	15302	16822	15302
query68	9031	630	654	630
query69	500	271	263	263
query70	1545	1359	1369	1359
query71	401	304	305	304
query72	6802	4868	4742	4742
query73	756	332	324	324
query74	6267	5851	5805	5805
query75	4535	3787	3654	3654
query76	4807	1106	1177	1106
query77	639	265	250	250
query78	12340	11869	11455	11455
query79	7137	634	648	634
query80	2589	400	386	386
query81	512	240	234	234
query82	1721	98	96	96
query83	179	130	134	130
query84	256	71	74	71
query85	1303	317	312	312
query86	363	299	311	299
query87	3230	3045	3087	3045
query88	4955	2283	2275	2275
query89	408	297	292	292
query90	1799	225	217	217
query91	174	135	125	125
query92	61	51	50	50
query93	5372	541	535	535
query94	873	211	204	204
query95	2125	1961	1974	1961
query96	644	329	323	323
query97	6457	6360	6320	6320
query98	214	203	202	202
query99	2744	920	878	878
Total cold run time: 316753 ms
Total hot run time: 211546 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.3 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 32d4321d73d462358cee535f4bf7576b11f3c35e, data reload: false

query1	0.02	0.02	0.02
query2	0.06	0.03	0.02
query3	0.25	0.05	0.05
query4	1.80	0.07	0.06
query5	0.55	0.53	0.52
query6	1.26	0.61	0.62
query7	0.02	0.01	0.01
query8	0.03	0.03	0.02
query9	0.51	0.49	0.47
query10	0.53	0.55	0.53
query11	0.11	0.09	0.09
query12	0.12	0.09	0.09
query13	0.62	0.61	0.61
query14	0.77	0.79	0.79
query15	0.78	0.76	0.78
query16	0.36	0.36	0.37
query17	1.03	1.03	1.02
query18	0.24	0.22	0.28
query19	1.92	1.83	1.83
query20	0.02	0.01	0.01
query21	15.46	0.55	0.55
query22	2.37	2.45	1.61
query23	17.02	0.87	0.90
query24	7.81	0.62	0.86
query25	0.39	0.10	0.05
query26	0.83	0.16	0.15
query27	0.05	0.04	0.03
query28	5.86	0.75	0.76
query29	12.72	2.23	2.38
query30	0.58	0.58	0.54
query31	2.82	0.39	0.37
query32	3.38	0.51	0.49
query33	3.12	3.07	3.07
query34	15.24	4.78	4.79
query35	4.84	4.83	4.85
query36	1.07	1.03	1.02
query37	0.06	0.05	0.04
query38	0.04	0.02	0.02
query39	0.02	0.01	0.02
query40	0.16	0.14	0.15
query41	0.07	0.01	0.01
query42	0.02	0.01	0.02
query43	0.03	0.02	0.02
Total cold run time: 104.96 s
Total hot run time: 30.3 s

@doris-robot
Copy link

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit 32d4321d73d462358cee535f4bf7576b11f3c35e with default session variables
Stream load json:         20 seconds loaded 2358488459 Bytes, about 112 MB/s
Stream load orc:          58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet:      31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select:       21.0 seconds inserted 10000000 Rows, about 476K ops/s

@Jibing-Li Jibing-Li merged commit b7fc188 into apache:branch-2.0 Sep 26, 2024
21 of 23 checks passed
@Jibing-Li Jibing-Li deleted the fix2.0 branch September 26, 2024 03:48
Jibing-Li added a commit that referenced this pull request Oct 8, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/planner Issues or PRs related to the query planner kind/test
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants