[fix](group commit) Fix replay wal core because undefined TLoadSourceType #30571

mymeiyi · 2024-01-30T08:26:46Z

Proposed changes

*** SIGSEGV address not mapped to object (@0x70) received by PID 3539569 (TID 3539924 OR 0x7fde59c8d700) from PID 112;
 stack trace: ***
 0# doris::signal::(anonymous namespace)::FailureSignalHandler(int, siginfo_t*, void*) at /mnt/disk2/meiyi/clion_workspace/doris_master2/be/src/common/signal_handler.h:417
 1# 0x00007FDFF97DB570 in /lib64/libc.so.6
 2# std::_Rb_tree<int, std::pair<int const, long>, std::_Select1st<std::pair<int const, long> >, std::less<int>, std::allocator<std::pair<int const, long> > >::begin() at /mnt/disk2/meiyi/soft/ldb_toolchain_2/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_tree.h:996
 3# std::map<int, long, std::less<int>, std::allocator<std::pair<int const, long> > >::begin() at /mnt/disk2/meiyi/soft/ldb_toolchain_2/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_map.h:357
 4# doris::KafkaLoadInfo::reset_offset() at /mnt/disk2/meiyi/clion_workspace/doris_master2/be/src/runtime/stream_load/stream_load_context.h:67
 5# doris::StreamLoadExecutor::execute_plan_fragment(std::shared_ptr<doris::StreamLoadContext>)::$_0::operator()(doris::RuntimeState*, doris::Status*) const at /mnt/disk2/meiyi/clion_workspace/doris_master2/be/src/runtime/stream_load/stream_load_executor.cpp:127
 6# void std::__invoke_impl<void, doris::StreamLoadExecutor::execute_plan_fragment(std::shared_ptr<doris::StreamLoadContext>)::$_0&, doris::RuntimeState*, doris::Status*>(std::__invoke_other, doris::StreamLoadExecutor::execute_plan_fragment(std::shared_ptr<doris::StreamLoadContext>)::$_0&, doris::RuntimeState*&&, doris::Status*&&) at /mnt/disk2/meiyi/soft/ldb_toolchain_2/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:61

Further comments

If this is a relatively large or complex change, kick off the discussion at [email protected] by explaining why you chose the solution you did and what alternatives you considered, etc...

mymeiyi · 2024-01-30T08:28:19Z

run buildall

github-actions · 2024-01-30T08:36:13Z

clang-tidy review says "All clean, LGTM! 👍"

doris-robot · 2024-01-30T08:47:27Z

TPC-H: Total hot run time: 36882 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit a7dd40f61c2c4358f7e11a85e9c3cef544a539a0, data reload: false

------ Round 1 ----------------------------------
q1	17656	4617	4393	4393
q2	2040	137	131	131
q3	10602	912	937	912
q4	4646	689	730	689
q5	7675	2847	2861	2847
q6	184	119	120	119
q7	1123	714	708	708
q8	9258	1991	2006	1991
q9	7172	6324	6343	6324
q10	8102	2432	2436	2432
q11	422	206	204	204
q12	740	266	271	266
q13	17989	3287	3301	3287
q14	273	260	240	240
q15	526	496	492	492
q16	464	411	423	411
q17	938	482	500	482
q18	6717	6095	5920	5920
q19	1554	1488	1361	1361
q20	612	336	310	310
q21	6711	3161	3053	3053
q22	803	310	310	310
Total cold run time: 106207 ms
Total hot run time: 36882 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4519	4383	4343	4343
q2	344	238	230	230
q3	3000	2852	2833	2833
q4	1809	1599	1672	1599
q5	5260	5322	5258	5258
q6	195	113	115	113
q7	2132	1806	1739	1739
q8	3112	3283	3262	3262
q9	8380	8348	8344	8344
q10	5793	3602	3574	3574
q11	539	459	473	459
q12	731	578	585	578
q13	5656	3123	3123	3123
q14	290	250	259	250
q15	531	486	491	486
q16	521	483	487	483
q17	1828	1695	1741	1695
q18	7981	7691	7600	7600
q19	6631	1509	1550	1509
q20	2124	1903	1902	1902
q21	4834	4608	4782	4608
q22	565	447	497	447
Total cold run time: 66775 ms
Total hot run time: 54435 ms

doris-robot · 2024-01-30T08:56:33Z

TeamCity be ut coverage result:
Function Coverage: 36.20% (8625/23827)
Line Coverage: 28.26% (70537/249625)
Region Coverage: 27.25% (36388/133546)
Branch Coverage: 24.05% (18648/77554)
Coverage Report: http://coverage.selectdb-in.cc/coverage/a7dd40f61c2c4358f7e11a85e9c3cef544a539a0_a7dd40f61c2c4358f7e11a85e9c3cef544a539a0/report/index.html

doris-robot · 2024-01-30T08:58:29Z

TPC-DS: Total hot run time: 172343 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit a7dd40f61c2c4358f7e11a85e9c3cef544a539a0, data reload: false

query1	943	336	339	336
query2	6557	2069	1878	1878
query3	6705	196	196	196
query4	28773	22163	22110	22110
query5	4444	407	349	349
query6	248	154	155	154
query7	4617	268	254	254
query8	259	171	178	171
query9	8957	2278	2261	2261
query10	415	208	197	197
query11	16603	15472	15332	15332
query12	114	68	63	63
query13	1682	368	370	368
query14	9273	6548	6404	6404
query15	207	168	182	168
query16	6168	265	247	247
query17	1810	467	472	467
query18	1772	249	250	249
query19	175	143	130	130
query20	68	62	63	62
query21	196	128	135	128
query22	4932	4638	4752	4638
query23	31214	30292	30333	30292
query24	12384	2753	2759	2753
query25	616	296	302	296
query26	2033	138	138	138
query27	3072	283	284	283
query28	7490	1845	1843	1843
query29	2314	627	619	619
query30	285	133	137	133
query31	901	723	717	717
query32	90	50	49	49
query33	714	210	200	200
query34	1186	463	463	463
query35	838	774	751	751
query36	1273	1180	1218	1180
query37	180	57	64	57
query38	3307	3157	3193	3157
query39	1329	1259	1258	1258
query40	338	87	82	82
query41	37	37	34	34
query42	84	77	84	77
query43	521	473	493	473
query44	1170	679	672	672
query45	188	173	170	170
query46	1038	656	640	640
query47	1517	1475	1450	1450
query48	380	324	292	292
query49	1279	272	276	272
query50	682	318	305	305
query51	5243	5099	5129	5099
query52	84	85	75	75
query53	321	252	252	252
query54	244	184	180	180
query55	80	74	70	70
query56	171	168	163	163
query57	992	927	978	927
query58	179	157	161	157
query59	2460	2441	2409	2409
query60	201	177	171	171
query61	83	83	82	82
query62	655	354	362	354
query63	271	262	261	261
query64	5642	1755	1709	1709
query65	3309	3215	3222	3215
query66	1487	317	327	317
query67	14372	13977	14427	13977
query68	3984	497	510	497
query69	448	312	308	308
query70	1535	1500	1461	1461
query71	305	208	202	202
query72	3727	2832	2831	2831
query73	688	311	301	301
query74	6630	6231	6186	6186
query75	2748	2294	2295	2294
query76	2941	958	900	900
query77	304	228	223	223
query78	9059	8710	8553	8553
query79	2284	493	489	489
query80	1327	320	305	305
query81	511	202	208	202
query82	664	84	78	78
query83	222	115	108	108
query84	231	75	77	75
query85	2136	339	319	319
query86	513	408	373	373
query87	3417	3273	3304	3273
query88	3913	2171	2171	2171
query89	405	361	353	353
query90	1963	179	176	176
query91	147	115	115	115
query92	57	47	45	45
query93	2009	449	410	410
query94	1355	155	153	153
query95	502	456	444	444
query96	613	318	304	304
query97	4254	4131	4125	4125
query98	212	193	189	189
query99	1027	644	667	644
Total cold run time: 277303 ms
Total hot run time: 172343 ms

doris-robot · 2024-01-30T09:04:21Z

ClickBench: Total hot run time: 31.16 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit a7dd40f61c2c4358f7e11a85e9c3cef544a539a0, data reload: false

query1	0.04	0.03	0.03
query2	0.06	0.02	0.03
query3	0.22	0.06	0.06
query4	1.68	0.10	0.09
query5	0.53	0.51	0.52
query6	1.18	0.63	0.63
query7	0.01	0.01	0.02
query8	0.04	0.02	0.02
query9	0.55	0.49	0.50
query10	0.54	0.54	0.56
query11	0.12	0.08	0.08
query12	0.11	0.09	0.09
query13	0.61	0.61	0.60
query14	0.79	0.80	0.79
query15	0.79	0.79	0.76
query16	0.39	0.39	0.39
query17	1.03	1.02	1.04
query18	0.19	0.28	0.27
query19	1.89	1.80	1.79
query20	0.01	0.01	0.01
query21	15.46	0.56	0.56
query22	3.19	2.03	2.02
query23	17.42	0.94	0.78
query24	2.77	0.94	0.61
query25	0.29	0.25	0.11
query26	0.49	0.13	0.14
query27	0.05	0.06	0.05
query28	12.70	0.84	0.88
query29	12.93	3.15	3.31
query30	0.62	0.55	0.53
query31	2.81	0.36	0.35
query32	3.48	0.47	0.48
query33	3.24	3.29	3.34
query34	17.44	4.49	4.54
query35	4.46	4.27	4.36
query36	1.23	1.05	1.05
query37	0.06	0.04	0.04
query38	0.03	0.03	0.03
query39	0.02	0.02	0.02
query40	0.16	0.12	0.12
query41	0.07	0.02	0.02
query42	0.03	0.01	0.02
query43	0.03	0.02	0.02
Total cold run time: 109.76 s
Total hot run time: 31.16 s

doris-robot · 2024-01-30T09:07:24Z

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit a7dd40f61c2c4358f7e11a85e9c3cef544a539a0 with default session variables
Stream load json:         18 seconds loaded 2358488459 Bytes, about 124 MB/s
Stream load orc:          58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet:      31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select:       13.6 seconds inserted 10000000 Rows, about 735K ops/s

mymeiyi · 2024-01-30T09:28:34Z

run p0

…Type

mymeiyi · 2024-01-30T10:42:52Z

run buildall

github-actions · 2024-01-30T10:52:09Z

clang-tidy review says "All clean, LGTM! 👍"

doris-robot · 2024-01-30T11:04:04Z

TeamCity be ut coverage result:
Function Coverage: 36.19% (8626/23835)
Line Coverage: 28.26% (70551/249687)
Region Coverage: 27.26% (36408/133580)
Branch Coverage: 24.04% (18647/77562)
Coverage Report: http://coverage.selectdb-in.cc/coverage/cc88205776e0561225e65732dcfdc3658423a48a_cc88205776e0561225e65732dcfdc3658423a48a/report/index.html

doris-robot · 2024-01-30T11:10:56Z

TPC-H: Total hot run time: 36616 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit cc88205776e0561225e65732dcfdc3658423a48a, data reload: false

------ Round 1 ----------------------------------
q1	17677	4467	4332	4332
q2	2038	140	132	132
q3	10600	904	891	891
q4	4681	719	704	704
q5	7653	2729	2790	2729
q6	181	120	119	119
q7	1139	708	702	702
q8	9310	2008	1982	1982
q9	7212	6351	6301	6301
q10	8074	2414	2418	2414
q11	414	216	194	194
q12	730	263	258	258
q13	18039	3282	3345	3282
q14	276	253	257	253
q15	525	482	499	482
q16	458	413	421	413
q17	928	502	497	497
q18	6727	5962	5966	5962
q19	1539	1386	1338	1338
q20	609	335	327	327
q21	6675	3013	3048	3013
q22	800	300	291	291
Total cold run time: 106285 ms
Total hot run time: 36616 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4470	4472	4267	4267
q2	345	225	235	225
q3	2965	2824	2802	2802
q4	1840	1616	1663	1616
q5	5155	5220	5176	5176
q6	189	112	113	112
q7	2110	1726	1755	1726
q8	3078	3193	3247	3193
q9	8361	8235	8262	8235
q10	5805	3591	3577	3577
q11	527	457	463	457
q12	733	589	567	567
q13	12123	3093	3058	3058
q14	285	243	256	243
q15	526	487	497	487
q16	516	460	462	460
q17	1838	1654	1700	1654
q18	8061	7629	7580	7580
q19	9621	1533	1532	1532
q20	2148	1910	1897	1897
q21	4918	4507	4602	4507
q22	567	500	459	459
Total cold run time: 76181 ms
Total hot run time: 53830 ms

doris-robot · 2024-01-30T11:21:45Z

TPC-DS: Total hot run time: 172888 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit cc88205776e0561225e65732dcfdc3658423a48a, data reload: false

query1	917	328	320	320
query2	6596	1959	2036	1959
query3	6694	214	202	202
query4	27340	22017	22026	22017
query5	4452	369	354	354
query6	243	154	173	154
query7	4600	266	263	263
query8	246	171	178	171
query9	9133	2225	2222	2222
query10	414	214	195	195
query11	17016	15358	15449	15358
query12	122	67	65	65
query13	1683	364	369	364
query14	9238	6468	6474	6468
query15	224	175	182	175
query16	6471	271	262	262
query17	1865	499	482	482
query18	1785	271	271	271
query19	174	134	138	134
query20	73	77	70	70
query21	197	128	136	128
query22	5173	4865	5035	4865
query23	30982	30217	30241	30217
query24	8609	2742	2754	2742
query25	539	319	307	307
query26	887	136	139	136
query27	2792	291	279	279
query28	5753	1807	1812	1807
query29	907	626	615	615
query30	277	134	133	133
query31	908	698	727	698
query32	89	49	49	49
query33	460	212	211	211
query34	833	458	463	458
query35	839	769	750	750
query36	1318	1213	1203	1203
query37	95	58	58	58
query38	3298	3153	3158	3153
query39	1318	1249	1231	1231
query40	192	87	80	80
query41	36	36	35	35
query42	94	79	88	79
query43	502	501	472	472
query44	1028	670	678	670
query45	191	177	172	172
query46	1058	636	646	636
query47	1474	1451	1499	1451
query48	409	301	309	301
query49	1118	285	282	282
query50	694	304	304	304
query51	5293	5153	5114	5114
query52	85	75	76	75
query53	327	257	256	256
query54	225	184	182	182
query55	79	76	74	74
query56	170	152	169	152
query57	1010	898	907	898
query58	183	162	161	161
query59	2389	2372	2450	2372
query60	201	179	178	178
query61	85	83	89	83
query62	585	352	352	352
query63	271	252	252	252
query64	4072	1779	1754	1754
query65	3288	3225	3229	3225
query66	1143	307	309	307
query67	14592	14096	14192	14096
query68	4027	501	516	501
query69	431	302	305	302
query70	1559	1496	1432	1432
query71	272	215	210	210
query72	3732	2835	2889	2835
query73	684	319	320	319
query74	6603	6203	6166	6166
query75	2806	2290	2324	2290
query76	2958	965	951	951
query77	348	232	236	232
query78	9027	8631	8676	8631
query79	2308	496	501	496
query80	1290	317	320	317
query81	539	191	200	191
query82	702	78	77	77
query83	244	117	113	113
query84	238	74	72	72
query85	1782	351	333	333
query86	503	399	405	399
query87	3386	3236	3293	3236
query88	3723	2261	2252	2252
query89	434	353	329	329
query90	1886	188	186	186
query91	150	117	115	115
query92	54	46	41	41
query93	2113	424	408	408
query94	1415	157	155	155
query95	494	458	452	452
query96	622	335	324	324
query97	4281	4150	4117	4117
query98	219	190	184	184
query99	1109	666	663	663
Total cold run time: 265293 ms
Total hot run time: 172888 ms

doris-robot · 2024-01-30T11:27:10Z

ClickBench: Total hot run time: 30.46 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit cc88205776e0561225e65732dcfdc3658423a48a, data reload: false

query1	0.04	0.04	0.03
query2	0.06	0.02	0.02
query3	0.22	0.06	0.06
query4	1.68	0.09	0.10
query5	0.52	0.53	0.52
query6	1.27	0.63	0.62
query7	0.02	0.01	0.01
query8	0.04	0.03	0.03
query9	0.57	0.51	0.49
query10	0.56	0.56	0.56
query11	0.12	0.09	0.09
query12	0.11	0.08	0.09
query13	0.60	0.63	0.60
query14	0.79	0.81	0.79
query15	0.80	0.76	0.78
query16	0.38	0.38	0.38
query17	1.02	1.01	1.03
query18	0.22	0.26	0.23
query19	1.87	1.74	1.73
query20	0.01	0.01	0.01
query21	15.40	0.56	0.58
query22	2.34	2.67	1.91
query23	17.32	0.69	0.79
query24	2.80	1.45	0.52
query25	0.36	0.09	0.15
query26	0.60	0.13	0.13
query27	0.04	0.06	0.05
query28	11.32	0.86	0.82
query29	12.52	3.25	3.11
query30	0.64	0.53	0.55
query31	2.77	0.33	0.36
query32	3.37	0.48	0.47
query33	3.22	3.24	3.24
query34	15.95	4.29	4.33
query35	4.35	4.33	4.30
query36	1.11	1.05	1.06
query37	0.06	0.05	0.05
query38	0.04	0.03	0.03
query39	0.02	0.01	0.01
query40	0.16	0.13	0.13
query41	0.07	0.01	0.02
query42	0.02	0.01	0.02
query43	0.03	0.02	0.02
Total cold run time: 105.41 s
Total hot run time: 30.46 s

doris-robot · 2024-01-30T11:30:16Z

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit cc88205776e0561225e65732dcfdc3658423a48a with default session variables
Stream load json:         19 seconds loaded 2358488459 Bytes, about 118 MB/s
Stream load orc:          58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet:      32 seconds loaded 861443392 Bytes, about 25 MB/s
Insert into select:       14.9 seconds inserted 10000000 Rows, about 671K ops/s

dataroaring

LGTM

github-actions · 2024-01-30T15:40:38Z

PR approved by at least one committer and no changes requested.

github-actions · 2024-01-30T15:40:40Z

PR approved by anyone and no changes requested.

…Type (#30571)

[fix](group commit) Fix replay wal core because undefined TLoadSource…

cc88205

…Type

mymeiyi force-pushed the fix-wal branch from a7dd40f to cc88205 Compare January 30, 2024 10:42

dataroaring approved these changes Jan 30, 2024

View reviewed changes

github-actions bot added the approved Indicates a PR has been approved by one committer. label Jan 30, 2024

github-actions bot added the reviewed label Jan 30, 2024

ByteYue approved these changes Jan 30, 2024

View reviewed changes

gavinchou approved these changes Jan 30, 2024

View reviewed changes

dataroaring approved these changes Jan 30, 2024

View reviewed changes

dataroaring merged commit 9a24e70 into apache:master Jan 30, 2024
29 of 32 checks passed

yiguolei pushed a commit that referenced this pull request Jan 31, 2024

[fix](group commit) Fix replay wal core because undefined TLoadSource…

2143c0a

…Type (#30571)

yiguolei pushed a commit that referenced this pull request Jan 31, 2024

[fix](group commit) Fix replay wal core because undefined TLoadSource…

1742be1

…Type (#30571)

yiguolei pushed a commit that referenced this pull request Jan 31, 2024

[fix](group commit) Fix replay wal core because undefined TLoadSource…

a1ccf34

…Type (#30571)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[fix](group commit) Fix replay wal core because undefined TLoadSourceType #30571

[fix](group commit) Fix replay wal core because undefined TLoadSourceType #30571

mymeiyi commented Jan 30, 2024 •

edited

Loading

mymeiyi commented Jan 30, 2024

github-actions bot commented Jan 30, 2024

doris-robot commented Jan 30, 2024

doris-robot commented Jan 30, 2024

doris-robot commented Jan 30, 2024

doris-robot commented Jan 30, 2024

doris-robot commented Jan 30, 2024

mymeiyi commented Jan 30, 2024

mymeiyi commented Jan 30, 2024

github-actions bot commented Jan 30, 2024

doris-robot commented Jan 30, 2024

doris-robot commented Jan 30, 2024

doris-robot commented Jan 30, 2024

doris-robot commented Jan 30, 2024

doris-robot commented Jan 30, 2024

dataroaring left a comment

github-actions bot commented Jan 30, 2024

github-actions bot commented Jan 30, 2024

[fix](group commit) Fix replay wal core because undefined TLoadSourceType #30571

[fix](group commit) Fix replay wal core because undefined TLoadSourceType #30571

Conversation

mymeiyi commented Jan 30, 2024 • edited Loading

Proposed changes

Further comments

mymeiyi commented Jan 30, 2024

github-actions bot commented Jan 30, 2024

doris-robot commented Jan 30, 2024

doris-robot commented Jan 30, 2024

doris-robot commented Jan 30, 2024

doris-robot commented Jan 30, 2024

doris-robot commented Jan 30, 2024

mymeiyi commented Jan 30, 2024

mymeiyi commented Jan 30, 2024

github-actions bot commented Jan 30, 2024

doris-robot commented Jan 30, 2024

doris-robot commented Jan 30, 2024

doris-robot commented Jan 30, 2024

doris-robot commented Jan 30, 2024

doris-robot commented Jan 30, 2024

dataroaring left a comment

Choose a reason for hiding this comment

github-actions bot commented Jan 30, 2024

github-actions bot commented Jan 30, 2024

mymeiyi commented Jan 30, 2024 •

edited

Loading