Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[scan](status) Finish execution if scanner failed #32966

Merged
merged 3 commits into from
Mar 29, 2024

Conversation

Gabriel39
Copy link
Contributor

Proposed changes

When scanner meet this error below

W20240327 19:08:40.074436 3465831 status.h:380] meet error status: [INTERNAL_ERROR]Merge block not match, self:[Nullable(Int64), Nullable(Int32), UInt8], input:[Nullable(Int64), Nullable(Int32)],

        0#  doris::Status doris::Status::Error<6, true, std::_cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >(std::basic_string_view<char, std::char_traits<char> >, std::cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&&, std::_cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&&) at /home/zcp/repo_center/doris_branch-2.1/doris/be/src/common/status.h:379
        1#  doris::Status doris::vectorized::MutableBlock::merge_impl<doris::vectorized::Block&>(doris::vectorized::Block&) at /home/zcp/repo_center/doris_branch-2.1/doris/be/src/vec/core/block.h:534
        2#  doris::Status doris::vectorized::MutableBlock::merge<doris::vectorized::Block&>(doris::vectorized::Block&) at /home/zcp/repo_center/doris_branch-2.1/doris/be/src/vec/core/block.h:512
        3#  doris::vectorized::ScannerScheduler::_scanner_scan(std::shared_ptr<doris::vectorized::ScannerContext>, std::shared_ptr<doris::vectorized::ScanTask>) at /home/zcp/repo_center/doris_branch-2.1/doris/be/src/vec/exec/scan/scanner_scheduler.cpp:278
        4#  doris::vectorized::ScannerScheduler::submit(std::shared_ptr<doris::vectorized::ScannerContext>, std::shared_ptr<doris::vectorized::ScanTask>)::$_1::operator()() const at /home/zcp/repo_center/doris_branch-2.1/doris/be/src/vec/exec/scan/scanner_scheduler.cpp:162

query will be hang because the error status is not processed by scanner.

Further comments

If this is a relatively large or complex change, kick off the discussion at [email protected] by explaining why you chose the solution you did and what alternatives you considered, etc...

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@Gabriel39
Copy link
Contributor Author

run buildall

Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 35.63% (8835/24797)
Line Coverage: 27.34% (72395/264820)
Region Coverage: 26.56% (37545/141355)
Branch Coverage: 23.37% (19152/81950)
Coverage Report: http://coverage.selectdb-in.cc/coverage/d835e9ebe6e11e8a430e68b0e0e9bb18a3c17008_d835e9ebe6e11e8a430e68b0e0e9bb18a3c17008/report/index.html

@doris-robot
Copy link

TPC-H: Total hot run time: 39014 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit d835e9ebe6e11e8a430e68b0e0e9bb18a3c17008, data reload: false

------ Round 1 ----------------------------------
q1	17622	4325	4248	4248
q2	2132	159	157	157
q3	10598	1236	1286	1236
q4	10458	813	816	813
q5	7499	3029	3028	3028
q6	220	124	124	124
q7	1080	621	609	609
q8	9346	2141	2230	2141
q9	7125	6496	6525	6496
q10	8433	3541	3611	3541
q11	445	232	224	224
q12	411	208	202	202
q13	17802	2861	2868	2861
q14	224	207	217	207
q15	516	471	466	466
q16	511	374	369	369
q17	1003	555	660	555
q18	7189	6533	6501	6501
q19	1709	1658	1567	1567
q20	578	261	259	259
q21	3732	3175	3109	3109
q22	351	301	302	301
Total cold run time: 108984 ms
Total hot run time: 39014 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4224	4219	4210	4210
q2	346	234	236	234
q3	3041	2900	2867	2867
q4	1888	1566	1546	1546
q5	5306	5312	5341	5312
q6	217	117	119	117
q7	2248	1812	1815	1812
q8	3379	3514	3579	3514
q9	8494	8501	8555	8501
q10	3890	3833	3870	3833
q11	559	449	440	440
q12	761	564	558	558
q13	16910	2874	2854	2854
q14	302	260	269	260
q15	524	470	460	460
q16	477	422	432	422
q17	1862	1553	1560	1553
q18	7433	7240	7236	7236
q19	1761	1741	1692	1692
q20	1963	1747	1717	1717
q21	4888	4637	4678	4637
q22	525	448	457	448
Total cold run time: 70998 ms
Total hot run time: 54223 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 181556 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit d835e9ebe6e11e8a430e68b0e0e9bb18a3c17008, data reload: false

query1	936	362	355	355
query2	6526	2012	1928	1928
query3	6703	212	207	207
query4	31887	21338	21300	21300
query5	4304	399	388	388
query6	276	175	168	168
query7	4621	293	290	290
query8	225	171	183	171
query9	9434	2338	2320	2320
query10	562	253	255	253
query11	14877	14243	14205	14205
query12	138	89	89	89
query13	1623	436	417	417
query14	8990	7897	7989	7897
query15	257	198	181	181
query16	8212	255	248	248
query17	1999	553	527	527
query18	2105	280	271	271
query19	350	145	148	145
query20	98	86	86	86
query21	205	128	127	127
query22	5095	4777	4844	4777
query23	33704	32935	32706	32706
query24	10575	2851	2859	2851
query25	658	385	406	385
query26	1578	159	151	151
query27	3003	345	353	345
query28	7586	1912	1892	1892
query29	934	635	604	604
query30	303	150	146	146
query31	985	722	751	722
query32	97	59	56	56
query33	765	254	251	251
query34	1071	500	482	482
query35	834	623	608	608
query36	998	897	901	897
query37	129	66	68	66
query38	3565	3401	3445	3401
query39	1478	1486	1426	1426
query40	213	114	116	114
query41	54	49	49	49
query42	106	96	98	96
query43	495	473	475	473
query44	1145	741	730	730
query45	278	258	273	258
query46	1115	693	697	693
query47	1917	1816	1848	1816
query48	429	374	362	362
query49	1150	336	360	336
query50	759	371	381	371
query51	6805	6828	6739	6739
query52	105	89	89	89
query53	341	273	267	267
query54	321	245	233	233
query55	83	80	79	79
query56	258	227	237	227
query57	1215	1127	1137	1127
query58	233	203	210	203
query59	2754	2594	2601	2594
query60	261	253	251	251
query61	116	117	116	116
query62	644	426	428	426
query63	305	281	278	278
query64	7023	4161	4062	4062
query65	3153	3053	3028	3028
query66	1443	387	373	373
query67	15435	14937	14835	14835
query68	8432	527	533	527
query69	628	403	384	384
query70	1260	1155	1152	1152
query71	499	273	273	273
query72	6395	2546	2345	2345
query73	743	317	315	315
query74	8019	6337	6430	6337
query75	3968	2199	2257	2199
query76	4896	878	925	878
query77	656	260	255	255
query78	10908	10102	10041	10041
query79	10782	516	526	516
query80	1954	381	368	368
query81	503	216	213	213
query82	808	90	84	84
query83	220	144	148	144
query84	287	75	77	75
query85	1433	313	307	307
query86	423	302	273	273
query87	3761	3504	3527	3504
query88	5286	2319	2310	2310
query89	499	365	369	365
query90	1988	175	180	175
query91	174	132	138	132
query92	67	48	49	48
query93	7223	503	482	482
query94	1176	174	185	174
query95	429	334	332	332
query96	627	266	271	266
query97	2702	2494	2512	2494
query98	225	215	203	203
query99	1241	821	826	821
Total cold run time: 314054 ms
Total hot run time: 181556 ms

@doris-robot
Copy link

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit d835e9ebe6e11e8a430e68b0e0e9bb18a3c17008 with default session variables
Stream load json:         18 seconds loaded 2358488459 Bytes, about 124 MB/s
Stream load orc:          59 seconds loaded 1101869774 Bytes, about 17 MB/s
Stream load parquet:      32 seconds loaded 861443392 Bytes, about 25 MB/s
Insert into select:       14.4 seconds inserted 10000000 Rows, about 694K ops/s

yiguolei
yiguolei previously approved these changes Mar 28, 2024
@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Mar 28, 2024
Copy link
Contributor

PR approved by at least one committer and no changes requested.

Copy link
Contributor

PR approved by anyone and no changes requested.

@Gabriel39
Copy link
Contributor Author

run buildall

@github-actions github-actions bot removed the approved Indicates a PR has been approved by one committer. label Mar 28, 2024
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

yiguolei
yiguolei previously approved these changes Mar 28, 2024
@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Mar 28, 2024
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 35.62% (8836/24808)
Line Coverage: 27.32% (72391/264959)
Region Coverage: 26.55% (37548/141411)
Branch Coverage: 23.35% (19144/81982)
Coverage Report: http://coverage.selectdb-in.cc/coverage/a262246da91529ec741a89f3ccbbf1509cdd309e_a262246da91529ec741a89f3ccbbf1509cdd309e/report/index.html

@doris-robot
Copy link

TPC-H: Total hot run time: 37741 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit a262246da91529ec741a89f3ccbbf1509cdd309e, data reload: false

------ Round 1 ----------------------------------
q1	17617	4376	4109	4109
q2	2116	161	151	151
q3	10582	1118	1214	1118
q4	10232	731	746	731
q5	7460	2964	2895	2895
q6	201	122	124	122
q7	1059	590	577	577
q8	9339	2087	2018	2018
q9	6911	6364	6348	6348
q10	8408	3450	3584	3450
q11	435	249	214	214
q12	422	197	192	192
q13	17785	2849	2871	2849
q14	250	198	204	198
q15	520	465	474	465
q16	505	372	377	372
q17	962	584	639	584
q18	7187	6456	6508	6456
q19	2135	1485	1444	1444
q20	568	249	242	242
q21	3626	2923	2951	2923
q22	344	283	284	283
Total cold run time: 108664 ms
Total hot run time: 37741 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4133	4056	4082	4056
q2	332	234	233	233
q3	2991	2876	2898	2876
q4	1842	1542	1561	1542
q5	5231	5215	5263	5215
q6	195	115	118	115
q7	2212	1838	1830	1830
q8	3180	3302	3310	3302
q9	8446	8448	8388	8388
q10	3808	3805	3980	3805
q11	560	455	477	455
q12	785	576	616	576
q13	16961	3098	3049	3049
q14	308	265	266	265
q15	506	486	471	471
q16	519	473	448	448
q17	1796	1568	1521	1521
q18	7968	7699	7646	7646
q19	7562	1610	1612	1610
q20	2038	1812	1817	1812
q21	5042	5024	4826	4826
q22	567	456	470	456
Total cold run time: 76982 ms
Total hot run time: 54497 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 182558 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit a262246da91529ec741a89f3ccbbf1509cdd309e, data reload: false

query1	917	345	352	345
query2	6335	2109	1939	1939
query3	6700	205	202	202
query4	31495	21477	21543	21477
query5	4307	403	404	403
query6	299	187	182	182
query7	4632	288	295	288
query8	236	180	178	178
query9	9287	2282	2279	2279
query10	453	247	276	247
query11	15378	14322	14293	14293
query12	144	90	91	90
query13	1630	420	414	414
query14	9949	7445	7669	7445
query15	247	183	187	183
query16	8200	256	272	256
query17	1939	564	529	529
query18	2115	298	274	274
query19	322	156	144	144
query20	89	86	88	86
query21	205	123	127	123
query22	5044	4834	4821	4821
query23	33322	33311	33006	33006
query24	10795	2982	2944	2944
query25	631	413	373	373
query26	1116	166	164	164
query27	2614	357	364	357
query28	7552	1900	1928	1900
query29	899	659	658	658
query30	334	161	156	156
query31	1032	771	759	759
query32	99	61	59	59
query33	787	267	261	261
query34	1332	504	539	504
query35	918	709	715	709
query36	1096	914	918	914
query37	120	69	68	68
query38	3750	3724	3610	3610
query39	1658	1596	1607	1596
query40	181	114	118	114
query41	51	52	49	49
query42	108	99	97	97
query43	492	462	459	459
query44	1179	752	750	750
query45	305	270	260	260
query46	1103	704	732	704
query47	2043	1933	1905	1905
query48	479	362	371	362
query49	909	361	351	351
query50	772	385	376	376
query51	6892	6782	6703	6703
query52	101	98	92	92
query53	350	287	279	279
query54	306	256	247	247
query55	93	80	80	80
query56	266	242	240	240
query57	1270	1219	1191	1191
query58	246	218	210	210
query59	2887	2519	2558	2519
query60	262	241	235	235
query61	92	93	91	91
query62	603	444	431	431
query63	297	276	271	271
query64	5196	3888	3957	3888
query65	3065	3027	3011	3011
query66	781	349	353	349
query67	15512	14912	14848	14848
query68	8595	534	527	527
query69	636	387	395	387
query70	1263	1181	1160	1160
query71	503	262	261	261
query72	6260	2523	2333	2333
query73	735	316	310	310
query74	8430	6468	6465	6465
query75	3769	2220	2224	2220
query76	5316	862	870	862
query77	608	251	256	251
query78	11070	10113	10138	10113
query79	10813	512	518	512
query80	2011	373	365	365
query81	527	217	218	217
query82	682	85	83	83
query83	211	149	150	149
query84	287	80	79	79
query85	1327	319	310	310
query86	411	304	305	304
query87	3754	3564	3528	3528
query88	4600	2322	2303	2303
query89	497	364	369	364
query90	1960	182	174	174
query91	168	138	131	131
query92	65	46	45	45
query93	6843	490	488	488
query94	1117	178	173	173
query95	402	308	307	307
query96	595	276	278	276
query97	2691	2491	2450	2450
query98	221	214	214	214
query99	1149	879	849	849
Total cold run time: 311335 ms
Total hot run time: 182558 ms

@doris-robot
Copy link

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit a262246da91529ec741a89f3ccbbf1509cdd309e with default session variables
Stream load json:         18 seconds loaded 2358488459 Bytes, about 124 MB/s
Stream load orc:          59 seconds loaded 1101869774 Bytes, about 17 MB/s
Stream load parquet:      31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select:       13.7 seconds inserted 10000000 Rows, about 729K ops/s

Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@eldenmoon
Copy link
Member

run buildall

@github-actions github-actions bot removed the approved Indicates a PR has been approved by one committer. label Mar 28, 2024
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@eldenmoon
Copy link
Member

run buildall

Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

TPC-H: Total hot run time: 37998 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 932c624f98c0750ef38d65e15325a762da1689b9, data reload: false

------ Round 1 ----------------------------------
q1	17639	4611	4138	4138
q2	2115	159	158	158
q3	10738	1189	1240	1189
q4	10871	810	753	753
q5	7653	2988	2932	2932
q6	212	129	127	127
q7	1076	606	608	606
q8	9410	2039	2027	2027
q9	7045	6444	6402	6402
q10	9826	3496	3549	3496
q11	432	221	216	216
q12	451	199	203	199
q13	17794	2871	2846	2846
q14	232	200	219	200
q15	508	480	466	466
q16	508	372	370	370
q17	962	528	618	528
q18	7135	6497	6505	6497
q19	1554	1468	1395	1395
q20	542	260	268	260
q21	3539	2934	2910	2910
q22	354	288	283	283
Total cold run time: 110596 ms
Total hot run time: 37998 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4123	4046	4054	4046
q2	331	233	232	232
q3	2955	2815	2813	2813
q4	1837	1574	1567	1567
q5	5215	5217	5232	5217
q6	195	116	119	116
q7	2233	1835	1822	1822
q8	3196	3333	3313	3313
q9	8436	8461	8384	8384
q10	3785	3780	3770	3770
q11	542	446	445	445
q12	710	535	557	535
q13	16906	2905	2840	2840
q14	289	266	247	247
q15	496	471	454	454
q16	490	415	433	415
q17	1730	1500	1480	1480
q18	7534	7219	7131	7131
q19	1614	1491	1529	1491
q20	1899	1733	1698	1698
q21	4652	4600	4755	4600
q22	523	463	450	450
Total cold run time: 69691 ms
Total hot run time: 53066 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 181609 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 932c624f98c0750ef38d65e15325a762da1689b9, data reload: false

query1	956	372	369	369
query2	6538	2101	1819	1819
query3	6701	226	223	223
query4	31794	21341	21315	21315
query5	4294	416	440	416
query6	274	178	177	177
query7	4640	307	298	298
query8	233	178	182	178
query9	9222	2241	2228	2228
query10	569	269	264	264
query11	15301	14221	14229	14221
query12	145	94	90	90
query13	1628	433	419	419
query14	11393	7817	7525	7525
query15	251	186	183	183
query16	8169	264	270	264
query17	1942	586	562	562
query18	2119	297	292	292
query19	318	168	165	165
query20	95	99	97	97
query21	205	128	123	123
query22	4969	4785	4880	4785
query23	33561	32997	32667	32667
query24	11996	2853	2880	2853
query25	637	374	372	372
query26	1817	155	160	155
query27	3033	330	319	319
query28	7500	1855	1846	1846
query29	1014	636	613	613
query30	308	147	146	146
query31	1002	726	743	726
query32	87	57	60	57
query33	759	266	264	264
query34	1103	490	497	490
query35	904	709	703	703
query36	1015	885	895	885
query37	270	67	63	63
query38	3564	3430	3455	3430
query39	1580	1558	1515	1515
query40	291	110	111	110
query41	50	45	47	45
query42	107	97	96	96
query43	470	443	448	443
query44	1226	732	734	732
query45	282	259	238	238
query46	1112	706	728	706
query47	1918	1822	1841	1822
query48	454	368	351	351
query49	1242	350	362	350
query50	768	382	391	382
query51	6703	6679	6662	6662
query52	108	93	92	92
query53	357	280	298	280
query54	324	245	247	245
query55	90	84	82	82
query56	265	243	239	239
query57	1215	1121	1142	1121
query58	237	219	218	218
query59	2818	2732	2515	2515
query60	275	249	256	249
query61	118	117	128	117
query62	665	436	425	425
query63	312	283	297	283
query64	6992	4181	4123	4123
query65	3093	3018	3031	3018
query66	1493	383	390	383
query67	15455	14829	15079	14829
query68	7253	549	530	530
query69	633	399	390	390
query70	1271	1212	1194	1194
query71	512	277	272	272
query72	6469	2590	2365	2365
query73	743	328	324	324
query74	7014	6510	6364	6364
query75	3528	2254	2246	2246
query76	4937	930	835	835
query77	621	260	271	260
query78	10964	10226	10140	10140
query79	12441	526	536	526
query80	2007	378	390	378
query81	512	212	215	212
query82	768	87	89	87
query83	218	148	148	148
query84	289	74	81	74
query85	1516	317	316	316
query86	484	306	300	300
query87	3772	3539	3509	3509
query88	5496	2366	2367	2366
query89	539	385	370	370
query90	1951	181	182	181
query91	179	149	138	138
query92	59	50	49	49
query93	7681	508	488	488
query94	1229	178	182	178
query95	406	316	320	316
query96	620	276	272	272
query97	2667	2494	2507	2494
query98	234	214	211	211
query99	1252	803	863	803
Total cold run time: 318611 ms
Total hot run time: 181609 ms

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 35.61% (8838/24816)
Line Coverage: 27.32% (72414/265022)
Region Coverage: 26.56% (37558/141433)
Branch Coverage: 23.36% (19152/81996)
Coverage Report: http://coverage.selectdb-in.cc/coverage/932c624f98c0750ef38d65e15325a762da1689b9_932c624f98c0750ef38d65e15325a762da1689b9/report/index.html

@doris-robot
Copy link

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit 932c624f98c0750ef38d65e15325a762da1689b9 with default session variables
Stream load json:         18 seconds loaded 2358488459 Bytes, about 124 MB/s
Stream load orc:          59 seconds loaded 1101869774 Bytes, about 17 MB/s
Stream load parquet:      31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select:       13.9 seconds inserted 10000000 Rows, about 719K ops/s

Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Mar 29, 2024
Copy link
Member

@eldenmoon eldenmoon left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@yiguolei yiguolei merged commit f381366 into apache:master Mar 29, 2024
26 of 30 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by one committer. reviewed
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants