Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[pipelineX](local exchange) Fix potential timeout problem #33022

Merged
merged 2 commits into from
Mar 29, 2024
Merged

[pipelineX](local exchange) Fix potential timeout problem #33022

merged 2 commits into from
Mar 29, 2024

Conversation

Gabriel39
Copy link
Contributor

Proposed changes

If pipeline task is cancelled, eos will be never to pass to sink operator. So local exchange will be never scheduled

Further comments

If this is a relatively large or complex change, kick off the discussion at [email protected] by explaining why you chose the solution you did and what alternatives you considered, etc...

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@Gabriel39
Copy link
Contributor Author

run buildall

github-actions[bot]
github-actions bot reviewed Mar 29, 2024
View reviewed changes
Copy link
Contributor

@github-actions github-actions bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

clang-tidy made some suggestions

be/src/pipeline/pipeline_x/local_exchange/local_exchange_sink_operator.cpp Show resolved Hide resolved
@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 35.58% (8839/24843)
Line Coverage: 27.30% (72446/265369)
Region Coverage: 26.51% (37519/141507)
Branch Coverage: 23.32% (19124/82016)
Coverage Report: http://coverage.selectdb-in.cc/coverage/e73362d145e549b92f815aa7531a385a31a0581d_e73362d145e549b92f815aa7531a385a31a0581d/report/index.html

@doris-robot
Copy link

TPC-H: Total hot run time: 38391 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit e73362d145e549b92f815aa7531a385a31a0581d, data reload: false

------ Round 1 ----------------------------------
q1	17615	4428	4419	4419
q2	1994	189	180	180
q3	10454	1069	1137	1069
q4	10189	770	758	758
q5	7472	2830	2705	2705
q6	211	127	129	127
q7	1013	607	582	582
q8	9220	2035	2028	2028
q9	7611	6574	6527	6527
q10	8586	3515	3568	3515
q11	464	241	241	241
q12	444	223	211	211
q13	18530	3023	2956	2956
q14	277	235	241	235
q15	501	471	468	468
q16	519	399	397	397
q17	968	586	657	586
q18	7269	6637	6656	6637
q19	5903	1467	1454	1454
q20	682	303	298	298
q21	3458	2857	2690	2690
q22	364	308	308	308
Total cold run time: 113744 ms
Total hot run time: 38391 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4318	4193	4194	4193
q2	376	268	264	264
q3	2971	2674	2682	2674
q4	1843	1527	1504	1504
q5	5246	5223	5193	5193
q6	205	121	121	121
q7	2183	1882	1892	1882
q8	3229	3273	3268	3268
q9	8558	8544	8565	8544
q10	3842	3692	3658	3658
q11	577	486	484	484
q12	733	588	572	572
q13	17465	2945	2981	2945
q14	306	275	279	275
q15	506	464	468	464
q16	495	445	419	419
q17	1790	1454	1442	1442
q18	7909	7981	7923	7923
q19	1678	1551	1599	1551
q20	2050	1839	1827	1827
q21	8438	4874	4941	4874
q22	540	473	481	473
Total cold run time: 75258 ms
Total hot run time: 54550 ms

@Gabriel39
Copy link
Contributor Author

run buildall

@github-actions GitHub Actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Mar 29, 2024
@github-actions GitHub Actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions GitHub Actions
Copy link
Contributor

PR approved by anyone and no changes requested.

@doris-robot
Copy link

TPC-H: Total hot run time: 38883 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 5c942d53dd2ab35913d3a2cc7e63e7940ddc09d7, data reload: false

------ Round 1 ----------------------------------
q1	17670	4979	4247	4247
q2	2079	191	203	191
q3	10508	1180	1170	1170
q4	10189	783	752	752
q5	7466	2813	2777	2777
q6	219	135	131	131
q7	958	530	541	530
q8	9210	1997	2015	1997
q9	8078	7125	7160	7125
q10	8471	3460	3528	3460
q11	443	233	229	229
q12	466	220	210	210
q13	17857	3028	2994	2994
q14	284	232	237	232
q15	519	477	460	460
q16	537	383	388	383
q17	982	627	579	579
q18	7467	6724	6726	6724
q19	1582	1460	1434	1434
q20	649	299	307	299
q21	3440	2658	2856	2658
q22	368	301	318	301
Total cold run time: 109442 ms
Total hot run time: 38883 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4327	4236	4131	4131
q2	370	270	259	259
q3	2973	2732	2702	2702
q4	1849	1484	1515	1484
q5	5295	5221	5259	5221
q6	212	125	127	125
q7	1763	1450	1418	1418
q8	3241	3356	3364	3356
q9	9319	9324	9310	9310
q10	3891	3616	3684	3616
q11	570	476	493	476
q12	735	587	597	587
q13	17533	2976	2992	2976
q14	307	277	277	277
q15	515	465	470	465
q16	478	428	423	423
q17	1924	1418	1428	1418
q18	7734	7445	7445	7445
q19	1671	1528	1508	1508
q20	1952	1733	1731	1731
q21	5021	4738	4767	4738
q22	541	481	441	441
Total cold run time: 72221 ms
Total hot run time: 54107 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 182216 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 5c942d53dd2ab35913d3a2cc7e63e7940ddc09d7, data reload: false

query1	910	1131	1123	1123
query2	6687	1939	1900	1900
query3	6652	218	213	213
query4	25089	21359	21318	21318
query5	4131	407	409	407
query6	271	182	177	177
query7	4600	298	296	296
query8	223	173	173	173
query9	8551	2278	2274	2274
query10	535	248	254	248
query11	14881	14312	14250	14250
query12	140	91	91	91
query13	1631	382	385	382
query14	9111	6863	6786	6786
query15	193	181	184	181
query16	7180	267	261	261
query17	1398	589	565	565
query18	1512	286	298	286
query19	207	162	159	159
query20	103	92	90	90
query21	197	125	126	125
query22	4964	4882	4768	4768
query23	33394	32858	32785	32785
query24	11928	2840	2830	2830
query25	649	396	393	393
query26	1650	158	155	155
query27	3100	317	323	317
query28	7358	1860	1854	1854
query29	914	634	627	627
query30	290	153	152	152
query31	987	704	728	704
query32	85	56	59	56
query33	698	260	250	250
query34	1058	487	481	481
query35	854	696	703	696
query36	1049	889	893	889
query37	129	69	69	69
query38	3609	3442	3430	3430
query39	1570	1556	1529	1529
query40	208	141	132	132
query41	51	46	54	46
query42	110	102	100	100
query43	496	448	462	448
query44	1142	752	731	731
query45	274	261	273	261
query46	1073	688	687	687
query47	1932	1850	1853	1850
query48	386	315	309	309
query49	1055	365	380	365
query50	771	389	399	389
query51	6906	6841	6712	6712
query52	107	97	93	93
query53	352	285	288	285
query54	303	245	240	240
query55	85	82	79	79
query56	257	236	238	236
query57	1235	1118	1125	1118
query58	234	215	224	215
query59	2806	2622	2672	2622
query60	266	247	252	247
query61	118	115	129	115
query62	670	451	442	442
query63	312	282	281	281
query64	6900	4186	4120	4120
query65	3131	3002	3029	3002
query66	1438	365	343	343
query67	15632	14887	14989	14887
query68	9444	549	555	549
query69	597	316	317	316
query70	1381	1141	1200	1141
query71	518	270	280	270
query72	6836	2613	2457	2457
query73	1677	335	341	335
query74	6927	6262	6391	6262
query75	3971	2311	2326	2311
query76	5549	995	1013	995
query77	610	259	267	259
query78	11072	10183	10191	10183
query79	12671	530	525	525
query80	1997	429	434	429
query81	538	223	218	218
query82	332	96	92	92
query83	228	174	174	174
query84	262	84	85	84
query85	1094	267	261	261
query86	351	294	293	293
query87	3779	3540	3506	3506
query88	5785	2432	2498	2432
query89	496	379	370	370
query90	2001	177	182	177
query91	132	100	104	100
query92	59	47	50	47
query93	6780	520	501	501
query94	1365	181	179	179
query95	417	314	309	309
query96	608	272	275	272
query97	2662	2470	2470	2470
query98	247	215	208	208
query99	1021	846	855	846
Total cold run time: 308145 ms
Total hot run time: 182216 ms

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 35.55% (8839/24862)
Line Coverage: 27.30% (72490/265571)
Region Coverage: 26.50% (37511/141545)
Branch Coverage: 23.31% (19129/82050)
Coverage Report: http://coverage.selectdb-in.cc/coverage/5c942d53dd2ab35913d3a2cc7e63e7940ddc09d7_5c942d53dd2ab35913d3a2cc7e63e7940ddc09d7/report/index.html

@doris-robot
Copy link

ClickBench: Total hot run time: 31.03 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 5c942d53dd2ab35913d3a2cc7e63e7940ddc09d7, data reload: false

query1	0.04	0.04	0.03
query2	0.08	0.04	0.04
query3	0.23	0.05	0.05
query4	1.67	0.08	0.09
query5	0.51	0.50	0.51
query6	1.15	0.66	0.66
query7	0.02	0.01	0.02
query8	0.05	0.04	0.04
query9	0.54	0.48	0.50
query10	0.55	0.56	0.54
query11	0.16	0.10	0.12
query12	0.14	0.12	0.11
query13	0.58	0.59	0.57
query14	0.75	0.78	0.77
query15	0.82	0.81	0.81
query16	0.35	0.37	0.36
query17	0.96	0.95	1.00
query18	0.22	0.23	0.27
query19	1.87	1.74	1.67
query20	0.01	0.01	0.01
query21	15.55	0.65	0.63
query22	2.51	3.30	3.42
query23	17.24	0.83	0.85
query24	1.16	0.22	0.24
query25	0.09	0.08	0.08
query26	0.24	0.17	0.17
query27	0.08	0.08	0.08
query28	14.01	0.93	0.92
query29	12.54	3.21	3.18
query30	0.27	0.08	0.10
query31	2.84	0.37	0.37
query32	3.27	0.46	0.45
query33	2.82	2.80	2.79
query34	16.32	4.42	4.40
query35	4.51	4.46	4.46
query36	0.63	0.46	0.47
query37	0.19	0.16	0.16
query38	0.16	0.14	0.14
query39	0.04	0.04	0.03
query40	0.17	0.15	0.15
query41	0.09	0.05	0.04
query42	0.06	0.05	0.04
query43	0.05	0.04	0.04
Total cold run time: 105.54 s
Total hot run time: 31.03 s

@doris-robot
Copy link

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit 5c942d53dd2ab35913d3a2cc7e63e7940ddc09d7 with default session variables
Stream load json:         19 seconds loaded 2358488459 Bytes, about 118 MB/s
Stream load orc:          59 seconds loaded 1101869774 Bytes, about 17 MB/s
Stream load parquet:      31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select:       13.8 seconds inserted 10000000 Rows, about 724K ops/s

@yiguolei yiguolei merged commit 52ed33f into apache:master Mar 29, 2024
26 of 30 checks passed
yiguolei pushed a commit that referenced this pull request Mar 29, 2024
@Gabriel39 @Doris-Extras
[pipelineX](local exchange) Fix potential timeout problem (#33022)
db3179e
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@github-actions github-actions[bot] github-actions[bot] left review comments

@yiguolei yiguolei yiguolei approved these changes

@Mryange Mryange Mryange approved these changes

Assignees
No one assigned
Labels
approved Indicates a PR has been approved by one committer. reviewed
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@Gabriel39 @doris-robot @yiguolei @Mryange