Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[log](pipeline)add more log in scan localstate #33062

Merged
merged 1 commit into from
Apr 9, 2024

Conversation

Mryange
Copy link
Contributor

@Mryange Mryange commented Mar 30, 2024

Proposed changes

error log

PipelineTask[this = 0x61901570a780, state = BLOCKED_FOR_SOURCE, dry run = false, elapse time = 599.986904548s], block dependency = OLAP_SCAN_OPERATOR_DEPENDENCY: id=4, block task = 1, ready=false, _always_ready=false, is cancelled=false, is running = false
operators: 
OLAP_SCAN_OPERATOR: id=4, parallel_tasks=1, _eos = false
  DATA_STREAM_SINK_OPERATOR: id=5, Sink Buffer: (_should_stop = false, _busy_channels = 0)
Read Dependency Information: 

parallel_tasks=1, _eos = false with out _scanner_ctx

Further comments

If this is a relatively large or complex change, kick off the discussion at [email protected] by explaining why you chose the solution you did and what alternatives you considered, etc...

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@Mryange
Copy link
Contributor Author

Mryange commented Mar 30, 2024

run buildall

Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

TPC-H: Total hot run time: 38555 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 9167b2c2dcafbf8c06e630270fd4cdcec651454a, data reload: false

------ Round 1 ----------------------------------
q1	17970	9316	4401	4401
q2	2584	201	192	192
q3	11072	1153	1153	1153
q4	10988	844	676	676
q5	7550	2863	2815	2815
q6	227	136	135	135
q7	1050	651	613	613
q8	9561	2076	2079	2076
q9	7618	6793	6609	6609
q10	8514	3504	3511	3504
q11	481	241	234	234
q12	473	213	214	213
q13	17793	2915	2956	2915
q14	276	234	241	234
q15	512	481	474	474
q16	536	386	390	386
q17	982	539	602	539
q18	7234	6711	6684	6684
q19	5108	1424	1448	1424
q20	662	307	308	307
q21	3523	2671	2752	2671
q22	362	300	311	300
Total cold run time: 115076 ms
Total hot run time: 38555 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4367	4228	4249	4228
q2	388	278	267	267
q3	3001	2736	2725	2725
q4	1834	1498	1552	1498
q5	5307	5249	5279	5249
q6	210	122	124	122
q7	2251	1881	1885	1881
q8	3223	3292	3252	3252
q9	8633	8508	8540	8508
q10	3899	3724	3734	3724
q11	568	484	472	472
q12	747	600	630	600
q13	17442	2939	2982	2939
q14	302	278	257	257
q15	518	462	470	462
q16	494	433	427	427
q17	1790	1436	1413	1413
q18	7676	7402	7395	7395
q19	1626	1533	1457	1457
q20	1939	1744	1741	1741
q21	5031	4893	4890	4890
q22	539	454	429	429
Total cold run time: 71785 ms
Total hot run time: 53936 ms

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 35.55% (8840/24864)
Line Coverage: 27.29% (72481/265608)
Region Coverage: 26.50% (37508/141564)
Branch Coverage: 23.31% (19127/82058)
Coverage Report: http://coverage.selectdb-in.cc/coverage/9167b2c2dcafbf8c06e630270fd4cdcec651454a_9167b2c2dcafbf8c06e630270fd4cdcec651454a/report/index.html

@doris-robot
Copy link

TPC-DS: Total hot run time: 182420 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 9167b2c2dcafbf8c06e630270fd4cdcec651454a, data reload: false

query1	897	1116	1132	1116
query2	7201	1898	1881	1881
query3	6653	207	206	206
query4	33118	21578	21333	21333
query5	4256	403	415	403
query6	223	182	168	168
query7	3976	293	284	284
query8	221	171	177	171
query9	5537	2315	2289	2289
query10	484	232	252	232
query11	14799	14268	14256	14256
query12	148	92	88	88
query13	970	368	372	368
query14	8758	6822	6892	6822
query15	227	173	193	173
query16	7571	263	258	258
query17	1581	576	564	564
query18	1906	290	280	280
query19	214	161	154	154
query20	95	86	90	86
query21	206	127	130	127
query22	5022	4880	4825	4825
query23	33355	33028	32874	32874
query24	12787	2803	2856	2803
query25	655	384	389	384
query26	1890	151	154	151
query27	3290	319	321	319
query28	7699	1834	1817	1817
query29	993	629	624	624
query30	284	151	149	149
query31	908	713	696	696
query32	63	55	53	53
query33	588	238	238	238
query34	947	466	468	466
query35	839	676	684	676
query36	1049	916	898	898
query37	262	66	70	66
query38	3573	3486	3433	3433
query39	1572	1556	1534	1534
query40	278	126	124	124
query41	47	46	43	43
query42	106	94	96	94
query43	490	451	447	447
query44	1105	720	699	699
query45	269	248	262	248
query46	1076	676	673	673
query47	1936	1879	1897	1879
query48	381	305	287	287
query49	1159	364	363	363
query50	768	379	370	370
query51	6767	6612	6658	6612
query52	108	91	95	91
query53	342	275	277	275
query54	272	230	243	230
query55	85	80	78	78
query56	241	216	222	216
query57	1221	1130	1142	1130
query58	228	207	204	204
query59	2835	2686	2641	2641
query60	257	237	241	237
query61	103	97	96	96
query62	635	448	451	448
query63	310	280	281	280
query64	6436	4154	4156	4154
query65	3122	3120	3041	3041
query66	1376	368	371	368
query67	15456	15299	15305	15299
query68	6428	531	517	517
query69	552	314	316	314
query70	1204	1131	1220	1131
query71	483	279	277	277
query72	6746	2771	2589	2589
query73	778	319	332	319
query74	6935	6476	6388	6388
query75	3466	2335	2382	2335
query76	4103	937	993	937
query77	678	265	256	256
query78	10897	10221	10180	10180
query79	8574	533	505	505
query80	2119	413	414	413
query81	529	218	211	211
query82	1427	94	90	90
query83	335	167	164	164
query84	256	83	83	83
query85	1675	264	258	258
query86	505	300	315	300
query87	3822	3543	3526	3526
query88	5404	2301	2299	2299
query89	538	358	362	358
query90	1950	179	178	178
query91	128	96	96	96
query92	63	46	48	46
query93	6864	512	498	498
query94	1300	182	178	178
query95	396	304	300	300
query96	589	266	266	266
query97	2644	2454	2471	2454
query98	249	228	212	212
query99	1298	850	854	850
Total cold run time: 305161 ms
Total hot run time: 182420 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.3 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 9167b2c2dcafbf8c06e630270fd4cdcec651454a, data reload: false

query1	0.05	0.04	0.03
query2	0.08	0.04	0.04
query3	0.22	0.05	0.05
query4	1.68	0.06	0.07
query5	0.50	0.49	0.50
query6	1.15	0.68	0.65
query7	0.02	0.01	0.02
query8	0.06	0.04	0.04
query9	0.55	0.49	0.49
query10	0.54	0.54	0.54
query11	0.16	0.10	0.11
query12	0.13	0.11	0.11
query13	0.59	0.58	0.58
query14	0.76	0.78	0.76
query15	0.83	0.80	0.81
query16	0.38	0.36	0.38
query17	0.97	0.97	0.97
query18	0.23	0.22	0.25
query19	1.87	1.75	1.66
query20	0.02	0.01	0.01
query21	15.56	0.64	0.64
query22	2.65	2.69	2.59
query23	17.24	0.98	0.80
query24	1.16	0.26	0.21
query25	0.10	0.08	0.08
query26	0.22	0.16	0.17
query27	0.08	0.08	0.07
query28	13.96	0.94	0.92
query29	12.54	3.20	3.19
query30	0.27	0.09	0.10
query31	2.80	0.38	0.38
query32	3.29	0.46	0.45
query33	2.82	2.81	2.81
query34	16.69	4.43	4.40
query35	4.48	4.49	4.52
query36	0.61	0.46	0.47
query37	0.18	0.15	0.15
query38	0.15	0.14	0.14
query39	0.06	0.03	0.04
query40	0.19	0.16	0.15
query41	0.08	0.05	0.05
query42	0.05	0.04	0.05
query43	0.05	0.04	0.03
Total cold run time: 106.02 s
Total hot run time: 30.3 s

@doris-robot
Copy link

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit 9167b2c2dcafbf8c06e630270fd4cdcec651454a with default session variables
Stream load json:         18 seconds loaded 2358488459 Bytes, about 124 MB/s
Stream load orc:          58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet:      31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select:       13.6 seconds inserted 10000000 Rows, about 735K ops/s

@Mryange
Copy link
Contributor Author

Mryange commented Apr 8, 2024

run buildall

Copy link
Contributor

github-actions bot commented Apr 8, 2024

PR approved by at least one committer and no changes requested.

@github-actions github-actions bot added approved Indicates a PR has been approved by one committer. reviewed labels Apr 8, 2024
Copy link
Contributor

github-actions bot commented Apr 8, 2024

PR approved by anyone and no changes requested.

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 35.58% (8887/24976)
Line Coverage: 27.33% (72975/267018)
Region Coverage: 26.50% (37718/142350)
Branch Coverage: 23.31% (19220/82468)
Coverage Report: http://coverage.selectdb-in.cc/coverage/9167b2c2dcafbf8c06e630270fd4cdcec651454a_9167b2c2dcafbf8c06e630270fd4cdcec651454a/report/index.html

@BiteTheDDDDt BiteTheDDDDt merged commit fb2ffe0 into apache:master Apr 9, 2024
27 of 31 checks passed
seawinde pushed a commit to seawinde/doris that referenced this pull request Apr 10, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by one committer. reviewed
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants