Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[profile](scanner) Fix wrong metrics #33965

Merged
merged 1 commit into from
Apr 22, 2024
Merged

Conversation

Gabriel39
Copy link
Contributor

Proposed changes

Now projections are done in parallel scanners. Timer for projections are cumulated to scan operator's exec time which is not correct.

Further comments

If this is a relatively large or complex change, kick off the discussion at [email protected] by explaining why you chose the solution you did and what alternatives you considered, etc...

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@Gabriel39
Copy link
Contributor Author

run buildall

Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 35.36% (8913/25209)
Line Coverage: 27.09% (73294/270592)
Region Coverage: 26.22% (37852/144361)
Branch Coverage: 23.03% (19275/83694)
Coverage Report: http://coverage.selectdb-in.cc/coverage/a0f8b22b595cf22634449714e3f881a7e89c9533_a0f8b22b595cf22634449714e3f881a7e89c9533/report/index.html

@doris-robot
Copy link

TPC-H: Total hot run time: 38692 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit a0f8b22b595cf22634449714e3f881a7e89c9533, data reload: false

------ Round 1 ----------------------------------
q1	17927	4595	4345	4345
q2	2643	207	192	192
q3	11780	1232	1172	1172
q4	10234	745	767	745
q5	7598	2711	2552	2552
q6	220	133	134	133
q7	1031	604	616	604
q8	9234	2285	2075	2075
q9	7369	7087	6581	6581
q10	9028	3695	3610	3610
q11	450	243	230	230
q12	397	232	222	222
q13	19251	3007	2924	2924
q14	263	228	230	228
q15	508	476	485	476
q16	514	414	377	377
q17	969	667	648	648
q18	7417	6896	6713	6713
q19	4637	1535	1492	1492
q20	638	320	306	306
q21	3588	2763	2885	2763
q22	364	304	307	304
Total cold run time: 116060 ms
Total hot run time: 38692 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4307	4211	4213	4211
q2	368	271	284	271
q3	3017	2796	2729	2729
q4	1883	1569	1597	1569
q5	5312	5301	5311	5301
q6	209	121	123	121
q7	2244	1894	1909	1894
q8	3205	3360	3356	3356
q9	8630	8609	8562	8562
q10	3846	3658	3679	3658
q11	573	481	470	470
q12	765	577	601	577
q13	16361	2954	2926	2926
q14	320	274	286	274
q15	527	471	480	471
q16	477	410	412	410
q17	1789	1494	1465	1465
q18	7802	7524	7376	7376
q19	1654	1528	1511	1511
q20	1967	1741	1750	1741
q21	4974	4684	4914	4684
q22	538	460	449	449
Total cold run time: 70768 ms
Total hot run time: 54026 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 184520 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit a0f8b22b595cf22634449714e3f881a7e89c9533, data reload: false

query1	901	372	359	359
query2	6490	2697	2383	2383
query3	6654	206	206	206
query4	23164	21337	21349	21337
query5	4160	402	411	402
query6	266	174	166	166
query7	4575	293	284	284
query8	246	194	179	179
query9	8482	2341	2336	2336
query10	581	261	260	260
query11	14763	14196	14334	14196
query12	140	95	88	88
query13	1638	352	351	351
query14	9532	7039	7850	7039
query15	302	186	184	184
query16	8155	263	252	252
query17	1923	587	539	539
query18	2093	271	271	271
query19	245	150	156	150
query20	92	86	82	82
query21	196	131	128	128
query22	4998	4855	4819	4819
query23	33627	33289	33200	33200
query24	12037	3032	2966	2966
query25	654	368	371	368
query26	1796	147	149	147
query27	3028	310	313	310
query28	7624	1979	1969	1969
query29	1024	640	633	633
query30	293	172	169	169
query31	959	730	747	730
query32	87	52	54	52
query33	744	243	234	234
query34	1054	484	486	484
query35	854	689	690	689
query36	1058	972	933	933
query37	255	68	70	68
query38	3405	3246	3149	3149
query39	1585	1524	1524	1524
query40	272	124	122	122
query41	48	43	46	43
query42	103	97	96	96
query43	572	525	542	525
query44	1224	722	719	719
query45	271	258	259	258
query46	1087	755	721	721
query47	1890	1846	1833	1833
query48	370	290	281	281
query49	1182	403	394	394
query50	754	368	370	368
query51	6708	6517	6546	6517
query52	112	90	92	90
query53	351	282	284	282
query54	337	240	232	232
query55	76	73	71	71
query56	249	237	230	230
query57	1210	1159	1105	1105
query58	228	203	201	201
query59	3506	3283	3192	3192
query60	263	242	240	240
query61	109	108	108	108
query62	661	453	446	446
query63	307	281	278	278
query64	6402	4066	4172	4066
query65	3157	3047	3054	3047
query66	1405	348	350	348
query67	15625	15028	14922	14922
query68	6304	529	530	529
query69	535	313	320	313
query70	1252	1230	1192	1192
query71	1456	1267	1274	1267
query72	6562	2738	2587	2587
query73	733	318	315	315
query74	6890	6334	6410	6334
query75	3649	2666	2607	2607
query76	3682	978	1007	978
query77	663	266	274	266
query78	10865	10132	10178	10132
query79	9094	506	524	506
query80	1968	456	443	443
query81	528	253	245	245
query82	1440	94	96	94
query83	308	167	171	167
query84	263	88	92	88
query85	1721	300	264	264
query86	472	307	306	306
query87	3465	3225	3270	3225
query88	5202	2304	2318	2304
query89	545	372	370	370
query90	1966	179	181	179
query91	123	101	97	97
query92	58	47	45	45
query93	7393	508	492	492
query94	1200	177	180	177
query95	379	301	292	292
query96	593	263	258	258
query97	3118	3002	3180	3002
query98	240	228	217	217
query99	1283	866	858	858
Total cold run time: 302109 ms
Total hot run time: 184520 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.44 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit a0f8b22b595cf22634449714e3f881a7e89c9533, data reload: false

query1	0.04	0.03	0.03
query2	0.07	0.04	0.04
query3	0.22	0.06	0.05
query4	1.66	0.10	0.10
query5	0.51	0.50	0.54
query6	1.47	0.72	0.72
query7	0.03	0.02	0.01
query8	0.05	0.04	0.04
query9	0.54	0.49	0.49
query10	0.55	0.55	0.53
query11	0.15	0.11	0.12
query12	0.14	0.13	0.12
query13	0.61	0.58	0.59
query14	0.77	0.76	0.76
query15	0.83	0.80	0.81
query16	0.36	0.34	0.36
query17	0.94	0.93	0.98
query18	0.20	0.24	0.22
query19	1.80	1.65	1.69
query20	0.02	0.01	0.01
query21	15.48	0.66	0.64
query22	4.48	6.85	2.19
query23	18.32	1.49	1.21
query24	1.82	0.22	0.25
query25	0.14	0.08	0.09
query26	0.25	0.16	0.17
query27	0.08	0.07	0.07
query28	13.27	1.00	0.97
query29	12.63	3.28	3.24
query30	0.25	0.07	0.06
query31	2.88	0.37	0.37
query32	3.27	0.46	0.46
query33	2.78	2.87	2.83
query34	17.12	4.40	4.44
query35	4.47	4.50	4.48
query36	0.65	0.46	0.45
query37	0.18	0.15	0.15
query38	0.16	0.15	0.14
query39	0.04	0.03	0.04
query40	0.18	0.15	0.14
query41	0.09	0.04	0.04
query42	0.06	0.05	0.05
query43	0.05	0.04	0.03
Total cold run time: 109.61 s
Total hot run time: 30.44 s

Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Apr 22, 2024
Copy link
Contributor

PR approved by anyone and no changes requested.

@Gabriel39 Gabriel39 merged commit f01053e into apache:master Apr 22, 2024
25 of 29 checks passed
yiguolei pushed a commit that referenced this pull request Apr 22, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by one committer. reviewed
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants