Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Refactor](statistics)Refactor of statistics buildConnectContext. #41553

Open
wants to merge 1 commit into
base: master
Choose a base branch
from

Conversation

Jibing-Li
Copy link
Contributor

@Jibing-Li Jibing-Li commented Oct 8, 2024

  1. Remove useless session variable enableScanRunSerial.
  2. Refactor StatisticsUtil.buildConnectContext
  3. Remove nested call of buildConnectContext.
  4. Not record metrics for internal statistics sql.

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@Jibing-Li Jibing-Li marked this pull request as ready for review October 8, 2024 08:18
@Jibing-Li
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 40786 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit a1d861008723c8e1cb37cf18664691fea2536efb, data reload: false

------ Round 1 ----------------------------------
q1	17591	7500	7268	7268
q2	2011	273	268	268
q3	12059	1062	1168	1062
q4	10565	751	672	672
q5	7776	2861	2853	2853
q6	235	150	151	150
q7	1023	633	606	606
q8	9529	1944	1933	1933
q9	7606	6358	6417	6358
q10	6957	2294	2315	2294
q11	436	252	250	250
q12	408	222	215	215
q13	17779	3009	2946	2946
q14	239	212	211	211
q15	574	516	516	516
q16	655	600	573	573
q17	969	597	519	519
q18	7281	6740	6697	6697
q19	1352	990	984	984
q20	478	213	202	202
q21	4173	3209	3349	3209
q22	1127	1000	1016	1000
Total cold run time: 110823 ms
Total hot run time: 40786 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7302	7302	7246	7246
q2	323	245	238	238
q3	3062	2931	2958	2931
q4	2059	1857	1798	1798
q5	5761	5749	5750	5749
q6	234	147	153	147
q7	2277	1880	1829	1829
q8	3367	3438	3497	3438
q9	8895	8960	8863	8863
q10	3602	3520	3517	3517
q11	591	486	492	486
q12	852	627	623	623
q13	7638	3176	3170	3170
q14	308	291	270	270
q15	580	527	511	511
q16	691	619	664	619
q17	1867	1624	1589	1589
q18	8186	7642	7453	7453
q19	1707	1426	1517	1426
q20	2096	1859	1921	1859
q21	5472	5343	5278	5278
q22	1157	1030	1059	1030
Total cold run time: 68027 ms
Total hot run time: 60070 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 190900 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit a1d861008723c8e1cb37cf18664691fea2536efb, data reload: false

query1	955	393	377	377
query2	6238	2069	2005	2005
query3	8689	195	206	195
query4	34120	23669	23443	23443
query5	3581	463	463	463
query6	266	163	158	158
query7	4187	302	309	302
query8	282	205	205	205
query9	9669	2696	2715	2696
query10	452	273	290	273
query11	17876	15104	15208	15104
query12	135	102	98	98
query13	1583	466	447	447
query14	9041	7502	6975	6975
query15	251	168	173	168
query16	7925	485	475	475
query17	1627	612	577	577
query18	2129	324	316	316
query19	272	151	167	151
query20	124	112	118	112
query21	210	104	104	104
query22	4846	4632	4782	4632
query23	34773	33960	33813	33813
query24	11187	2862	2848	2848
query25	672	401	406	401
query26	1404	160	163	160
query27	2673	307	297	297
query28	7666	2432	2407	2407
query29	940	435	437	435
query30	281	151	150	150
query31	1043	785	798	785
query32	104	55	55	55
query33	776	299	307	299
query34	950	520	517	517
query35	864	717	719	717
query36	1115	958	931	931
query37	165	83	84	83
query38	4059	3882	3856	3856
query39	1490	1467	1451	1451
query40	258	98	96	96
query41	46	46	42	42
query42	117	95	94	94
query43	535	507	485	485
query44	1278	800	804	800
query45	193	164	165	164
query46	1159	717	716	716
query47	1909	1825	1815	1815
query48	445	349	357	349
query49	989	412	397	397
query50	836	406	430	406
query51	7047	6871	6942	6871
query52	98	89	86	86
query53	257	186	178	178
query54	1335	471	482	471
query55	74	78	75	75
query56	282	271	262	262
query57	1268	1180	1144	1144
query58	227	224	225	224
query59	3255	3003	2909	2909
query60	292	261	259	259
query61	104	96	104	96
query62	866	674	639	639
query63	232	184	187	184
query64	5010	631	598	598
query65	3245	3192	3175	3175
query66	1380	300	297	297
query67	15842	15574	15608	15574
query68	4416	579	560	560
query69	474	300	287	287
query70	1190	1095	1113	1095
query71	363	265	274	265
query72	7370	3973	3839	3839
query73	765	342	346	342
query74	10299	8924	8988	8924
query75	3416	2633	2667	2633
query76	2927	884	858	858
query77	543	302	307	302
query78	10379	9644	9510	9510
query79	2326	587	586	586
query80	2244	446	441	441
query81	593	236	236	236
query82	707	140	140	140
query83	283	133	134	133
query84	272	81	69	69
query85	1434	310	283	283
query86	422	314	255	255
query87	4341	4314	4268	4268
query88	3519	2463	2393	2393
query89	411	281	285	281
query90	1937	182	178	178
query91	153	109	108	108
query92	64	45	47	45
query93	1609	547	547	547
query94	1033	294	300	294
query95	345	242	258	242
query96	618	282	279	279
query97	3315	3160	3149	3149
query98	212	204	202	202
query99	1511	1296	1269	1269
Total cold run time: 302690 ms
Total hot run time: 190900 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 32.81 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit a1d861008723c8e1cb37cf18664691fea2536efb, data reload: false

query1	0.05	0.05	0.04
query2	0.07	0.03	0.03
query3	0.24	0.07	0.07
query4	1.64	0.10	0.10
query5	0.53	0.50	0.50
query6	1.13	0.74	0.72
query7	0.02	0.02	0.03
query8	0.04	0.04	0.03
query9	0.55	0.51	0.49
query10	0.56	0.57	0.54
query11	0.14	0.11	0.11
query12	0.13	0.10	0.10
query13	0.63	0.59	0.59
query14	2.69	2.74	2.85
query15	0.91	0.82	0.82
query16	0.38	0.39	0.40
query17	1.06	1.05	1.06
query18	0.20	0.19	0.20
query19	1.97	1.80	2.01
query20	0.02	0.01	0.01
query21	15.35	0.60	0.58
query22	2.28	2.68	1.62
query23	17.18	0.85	0.82
query24	2.99	1.26	1.68
query25	0.19	0.13	0.12
query26	0.38	0.13	0.14
query27	0.04	0.05	0.03
query28	9.88	1.09	1.07
query29	12.65	3.21	3.21
query30	0.24	0.06	0.06
query31	2.87	0.40	0.39
query32	3.25	0.46	0.46
query33	2.98	3.02	3.01
query34	16.74	4.49	4.48
query35	4.50	4.46	4.56
query36	0.66	0.47	0.50
query37	0.08	0.06	0.06
query38	0.05	0.03	0.03
query39	0.03	0.02	0.02
query40	0.16	0.13	0.12
query41	0.08	0.02	0.02
query42	0.03	0.02	0.02
query43	0.03	0.03	0.03
Total cold run time: 105.6 s
Total hot run time: 32.81 s

@Jibing-Li
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 40579 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 02936cf19c235bae4feada46d36b3c9e52375c19, data reload: false

------ Round 1 ----------------------------------
q1	17579	7340	7189	7189
q2	2009	275	271	271
q3	12217	1067	1108	1067
q4	10560	789	730	730
q5	7777	2859	2750	2750
q6	240	153	147	147
q7	1010	659	633	633
q8	9377	1950	1995	1950
q9	6559	6419	6326	6326
q10	6948	2284	2291	2284
q11	433	243	240	240
q12	406	227	215	215
q13	17782	2980	2959	2959
q14	259	211	208	208
q15	560	525	516	516
q16	667	571	598	571
q17	954	546	611	546
q18	7178	6643	6643	6643
q19	1341	1006	978	978
q20	485	194	189	189
q21	3940	3181	3180	3180
q22	1125	987	996	987
Total cold run time: 109406 ms
Total hot run time: 40579 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7251	7209	7150	7150
q2	326	236	228	228
q3	2960	2931	2944	2931
q4	2036	1819	1791	1791
q5	5797	5712	5754	5712
q6	232	144	141	141
q7	2233	1819	1847	1819
q8	3324	3619	3430	3430
q9	8933	8937	8841	8841
q10	3575	3564	3531	3531
q11	594	493	490	490
q12	843	635	616	616
q13	10826	3185	3133	3133
q14	320	290	289	289
q15	572	515	520	515
q16	695	657	630	630
q17	1815	1620	1604	1604
q18	8275	7735	7509	7509
q19	1739	1491	1448	1448
q20	2140	1879	1903	1879
q21	5508	5385	5381	5381
q22	1124	1079	1048	1048
Total cold run time: 71118 ms
Total hot run time: 60116 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 192512 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 02936cf19c235bae4feada46d36b3c9e52375c19, data reload: false

query1	936	404	410	404
query2	6241	2140	2086	2086
query3	8681	198	209	198
query4	34197	23597	23407	23407
query5	3378	475	472	472
query6	278	168	156	156
query7	4182	305	311	305
query8	275	218	227	218
query9	9564	2721	2713	2713
query10	469	277	285	277
query11	17706	15121	15142	15121
query12	135	100	97	97
query13	1575	467	434	434
query14	9862	7635	7442	7442
query15	256	176	192	176
query16	7784	467	455	455
query17	1613	619	613	613
query18	1591	334	320	320
query19	359	155	157	155
query20	130	116	112	112
query21	230	119	108	108
query22	4903	4793	4735	4735
query23	34766	34056	33968	33968
query24	11122	2921	2842	2842
query25	607	413	394	394
query26	1087	164	165	164
query27	2084	302	312	302
query28	6645	2410	2412	2410
query29	742	439	435	435
query30	265	150	159	150
query31	1022	790	818	790
query32	102	53	56	53
query33	772	285	299	285
query34	922	523	507	507
query35	868	737	747	737
query36	1119	921	975	921
query37	150	93	89	89
query38	3932	3870	3870	3870
query39	1476	1422	1423	1422
query40	209	98	97	97
query41	47	45	44	44
query42	117	98	98	98
query43	546	498	491	491
query44	1309	808	816	808
query45	200	166	166	166
query46	1180	706	727	706
query47	1949	1840	1833	1833
query48	449	357	347	347
query49	933	422	397	397
query50	826	424	432	424
query51	7152	6937	7109	6937
query52	100	88	91	88
query53	255	181	187	181
query54	1217	478	482	478
query55	81	78	84	78
query56	285	283	261	261
query57	1259	1170	1146	1146
query58	229	229	236	229
query59	3194	2937	2972	2937
query60	307	265	271	265
query61	109	98	100	98
query62	885	667	666	666
query63	218	192	190	190
query64	3727	639	624	624
query65	3237	3197	3209	3197
query66	824	302	302	302
query67	15864	15602	15695	15602
query68	4464	554	554	554
query69	510	293	297	293
query70	1229	1127	1117	1117
query71	364	265	273	265
query72	7415	3981	3982	3981
query73	760	344	358	344
query74	10182	8936	9053	8936
query75	3453	2697	2676	2676
query76	2837	937	935	935
query77	617	292	295	292
query78	10478	9557	9613	9557
query79	1792	592	595	592
query80	1787	428	445	428
query81	589	243	236	236
query82	704	140	135	135
query83	311	135	136	135
query84	275	79	75	75
query85	1581	292	280	280
query86	441	303	308	303
query87	4698	4348	4269	4269
query88	3220	2504	2462	2462
query89	411	292	289	289
query90	2077	198	189	189
query91	142	108	104	104
query92	62	47	46	46
query93	2098	544	542	542
query94	1172	297	278	278
query95	359	252	249	249
query96	617	289	290	289
query97	3345	3193	3176	3176
query98	216	197	208	197
query99	1736	1304	1255	1255
Total cold run time: 298676 ms
Total hot run time: 192512 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 32.73 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 02936cf19c235bae4feada46d36b3c9e52375c19, data reload: false

query1	0.06	0.04	0.04
query2	0.07	0.03	0.02
query3	0.23	0.07	0.07
query4	1.65	0.10	0.09
query5	0.53	0.50	0.51
query6	1.13	0.72	0.72
query7	0.02	0.02	0.02
query8	0.03	0.03	0.02
query9	0.55	0.50	0.49
query10	0.55	0.55	0.55
query11	0.13	0.10	0.10
query12	0.13	0.11	0.11
query13	0.61	0.60	0.60
query14	2.84	2.67	2.82
query15	0.92	0.82	0.83
query16	0.38	0.39	0.39
query17	1.02	1.03	0.98
query18	0.20	0.19	0.19
query19	1.88	1.76	1.88
query20	0.01	0.01	0.01
query21	15.35	0.59	0.60
query22	2.60	1.81	2.65
query23	17.14	0.97	0.82
query24	2.56	1.39	1.77
query25	0.26	0.10	0.04
query26	0.55	0.14	0.13
query27	0.04	0.03	0.04
query28	10.24	1.09	1.06
query29	12.54	3.23	3.19
query30	0.24	0.06	0.06
query31	2.87	0.40	0.39
query32	3.26	0.46	0.47
query33	2.95	2.98	3.04
query34	17.01	4.46	4.41
query35	4.51	4.51	4.47
query36	0.68	0.51	0.48
query37	0.08	0.05	0.05
query38	0.04	0.03	0.04
query39	0.03	0.02	0.02
query40	0.15	0.12	0.12
query41	0.08	0.02	0.02
query42	0.03	0.02	0.01
query43	0.03	0.02	0.03
Total cold run time: 106.18 s
Total hot run time: 32.73 s

@Jibing-Li
Copy link
Contributor Author

run p0

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants