Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[debug](log) Add debug logs #30566

Closed
wants to merge 4 commits into from
Closed

[debug](log) Add debug logs #30566

wants to merge 4 commits into from

Conversation

Gabriel39
Copy link
Contributor

Proposed changes

Issue Number: close #xxx

Further comments

If this is a relatively large or complex change, kick off the discussion at [email protected] by explaining why you chose the solution you did and what alternatives you considered, etc...

@Gabriel39
Copy link
Contributor Author

run buildall

@github-actions GitHub Actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

yiguolei
yiguolei previously approved these changes Jan 30, 2024
View reviewed changes
@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Jan 30, 2024
@github-actions GitHub Actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions GitHub Actions
Copy link
Contributor

PR approved by anyone and no changes requested.

@doris-robot
Copy link

TPC-H: Total hot run time: 37169 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit b238d3973d12b2ca581a2d2f7b9eb2422b973ed6, data reload: false

------ Round 1 ----------------------------------
q1	17882	4522	4568	4522
q2	2463	143	141	141
q3	11738	941	938	938
q4	5799	761	752	752
q5	7691	2925	2998	2925
q6	190	120	120	120
q7	1095	716	711	711
q8	9278	2033	1983	1983
q9	7197	6319	6291	6291
q10	8125	2402	2422	2402
q11	424	204	211	204
q12	779	290	282	282
q13	18005	3313	3294	3294
q14	279	245	243	243
q15	526	480	487	480
q16	468	418	400	400
q17	925	513	598	513
q18	6831	6039	6030	6030
q19	1577	1335	1451	1335
q20	581	319	326	319
q21	6816	2985	3178	2985
q22	803	299	317	299
Total cold run time: 109472 ms
Total hot run time: 37169 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4436	4352	4437	4352
q2	325	222	229	222
q3	2976	2856	2789	2789
q4	1826	1541	1620	1541
q5	5226	5191	5271	5191
q6	195	113	116	113
q7	2125	1780	1743	1743
q8	3061	3235	3234	3234
q9	8274	8323	8292	8292
q10	5871	3568	3526	3526
q11	548	460	459	459
q12	770	576	561	561
q13	13214	3113	3042	3042
q14	282	267	269	267
q15	535	486	492	486
q16	513	476	483	476
q17	1857	1695	1699	1695
q18	8002	7860	7501	7501
q19	9652	1515	1498	1498
q20	2115	1922	1912	1912
q21	4671	4662	4480	4480
q22	562	451	485	451
Total cold run time: 77036 ms
Total hot run time: 53831 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 172794 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit b238d3973d12b2ca581a2d2f7b9eb2422b973ed6, data reload: false

query1	932	337	329	329
query2	6547	1965	1887	1887
query3	6699	206	196	196
query4	27625	22084	21980	21980
query5	4449	354	350	350
query6	242	155	154	154
query7	4620	266	259	259
query8	241	185	174	174
query9	9078	2295	2276	2276
query10	414	198	196	196
query11	16825	15481	15348	15348
query12	126	64	61	61
query13	1682	369	366	366
query14	9334	6429	6540	6429
query15	224	184	185	184
query16	6364	255	251	251
query17	1891	473	470	470
query18	1913	259	253	253
query19	177	129	129	129
query20	72	77	70	70
query21	195	132	135	132
query22	4884	4760	4714	4714
query23	31812	30287	30298	30287
query24	11379	2785	2766	2766
query25	600	309	312	309
query26	1795	142	144	142
query27	2932	282	285	282
query28	7618	1871	1843	1843
query29	1464	642	626	626
query30	281	138	144	138
query31	924	701	710	701
query32	88	49	49	49
query33	706	203	203	203
query34	1091	453	470	453
query35	878	776	745	745
query36	1367	1238	1200	1200
query37	97	57	58	57
query38	3347	3143	3119	3119
query39	1299	1264	1262	1262
query40	207	90	81	81
query41	39	36	35	35
query42	93	78	83	78
query43	522	471	496	471
query44	1064	681	677	677
query45	200	181	183	181
query46	1063	648	650	648
query47	1532	1481	1505	1481
query48	386	302	303	302
query49	1156	278	277	277
query50	673	309	302	302
query51	5286	5097	5092	5092
query52	88	85	75	75
query53	320	262	250	250
query54	265	188	181	181
query55	81	81	73	73
query56	184	168	163	163
query57	1004	915	907	907
query58	175	150	158	150
query59	2352	2284	2301	2284
query60	218	189	172	172
query61	82	88	85	85
query62	665	371	360	360
query63	269	261	257	257
query64	4672	1786	1780	1780
query65	3303	3222	3192	3192
query66	1160	322	300	300
query67	14221	14223	14017	14017
query68	4003	500	510	500
query69	442	321	312	312
query70	1612	1500	1571	1500
query71	295	211	212	211
query72	3744	2849	2873	2849
query73	697	321	307	307
query74	6665	6356	6398	6356
query75	2805	2349	2288	2288
query76	2866	977	925	925
query77	370	235	221	221
query78	9079	8864	8554	8554
query79	2333	502	510	502
query80	1613	323	311	311
query81	538	194	199	194
query82	677	88	81	81
query83	230	115	117	115
query84	234	73	70	70
query85	2094	358	337	337
query86	535	422	393	393
query87	3479	3296	3280	3280
query88	3789	2192	2186	2186
query89	404	369	339	339
query90	1925	184	183	183
query91	146	117	121	117
query92	54	43	42	42
query93	1999	453	444	444
query94	1310	162	160	160
query95	505	467	457	457
query96	619	320	322	320
query97	4272	4181	4097	4097
query98	213	198	189	189
query99	1018	693	727	693
Total cold run time: 274057 ms
Total hot run time: 172794 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.1 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit b238d3973d12b2ca581a2d2f7b9eb2422b973ed6, data reload: false

query1	0.04	0.03	0.03
query2	0.06	0.02	0.02
query3	0.23	0.06	0.07
query4	1.68	0.10	0.10
query5	0.52	0.51	0.52
query6	1.22	0.64	0.62
query7	0.02	0.01	0.02
query8	0.03	0.02	0.02
query9	0.53	0.47	0.51
query10	0.57	0.57	0.55
query11	0.11	0.09	0.09
query12	0.11	0.09	0.09
query13	0.60	0.62	0.61
query14	0.79	0.81	0.79
query15	0.81	0.78	0.78
query16	0.38	0.38	0.38
query17	1.04	1.04	1.04
query18	0.21	0.30	0.21
query19	1.84	1.76	1.83
query20	0.02	0.01	0.01
query21	15.41	0.57	0.57
query22	2.53	2.78	1.59
query23	17.02	0.75	0.77
query24	2.54	0.26	1.38
query25	0.36	0.18	0.16
query26	0.47	0.14	0.15
query27	0.06	0.05	0.04
query28	12.15	0.87	0.83
query29	12.54	3.12	3.12
query30	0.68	0.61	0.54
query31	2.80	0.35	0.36
query32	3.36	0.49	0.48
query33	3.20	3.24	3.24
query34	15.80	4.27	4.29
query35	4.30	4.27	4.33
query36	1.11	1.06	1.08
query37	0.07	0.05	0.05
query38	0.04	0.03	0.03
query39	0.02	0.02	0.02
query40	0.16	0.13	0.12
query41	0.07	0.02	0.02
query42	0.02	0.02	0.02
query43	0.02	0.02	0.02
Total cold run time: 105.54 s
Total hot run time: 30.1 s

@doris-robot
Copy link

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit b238d3973d12b2ca581a2d2f7b9eb2422b973ed6 with default session variables
Stream load json:         18 seconds loaded 2358488459 Bytes, about 124 MB/s
Stream load orc:          58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet:      31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select:       13.4 seconds inserted 10000000 Rows, about 746K ops/s

@Gabriel39
Copy link
Contributor Author

run buildall

@github-actions github-actions bot removed the approved Indicates a PR has been approved by one committer. label Jan 30, 2024
@github-actions GitHub Actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

TPC-H: Total hot run time: 37094 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit bbda42c233baf91ffda6ed4aa9265278226dbf1f, data reload: false

------ Round 1 ----------------------------------
q1	17825	4574	4617	4574
q2	2173	141	135	135
q3	11245	935	932	932
q4	5069	771	739	739
q5	7907	2850	2864	2850
q6	185	119	122	119
q7	1172	752	732	732
q8	9540	2027	2031	2027
q9	7330	6375	6313	6313
q10	8090	2443	2395	2395
q11	424	216	213	213
q12	765	267	255	255
q13	18021	3358	3292	3292
q14	281	239	251	239
q15	525	485	485	485
q16	474	401	414	401
q17	946	533	515	515
q18	6700	5978	5865	5865
q19	1549	1417	1388	1388
q20	569	347	306	306
q21	6852	3011	3034	3011
q22	808	315	308	308
Total cold run time: 108450 ms
Total hot run time: 37094 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4527	4465	4407	4407
q2	325	231	232	231
q3	3001	2858	2826	2826
q4	1814	1612	1615	1612
q5	5211	5200	5221	5200
q6	188	112	114	112
q7	2146	1796	1823	1796
q8	3108	3221	3275	3221
q9	8326	8268	8300	8268
q10	5854	3571	3539	3539
q11	535	450	452	450
q12	739	569	563	563
q13	8983	3123	3052	3052
q14	266	241	270	241
q15	530	490	484	484
q16	522	467	463	463
q17	1838	1705	1694	1694
q18	7895	7755	7486	7486
q19	9320	1538	1501	1501
q20	2126	1897	1929	1897
q21	4931	4651	4491	4491
q22	515	484	456	456
Total cold run time: 72700 ms
Total hot run time: 53990 ms

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 36.19% (8622/23827)
Line Coverage: 28.25% (70511/249630)
Region Coverage: 27.24% (36380/133551)
Branch Coverage: 24.04% (18643/77554)
Coverage Report: http://coverage.selectdb-in.cc/coverage/bbda42c233baf91ffda6ed4aa9265278226dbf1f_bbda42c233baf91ffda6ed4aa9265278226dbf1f/report/index.html

@doris-robot
Copy link

TPC-DS: Total hot run time: 174200 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit bbda42c233baf91ffda6ed4aa9265278226dbf1f, data reload: false

query1	948	339	330	330
query2	6543	2019	1930	1930
query3	6703	202	192	192
query4	27082	22171	22078	22078
query5	4459	429	406	406
query6	252	166	157	157
query7	4598	273	261	261
query8	248	174	176	174
query9	9063	2280	2276	2276
query10	422	195	195	195
query11	16554	15485	15309	15309
query12	126	72	65	65
query13	1679	406	392	392
query14	9210	6556	6375	6375
query15	233	178	180	178
query16	6506	262	250	250
query17	1896	468	464	464
query18	1936	255	247	247
query19	171	132	132	132
query20	72	71	69	69
query21	189	128	123	123
query22	4852	4681	4766	4681
query23	31089	30438	30525	30438
query24	11554	2786	2804	2786
query25	591	314	305	305
query26	1880	143	144	143
query27	3012	284	283	283
query28	7709	1855	1853	1853
query29	1754	621	621	621
query30	280	138	139	138
query31	918	719	735	719
query32	91	52	55	52
query33	701	205	203	203
query34	1088	442	473	442
query35	934	775	752	752
query36	1343	1218	1218	1218
query37	93	60	58	58
query38	3392	3241	3234	3234
query39	1332	1259	1271	1259
query40	956	87	84	84
query41	37	35	34	34
query42	90	79	87	79
query43	504	481	531	481
query44	1079	678	678	678
query45	186	177	177	177
query46	1041	647	697	647
query47	1609	1606	1526	1526
query48	382	314	325	314
query49	1230	282	281	281
query50	683	314	306	306
query51	5233	5480	5378	5378
query52	102	86	86	86
query53	336	263	271	263
query54	256	203	192	192
query55	86	82	86	82
query56	191	169	189	169
query57	1104	977	1014	977
query58	193	162	194	162
query59	2745	2553	2551	2551
query60	204	190	186	186
query61	84	85	84	84
query62	619	379	367	367
query63	288	273	260	260
query64	6544	1799	1726	1726
query65	3408	3286	3293	3286
query66	4020	332	338	332
query67	14561	14238	14289	14238
query68	5413	510	482	482
query69	471	316	307	307
query70	1559	1557	1559	1557
query71	317	205	208	205
query72	3920	2812	2848	2812
query73	686	307	316	307
query74	6913	6295	6324	6295
query75	2768	2340	2270	2270
query76	3892	961	920	920
query77	379	227	227	227
query78	9096	8673	8663	8663
query79	2080	486	484	484
query80	1284	330	311	311
query81	501	199	201	199
query82	679	81	77	77
query83	205	115	120	115
query84	222	76	78	76
query85	2078	363	322	322
query86	514	404	420	404
query87	3434	3354	3306	3306
query88	3791	2159	2152	2152
query89	403	387	354	354
query90	1931	182	181	181
query91	145	119	116	116
query92	55	47	44	44
query93	1955	442	408	408
query94	1340	158	157	157
query95	510	451	446	446
query96	610	310	313	310
query97	4238	4118	4163	4118
query98	205	187	186	186
query99	961	655	718	655
Total cold run time: 281829 ms
Total hot run time: 174200 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.13 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit bbda42c233baf91ffda6ed4aa9265278226dbf1f, data reload: false

query1	0.04	0.04	0.03
query2	0.05	0.03	0.03
query3	0.23	0.06	0.06
query4	1.69	0.09	0.10
query5	0.53	0.51	0.51
query6	1.21	0.65	0.64
query7	0.01	0.01	0.02
query8	0.04	0.02	0.02
query9	0.57	0.51	0.48
query10	0.55	0.55	0.57
query11	0.12	0.08	0.08
query12	0.11	0.09	0.09
query13	0.61	0.61	0.60
query14	0.79	0.79	0.80
query15	0.80	0.80	0.79
query16	0.37	0.40	0.41
query17	1.01	1.02	1.03
query18	0.22	0.24	0.23
query19	1.91	1.79	1.71
query20	0.01	0.01	0.02
query21	15.41	0.59	0.60
query22	2.52	2.27	1.57
query23	17.42	0.74	0.76
query24	2.39	0.66	0.43
query25	0.37	0.20	0.06
query26	0.46	0.14	0.14
query27	0.06	0.06	0.05
query28	12.01	0.86	0.85
query29	12.56	3.24	3.13
query30	0.68	0.52	0.54
query31	2.78	0.36	0.35
query32	3.35	0.48	0.49
query33	3.25	3.22	3.24
query34	15.85	4.31	4.30
query35	4.25	4.28	4.26
query36	1.10	1.05	1.06
query37	0.06	0.04	0.04
query38	0.04	0.03	0.03
query39	0.02	0.01	0.02
query40	0.16	0.13	0.14
query41	0.08	0.02	0.01
query42	0.03	0.01	0.02
query43	0.02	0.02	0.02
Total cold run time: 105.74 s
Total hot run time: 30.13 s

@doris-robot
Copy link

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit bbda42c233baf91ffda6ed4aa9265278226dbf1f with default session variables
Stream load json:         18 seconds loaded 2358488459 Bytes, about 124 MB/s
Stream load orc:          58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet:      31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select:       14.2 seconds inserted 10000000 Rows, about 704K ops/s

@Gabriel39
Copy link
Contributor Author

run buildall

@github-actions GitHub Actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

TPC-H: Total hot run time: 36947 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 15825ae80533a7fd120e9a652a64afdcfb5996f6, data reload: false

------ Round 1 ----------------------------------
q1	17626	4820	4488	4488
q2	2040	144	135	135
q3	10852	936	908	908
q4	4680	689	705	689
q5	7692	2888	2793	2793
q6	189	123	122	122
q7	1144	719	707	707
q8	9301	2027	2048	2027
q9	7262	6334	6359	6334
q10	8103	2429	2420	2420
q11	420	210	203	203
q12	739	270	273	270
q13	18018	3308	3290	3290
q14	283	250	252	250
q15	538	495	488	488
q16	459	412	402	402
q17	946	667	491	491
q18	6948	5987	6087	5987
q19	1558	1368	1263	1263
q20	563	357	320	320
q21	6768	3060	3055	3055
q22	794	307	305	305
Total cold run time: 106923 ms
Total hot run time: 36947 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4494	4397	4391	4391
q2	320	237	234	234
q3	3021	2860	2848	2848
q4	1871	1682	1644	1644
q5	5215	5216	5263	5216
q6	195	114	115	114
q7	2139	1735	1798	1735
q8	3125	3211	3271	3211
q9	8420	8319	8307	8307
q10	5725	3490	3480	3480
q11	549	473	458	458
q12	762	568	588	568
q13	12783	3076	3072	3072
q14	276	252	247	247
q15	535	496	498	496
q16	517	478	472	472
q17	1843	1615	1633	1615
q18	7875	7649	7620	7620
q19	10583	1552	1496	1496
q20	2159	1905	1915	1905
q21	4794	4541	4556	4541
q22	596	475	489	475
Total cold run time: 77797 ms
Total hot run time: 54145 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 174522 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 15825ae80533a7fd120e9a652a64afdcfb5996f6, data reload: false

query1	937	331	332	331
query2	6551	2052	1874	1874
query3	6699	229	212	212
query4	26342	22130	22028	22028
query5	4453	449	445	445
query6	260	187	181	181
query7	4604	280	271	271
query8	245	196	216	196
query9	9216	2439	2405	2405
query10	424	257	242	242
query11	16769	15343	15393	15343
query12	136	85	83	83
query13	1705	384	378	378
query14	9319	6580	6515	6515
query15	214	182	182	182
query16	6184	279	271	271
query17	1403	498	481	481
query18	1781	263	257	257
query19	184	145	149	145
query20	83	81	78	78
query21	203	115	123	115
query22	4814	4745	4615	4615
query23	31223	30335	30479	30335
query24	7327	2745	2783	2745
query25	522	327	330	327
query26	860	150	151	150
query27	2511	298	296	296
query28	4072	1908	1893	1893
query29	931	648	653	648
query30	210	139	133	133
query31	917	711	721	711
query32	97	61	61	61
query33	448	252	259	252
query34	830	473	482	473
query35	841	767	760	760
query36	1288	1188	1169	1169
query37	111	77	75	75
query38	3320	3173	3132	3132
query39	1321	1266	1252	1252
query40	191	99	94	94
query41	41	35	36	35
query42	107	98	101	98
query43	526	491	484	484
query44	1067	723	726	723
query45	192	183	187	183
query46	1043	631	634	631
query47	1674	1534	1474	1474
query48	404	326	328	326
query49	639	313	326	313
query50	695	313	315	313
query51	5255	5172	5176	5172
query52	119	97	100	97
query53	320	273	264	264
query54	284	246	234	234
query55	97	90	95	90
query56	224	220	218	218
query57	980	925	931	925
query58	234	211	210	210
query59	2509	2376	2376	2376
query60	247	219	212	212
query61	84	90	85	85
query62	499	375	361	361
query63	297	273	280	273
query64	3649	1786	1798	1786
query65	3297	3245	3245	3245
query66	1140	340	326	326
query67	14220	13837	14379	13837
query68	3983	505	493	493
query69	486	355	336	336
query70	1629	1522	1503	1503
query71	307	259	248	248
query72	3797	2863	2848	2848
query73	692	327	318	318
query74	6619	6288	6321	6288
query75	2814	2331	2381	2331
query76	2933	955	906	906
query77	379	275	264	264
query78	9018	8715	8661	8661
query79	2579	509	507	507
query80	1395	361	362	361
query81	496	198	200	198
query82	907	104	104	104
query83	222	126	129	126
query84	235	79	76	76
query85	1110	368	370	368
query86	452	367	356	356
query87	3411	3281	3290	3281
query88	3841	2357	2329	2329
query89	434	374	352	352
query90	2152	214	226	214
query91	149	137	121	121
query92	74	66	67	66
query93	1704	423	413	413
query94	1331	195	194	194
query95	546	502	489	489
query96	651	339	346	339
query97	4252	4158	4139	4139
query98	221	204	202	202
query99	1101	716	646	646
Total cold run time: 259309 ms
Total hot run time: 174522 ms

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 36.19% (8625/23835)
Line Coverage: 28.25% (70538/249701)
Region Coverage: 27.25% (36401/133585)
Branch Coverage: 24.03% (18642/77562)
Coverage Report: http://coverage.selectdb-in.cc/coverage/15825ae80533a7fd120e9a652a64afdcfb5996f6_15825ae80533a7fd120e9a652a64afdcfb5996f6/report/index.html

@doris-robot
Copy link

ClickBench: Total hot run time: 31.81 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 15825ae80533a7fd120e9a652a64afdcfb5996f6, data reload: false

query1	0.04	0.04	0.03
query2	0.06	0.03	0.03
query3	0.23	0.06	0.06
query4	1.67	0.10	0.10
query5	0.52	0.51	0.52
query6	1.19	0.65	0.65
query7	0.01	0.01	0.01
query8	0.04	0.02	0.03
query9	0.55	0.52	0.48
query10	0.58	0.55	0.54
query11	0.11	0.08	0.08
query12	0.11	0.09	0.09
query13	0.61	0.62	0.60
query14	0.78	0.82	0.80
query15	0.79	0.77	0.77
query16	0.38	0.37	0.38
query17	1.00	1.02	1.02
query18	0.22	0.27	0.24
query19	1.84	1.82	1.74
query20	0.01	0.01	0.02
query21	15.40	0.56	0.58
query22	2.67	2.54	2.54
query23	17.20	0.84	0.82
query24	2.36	1.05	1.29
query25	0.32	0.20	0.15
query26	0.60	0.15	0.14
query27	0.05	0.05	0.05
query28	11.87	0.82	0.84
query29	12.48	3.12	3.21
query30	0.63	0.56	0.48
query31	2.79	0.35	0.35
query32	3.35	0.48	0.48
query33	3.22	3.20	3.23
query34	16.15	4.26	4.33
query35	4.31	4.32	4.37
query36	1.09	1.06	1.06
query37	0.07	0.05	0.05
query38	0.04	0.03	0.03
query39	0.02	0.01	0.02
query40	0.16	0.13	0.12
query41	0.07	0.02	0.02
query42	0.03	0.02	0.01
query43	0.03	0.02	0.02
Total cold run time: 105.65 s
Total hot run time: 31.81 s

@doris-robot
Copy link

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit 15825ae80533a7fd120e9a652a64afdcfb5996f6 with default session variables
Stream load json:         19 seconds loaded 2358488459 Bytes, about 118 MB/s
Stream load orc:          58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet:      32 seconds loaded 861443392 Bytes, about 25 MB/s
Insert into select:       13.2 seconds inserted 10000000 Rows, about 757K ops/s

@Gabriel39
Copy link
Contributor Author

run buildall

@Gabriel39 Gabriel39 changed the title [pipelineX](profile) Fix Tablet counter on pipelineX engine [debug](log) Add debug logs Jan 31, 2024
@Gabriel39 Gabriel39 marked this pull request as draft January 31, 2024 03:43
@github-actions GitHub Actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 36.18% (8623/23834)
Line Coverage: 28.24% (70526/249722)
Region Coverage: 27.24% (36392/133597)
Branch Coverage: 24.03% (18637/77566)
Coverage Report: http://coverage.selectdb-in.cc/coverage/8fdf92fc65490f82f85740c9d46aaad91a0cb257_8fdf92fc65490f82f85740c9d46aaad91a0cb257/report/index.html

@doris-robot
Copy link

TPC-H: Total hot run time: 36795 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 8fdf92fc65490f82f85740c9d46aaad91a0cb257, data reload: false

------ Round 1 ----------------------------------
q1	17654	4630	4405	4405
q2	2031	138	132	132
q3	10613	901	915	901
q4	4648	705	667	667
q5	7663	2802	2800	2800
q6	189	120	120	120
q7	1110	721	706	706
q8	9301	1995	2028	1995
q9	7256	6278	6322	6278
q10	8102	2411	2408	2408
q11	417	208	202	202
q12	814	277	286	277
q13	18031	3287	3305	3287
q14	283	262	252	252
q15	536	501	503	501
q16	478	416	415	415
q17	936	573	517	517
q18	6738	6017	5859	5859
q19	1550	1334	1244	1244
q20	629	340	328	328
q21	6807	3191	3209	3191
q22	816	333	310	310
Total cold run time: 106602 ms
Total hot run time: 36795 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4547	4445	4352	4352
q2	332	234	234	234
q3	2967	2801	2828	2801
q4	1935	1633	1676	1633
q5	5183	5224	5290	5224
q6	192	115	115	115
q7	2191	1778	1760	1760
q8	3121	3231	3283	3231
q9	8393	8296	8254	8254
q10	5754	3557	3463	3463
q11	530	459	449	449
q12	776	566	577	566
q13	12472	3069	3148	3069
q14	275	270	258	258
q15	538	495	489	489
q16	518	451	456	451
q17	1865	1592	1671	1592
q18	7940	7747	7538	7538
q19	9905	1528	1514	1514
q20	2142	1900	1888	1888
q21	4836	4569	4555	4555
q22	561	488	455	455
Total cold run time: 76973 ms
Total hot run time: 53891 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 174573 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 8fdf92fc65490f82f85740c9d46aaad91a0cb257, data reload: false

query1	944	340	324	324
query2	6565	1953	1996	1953
query3	6708	214	214	214
query4	26573	22010	22186	22010
query5	4467	388	448	388
query6	251	172	177	172
query7	4609	277	266	266
query8	273	199	188	188
query9	9103	2508	2495	2495
query10	415	251	242	242
query11	16886	15421	15463	15421
query12	133	92	85	85
query13	1674	388	393	388
query14	9384	6560	6438	6438
query15	217	178	180	178
query16	6213	284	270	270
query17	1797	498	472	472
query18	1790	278	264	264
query19	183	142	142	142
query20	85	84	76	76
query21	204	121	129	121
query22	4842	4680	4658	4658
query23	31233	30464	30385	30385
query24	8878	2754	2802	2754
query25	575	340	324	324
query26	906	152	145	145
query27	2741	305	299	299
query28	6113	1937	1895	1895
query29	953	656	632	632
query30	276	139	132	132
query31	929	702	729	702
query32	97	61	59	59
query33	552	255	246	246
query34	834	457	474	457
query35	854	770	751	751
query36	1303	1175	1107	1107
query37	102	72	73	72
query38	3286	3159	3215	3159
query39	1331	1257	1256	1256
query40	193	100	97	97
query41	38	35	35	35
query42	106	97	96	96
query43	535	485	504	485
query44	1061	736	742	736
query45	189	181	177	177
query46	1030	629	628	628
query47	1505	1523	1538	1523
query48	410	326	329	326
query49	1137	328	328	328
query50	680	317	309	309
query51	5279	5146	5155	5146
query52	110	91	102	91
query53	331	272	277	272
query54	275	234	238	234
query55	95	98	86	86
query56	231	210	211	210
query57	988	912	917	912
query58	225	210	210	210
query59	2410	2419	2286	2286
query60	259	231	233	231
query61	82	82	80	80
query62	607	375	360	360
query63	289	281	266	266
query64	4070	1779	1738	1738
query65	3353	3264	3214	3214
query66	1143	330	326	326
query67	14384	14138	14299	14138
query68	4012	512	503	503
query69	470	354	340	340
query70	1571	1568	1502	1502
query71	288	249	252	249
query72	3822	2845	2858	2845
query73	707	324	323	323
query74	6672	6251	6248	6248
query75	2918	2359	2280	2280
query76	3018	991	956	956
query77	367	262	249	249
query78	9086	8786	8557	8557
query79	2494	486	511	486
query80	1950	363	356	356
query81	532	194	200	194
query82	730	114	104	104
query83	252	129	118	118
query84	293	73	74	73
query85	2088	336	351	336
query86	527	390	397	390
query87	3532	3244	3294	3244
query88	3904	2286	2263	2263
query89	438	357	360	357
query90	1928	217	219	217
query91	168	119	115	115
query92	75	68	65	65
query93	2222	444	455	444
query94	1328	197	191	191
query95	540	510	499	499
query96	637	332	327	327
query97	4279	4143	4212	4143
query98	241	202	194	194
query99	980	730	725	725
Total cold run time: 267363 ms
Total hot run time: 174573 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 31.28 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 8fdf92fc65490f82f85740c9d46aaad91a0cb257, data reload: false

query1	0.04	0.03	0.03
query2	0.06	0.03	0.03
query3	0.23	0.06	0.05
query4	1.69	0.10	0.09
query5	0.53	0.52	0.51
query6	1.19	0.63	0.67
query7	0.02	0.02	0.01
query8	0.04	0.03	0.02
query9	0.55	0.50	0.51
query10	0.55	0.57	0.55
query11	0.13	0.08	0.08
query12	0.10	0.09	0.10
query13	0.59	0.61	0.60
query14	0.81	0.78	0.81
query15	0.80	0.78	0.78
query16	0.40	0.39	0.40
query17	1.04	1.00	0.98
query18	0.25	0.23	0.25
query19	1.90	1.75	1.77
query20	0.01	0.01	0.01
query21	15.42	0.55	0.55
query22	2.54	2.59	2.20
query23	17.13	0.84	0.84
query24	2.75	1.63	0.89
query25	0.43	0.22	0.10
query26	0.55	0.13	0.13
query27	0.05	0.05	0.05
query28	10.80	0.86	0.85
query29	12.74	3.22	3.15
query30	0.65	0.53	0.48
query31	2.79	0.34	0.35
query32	3.38	0.48	0.50
query33	3.16	3.24	3.23
query34	15.87	4.29	4.33
query35	4.32	4.28	4.29
query36	1.10	1.07	1.06
query37	0.06	0.04	0.05
query38	0.04	0.03	0.02
query39	0.02	0.02	0.01
query40	0.16	0.13	0.13
query41	0.06	0.01	0.02
query42	0.02	0.02	0.02
query43	0.03	0.02	0.02
Total cold run time: 105 s
Total hot run time: 31.28 s

@doris-robot
Copy link

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit 8fdf92fc65490f82f85740c9d46aaad91a0cb257 with default session variables
Stream load json:         19 seconds loaded 2358488459 Bytes, about 118 MB/s
Stream load orc:          58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet:      32 seconds loaded 861443392 Bytes, about 25 MB/s
Insert into select:       13.6 seconds inserted 10000000 Rows, about 735K ops/s

@Gabriel39 Gabriel39 closed this Mar 5, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@yiguolei yiguolei yiguolei left review comments

Assignees
No one assigned
Labels
reviewed
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@Gabriel39 @doris-robot @yiguolei