Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[pipelinex](profile) improve memory counter of pipelineX #30538

Merged
merged 1 commit into from
Jan 31, 2024
Merged

[pipelinex](profile) improve memory counter of pipelineX #30538

merged 1 commit into from
Jan 31, 2024

Conversation

jacktengg
Copy link
Contributor

Proposed changes

Issue Number: close #xxx
Improve memory counters for operators, e.g.:
scanner merged:

               VScanner:
                  - MemoryUsage: sum , avg , max , min 
                    - FreeBlocks: sum 1.98 GB, avg 42.16 MB, max 75.55 MB, min 14.25 MB
                    - QueuedBlocks: sum 2.35 GB, avg 50.13 MB, max 81.64 MB, min 27.02 MB

scanner:

VScanner:
                 - MemoryUsage: 
                   - FreeBlocks: 15.53 MB
                   - QueuedBlocks: 36.12 MB

hash join sink merged:

           HASH_JOIN_SINK_OPERATOR (id=44):
              - MemoryUsage: sum , avg , max , min 
                - BuildBlocks: sum 31.28 MB, avg 667.33 KB, max 674.78 KB, min 659.93 KB
                - BuildKeyArena: sum 17.19 MB, avg 366.75 KB, max 372.00 KB, min 364.00 KB
                - HashTable: sum 9.38 MB, avg 200.15 KB, max 200.96 KB, min 199.35 KB
                - PeakMemoryUsage: sum 57.67 MB, avg 1.20 MB, max 1.21 MB, min 1.19 MB

hash join sink:

          HASH_JOIN_SINK_OPERATOR (id=43):
             - MemoryUsage: 
               - BuildBlocks: 681.86 KB
               - BuildKeyArena: 376.00 KB
               - HashTable: 201.72 KB
               - PeakMemoryUsage: 1.23 MB

sort sink merged:

           SORT_SINK_OPERATOR (id=45):
              - MemoryUsage: sum , avg , max , min 
                - PeakMemoryUsage: sum 5.73 MB, avg 122.22 KB, max 129.96 KB, min 114.84 KB
                - SortBlocks: sum 5.73 MB, avg 122.22 KB, max 129.96 KB, min 114.84 KB

sort sink:

          SORT_SINK_OPERATOR (id=45):
             - MemoryUsage: 
               - PeakMemoryUsage: 121.77 KB
               - SortBlocks: 121.77 KB

Further comments

If this is a relatively large or complex change, kick off the discussion at [email protected] by explaining why you chose the solution you did and what alternatives you considered, etc...

@jacktengg
Copy link
Contributor Author

run buildall

@github-actions GitHub Actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

1 similar comment
@github-actions GitHub Actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

TPC-H: Total hot run time: 36868 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 805519b3e81a452118fcb1fb1188226d5e9ff708, data reload: false

------ Round 1 ----------------------------------
q1	17657	4748	4491	4491
q2	2050	138	132	132
q3	10608	924	948	924
q4	4655	685	721	685
q5	7656	2835	2827	2827
q6	187	120	117	117
q7	1123	726	712	712
q8	9260	2015	2018	2015
q9	7260	6320	6353	6320
q10	8120	2432	2466	2432
q11	413	211	194	194
q12	703	267	262	262
q13	18030	3287	3279	3279
q14	277	253	243	243
q15	526	525	482	482
q16	478	405	418	405
q17	946	559	494	494
q18	6764	5959	5862	5862
q19	1573	1372	1317	1317
q20	586	334	342	334
q21	6643	3086	3058	3058
q22	797	294	283	283
Total cold run time: 106312 ms
Total hot run time: 36868 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4521	4427	4394	4394
q2	350	240	234	234
q3	3010	2856	2913	2856
q4	1831	1729	1663	1663
q5	5236	5329	5236	5236
q6	193	112	111	111
q7	2131	1793	1774	1774
q8	3141	3219	3243	3219
q9	8385	8371	8371	8371
q10	5778	3571	3624	3571
q11	560	468	453	453
q12	752	578	599	578
q13	12407	3102	3075	3075
q14	280	260	253	253
q15	545	485	487	485
q16	538	493	484	484
q17	1866	1734	1710	1710
q18	8081	7613	7428	7428
q19	7786	1548	1594	1548
q20	2141	1902	1884	1884
q21	4862	4622	4591	4591
q22	535	462	478	462
Total cold run time: 74929 ms
Total hot run time: 54380 ms

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 36.18% (8621/23827)
Line Coverage: 28.25% (70507/249624)
Region Coverage: 27.24% (36380/133538)
Branch Coverage: 24.04% (18642/77552)
Coverage Report: http://coverage.selectdb-in.cc/coverage/805519b3e81a452118fcb1fb1188226d5e9ff708_805519b3e81a452118fcb1fb1188226d5e9ff708/report/index.html

@doris-robot
Copy link

TPC-DS: Total hot run time: 172244 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 805519b3e81a452118fcb1fb1188226d5e9ff708, data reload: false

query1	939	334	322	322
query2	6570	1851	1904	1851
query3	6732	203	198	198
query4	27119	22072	22023	22023
query5	4451	351	347	347
query6	252	151	164	151
query7	4609	260	248	248
query8	242	170	177	170
query9	8917	2291	2264	2264
query10	411	205	201	201
query11	16607	15301	15528	15301
query12	125	66	59	59
query13	1674	366	364	364
query14	9368	6682	6471	6471
query15	211	175	180	175
query16	6427	248	247	247
query17	1920	485	470	470
query18	1928	265	247	247
query19	173	131	133	131
query20	76	71	67	67
query21	200	128	135	128
query22	4909	4769	4790	4769
query23	31138	30191	30227	30191
query24	8142	2782	2704	2704
query25	544	301	298	298
query26	828	139	134	134
query27	2762	284	283	283
query28	5756	1852	1835	1835
query29	902	601	605	601
query30	284	136	137	136
query31	915	705	716	705
query32	90	51	49	49
query33	447	216	205	205
query34	807	450	450	450
query35	833	786	756	756
query36	1285	1156	1192	1156
query37	91	56	58	56
query38	3307	3196	3174	3174
query39	1298	1247	1241	1241
query40	195	85	81	81
query41	36	35	34	34
query42	90	83	81	81
query43	536	470	483	470
query44	1014	685	678	678
query45	192	178	173	173
query46	1059	641	611	611
query47	1511	1549	1478	1478
query48	378	302	318	302
query49	1105	283	274	274
query50	668	297	302	297
query51	5258	5152	5127	5127
query52	84	80	71	71
query53	327	255	254	254
query54	227	181	182	181
query55	76	72	76	72
query56	179	173	163	163
query57	968	910	925	910
query58	180	157	151	151
query59	2378	2335	2234	2234
query60	198	170	173	170
query61	82	87	79	79
query62	583	362	348	348
query63	270	250	265	250
query64	4001	1716	1706	1706
query65	3302	3238	3178	3178
query66	1136	314	328	314
query67	14483	14127	14556	14127
query68	4092	489	478	478
query69	440	308	303	303
query70	1429	1527	1496	1496
query71	278	216	220	216
query72	3765	2817	2802	2802
query73	684	308	308	308
query74	6738	6216	6336	6216
query75	2774	2356	2324	2324
query76	3001	961	964	961
query77	346	219	218	218
query78	9125	8928	8629	8629
query79	2465	481	477	477
query80	1296	312	301	301
query81	513	194	196	194
query82	874	78	74	74
query83	210	116	115	115
query84	235	71	76	71
query85	1076	316	311	311
query86	463	378	388	378
query87	3421	3307	3284	3284
query88	3816	2167	2148	2148
query89	408	326	369	326
query90	2003	182	177	177
query91	148	116	118	116
query92	55	40	45	40
query93	1759	443	431	431
query94	1184	158	156	156
query95	484	478	439	439
query96	606	318	317	317
query97	4238	4107	4109	4107
query98	198	195	178	178
query99	1063	661	712	661
Total cold run time: 262992 ms
Total hot run time: 172244 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 31.07 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 805519b3e81a452118fcb1fb1188226d5e9ff708, data reload: false

query1	0.04	0.03	0.02
query2	0.05	0.02	0.02
query3	0.22	0.06	0.06
query4	1.69	0.10	0.10
query5	0.52	0.51	0.52
query6	1.21	0.63	0.64
query7	0.02	0.02	0.01
query8	0.03	0.03	0.03
query9	0.56	0.51	0.51
query10	0.57	0.56	0.58
query11	0.12	0.08	0.08
query12	0.12	0.09	0.09
query13	0.60	0.61	0.60
query14	0.79	0.81	0.81
query15	0.80	0.78	0.77
query16	0.38	0.40	0.39
query17	1.04	1.04	1.04
query18	0.21	0.26	0.23
query19	1.93	1.78	1.81
query20	0.02	0.01	0.01
query21	15.42	0.56	0.56
query22	2.26	2.53	1.55
query23	17.41	0.76	0.81
query24	2.40	1.57	1.29
query25	0.44	0.17	0.07
query26	0.61	0.15	0.14
query27	0.05	0.06	0.05
query28	10.91	0.86	0.83
query29	12.65	3.20	3.20
query30	0.64	0.52	0.53
query31	2.79	0.35	0.36
query32	3.35	0.48	0.48
query33	3.23	3.19	3.31
query34	15.92	4.31	4.23
query35	4.30	4.30	4.30
query36	1.10	1.05	1.08
query37	0.06	0.04	0.04
query38	0.04	0.03	0.03
query39	0.02	0.01	0.02
query40	0.17	0.13	0.13
query41	0.07	0.01	0.02
query42	0.02	0.02	0.01
query43	0.03	0.02	0.02
Total cold run time: 104.81 s
Total hot run time: 31.07 s

@doris-robot
Copy link

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit 805519b3e81a452118fcb1fb1188226d5e9ff708 with default session variables
Stream load json:         18 seconds loaded 2358488459 Bytes, about 124 MB/s
Stream load orc:          58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet:      32 seconds loaded 861443392 Bytes, about 25 MB/s
Insert into select:       13.5 seconds inserted 10000000 Rows, about 740K ops/s

@jacktengg
Copy link
Contributor Author

run pipelinex_p0

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Jan 30, 2024
@github-actions GitHub Actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions GitHub Actions
Copy link
Contributor

PR approved by anyone and no changes requested.

@jacktengg
Copy link
Contributor Author

run buildall

@github-actions GitHub Actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 36.19% (8625/23831)
Line Coverage: 28.26% (70546/249653)
Region Coverage: 27.25% (36399/133554)
Branch Coverage: 24.06% (18657/77558)
Coverage Report: http://coverage.selectdb-in.cc/coverage/c102a6c0e89da6dbb2faa7672a8b59d9dc932ddb_c102a6c0e89da6dbb2faa7672a8b59d9dc932ddb/report/index.html

@doris-robot
Copy link

TPC-H: Total hot run time: 36672 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit c102a6c0e89da6dbb2faa7672a8b59d9dc932ddb, data reload: false

------ Round 1 ----------------------------------
q1	17641	4424	4443	4424
q2	2034	137	134	134
q3	10593	923	937	923
q4	4647	730	694	694
q5	7662	2777	2842	2777
q6	183	118	118	118
q7	1160	721	723	721
q8	9252	1973	1993	1973
q9	7207	6282	6326	6282
q10	8084	2457	2391	2391
q11	425	204	196	196
q12	743	265	281	265
q13	18020	3276	3263	3263
q14	267	249	257	249
q15	523	516	497	497
q16	463	395	405	395
q17	927	504	500	500
q18	6704	5987	5848	5848
q19	1560	1452	1382	1382
q20	570	333	323	323
q21	7093	3062	3029	3029
q22	822	317	288	288
Total cold run time: 106580 ms
Total hot run time: 36672 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4471	4398	4362	4362
q2	327	232	230	230
q3	3027	2837	2840	2837
q4	1842	1579	1654	1579
q5	5208	5272	5251	5251
q6	192	115	112	112
q7	2174	1730	1748	1730
q8	3115	3241	3213	3213
q9	8311	8343	8287	8287
q10	5873	3609	3506	3506
q11	534	473	457	457
q12	729	570	595	570
q13	14598	3037	3072	3037
q14	271	255	261	255
q15	534	494	494	494
q16	529	474	458	458
q17	1823	1669	1707	1669
q18	7908	7764	7516	7516
q19	10474	1545	1524	1524
q20	2156	1903	1890	1890
q21	4874	4691	4598	4598
q22	533	457	493	457
Total cold run time: 79503 ms
Total hot run time: 54032 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 172729 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit c102a6c0e89da6dbb2faa7672a8b59d9dc932ddb, data reload: false

query1	921	333	330	330
query2	6568	1965	1944	1944
query3	6697	199	197	197
query4	26921	22065	22082	22065
query5	4453	353	353	353
query6	256	159	146	146
query7	4622	262	261	261
query8	250	174	178	174
query9	9035	2331	2298	2298
query10	409	201	201	201
query11	16848	15426	15261	15261
query12	129	66	64	64
query13	1701	375	375	375
query14	9252	6573	6452	6452
query15	225	188	191	188
query16	6314	253	243	243
query17	1868	477	459	459
query18	1827	258	257	257
query19	180	126	126	126
query20	73	73	68	68
query21	197	131	132	131
query22	4866	4782	4746	4746
query23	31002	30268	30262	30262
query24	7053	2735	2717	2717
query25	484	315	297	297
query26	818	134	134	134
query27	2473	283	285	283
query28	4032	1886	1855	1855
query29	904	606	629	606
query30	202	135	134	134
query31	900	709	717	709
query32	81	49	52	49
query33	421	212	208	208
query34	810	458	462	458
query35	819	751	783	751
query36	1260	1201	1189	1189
query37	96	59	59	59
query38	3246	3176	3135	3135
query39	1308	1260	1265	1260
query40	194	87	80	80
query41	37	36	33	33
query42	94	80	85	80
query43	478	490	494	490
query44	1089	687	691	687
query45	198	176	174	174
query46	1033	657	643	643
query47	1556	1428	1496	1428
query48	382	302	326	302
query49	624	272	285	272
query50	684	310	304	304
query51	5279	5211	5148	5148
query52	91	80	77	77
query53	346	258	272	258
query54	222	185	178	178
query55	78	72	75	72
query56	177	165	163	163
query57	987	917	885	885
query58	186	145	154	145
query59	2294	2404	2268	2268
query60	198	177	178	177
query61	81	80	80	80
query62	443	370	342	342
query63	289	268	277	268
query64	3485	1742	1712	1712
query65	3289	3217	3217	3217
query66	1088	312	322	312
query67	14637	14500	14214	14214
query68	4002	504	478	478
query69	451	300	317	300
query70	1531	1541	1538	1538
query71	296	210	213	210
query72	3669	2793	2805	2793
query73	678	311	318	311
query74	6596	6243	6258	6243
query75	2769	2335	2297	2297
query76	2937	960	963	960
query77	375	227	222	222
query78	9117	8770	8512	8512
query79	2502	485	493	485
query80	2118	319	313	313
query81	512	200	196	196
query82	711	81	81	81
query83	245	114	118	114
query84	279	78	69	69
query85	1955	342	314	314
query86	508	397	376	376
query87	3409	3295	3313	3295
query88	3580	2202	2167	2167
query89	433	347	364	347
query90	2031	185	188	185
query91	149	120	114	114
query92	52	42	46	42
query93	2475	420	437	420
query94	1348	155	156	155
query95	498	453	449	449
query96	625	310	308	308
query97	4217	4134	4134	4134
query98	216	194	183	183
query99	1000	678	664	664
Total cold run time: 260344 ms
Total hot run time: 172729 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.44 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit c102a6c0e89da6dbb2faa7672a8b59d9dc932ddb, data reload: false

query1	0.04	0.03	0.03
query2	0.06	0.03	0.02
query3	0.23	0.07	0.06
query4	1.65	0.10	0.10
query5	0.56	0.52	0.52
query6	1.19	0.64	0.65
query7	0.01	0.02	0.01
query8	0.04	0.03	0.03
query9	0.55	0.52	0.50
query10	0.56	0.55	0.55
query11	0.12	0.08	0.08
query12	0.11	0.08	0.09
query13	0.60	0.61	0.61
query14	0.78	0.82	0.78
query15	0.79	0.79	0.78
query16	0.37	0.39	0.38
query17	1.01	0.99	1.02
query18	0.21	0.27	0.24
query19	1.90	1.80	1.74
query20	0.02	0.01	0.01
query21	15.41	0.57	0.58
query22	2.45	3.14	1.91
query23	17.37	0.75	0.90
query24	2.54	0.31	1.38
query25	0.31	0.24	0.20
query26	0.44	0.14	0.14
query27	0.05	0.05	0.04
query28	11.59	0.85	0.83
query29	12.59	3.13	3.21
query30	0.61	0.57	0.56
query31	2.78	0.35	0.36
query32	3.32	0.49	0.48
query33	3.21	3.20	3.25
query34	15.69	4.24	4.29
query35	4.38	4.34	4.27
query36	1.09	1.05	1.06
query37	0.07	0.05	0.05
query38	0.03	0.03	0.03
query39	0.02	0.01	0.02
query40	0.16	0.13	0.13
query41	0.07	0.02	0.01
query42	0.03	0.01	0.02
query43	0.03	0.02	0.02
Total cold run time: 105.04 s
Total hot run time: 30.44 s

@doris-robot
Copy link

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit c102a6c0e89da6dbb2faa7672a8b59d9dc932ddb with default session variables
Stream load json:         20 seconds loaded 2358488459 Bytes, about 112 MB/s
Stream load orc:          58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet:      32 seconds loaded 861443392 Bytes, about 25 MB/s
Insert into select:       13.5 seconds inserted 10000000 Rows, about 740K ops/s

mrhhsg
mrhhsg approved these changes Jan 30, 2024
View reviewed changes
Copy link
Member

@mrhhsg mrhhsg left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

@github-actions GitHub Actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@yiguolei
Copy link
Contributor

run buildall

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 36.18% (8623/23835)
Line Coverage: 28.24% (70509/249665)
Region Coverage: 27.24% (36384/133574)
Branch Coverage: 24.02% (18627/77560)
Coverage Report: http://coverage.selectdb-in.cc/coverage/57bc76687bad7a4a442ff594d877327237baf654_57bc76687bad7a4a442ff594d877327237baf654/report/index.html

@doris-robot
Copy link

TPC-H: Total hot run time: 37007 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 57bc76687bad7a4a442ff594d877327237baf654, data reload: false

------ Round 1 ----------------------------------
q1	17648	4609	4522	4522
q2	2055	159	135	135
q3	10575	895	909	895
q4	4651	819	689	689
q5	7676	2865	2799	2799
q6	183	120	121	120
q7	1147	723	705	705
q8	9307	2005	2022	2005
q9	7240	6393	6311	6311
q10	8114	2468	2423	2423
q11	414	199	210	199
q12	755	275	273	273
q13	18002	3325	3291	3291
q14	281	236	252	236
q15	528	501	485	485
q16	460	430	426	426
q17	949	526	534	526
q18	6944	6003	5894	5894
q19	1574	1504	1388	1388
q20	611	367	333	333
q21	6922	3064	3056	3056
q22	803	299	296	296
Total cold run time: 106839 ms
Total hot run time: 37007 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4486	4441	4436	4436
q2	336	237	232	232
q3	3021	2900	2830	2830
q4	1906	1683	1601	1601
q5	5253	5247	5237	5237
q6	191	115	115	115
q7	2208	1847	1772	1772
q8	3126	3244	3256	3244
q9	8438	8331	8343	8331
q10	5859	3647	3567	3567
q11	551	463	468	463
q12	739	558	614	558
q13	13015	3094	3082	3082
q14	279	248	257	248
q15	531	497	491	491
q16	522	479	460	460
q17	1810	1679	1679	1679
q18	8197	7606	7611	7606
q19	7523	1517	1553	1517
q20	2138	1929	1906	1906
q21	4741	4696	4622	4622
q22	551	460	448	448
Total cold run time: 75421 ms
Total hot run time: 54445 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 173086 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 57bc76687bad7a4a442ff594d877327237baf654, data reload: false

query1	947	337	330	330
query2	6561	1948	1898	1898
query3	6701	202	199	199
query4	27820	22117	22102	22102
query5	4452	445	345	345
query6	250	164	152	152
query7	4607	259	261	259
query8	242	175	177	175
query9	9183	2297	2302	2297
query10	416	206	212	206
query11	16709	15478	15390	15390
query12	116	66	63	63
query13	1671	370	370	370
query14	9261	6599	6508	6508
query15	219	177	184	177
query16	6376	258	253	253
query17	1895	484	467	467
query18	1893	252	254	252
query19	179	137	133	133
query20	71	75	67	67
query21	196	135	129	129
query22	4826	4699	4677	4677
query23	31164	30444	30216	30216
query24	11056	2767	2762	2762
query25	553	308	307	307
query26	825	131	135	131
query27	2803	284	282	282
query28	6122	1879	1856	1856
query29	951	604	626	604
query30	283	132	136	132
query31	907	709	710	709
query32	85	50	51	50
query33	602	205	206	205
query34	810	456	464	456
query35	833	745	743	743
query36	1284	1267	1214	1214
query37	96	62	63	62
query38	3306	3204	3178	3178
query39	1325	1236	1243	1236
query40	195	94	81	81
query41	37	34	33	33
query42	92	84	84	84
query43	492	490	500	490
query44	1058	699	699	699
query45	191	177	174	174
query46	1053	642	636	636
query47	1562	1452	1507	1452
query48	389	307	317	307
query49	1110	281	282	281
query50	701	303	312	303
query51	5302	5218	5087	5087
query52	100	76	76	76
query53	316	258	261	258
query54	246	191	178	178
query55	81	74	74	74
query56	182	158	166	158
query57	987	868	933	868
query58	179	147	155	147
query59	2345	2319	2398	2319
query60	207	177	176	176
query61	84	86	83	83
query62	586	364	367	364
query63	275	262	264	262
query64	4030	1756	1772	1756
query65	3270	3224	3240	3224
query66	1137	328	321	321
query67	14549	14113	14132	14113
query68	3840	498	486	486
query69	431	309	314	309
query70	1569	1512	1552	1512
query71	275	220	210	210
query72	3704	2871	2835	2835
query73	690	316	313	313
query74	6565	6222	6315	6222
query75	2816	2342	2296	2296
query76	2933	985	928	928
query77	381	229	232	229
query78	9052	8712	8623	8623
query79	2255	496	503	496
query80	1379	321	310	310
query81	538	199	197	197
query82	663	83	76	76
query83	213	121	121	121
query84	229	75	70	70
query85	1906	347	340	340
query86	515	368	397	368
query87	3427	3288	3297	3288
query88	3911	2195	2225	2195
query89	416	355	348	348
query90	1933	184	186	184
query91	149	116	138	116
query92	55	44	42	42
query93	2133	458	441	441
query94	1288	163	157	157
query95	491	446	453	446
query96	609	316	322	316
query97	4270	4122	4125	4122
query98	209	194	193	193
query99	951	701	704	701
Total cold run time: 268148 ms
Total hot run time: 173086 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.46 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 57bc76687bad7a4a442ff594d877327237baf654, data reload: false

query1	0.04	0.04	0.03
query2	0.06	0.02	0.03
query3	0.22	0.06	0.06
query4	1.68	0.09	0.10
query5	0.53	0.52	0.52
query6	1.18	0.63	0.64
query7	0.02	0.02	0.01
query8	0.04	0.03	0.03
query9	0.56	0.50	0.49
query10	0.56	0.55	0.54
query11	0.12	0.09	0.08
query12	0.10	0.08	0.09
query13	0.61	0.60	0.61
query14	0.79	0.78	0.80
query15	0.79	0.78	0.78
query16	0.38	0.38	0.39
query17	1.02	0.99	1.01
query18	0.22	0.27	0.26
query19	1.91	1.78	1.81
query20	0.02	0.01	0.01
query21	15.41	0.56	0.57
query22	2.63	2.78	1.69
query23	17.38	0.81	0.74
query24	2.34	1.18	0.45
query25	0.43	0.16	0.12
query26	0.48	0.14	0.14
query27	0.06	0.05	0.04
query28	11.68	0.85	0.85
query29	12.55	3.31	3.26
query30	0.67	0.54	0.55
query31	2.78	0.35	0.34
query32	3.38	0.49	0.50
query33	3.21	3.25	3.19
query34	15.76	4.32	4.26
query35	4.33	4.32	4.33
query36	1.09	1.06	1.07
query37	0.06	0.04	0.05
query38	0.03	0.03	0.02
query39	0.02	0.02	0.02
query40	0.16	0.13	0.13
query41	0.07	0.02	0.01
query42	0.02	0.01	0.02
query43	0.03	0.02	0.02
Total cold run time: 105.42 s
Total hot run time: 30.46 s

@doris-robot
Copy link

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit 57bc76687bad7a4a442ff594d877327237baf654 with default session variables
Stream load json:         18 seconds loaded 2358488459 Bytes, about 124 MB/s
Stream load orc:          58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet:      32 seconds loaded 861443392 Bytes, about 25 MB/s
Insert into select:       13.6 seconds inserted 10000000 Rows, about 735K ops/s

@HappenLee HappenLee merged commit 2cec4ca into apache:master Jan 31, 2024
24 of 26 checks passed
yiguolei pushed a commit that referenced this pull request Jan 31, 2024
@jacktengg @Doris-Extras
[pipelinex](profile) improve memory counter of pipelineX (#30538)
9012098
yiguolei pushed a commit that referenced this pull request Jan 31, 2024
@jacktengg @Doris-Extras
[pipelinex](profile) improve memory counter of pipelineX (#30538)
ef8d9ad
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@mrhhsg mrhhsg mrhhsg approved these changes

@yiguolei yiguolei yiguolei approved these changes

@zclllyybb zclllyybb zclllyybb approved these changes

Assignees
No one assigned
Labels
approved Indicates a PR has been approved by one committer. reviewed
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
6 participants
@jacktengg @doris-robot @yiguolei @mrhhsg @zclllyybb @HappenLee