Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[PipelineX](improvement) Prepare tasks in parallel #32789

Closed
wants to merge 9 commits into from
Closed

[PipelineX](improvement) Prepare tasks in parallel #32789

wants to merge 9 commits into from

Conversation

Gabriel39
Copy link
Contributor

@Gabriel39 Gabriel39 commented Mar 25, 2024 •
edited
Loading

Proposed changes

Now pipeline tasks are prepared in serial mode which has high overhead for short queries. This PR change it to parallel preparation.

Further comments

If this is a relatively large or complex change, kick off the discussion at [email protected] by explaining why you chose the solution you did and what alternatives you considered, etc...

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@Gabriel39
Copy link
Contributor Author

run buildall

github-actions[bot]
github-actions bot reviewed Mar 25, 2024
View reviewed changes
Copy link
Contributor

@github-actions github-actions bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

clang-tidy made some suggestions

be/src/pipeline/pipeline_x/pipeline_x_fragment_context.cpp
@@ -162,7 +162,8 @@ void PipelineXFragmentContext::cancel(const PPlanFragmentCancelReason& reason,
}
}

Status PipelineXFragmentContext::prepare(const doris::TPipelineFragmentParams& request) {
Status PipelineXFragmentContext::prepare(const doris::TPipelineFragmentParams& request,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

warning: function 'prepare' exceeds recommended size/complexity thresholds [readability-function-size]

Status PipelineXFragmentContext::prepare(const doris::TPipelineFragmentParams& request,
                                 ^
Additional context

be/src/pipeline/pipeline_x/pipeline_x_fragment_context.cpp:164: 107 lines including whitespace and comments (threshold 80)

Status PipelineXFragmentContext::prepare(const doris::TPipelineFragmentParams& request,
                                 ^

be/src/pipeline/pipeline_x/pipeline_x_fragment_context.cpp
@@ -479,11 +480,14 @@
return Status::OK();
}

Status PipelineXFragmentContext::_build_pipeline_tasks(
const doris::TPipelineFragmentParams& request) {
Status PipelineXFragmentContext::_build_pipeline_x_tasks(
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

warning: function '_build_pipeline_x_tasks' exceeds recommended size/complexity thresholds [readability-function-size]

Status PipelineXFragmentContext::_build_pipeline_x_tasks(
                                 ^
Additional context

be/src/pipeline/pipeline_x/pipeline_x_fragment_context.cpp:482: 220 lines including whitespace and comments (threshold 80)

Status PipelineXFragmentContext::_build_pipeline_x_tasks(
                                 ^

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 35.25% (8732/24770)
Line Coverage: 27.03% (71503/264501)
Region Coverage: 26.29% (37098/141135)
Branch Coverage: 23.18% (18969/81842)
Coverage Report: http://coverage.selectdb-in.cc/coverage/dcf772d4ab553c4f7805fad0221ae5aa57267747_dcf772d4ab553c4f7805fad0221ae5aa57267747/report/index.html

@Gabriel39
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 35.25% (8732/24770)
Line Coverage: 27.04% (71526/264501)
Region Coverage: 26.29% (37100/141137)
Branch Coverage: 23.19% (18977/81844)
Coverage Report: http://coverage.selectdb-in.cc/coverage/b2b1a8fd463b5ccd4aff4a269f31a23b79744f4f_b2b1a8fd463b5ccd4aff4a269f31a23b79744f4f/report/index.html

@Gabriel39
Copy link
Contributor Author

run buildall

github-actions[bot]
github-actions bot reviewed Mar 26, 2024
View reviewed changes
Copy link
Contributor

@github-actions github-actions bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

clang-tidy made some suggestions

be/src/pipeline/pipeline_x/pipeline_x_fragment_context.cpp
@@ -162,7 +164,8 @@ void PipelineXFragmentContext::cancel(const PPlanFragmentCancelReason& reason,
}
}

Status PipelineXFragmentContext::prepare(const doris::TPipelineFragmentParams& request) {
Status PipelineXFragmentContext::prepare(const doris::TPipelineFragmentParams& request,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

warning: function 'prepare' exceeds recommended size/complexity thresholds [readability-function-size]

Status PipelineXFragmentContext::prepare(const doris::TPipelineFragmentParams& request,
                                 ^
Additional context

be/src/pipeline/pipeline_x/pipeline_x_fragment_context.cpp:166: 107 lines including whitespace and comments (threshold 80)

Status PipelineXFragmentContext::prepare(const doris::TPipelineFragmentParams& request,
                                 ^

be/src/pipeline/pipeline_x/pipeline_x_fragment_context.cpp
@@ -479,11 +482,14 @@
return Status::OK();
}

Status PipelineXFragmentContext::_build_pipeline_tasks(
const doris::TPipelineFragmentParams& request) {
Status PipelineXFragmentContext::_build_pipeline_x_tasks(
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

warning: function '_build_pipeline_x_tasks' exceeds recommended size/complexity thresholds [readability-function-size]

Status PipelineXFragmentContext::_build_pipeline_x_tasks(
                                 ^
Additional context

be/src/pipeline/pipeline_x/pipeline_x_fragment_context.cpp:484: 216 lines including whitespace and comments (threshold 80)

Status PipelineXFragmentContext::_build_pipeline_x_tasks(
                                 ^

@Gabriel39
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 35.25% (8735/24777)
Line Coverage: 27.03% (71531/264661)
Region Coverage: 26.28% (37107/141188)
Branch Coverage: 23.17% (18975/81884)
Coverage Report: http://coverage.selectdb-in.cc/coverage/62324e15b00b1d474b0e1ef47759ecdce5d79221_62324e15b00b1d474b0e1ef47759ecdce5d79221/report/index.html

@Gabriel39
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 35.26% (8737/24782)
Line Coverage: 27.02% (71524/264733)
Region Coverage: 26.27% (37114/141260)
Branch Coverage: 23.17% (18975/81908)
Coverage Report: http://coverage.selectdb-in.cc/coverage/62324e15b00b1d474b0e1ef47759ecdce5d79221_62324e15b00b1d474b0e1ef47759ecdce5d79221/report/index.html

@Gabriel39
Copy link
Contributor Author

run buildall

github-actions[bot]
github-actions bot reviewed Apr 11, 2024
View reviewed changes
Copy link
Contributor

@github-actions github-actions bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

clang-tidy made some suggestions

be/src/pipeline/pipeline_x/pipeline_x_fragment_context.cpp
@@ -164,7 +166,8 @@ void PipelineXFragmentContext::cancel(const PPlanFragmentCancelReason& reason,
}
}

Status PipelineXFragmentContext::prepare(const doris::TPipelineFragmentParams& request) {
Status PipelineXFragmentContext::prepare(const doris::TPipelineFragmentParams& request,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

warning: function 'prepare' exceeds recommended size/complexity thresholds [readability-function-size]

Status PipelineXFragmentContext::prepare(const doris::TPipelineFragmentParams& request,
                                 ^
Additional context

be/src/pipeline/pipeline_x/pipeline_x_fragment_context.cpp:168: 106 lines including whitespace and comments (threshold 80)

Status PipelineXFragmentContext::prepare(const doris::TPipelineFragmentParams& request,
                                 ^

be/src/pipeline/pipeline_x/pipeline_x_fragment_context.cpp
@@ -480,11 +483,14 @@
return Status::OK();
}

Status PipelineXFragmentContext::_build_pipeline_tasks(
const doris::TPipelineFragmentParams& request) {
Status PipelineXFragmentContext::_build_pipeline_x_tasks(
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

warning: function '_build_pipeline_x_tasks' exceeds recommended size/complexity thresholds [readability-function-size]

Status PipelineXFragmentContext::_build_pipeline_x_tasks(
                                 ^
Additional context

be/src/pipeline/pipeline_x/pipeline_x_fragment_context.cpp:485: 216 lines including whitespace and comments (threshold 80)

Status PipelineXFragmentContext::_build_pipeline_x_tasks(
                                 ^

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 35.60% (8904/25011)
Line Coverage: 27.30% (73092/267776)
Region Coverage: 26.44% (37797/142963)
Branch Coverage: 23.20% (19264/83048)
Coverage Report: http://coverage.selectdb-in.cc/coverage/36d531d736fc8d54873f82a3bbef5f4ad8ae8731_36d531d736fc8d54873f82a3bbef5f4ad8ae8731/report/index.html

github-actions[bot]
github-actions bot reviewed Apr 19, 2024
View reviewed changes
Copy link
Contributor

@github-actions github-actions bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

clang-tidy made some suggestions

be/src/pipeline/pipeline_x/pipeline_x_fragment_context.cpp
@@ -486,11 +491,14 @@ Status PipelineXFragmentContext::_create_data_sink(ObjectPool* pool, const TData
return Status::OK();
}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

warning: function '_build_pipeline_x_tasks' exceeds recommended size/complexity thresholds [readability-function-size]

Status PipelineXFragmentContext::_build_pipeline_x_tasks(
                                 ^
Additional context

be/src/pipeline/pipeline_x/pipeline_x_fragment_context.cpp:491: 216 lines including whitespace and comments (threshold 80)

Status PipelineXFragmentContext::_build_pipeline_x_tasks(
                                 ^

@Gabriel39
Copy link
Contributor Author

run buildall

github-actions[bot]
github-actions bot reviewed Apr 19, 2024
View reviewed changes
Copy link
Contributor

@github-actions github-actions bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

clang-tidy made some suggestions

be/src/pipeline/pipeline_x/pipeline_x_fragment_context.cpp
@@ -164,7 +166,8 @@ void PipelineXFragmentContext::cancel(const PPlanFragmentCancelReason& reason,
}
}

Status PipelineXFragmentContext::prepare(const doris::TPipelineFragmentParams& request) {
Status PipelineXFragmentContext::prepare(const doris::TPipelineFragmentParams& request,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

warning: function 'prepare' exceeds recommended size/complexity thresholds [readability-function-size]

Status PipelineXFragmentContext::prepare(const doris::TPipelineFragmentParams& request,
                                 ^
Additional context

be/src/pipeline/pipeline_x/pipeline_x_fragment_context.cpp:168: 108 lines including whitespace and comments (threshold 80)

Status PipelineXFragmentContext::prepare(const doris::TPipelineFragmentParams& request,
                                 ^

be/src/pipeline/pipeline_x/pipeline_x_fragment_context.cpp
@@ -486,11 +491,14 @@
return Status::OK();
}

Status PipelineXFragmentContext::_build_pipeline_tasks(
const doris::TPipelineFragmentParams& request) {
Status PipelineXFragmentContext::_build_pipeline_x_tasks(
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

warning: function '_build_pipeline_x_tasks' exceeds recommended size/complexity thresholds [readability-function-size]

Status PipelineXFragmentContext::_build_pipeline_x_tasks(
                                 ^
Additional context

be/src/pipeline/pipeline_x/pipeline_x_fragment_context.cpp:493: 216 lines including whitespace and comments (threshold 80)

Status PipelineXFragmentContext::_build_pipeline_x_tasks(
                                 ^

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 35.42% (8920/25184)
Line Coverage: 27.13% (73289/270140)
Region Coverage: 26.25% (37835/144124)
Branch Coverage: 23.06% (19277/83586)
Coverage Report: http://coverage.selectdb-in.cc/coverage/6047d6d3ee97a21b3d36461df074d3daababe733_6047d6d3ee97a21b3d36461df074d3daababe733/report/index.html

@Gabriel39
Copy link
Contributor Author

run buildall

github-actions[bot]
github-actions bot reviewed Apr 19, 2024
View reviewed changes
Copy link
Contributor

@github-actions github-actions bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

clang-tidy made some suggestions

be/src/pipeline/pipeline_x/pipeline_x_fragment_context.cpp
@@ -486,11 +491,14 @@ Status PipelineXFragmentContext::_create_data_sink(ObjectPool* pool, const TData
return Status::OK();
}

Status PipelineXFragmentContext::_build_pipeline_tasks(
const doris::TPipelineFragmentParams& request) {
Status PipelineXFragmentContext::_build_pipeline_x_tasks(
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

warning: function '_build_pipeline_x_tasks' exceeds recommended size/complexity thresholds [readability-function-size]

Status PipelineXFragmentContext::_build_pipeline_x_tasks(
                                 ^
Additional context

be/src/pipeline/pipeline_x/pipeline_x_fragment_context.cpp:493: 217 lines including whitespace and comments (threshold 80)

Status PipelineXFragmentContext::_build_pipeline_x_tasks(
                                 ^

@Gabriel39
Copy link
Contributor Author

run buildall

github-actions[bot]
github-actions bot reviewed Apr 19, 2024
View reviewed changes
Copy link
Contributor

@github-actions github-actions bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

clang-tidy made some suggestions

be/src/pipeline/pipeline_x/pipeline_x_fragment_context.cpp
@@ -486,11 +491,14 @@ Status PipelineXFragmentContext::_create_data_sink(ObjectPool* pool, const TData
return Status::OK();
}

Status PipelineXFragmentContext::_build_pipeline_tasks(
const doris::TPipelineFragmentParams& request) {
Status PipelineXFragmentContext::_build_pipeline_x_tasks(
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

warning: function '_build_pipeline_x_tasks' exceeds recommended size/complexity thresholds [readability-function-size]

Status PipelineXFragmentContext::_build_pipeline_x_tasks(
                                 ^
Additional context

be/src/pipeline/pipeline_x/pipeline_x_fragment_context.cpp:493: 215 lines including whitespace and comments (threshold 80)

Status PipelineXFragmentContext::_build_pipeline_x_tasks(
                                 ^

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 35.42% (8920/25184)
Line Coverage: 27.13% (73288/270145)
Region Coverage: 26.25% (37838/144126)
Branch Coverage: 23.07% (19284/83588)
Coverage Report: http://coverage.selectdb-in.cc/coverage/e7c9d215f45eff0d56edb5487c5f00a85f3696af_e7c9d215f45eff0d56edb5487c5f00a85f3696af/report/index.html

@Gabriel39
Copy link
Contributor Author

run buildall

github-actions[bot]
github-actions bot reviewed Apr 19, 2024
View reviewed changes
Copy link
Contributor

@github-actions github-actions bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

clang-tidy made some suggestions

be/src/pipeline/pipeline_x/pipeline_x_fragment_context.cpp
@@ -164,7 +168,8 @@ void PipelineXFragmentContext::cancel(const PPlanFragmentCancelReason& reason,
}
}

Status PipelineXFragmentContext::prepare(const doris::TPipelineFragmentParams& request) {
Status PipelineXFragmentContext::prepare(const doris::TPipelineFragmentParams& request,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

warning: function 'prepare' exceeds recommended size/complexity thresholds [readability-function-size]

Status PipelineXFragmentContext::prepare(const doris::TPipelineFragmentParams& request,
                                 ^
Additional context

be/src/pipeline/pipeline_x/pipeline_x_fragment_context.cpp:170: 105 lines including whitespace and comments (threshold 80)

Status PipelineXFragmentContext::prepare(const doris::TPipelineFragmentParams& request,
                                 ^

be/src/pipeline/pipeline_x/pipeline_x_fragment_context.cpp
@@ -486,11 +490,17 @@
return Status::OK();
}

Status PipelineXFragmentContext::_build_pipeline_tasks(
const doris::TPipelineFragmentParams& request) {
Status PipelineXFragmentContext::_build_pipeline_x_tasks(
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

warning: function '_build_pipeline_x_tasks' exceeds recommended size/complexity thresholds [readability-function-size]

Status PipelineXFragmentContext::_build_pipeline_x_tasks(
                                 ^
Additional context

be/src/pipeline/pipeline_x/pipeline_x_fragment_context.cpp:492: 222 lines including whitespace and comments (threshold 80)

Status PipelineXFragmentContext::_build_pipeline_x_tasks(
                                 ^

@doris-robot
Copy link

TPC-H: Total hot run time: 38130 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit a018575f4cf3da1a5f4d7309606077cdb9e9191b, data reload: false

------ Round 1 ----------------------------------
q1	17612	4352	4229	4229
q2	2006	183	173	173
q3	10454	1226	1203	1203
q4	10196	811	751	751
q5	7535	2612	2627	2612
q6	213	130	131	130
q7	1003	595	572	572
q8	9220	2034	2020	2020
q9	7317	6598	6512	6512
q10	8529	3487	3529	3487
q11	430	222	212	212
q12	462	211	207	207
q13	18940	2916	2932	2916
q14	260	223	229	223
q15	531	484	461	461
q16	523	370	370	370
q17	955	633	679	633
q18	7385	6661	6613	6613
q19	5889	1545	1475	1475
q20	625	304	291	291
q21	3428	2779	2744	2744
q22	353	296	308	296
Total cold run time: 113866 ms
Total hot run time: 38130 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4400	4217	4224	4217
q2	358	252	260	252
q3	2958	2670	2739	2670
q4	1863	1596	1536	1536
q5	5329	5272	5304	5272
q6	203	122	122	122
q7	2207	1852	1839	1839
q8	3197	3321	3292	3292
q9	8562	8574	8548	8548
q10	4071	3917	3953	3917
q11	636	485	496	485
q12	820	620	592	592
q13	16317	3230	3205	3205
q14	323	284	274	274
q15	501	471	490	471
q16	485	415	445	415
q17	1823	1531	1535	1531
q18	8126	7767	7803	7767
q19	1731	1558	1539	1539
q20	1944	1797	1866	1797
q21	5196	4881	4895	4881
q22	521	458	452	452
Total cold run time: 71571 ms
Total hot run time: 55074 ms

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 35.42% (8922/25188)
Line Coverage: 27.10% (73276/270381)
Region Coverage: 26.25% (37852/144223)
Branch Coverage: 23.05% (19279/83632)
Coverage Report: http://coverage.selectdb-in.cc/coverage/a018575f4cf3da1a5f4d7309606077cdb9e9191b_a018575f4cf3da1a5f4d7309606077cdb9e9191b/report/index.html

@doris-robot
Copy link

TPC-DS: Total hot run time: 183049 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit a018575f4cf3da1a5f4d7309606077cdb9e9191b, data reload: false

query1	884	362	345	345
query2	6704	2493	2415	2415
query3	6639	199	195	195
query4	23584	21314	21215	21215
query5	4094	379	376	376
query6	253	167	161	161
query7	4565	274	280	274
query8	233	183	170	170
query9	8398	2287	2243	2243
query10	405	219	231	219
query11	14767	14180	14113	14113
query12	128	82	76	76
query13	1626	349	347	347
query14	9320	7062	7033	7033
query15	213	177	177	177
query16	7860	247	250	247
query17	1944	587	572	572
query18	1951	274	272	272
query19	194	144	150	144
query20	85	76	82	76
query21	200	131	117	117
query22	5044	4804	4815	4804
query23	33530	32807	32862	32807
query24	11185	3008	3062	3008
query25	589	385	377	377
query26	684	154	151	151
query27	2465	366	356	356
query28	5657	2046	2038	2038
query29	888	626	608	608
query30	328	172	170	170
query31	983	759	734	734
query32	96	49	50	49
query33	619	224	227	224
query34	1024	490	493	490
query35	846	720	701	701
query36	1088	913	916	913
query37	101	68	67	67
query38	3528	3391	3299	3299
query39	1661	1586	1572	1572
query40	173	122	118	118
query41	50	45	41	41
query42	102	93	96	93
query43	531	531	525	525
query44	1152	731	755	731
query45	282	254	262	254
query46	1130	777	739	739
query47	2024	1957	1933	1933
query48	360	285	301	285
query49	851	384	385	384
query50	775	391	388	388
query51	6875	6809	6944	6809
query52	105	85	87	85
query53	344	271	273	271
query54	302	218	225	218
query55	141	69	68	68
query56	216	194	201	194
query57	1294	1197	1183	1183
query58	194	178	184	178
query59	3531	3439	3208	3208
query60	236	221	197	197
query61	71	71	87	71
query62	590	423	439	423
query63	292	264	262	262
query64	4728	2870	3744	2870
query65	3036	2984	2972	2972
query66	728	301	307	301
query67	15637	14981	14907	14907
query68	9796	581	541	541
query69	557	285	278	278
query70	1416	1140	1127	1127
query71	1462	1262	1257	1257
query72	6853	2598	2395	2395
query73	1586	308	314	308
query74	6843	6419	6445	6419
query75	4724	2534	2604	2534
query76	5417	985	951	951
query77	593	229	232	229
query78	11006	10142	10174	10142
query79	10311	512	508	508
query80	2060	390	399	390
query81	502	223	227	223
query82	162	85	82	82
query83	198	137	135	135
query84	259	75	73	73
query85	1073	247	240	240
query86	330	256	274	256
query87	3487	3285	3223	3223
query88	5870	2263	2269	2263
query89	474	355	349	349
query90	2451	166	164	164
query91	112	83	86	83
query92	51	42	47	42
query93	5475	505	505	505
query94	1574	166	163	163
query95	1087	1114	1080	1080
query96	607	263	253	253
query97	3102	2918	2965	2918
query98	226	217	207	207
query99	1081	853	848	848
Total cold run time: 300858 ms
Total hot run time: 183049 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.21 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit a018575f4cf3da1a5f4d7309606077cdb9e9191b, data reload: false

query1	0.03	0.03	0.03
query2	0.08	0.04	0.04
query3	0.23	0.04	0.04
query4	1.69	0.09	0.10
query5	0.53	0.51	0.49
query6	1.46	0.73	0.71
query7	0.02	0.01	0.02
query8	0.05	0.04	0.03
query9	0.54	0.49	0.49
query10	0.54	0.55	0.54
query11	0.15	0.12	0.12
query12	0.14	0.12	0.11
query13	0.60	0.59	0.58
query14	0.74	0.77	0.76
query15	0.84	0.80	0.80
query16	0.36	0.35	0.34
query17	1.00	0.94	0.99
query18	0.21	0.23	0.24
query19	1.77	1.71	1.71
query20	0.01	0.01	0.01
query21	15.40	0.66	0.65
query22	4.97	6.86	1.73
query23	18.33	1.40	1.39
query24	1.96	0.24	0.21
query25	0.16	0.08	0.08
query26	0.27	0.16	0.17
query27	0.07	0.07	0.08
query28	13.26	1.01	0.99
query29	12.98	3.26	3.26
query30	0.26	0.09	0.06
query31	2.83	0.38	0.36
query32	3.32	0.46	0.45
query33	2.80	2.84	2.85
query34	17.16	4.42	4.43
query35	4.43	4.46	4.48
query36	0.63	0.46	0.44
query37	0.18	0.15	0.15
query38	0.15	0.15	0.14
query39	0.05	0.03	0.03
query40	0.17	0.12	0.13
query41	0.08	0.04	0.05
query42	0.06	0.05	0.05
query43	0.04	0.04	0.04
Total cold run time: 110.55 s
Total hot run time: 30.21 s

@doris-robot
Copy link

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit a018575f4cf3da1a5f4d7309606077cdb9e9191b with default session variables
Stream load json:         19 seconds loaded 2358488459 Bytes, about 118 MB/s
Stream load orc:          58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet:      32 seconds loaded 861443392 Bytes, about 25 MB/s
Insert into select:       13.5 seconds inserted 10000000 Rows, about 740K ops/s

@Gabriel39 Gabriel39 closed this Apr 26, 2024
@Gabriel39 Gabriel39 reopened this Apr 30, 2024
@Gabriel39 Gabriel39 closed this Sep 18, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@github-actions github-actions[bot] github-actions[bot] left review comments

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@Gabriel39 @doris-robot