[improvement](spill) spill trigger improvement #32641
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Thank you for your contribution to Apache Doris. Since 2024-03-18, the Document has been moved to doris-website. |
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
| @@ -48,15 +48,25 @@ Status PartitionedHashJoinSinkLocalState::open(RuntimeState* state) { | |||
| RETURN_IF_ERROR(PipelineXSinkLocalState::open(state)); | |||
| return _partitioner->open(state); | |||
| } | |||
| Status PartitionedHashJoinSinkLocalState::close(RuntimeState* state, Status exec_status) { | |||
There was a problem hiding this comment.
warning: method 'close' can be made static [readability-convert-member-functions-to-static]
be/src/pipeline/exec/partitioned_hash_join_sink_operator.h:48:
- Status close(RuntimeState* state, Status exec_status) override;
+ static Status close(RuntimeState* state, Status exec_status) override;| _weighted_consumption = _consumption->current_value() * ratio; | ||
| _weighted_limit = weighted_limit; | ||
| } | ||
| void get_weighted_mem_info(int64_t& weighted_limit, int64_t& weighted_consumption) { |
There was a problem hiding this comment.
warning: method 'get_weighted_mem_info' can be made const [readability-make-member-function-const]
Suggested change
| void get_weighted_mem_info(int64_t& weighted_limit, int64_t& weighted_consumption) { | |
| void get_weighted_mem_info(int64_t& weighted_limit, int64_t& weighted_consumption) const { |
| bool is_high_wartermark = false; | ||
| double mem_used_ratio = 0; | ||
| }; | ||
| void WorkloadGroupMgr::refresh_wg_used_memory() { |
There was a problem hiding this comment.
warning: method 'refresh_wg_used_memory' can be made static [readability-convert-member-functions-to-static]
be/src/runtime/workload_group/workload_group_manager.h:56:
- void refresh_wg_used_memory();
+ static void refresh_wg_used_memory();| bool is_high_wartermark = false; | ||
| double mem_used_ratio = 0; | ||
| }; | ||
| void WorkloadGroupMgr::refresh_wg_used_memory() { |
There was a problem hiding this comment.
warning: function 'refresh_wg_used_memory' exceeds recommended size/complexity thresholds [readability-function-size]
void WorkloadGroupMgr::refresh_wg_used_memory() {
^Additional context
be/src/runtime/workload_group/workload_group_manager.cpp:146: 106 lines including whitespace and comments (threshold 80)
void WorkloadGroupMgr::refresh_wg_used_memory() {
^
yiguolei
reviewed
Mar 21, 2024
be/src/vec/spill/spill_writer.cpp
Outdated
| @@ -35,13 +35,19 @@ Status SpillWriter::open() { | |||
| return Status::OK(); | |||
| } | |||
|
|
|||
| SpillWriter::~SpillWriter() { | |||
| if (!closed_) { | |||
| LOG(WARNING) << "spill writer not closed correctly:\n" << boost::stacktrace::stacktrace(); | |||
There was a problem hiding this comment.
not use this way, boost has a static global lock.
Using doris::Status wrapped error stack, you could use it.
yiguolei
reviewed
Mar 21, 2024
be/src/vec/exprs/vexpr.cpp
Outdated
| @@ -97,11 +97,11 @@ TExprNode create_texpr_node_from(const void* data, const PrimitiveType& type, in | |||
| break; | |||
| } | |||
| case TYPE_DATEV2: { | |||
| THROW_IF_ERROR(create_texpr_literal_node<TYPE_DATEV2>(data, &node)); | |||
| THROW_IF_ERROR(create_texpr_literal_node<TYPE_DATEV2>(data, &node, precision, scale)); | |||
There was a problem hiding this comment.
These code is not related with this PR.
yiguolei
reviewed
Mar 21, 2024
| @@ -462,6 +472,10 @@ Status PartitionedHashJoinProbeOperatorX::open(RuntimeState* state) { | |||
| Status PartitionedHashJoinProbeOperatorX::push(RuntimeState* state, vectorized::Block* input_block, | |||
| bool eos) const { | |||
| auto& local_state = get_local_state(state); | |||
| if (!local_state._is_running) { | |||
There was a problem hiding this comment.
You could implement 2 method in parent class:
inc_running_big_mem_op_num(){
if (not add) {
get_query_ctx()->inc_running_big_mem_op_num;
not_add=true;
}
}
dec_big_mem_op_num(){
if (not_add) {
log_fatal<<} else if (!_dec) {
query_ctx->dec;
_dec=true;
}
}
then all subclass could use this method.
jacktengg
force-pushed
the
new-spill-to-disk-trigger-improve
branch
from
c893b72
to
d29b975
Compare
jacktengg
force-pushed
the
new-spill-to-disk-trigger-improve
branch
2 times, most recently
from
f6e85b1
to
0e279b3
Compare
|
run buildall |
TPC-H: Total hot run time: 38070 ms |
|
TeamCity be ut coverage result: |
TPC-DS: Total hot run time: 186969 ms |
|
Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G' |
jacktengg
force-pushed
the
new-spill-to-disk-trigger-improve
branch
2 times, most recently
from
074178e
to
e178335
Compare
|
run buildall |
TPC-H: Total hot run time: 38470 ms |
TPC-DS: Total hot run time: 181926 ms |
|
TeamCity be ut coverage result: |
|
Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G' |
jacktengg
force-pushed
the
new-spill-to-disk-trigger-improve
branch
from
e178335
to
615fff1
Compare
|
run buildall |
TPC-H: Total hot run time: 37958 ms |
TPC-DS: Total hot run time: 181686 ms |
|
TeamCity be ut coverage result: |
|
Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G' |
yiguolei
reviewed
Mar 28, 2024
| @@ -128,6 +129,7 @@ Status PartitionedAggSourceOperatorX::close(RuntimeState* state) { | |||
| Status PartitionedAggSourceOperatorX::get_block(RuntimeState* state, vectorized::Block* block, | |||
| bool* eos) { | |||
| auto& local_state = get_local_state(state); | |||
| local_state.inc_running_big_mem_op_num(state); | |||
There was a problem hiding this comment.
why do we need check local state's count.
Only operator is ok?
yiguolei
reviewed
Mar 28, 2024
|
|
||
| const auto min_revocable_mem_bytes = state->min_revocable_mem(); | ||
|
|
||
| auto tg = query_ctx->workload_group(); |
There was a problem hiding this comment.
If tg == nullptr then return false; to avoid core when there is no workload group bind to this query. And print log in this case.
There was a problem hiding this comment.
do not use tg. tg is old name "task group", use wg
yiguolei
previously approved these changes
Mar 29, 2024
github-actions
bot
added
the
approved
|
PR approved by at least one committer and no changes requested. |
|
PR approved by anyone and no changes requested. |
jacktengg
force-pushed
the
new-spill-to-disk-trigger-improve
branch
from
df0ddde
to
243b57b
Compare
github-actions
bot
removed
the
approved
jacktengg
force-pushed
the
new-spill-to-disk-trigger-improve
branch
from
243b57b
to
7943882
Compare
jacktengg
force-pushed
the
new-spill-to-disk-trigger-improve
branch
from
7943882
to
58807ff
Compare
jacktengg
force-pushed
the
new-spill-to-disk-trigger-improve
branch
from
58807ff
to
c8f6c47
Compare
|
run buildall |
TPC-H: Total hot run time: 37970 ms |
TPC-DS: Total hot run time: 182069 ms |
|
TeamCity be ut coverage result: |
ClickBench: Total hot run time: 29.21 s |
|
Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G' |
github-actions
bot
added
the
approved
|
PR approved by at least one committer and no changes requested. |
yiguolei
approved these changes
Mar 29, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Proposed changes
Issue Number: close #xxx
Further comments
If this is a relatively large or complex change, kick off the discussion at [email protected] by explaining why you chose the solution you did and what alternatives you considered, etc...