[enhencement](segcompaction) cancel inflight segcompaction tasks faster when load finish #28901
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…er when load finish [Goal] When building the rowset writer, avoid waiting for inflight segcompaction to elimite long tail latency for load. [Current situation] 1. The segcompaction of a rowset is executed serially. During the build phase, we need to wait for the completion of the inflight segcompaction task. 2. If the rowset writer finishes writing and starts building meta, then segments that have not been compacted will not be submitted to segcompaction worker. We simply ignore them to accelerate the build process. 3. But this is not enough. If a segcompaction task has already been submitted to the worker thread pool, we will set a cancelled flag for the worker, and nothing will be done during execution to complete the task ASAP. 4. But this is still not enough. Although the latency of the segcompaction task has been shortened by aforemetioned method, tasks may still be queuing in the thread pool. [Solution] We can increase the worker thread pool to avoid queuing congestion, but this is not the best solution. Segcompaction should be a best effort work, and should not use too many CPU and memory resources. So we adopted the strategy of unbinding build and segcompaction, specifically: 1. For the segcompaction task that is performing compaction operations, we should not interrupt it, otherwise it may cause file corruption 2. For those tasks still queued, we no longer care about their results (because these tasks will know they are cancelled and will not perform any actual operations), so we just ignore them and continue with the subsequent rowset build process Signed-off-by: freemandealer <[email protected]>
|
run buildall |
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
| @@ -997,17 +997,22 @@ Status StorageEngine::submit_compaction_task(TabletSharedPtr tablet, CompactionT | |||
| return _submit_compaction_task(tablet, compaction_type, force); | |||
| } | |||
|
|
|||
| Status StorageEngine::_handle_seg_compaction(SegcompactionWorker* worker, | |||
| SegCompactionCandidatesSharedPtr segments) { | |||
| Status StorageEngine::_handle_seg_compaction(std::shared_ptr<SegcompactionWorker> worker, | |||
There was a problem hiding this comment.
warning: method '_handle_seg_compaction' can be made static [readability-convert-member-functions-to-static]
Suggested change
| Status StorageEngine::_handle_seg_compaction(std::shared_ptr<SegcompactionWorker> worker, | |
| static Status StorageEngine::_handle_seg_compaction(std::shared_ptr<SegcompactionWorker> worker, |
| worker->compact_segments(segments); | ||
| // return OK here. error will be reported via BetaRowsetWriter::_segcompaction_status | ||
| return Status::OK(); | ||
| } | ||
|
|
||
| Status StorageEngine::submit_seg_compaction_task(SegcompactionWorker* worker, | ||
| Status StorageEngine::submit_seg_compaction_task(std::shared_ptr<SegcompactionWorker> worker, |
There was a problem hiding this comment.
warning: method 'submit_seg_compaction_task' can be made static [readability-convert-member-functions-to-static]
Suggested change
| Status StorageEngine::submit_seg_compaction_task(std::shared_ptr<SegcompactionWorker> worker, | |
| static Status StorageEngine::submit_seg_compaction_task(std::shared_ptr<SegcompactionWorker> worker, |
|
TeamCity be ut coverage result: |
|
(From new machine)TeamCity pipeline, clickbench performance test result: |
|
TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G' |
github-actions
bot
added
the
approved
|
PR approved by at least one committer and no changes requested. |
|
PR approved by anyone and no changes requested. |
CalvinKirs
approved these changes
Dec 28, 2023
HappenLee
pushed a commit
to HappenLee/incubator-doris
that referenced
this pull request
Jan 12, 2024
…er when load finish (apache#28901) [Goal] When building the rowset writer, avoid waiting for inflight segcompaction to elimite long tail latency for load. [Current situation] 1. The segcompaction of a rowset is executed serially. During the build phase, we need to wait for the completion of the inflight segcompaction task. 2. If the rowset writer finishes writing and starts building meta, then segments that have not been compacted will not be submitted to segcompaction worker. We simply ignore them to accelerate the build process. 3. But this is not enough. If a segcompaction task has already been submitted to the worker thread pool, we will set a cancelled flag for the worker, and nothing will be done during execution to complete the task ASAP. 4. But this is still not enough. Although the latency of the segcompaction task has been shortened by aforemetioned method, tasks may still be queuing in the thread pool. [Solution] We can increase the worker thread pool to avoid queuing congestion, but this is not the best solution. Segcompaction should be a best effort work, and should not use too many CPU and memory resources. So we adopted the strategy of unbinding build and segcompaction, specifically: 1. For the segcompaction task that is performing compaction operations, we should not interrupt it, otherwise it may cause file corruption 2. For those tasks still queued, we no longer care about their results (because these tasks will know they are cancelled and will not perform any actual operations), so we just ignore them and continue with the subsequent rowset build process Signed-off-by: freemandealer <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
[Goal]
When building the rowset writer, avoid waiting for inflight segcompaction to elimite long tail latency for load.
[Current situation]
The segcompaction of a rowset is executed serially. During the build phase, we need to wait for the completion of the inflight segcompaction task.
If the rowset writer finishes writing and starts building meta, then segments that have not been compacted will not be submitted to segcompaction worker. We simply ignore them to accelerate the build process.
But this is not enough. If a segcompaction task has already been submitted to the worker thread pool, we will set a cancelled flag for the worker, and nothing will be done during execution to complete the task ASAP.
But this is still not enough. Although the latency of the segcompaction task has been shortened by aforemetioned method, tasks may still be queuing in the thread pool.
[Solution]
We can increase the worker thread pool to avoid queuing congestion, but this is not the best solution.
Segcompaction should be a best effort work, and should not use too many CPU and memory resources. So we adopted the strategy of unbinding build and segcompaction, specifically:
For the segcompaction task that is performing compaction operations, we should not interrupt it, otherwise it may cause file corruption
For those tasks still queued, we no longer care about their results (because these tasks will know they are cancelled and will not perform any actual operations), so we just ignore them and continue with the subsequent rowset build process
Proposed changes
Issue Number: close #xxx
Further comments
If this is a relatively large or complex change, kick off the discussion at [email protected] by explaining why you chose the solution you did and what alternatives you considered, etc...