Commits on Dec 22, 2023

1. [enhencement](segcompaction) cancel inflight segcompaction tasks fast… …

   …er when load finish
   
   [Goal]
   When building the rowset writer, avoid waiting for inflight segcompaction
   to elimite long tail latency for load.
   
   [Current situation]
   1. The segcompaction of a rowset is executed serially. During the build phase,
   we need to wait for the completion of the inflight segcompaction task.
   
   2. If the rowset writer finishes writing and starts building meta, then segments
   that have not been compacted will not be submitted to segcompaction worker.
   We simply ignore them to accelerate the build process.
   
   3. But this is not enough. If a segcompaction task has already been submitted to
   the worker thread pool, we will set a cancelled flag for the worker,
   and nothing will be done during execution to complete the task ASAP.
   
   4. But this is still not enough. Although the latency of the segcompaction task
   has been shortened by aforemetioned method, tasks may still be queuing in the
   thread pool.
   
   [Solution]
   We can increase the worker thread pool to avoid queuing congestion, but this is
   not the best solution.
   Segcompaction should be a best effort work, and should not use too many CPU and
   memory resources. So we adopted the strategy of unbinding build and segcompaction,
   specifically:
   
   1. For the segcompaction task that is performing compaction operations, we should
   not interrupt it, otherwise it may cause file corruption
   
   2. For those tasks still queued, we no longer care about their results (because
   these tasks will know they are cancelled and will not perform any actual operations),
   so we just ignore them and continue with the subsequent rowset build process
   
   Signed-off-by: freemandealer <[email protected]>

   

   freemandealer committed Dec 22, 2023
   Configuration menu

   • View commit details
   • Copy full SHA for cef2636
   • Browse repository at this point

   Copy the full SHA

   cef2636 View commit details

   Browse the repository at this point in the history