[enhancement](MOW) refactor delete bitmap calculator and enhance the deletion bitmap calculation for full duplicate load

[jixxiong]

Proposed changes

Issue Number: close #xxx

We propose a new approach based on multi-way merge sort to calculate the delete bitmap between new and old records. Specifically, under the Unique-Key model, in the previous implementation, we would individually search for each newly added record within the existing rowsets using a primary key index to identify records that needed to be marked for deletion. However, this method's efficiency is significantly compromised when performing full duplicate loads especially when the number of rowset gets larger.

This patch does not exhibit significant performance degradation on average, but in the context of 't+1 load' scenarios (where multiple minor updates are performed on a substantial existing dataset before a full-scale replacement update), there is a notable performance improvement.

For example, let's say we create a unique-key table with a composite key of 3 integers. We then adopt the following import approach: first, perform a full import of all the data (40M rows), followed by several, say 20, smaller-scale imports (1M rows), and then another full import (40M rows). In the final import, on our machine, we observed that the unoptimized code took 89 seconds to import, whereas after implementing optimizations, the code only required 50 seconds.

Further comments

If this is a relatively large or complex change, kick off the discussion at [email protected] by explaining why you chose the solution you did and what alternatives you considered, etc...

[github-actions]

clang-tidy review says "All clean, LGTM! 👍"

[github-actions]

clang-tidy review says "All clean, LGTM! 👍"

[github-actions]

clang-tidy review says "All clean, LGTM! 👍"

[github-actions]

clang-tidy review says "All clean, LGTM! 👍"

[jixxiong]

run buildall

[hello-stephen]

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 48.37 seconds
stream load tsv: 509 seconds loaded 74807831229 Bytes, about 140 MB/s
stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s
stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s
insert into select: 29.1 seconds inserted 10000000 Rows, about 343K ops/s
storage size: 17162308626 Bytes

[github-actions]

clang-tidy review says "All clean, LGTM! 👍"

[jixxiong]

run buildall

[hello-stephen]

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 45.22 seconds
stream load tsv: 514 seconds loaded 74807831229 Bytes, about 138 MB/s
stream load json: 21 seconds loaded 2358488459 Bytes, about 107 MB/s
stream load orc: 64 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 29.1 seconds inserted 10000000 Rows, about 343K ops/s
storage size: 17162229976 Bytes

[github-actions]

clang-tidy review says "All clean, LGTM! 👍"

[github-actions]

clang-tidy review says "All clean, LGTM! 👍"

[github-actions]

clang-tidy review says "All clean, LGTM! 👍"

[github-actions]

clang-tidy review says "All clean, LGTM! 👍"

[github-actions]

clang-tidy review says "All clean, LGTM! 👍"

[github-actions]

clang-tidy review says "All clean, LGTM! 👍"

[github-actions]

clang-tidy review says "All clean, LGTM! 👍"

[jixxiong]

run buildall

[hello-stephen]

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 46.46 seconds
stream load tsv: 510 seconds loaded 74807831229 Bytes, about 139 MB/s
stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s
stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 30 seconds loaded 861443392 Bytes, about 27 MB/s
insert into select: 29.0 seconds inserted 10000000 Rows, about 344K ops/s
storage size: 17162165275 Bytes

[github-actions]

clang-tidy review says "All clean, LGTM! 👍"

[jixxiong]

run buildall

[hello-stephen]

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 45.66 seconds
stream load tsv: 510 seconds loaded 74807831229 Bytes, about 139 MB/s
stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s
stream load orc: 66 seconds loaded 1101869774 Bytes, about 15 MB/s
stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s
insert into select: 29.3 seconds inserted 10000000 Rows, about 341K ops/s
storage size: 17162109814 Bytes

[github-actions]

clang-tidy review says "All clean, LGTM! 👍"

[github-actions]

clang-tidy review says "All clean, LGTM! 👍"

[jixxiong]

run buildall

[hello-stephen]

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 47.79 seconds
stream load tsv: 512 seconds loaded 74807831229 Bytes, about 139 MB/s
stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s
stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 30 seconds loaded 861443392 Bytes, about 27 MB/s
insert into select: 29.2 seconds inserted 10000000 Rows, about 342K ops/s
storage size: 17162059115 Bytes

[jixxiong]

run buildall

[github-actions]

clang-tidy review says "All clean, LGTM! 👍"

[hello-stephen]

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 47.32 seconds
stream load tsv: 517 seconds loaded 74807831229 Bytes, about 137 MB/s
stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s
stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s
insert into select: 29.4 seconds inserted 10000000 Rows, about 340K ops/s
storage size: 17162182728 Bytes

[jixxiong]

run p0

[jixxiong]

run buildall

[hello-stephen]

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 45.83 seconds
stream load tsv: 517 seconds loaded 74807831229 Bytes, about 137 MB/s
stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s
stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s
insert into select: 29.1 seconds inserted 10000000 Rows, about 343K ops/s
storage size: 17162304040 Bytes

[github-actions]

clang-tidy review says "All clean, LGTM! 👍"

[jixxiong]

run buildall

[github-actions]

We're closing this PR because it hasn't been updated in a while.
This isn't a judgement on the merit of the PR in any way. It's just a way of keeping the PR queue manageable.
If you'd like to revive this PR, please reopen it and feel free a maintainer to remove the Stale tag!