Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
#Centipede Seeder: More RAM-efficient source element sampling
Avoid a spike in peak RSS & VSize usage and speed up source shards processing. The previous version std::sampled the elements themselves, inserting copies into the destination. The new version std::shuffles and resizes a source's elements' indices to achieve the same effect of sampling, then moves selected elements from the source's vector to the destination vector. Example rusage profile for one large shard shows RSS reduction of 30%, from 36GB to 24GB, and sampling time decrease from 40 seconds to 2. BEFORE: ``` RSS: seed_corpus_maker_lib.cc:254 @ 14:49:18.95 [P.0:S.80 Done reading ] 25.22G [#################################-----------------] seed_corpus_maker_lib.cc:271 @ 14:49:20.07 [P.0:S.81 Done merging ] 25.22G [#################################-----------------] seed_corpus_maker_lib.cc:154 @ 14:49:24.50 [P.0:S.82 Timelapse ] 26.22G [###################################---------------] seed_corpus_maker_lib.cc:154 @ 14:49:34.50 [P.0:S.83 Timelapse ] 29.22G [#######################################-----------] seed_corpus_maker_lib.cc:154 @ 14:49:44.50 [P.0:S.84 Timelapse ] 32.22G [############################################------] seed_corpus_maker_lib.cc:154 @ 14:49:54.50 [P.0:S.85 Timelapse ] 35.22G [################################################--] seed_corpus_maker_lib.cc:307 @ 14:49:59.00 [P.0:S.86 Done sampling] 36.22G [##################################################] seed_corpus_maker_lib.cc:154 @ 14:50:04.50 [P.0:S.87 Timelapse ] 36.22G [##################################################] seed_corpus_maker_lib.cc:154 @ 14:50:14.50 [P.0:S.88 Timelapse ] 36.22G [##################################################] seed_corpus_maker_lib.cc:154 @ 14:50:24.50 [P.0:S.89 Timelapse ] 36.22G [##################################################] seed_corpus_maker_lib.cc:154 @ 14:50:34.50 [P.0:S.90 Timelapse ] 36.22G [##################################################] seed_corpus_maker_lib.cc:154 @ 14:50:44.50 [P.0:S.91 FINAL ] 36.22G [##################################################] ``` AFTER: ``` RSS: seed_corpus_maker_lib.cc:256 @ 21:05:28.54 [P.0:S.74 Done reading ] 24.07G [#################################################-] seed_corpus_maker_lib.cc:276 @ 21:05:29.61 [P.0:S.75 Done merging ] 24.26G [#################################################-] seed_corpus_maker_lib.cc:316 @ 21:05:31.14 [P.0:S.76 Done sampling ] 24.26G [#################################################-] seed_corpus_maker_lib.cc:324 @ 21:05:31.57 [P.0:S.77 Done appending] 24.26G [#################################################-] seed_corpus_maker_lib.cc:156 @ 21:05:33.96 [P.0:S.78 Timelapse ] 24.26G [#################################################-] seed_corpus_maker_lib.cc:156 @ 21:05:43.96 [P.0:S.79 Timelapse ] 24.26G [#################################################-] seed_corpus_maker_lib.cc:156 @ 21:05:53.96 [P.0:S.80 FINAL ] 24.26G [##################################################] ``` PiperOrigin-RevId: 592309603
- Loading branch information