Failed to build delete range aggregator Magic number mismatch: expected

[kwannoel]

2022-08-24T01:04:14.584219Z  WARN node{id=13 name="compactor-2"}:task{id=6083}: risingwave_storage::hummock::compactor: Failed to build delete range aggregator Magic number mismatch: expected 1468377971, found: 23981349.

See https://gist.github.com/kwannoel/7f34fff04ee8f3eecdd8d10ab25bdc7f?permalink_comment_id=4608036#gistcomment-4608036 for more info.

[hzxa21]

I can successfully reproduced this issue locally and verified several things:

1. The encoding size and meta offset are both correct in the writer and reader side.
2. The content seen by the writer and the reader are indeed different.

I suspect there is something wrong in madsim s3 simulator so I turned on debug log in madsim and saw these debug logs in madsim relevant to the problematic object (id=93):

2022-08-24T01:04:09.509693Z DEBUG node{id=3 name="s3"}:task{id=4273}: madsim_aws_sdk_s3::sim::server::service: upload_part bucket="hummock001" key="hummock_001/32/93.data" upload_id="121596723" part_number=2
2022-08-24T01:04:09.511002Z DEBUG node{id=3 name="s3"}:task{id=4274}: madsim_aws_sdk_s3::sim::server::service: upload_part bucket="hummock001" key="hummock_001/32/93.data" upload_id="121596723" part_number=1
2022-08-24T01:04:09.528232Z DEBUG node{id=3 name="s3"}:task{id=4280}: madsim_aws_sdk_s3::sim::server::service: complete_multipart_upload bucket="hummock001" key="hummock_001/32/93.data" upload_id="121596723"

We can see that UploadPart for part_number=2 finishes before UploadPart for part_number=1, which is a valid behavior for S3 multipart upload since parts can be uploaded concurrently in different connections. However, after checking the madsim s3 simulator codes, the simulator seems to miss the ordering and put part2 before part1 in the object body.

Root cause: although the simulator sorts the parts by number at first and populates a selection idx, it also sorts the selection idx afterwards. This causes the selection idx ordering to be identical with the parts original ordering and thus miss to order the parts by part number.

[kwannoel]

cc @wangrunji0408

[hzxa21]

Potential fix: madsim-rs/madsim#149

[kwannoel]

fuzzing-37.log

Similar error encountered today, but from monitored_store iter instead:

risingwave_storage::monitor::monitored_store: Failed in iter: Hummock error: Magic number mismatch: expected 1468377971, found: 564084722.
  backtrace of inner error:

https://buildkite.com/risingwavelabs/generate-sqlsmith-snapshots-weekly/builds/108#0188f428-2377-4dcf-a342-456063a88aa3

[hzxa21]

fuzzing-37.log

Similar error encountered today, but from monitored_store iter instead:

risingwave_storage::monitor::monitored_store: Failed in iter: Hummock error: Magic number mismatch: expected 1468377971, found: 564084722.
  backtrace of inner error:

https://buildkite.com/risingwavelabs/generate-sqlsmith-snapshots-weekly/builds/108#0188f428-2377-4dcf-a342-456063a88aa3

@kwannoel Is the issue resolved after the madsim fix?

[kwannoel]

Yup, tested this issue + fuzzing-37 locally, in #10700.
Passes both. Thanks for your help!