[Filestore] Invalid CollectGarbage requests to blobstorage.

[debnatkh]

Errors like following stared causing IndexTablet to restart

NFS_SERVER[578573]: 2024-03-05T15:06:36.674043Z :NFS_TABLET ERROR: [f:***][t:***] CollectGarbage failed: SEVERITY_ERROR | FACILITY_KIKIMR | 1 Processed status# ERROR from VDisk# [8200021d:2:0:0:0] incarnationGuid# empty QuorumTracker status# ERROR

Looks like CollectGarbage requests sent by TIndexTablet does not guarantee the increasing order of (gen, step)

Started seeing this error much more often after enabling vhost-side reads on the whole cluster

[debnatkh]

1. CollectGarbage is executed with commitId = GetCurrentCommitId() = 42
2. Cleanup is started. It acquires a collect barrier with commitId = 42:

nbs/cloud/filestore/libs/storage/tablet/tablet_actor_cleanup.cpp

Lines 68 to 72 in 0555c7d

	ExecuteTx<TCleanup>(
	ctx,
	std::move(requestInfo),
	msg->RangeId,
	GetCurrentCommitId());

nbs/cloud/filestore/libs/storage/tablet/tablet_actor_cleanup.cpp

Line 100 in 0555c7d

AcquireCollectBarrier(args.CollectBarrier);

3. Before the collect barrier is released on completing the Cleanup transaction, another CollectGarbage is exeсuted. CollectCommitId is selected as follows:

nbs/cloud/filestore/libs/storage/tablet/tablet_state_data.cpp

Lines 935 to 939 in 0555c7d

	ui64 TIndexTabletState::GetCollectCommitId() const
	{
	// should not collect after any barrier
	return Min(GetCurrentCommitId(), Impl->GarbageQueue.GetCollectCommitId());
	}

4. GarbageQueue.GetCollectCommitId():

nbs/cloud/filestore/libs/storage/tablet/model/garbage_queue.cpp

Lines 224 to 232 in 0555c7d

	ui64 TGarbageQueue::GetCollectCommitId() const
	{
	if (Impl->Barriers) {
	const auto& barrier = *Impl->Barriers.begin();
	return barrier.CommitId - 1;
	}
	return InvalidCommitId;
	}

5. There is an unreleased collect barrier with commitId = 42, thus CollectCommitId will be equal to 41, which is less than LastCollectCommitId

---

Generating a new CommitId on the Cleanup execution will solve the issue

[debnatkh]

The main problem is that FlushBytes acquires collect barrier, which is less than the last collect commit id:

---

1. Consider that there were the following sequence of writes:

Write(0,       256 KiB, 'a') -> Blob(commitId = 42)
Write(256 KiB, 256 KiB, 'b') -> Blob(commitId = 43)
Write(512 KiB, 1,       'f') -> FreshBytes(commitId = 44)
Write(0,       256 KiB, 'c') -> Blob(commitId = 45)

This will lead to the following file layout: [ccccccc][bbbbbbb][f]

2. After execution of the CollectGarbage, all three new blobs will get a KeepFlag and the last collect commit id will be equal to 44

CommitId:     41      42        43       44
            Blob(a) Blob(b) FreshBytes Blob(c)
                                          |
                                 LastCollectCommitId

3. After execution of the Cleanup operation, the first blob will be marked as garbage

4. Let us execute FlushBytes operation, It will acquire collect barrier, equal to the minimal commitId, associated with FreshBlobs:
   

   nbs/cloud/filestore/libs/storage/tablet/tablet_actor_flush_bytes.cpp

   Lines 669 to 671 in 836a516

   	for (const auto& bytes: args.Bytes) {
   	args.CollectCommitId = Min(args.CollectCommitId, bytes.MinCommitId);
   	}

After this acquisition there will be one barrier, equal to 43:

CommitId:     41      42        43       44
            Blob(a) Blob(b) FreshBytes Blob(c)
                                |         |
                             Barrier  LastCollectCommitId

5. When the next CollectGarbage operation is to be executed, it will choose 42 as a collectCommitId:

nbs/cloud/filestore/libs/storage/tablet/model/garbage_queue.cpp

Lines 227 to 228 in 836a516

	const auto& barrier = *Impl->Barriers.begin();
	return barrier.CommitId - 1;

After it the CollectGarbage request with one new grabage will be sent, leading to a decrease in collectCommitIds sequence: 42 after 44

[debnatkh]

To reproduce the issue, one can use fio:

fio --name=random-write-test \
    --ioengine=libaio \
    --rw=randwrite \
    --bs=512-4k \
    --size=1G \
    --direct=1 \
    --iodepth=16 \
    --numjobs=4 \
    --offset_increment=512 \
    --do_verify=0 \
    --time_based \
    --runtime=$[120*60*60]

AppCriticalEvents/CollectGarbageError errors after starting afformentioned fio:

AppCriticalEvents/CollectGarbageError errors after deploying fix #1919: