[Remote Store] Translog fetch logic during replica promotion to primary & other cases

[ashking94]

Is your feature request related to a problem? Please describe.
As part of Remote store work, the translogs would be stored in remote store in the following path - Base64UUID/IndexUUID/ShardID/PrimaryTerm. With segment replication, there is a possibility that the replicas can be lagging in terms of the segment generation. Now, given there is no replicas available locally due to storing translogs in the replicas, there is also a need to fetch the translogs given any sequence number. There will also be cases where stale isolated primaries can persist the translog in remote store, but the call would have been unacknowledged or failed during the primary term validation. The discussion to avoid the divergent writes and any overwrite of data is being discussed in #3706.

Describe the solution you'd like
We can determine the min sequence number either by going through the .tlog / .ckp files or by keeping another file per primary term which holds the min sequence number. Using the min sequence number per primary term we can construct a map which can be used for creating the abstraction for fetching the translogs from a certain sequence number.

primary term	min
0	0
1	102
2	207
3	309
4	441

Using the above information, we can create a reverse map and use the same to find the translogs that needs to be fetched. This way we can avoid reading the unacknowledged/failed write requests.

Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.

Additional context
The same logic can as well be used for translog pruning (#3766 ) on lower levels once we have decided upto what sequence number should the translogs be purged.

[gbbafna]

Using the above information, we can create a reverse map and use the same to find the translogs that needs to be fetched. This way we can avoid reading the unacknowledged/failed write requests.

I think reading the unacknowledged/failed write requests is fine. But we should not lose the acknowledged writes.

[ashking94]

Using the above information, we can create a reverse map and use the same to find the translogs that needs to be fetched. This way we can avoid reading the unacknowledged/failed write requests.

I think reading the unacknowledged/failed write requests is fine. But we should not lose the acknowledged writes.

That's correct. We, however, should make sure that the sequencing happens correctly not leading to inconsistent state.