Round-robin updateMember may cause the timestamp to fallback

[JmPotato]

Bug Report

We use a round-robin way to request the TSO server with FindGroupByKeyspaceID gRPC call.

pd/client/tso_service_discovery.go

Lines 418 to 441 in b176a0a

	func (c *tsoServiceDiscovery) updateMember() error {
	// The keyspace membership or the primary serving address of the keyspace group, to which this
	// keyspace belongs, might have been changed. We need to query tso servers to get the latest info.
	tsoServerAddr, err := c.getTSOServer(c.apiSvcDiscovery)
	if err != nil {
	log.Error("[tso] failed to get tso server", errs.ZapError(err))
	return err
	}
	keyspaceID := c.GetKeyspaceID()
	var keyspaceGroup *tsopb.KeyspaceGroup
	if len(tsoServerAddr) > 0 {
	keyspaceGroup, err = c.findGroupByKeyspaceID(keyspaceID, tsoServerAddr, updateMemberTimeout)
	if err != nil {
	if c.tsoServerDiscovery.countFailure() {
	log.Error("[tso] failed to find the keyspace group",
	zap.Uint32("keyspace-id-in-request", keyspaceID),
	zap.String("tso-server-addr", tsoServerAddr),
	errs.ZapError(err))
	}
	return err
	}
	c.tsoServerDiscovery.resetFailure()
	} else {

However, if the requested TSO server satisfies this kind of case, it may cause the timestamp to fallback.

• TSO node A initialized group 1.
• TSO node B has not initialized group 1 yet.

If we do updateMember quickly while requesting the TSO, it may cause the keyspace group in the request to change between the default group and group 1, resulting in the timestamp fallback.

[JmPotato]

After some investigation, I found the following minimum reproduction steps, presume we have a keyspace 1 belonging to group 1.

1. TSO node A initializes group 1 and is elected as the primary of group 1.
2. Keyspace 1 client executes updateMember and finds that the TSO node A is the primary of group 1.
3. As the primary of group 1, TSO node A responds successfully to the TSO request of the keyspace 1 client at least once.
4. The gRPC stream between keyspace 1 client and TSO node A occurs an error and is destroyed.
5. TSO node B has not initialized group 1 yet at this time.
6. Keyspace 1 client executes updateMember and finds that the TSO node B doesn't have any info about keyspace 1, so the default group is chosen as the served group.
7. As the primary of the default group, TSO node B responds successfully to the TSO request of the keyspace 1 client at least once.
8. If the TSO of the default group is less than group 1, panic will be triggered.

The critical point here is the order in which steps 4 and 5 occur, that is, the reconstruction of the stream needs to be triggered before the TSO node of the next polling in round-robin does not initialize this group at this time.

[JmPotato]

Because the trigger conditions are complicated, we consider not fixing this problem for the time being. However, due to the lack of some constraints in the current client-side group switching mechanism, it is still worthwhile to introduce some protection mechanisms to limit the occurrence of this situation.

[nolouch]

https://github.com/tikv/pd/actions/runs/6609371269/job/17949420257?pr=7243 ref
panic.txt

[lhy1024]

https://github.com/tikv/pd/actions/runs/6649398798/job/18069320913?pr=7233 is it the same?

[lhy1024]

https://github.com/tikv/pd/actions/runs/6779422232/job/18427043934?pr=7278 is it the same?

[CabinfeverB]

also see https://github.com/tikv/pd/actions/runs/8107182719/job/22158279341?pr=7862#step:5:38038