Client occasionally crashes with index out of bounds when getting a node from a partition

[asecondo]

Occasionally when writing to Aerospike, a service crashes in the Aerospike client when attempting to get a node from the current partition. The length of the array is always 0 when the crash occurs.

The code here could be modified to ensure it's not getting exceeding the bounds of replicas, but I'm trying to understand why this is happening. Any thoughts?

Client version: 5.7.0

Example stack trace (excluding service specific code):

github.com/aerospike/aerospike-client-go/v5.(*Partition).getSequenceNode(0x46d5d3, 0x61f9685e)
	/go/pkg/mod/github.com/aerospike/aerospike-client-go/[email protected]/partition.go:195 +0xfc
github.com/aerospike/aerospike-client-go/v5.(*Partition).GetNodeWrite(0xc01b79eef8, 0x48c6d7)
	/go/pkg/mod/github.com/aerospike/aerospike-client-go/[email protected]/partition.go:165 +0x53
github.com/aerospike/aerospike-client-go/v5.(*operateCommand).getNode(0xc07677f782a295c3, {0x4e94324576a4, 0x1d0a1c0})
	/go/pkg/mod/github.com/aerospike/aerospike-client-go/[email protected]/operate_command.go:43 +0x2d
github.com/aerospike/aerospike-client-go/v5.(*baseCommand).executeAt(0xc03018ae10, {0x13db778, 0xc03018ae10}, 0xc001fabb90, 0x80, {0xc001fabb90, 0x0, 0x1d0a1c0}, 0x0, 0x0)
	/go/pkg/mod/github.com/aerospike/aerospike-client-go/[email protected]/command.go:2058 +0x532
github.com/aerospike/aerospike-client-go/v5.(*baseCommand).execute(0xc001fabb90, {0x13db778, 0xc03018ae10}, 0x4)
	/go/pkg/mod/github.com/aerospike/aerospike-client-go/[email protected]/command.go:1991 +0x8a
github.com/aerospike/aerospike-client-go/v5.(*operateCommand).Execute(...)
	/go/pkg/mod/github.com/aerospike/aerospike-client-go/[email protected]/operate_command.go:60
github.com/aerospike/aerospike-client-go/v5.(*Client).Operate(0xc0001fc840, 0x0, 0x0, {0xc01cf073e0, 0xfc4500, 0xc01cf13eb0})
	/go/pkg/mod/github.com/aerospike/aerospike-client-go/[email protected]/client.go:515 +0x3be

[khaf]

This is very surprising. Are you monitoring your cluster? Does a partition or major network issue happen at the same time as these errors?

[asecondo]

@khaf We are monitoring our clusters, but I wasn't able to find any direct indicators that a partition or major network issue happened during the time this error popped up. Do you have any suggestions for anything to specifically look for?

For a bit more context, this has only happened ~75 times in the past 30 days, and 1 time in the past 2 weeks. It's likely that this is related to some partition or network issue as you described since there's no clear pattern that I've been able to identify yet.