TestGoBGPLifecycle failed occasionally

[tnqn]

Describe the bug

See https://github.com/antrea-io/antrea/actions/runs/9754329776/job/26921085868

=== RUN   TestGoBGPLifecycle
    gobgp_test.go:65: Starting all BGP servers
    gobgp_test.go:69: Started all BGP servers
    gobgp_test.go:108: Adding BGP peers for BGP server1
I0702 03:08:49.328791   11716 logs.go:48] "Add a peer configuration" Key="127.0.0.1" Topic="Peer"
I0702 03:08:49.328915   11716 logs.go:48] "Add a peer configuration" Key="::1" Topic="Peer"
    gobgp_test.go:111: Added BGP peers for BGP server1
    gobgp_test.go:113: Adding BGP peers for BGP server2
I0702 03:08:49.329028   11716 logs.go:48] "Add a peer configuration" Key="127.0.0.1" Topic="Peer"
    gobgp_test.go:115: Added BGP peers for BGP server2
    gobgp_test.go:117: Adding BGP peers for BGP server3
I0702 03:08:49.329170   11716 logs.go:48] "Add a peer configuration" Key="::1" Topic="Peer"
    gobgp_test.go:119: Added BGP peers for BGP server3
    gobgp_test.go:136: Getting peers of BGP server1 and verifying them
W0702 03:08:49.355965   11716 interface_configuration_linux.go:327] Failed to send gratuitous ARP #2: no such device or address
I0702 03:08:56.329568   11716 logs.go:48] "Peer Up" Key="::1" State="BGP_FSM_OPENCONFIRM" Topic="Peer"
I0702 03:08:56.329584   11716 logs.go:48] "Peer Up" Key="::1" State="BGP_FSM_OPENCONFIRM" Topic="Peer"
I0702 03:08:57.330204   11716 logs.go:44] "Closed an accepted connection" Key="127.0.0.1" State="BGP_FSM_OPENSENT" Topic="Peer"
I0702 03:08:57.330214   11716 logs.go:44] "Closed an accepted connection" Key="127.0.0.1" State="BGP_FSM_OPENSENT" Topic="Peer"
I0702 03:09:08.331918   11716 logs.go:44] "Closed an accepted connection" Key="127.0.0.1" State="BGP_FSM_OPENSENT" Topic="Peer"
I0702 03:09:08.331950   11716 logs.go:44] "Closed an accepted connection" Key="127.0.0.1" State="BGP_FSM_OPENSENT" Topic="Peer"
    gobgp_test.go:137: 
        	Error Trace:	/usr/src/antrea.io/antrea/test/integration/agent/gobgp_test.go:140
        	            				/usr/local/go/src/runtime/asm_amd64.s:1650
        	Error:      	Not equal: 
        	            	expected: sets.Set[string]{"127.0.0.1-62179":sets.Empty{}, "::1-63179":sets.Empty{}}
        	            	actual  : sets.Set[string]{"::1-63179":sets.Empty{}}
        	            	
        	            	Diff:
        	            	--- Expected
        	            	+++ Actual
        	            	@@ -1,4 +1,2 @@
        	            	-(sets.Set[string]) (len=2) {
        	            	- (string) (len=15) "127.0.0.1-62179": (sets.Empty) {
        	            	- },
        	            	+(sets.Set[string]) (len=1) {
        	            	  (string) (len=9) "::1-63179": (sets.Empty) {
    gobgp_test.go:137: 
        	Error Trace:	/usr/src/antrea.io/antrea/test/integration/agent/gobgp_test.go:137
        	Error:      	Condition never satisfied
        	Test:       	TestGoBGPLifecycle
    gobgp_test.go:142: Got peers of BGP server1 and verified them
    gobgp_test.go:144: Getting peers of BGP server2 and verifying them

[hongliangl]

Will resolve it.