FRR: iBGP neighborship not forming with non-connected neighbor ip (loopback ip)

[Madhu-api]

FRRouting 9.2-dev-MyOwnFRRVersion-gf6d708bc8 on Linux(5.4.0-166-generic).

lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 20.04.6 LTS
Release: 20.04
Codename: focal

Two FRR routers (VMs) are connected in IBGP mode

lo:11 VM1 =(enp0s8)=================(enp0s8)=VM2 lo:22
11.11.11.11/32 192.168.62.222 192.168.62.220 22.22.22.1/32

VM1 and VM2 are in AS 100

VM1 is having BGP neighbor ip as VM2's loopback ip (i.e.,22.22.22.1)
VM2 is having BGP neighbor ip as VM1's loopback ip (i.e.,11.11.11.1)
VM2 is having a static route to reach VM2's loopback ip
VM2 is having a static route to reach VM1's loopback ip

neighbor ip's are reachable and pingable from each VM's

Problem is that iBGP neighborship is NOT forming (stays in 'Active' mode)

running-config from each router is attached below.
Please let me know if any config. is missing out and why iBGP neighborship is not establishing.

VM1

madhuVM3# show running-config
Building configuration...

Current configuration:
!
frr version 9.2-dev-MyOwnFRRVersion
frr defaults traditional
hostname madhuVM3
log syslog informational
no ipv6 forwarding
service integrated-vtysh-config
!
debug zebra events
!
ip route 22.22.22.1/32 192.168.62.220
!
router bgp 100
 bgp log-neighbor-changes
 neighbor 22.22.22.1 remote-as 100
 neighbor 22.22.22.1 update-source lo:11
 !
 address-family ipv4 unicast
  neighbor 22.22.22.1 soft-reconfiguration inbound
 exit-address-family
exit
!
end
madhuVM3#

VM2

madhuVM4# show running-config
Building configuration...

Current configuration:
!
frr version 9.2-dev-MyOwnFRRVersion
frr defaults traditional
hostname madhuVM4
log syslog informational
no ipv6 forwarding
service integrated-vtysh-config
!
ip route 11.11.11.1/32 192.168.62.222
!
router bgp 100
 bgp log-neighbor-changes
 neighbor 11.11.11.1 remote-as 100
 neighbor 11.11.11.1 update-source lo:22
 !
 address-family ipv4 unicast
  neighbor 11.11.11.1 soft-reconfiguration inbound
 exit-address-family
exit
!
end
madhuVM4#

Ping from VM1

madhuVM3# ping 22.22.22.1
PING 22.22.22.1 (22.22.22.1) 56(84) bytes of data.
64 bytes from 22.22.22.1: icmp_seq=1 ttl=64 time=0.607 ms
64 bytes from 22.22.22.1: icmp_seq=2 ttl=64 time=0.747 ms
64 bytes from 22.22.22.1: icmp_seq=3 ttl=64 time=0.767 ms
64 bytes from 22.22.22.1: icmp_seq=4 ttl=64 time=0.571 ms
64 bytes from 22.22.22.1: icmp_seq=5 ttl=64 time=0.690 ms
64 bytes from 22.22.22.1: icmp_seq=6 ttl=64 time=0.526 ms
64 bytes from 22.22.22.1: icmp_seq=7 ttl=64 time=0.489 ms
64 bytes from 22.22.22.1: icmp_seq=8 ttl=64 time=0.714 ms
^C
--- 22.22.22.1 ping statistics ---
8 packets transmitted, 8 received, 0% packet loss, time 7227ms
rtt min/avg/max/mdev = 0.489/0.638/0.767/0.098 ms
madhuVM3#

Ping from VM2

madhuVM4# ping 11.11.11.1
PING 11.11.11.1 (11.11.11.1) 56(84) bytes of data.
64 bytes from 11.11.11.1: icmp_seq=1 ttl=64 time=0.595 ms
64 bytes from 11.11.11.1: icmp_seq=2 ttl=64 time=0.648 ms
64 bytes from 11.11.11.1: icmp_seq=3 ttl=64 time=0.615 ms
64 bytes from 11.11.11.1: icmp_seq=4 ttl=64 time=0.610 ms
64 bytes from 11.11.11.1: icmp_seq=5 ttl=64 time=0.723 ms
64 bytes from 11.11.11.1: icmp_seq=6 ttl=64 time=0.655 ms
64 bytes from 11.11.11.1: icmp_seq=7 ttl=64 time=0.722 ms
64 bytes from 11.11.11.1: icmp_seq=8 ttl=64 time=0.551 ms
64 bytes from 11.11.11.1: icmp_seq=9 ttl=64 time=0.602 ms
64 bytes from 11.11.11.1: icmp_seq=10 ttl=64 time=0.713 ms
^C
--- 11.11.11.1 ping statistics ---
10 packets transmitted, 10 received, 0% packet loss, time 9225ms
rtt min/avg/max/mdev = 0.551/0.643/0.723/0.056 ms
madhuVM4#

neighbor status in VM1:

madhuVM3# show ip bgp summary

IPv4 Unicast Summary (VRF default):
BGP router identifier 11.11.11.1, local AS number 100 vrf-id 0
BGP table version 0
RIB entries 0, using 0 bytes of memory
Peers 1, using 20 KiB of memory

Neighbor        V         AS   MsgRcvd   MsgSent   TblVer  InQ OutQ  Up/Down State/PfxRcd   PfxSnt Desc
22.22.22.1      4        100         0         1        0    0    0    never       **Active**        0 N/A

Total number of neighbors 1
madhuVM3#

neighbor status in VM2:

madhuVM4# show ip bgp summary

IPv4 Unicast Summary (VRF default):
BGP router identifier 22.22.22.1, local AS number 100 vrf-id 0
BGP table version 0
RIB entries 0, using 0 bytes of memory
Peers 1, using 20 KiB of memory

Neighbor        V         AS   MsgRcvd   MsgSent   TblVer  InQ OutQ  Up/Down State/PfxRcd   PfxSnt Desc
11.11.11.1      4        100         0         0        0    0    0    never       **Active**        0 N/A

Total number of neighbors 1
madhuVM4#

Events

2023-11-08 13:11:04.151 [DEBG] bgpd: [ZWCSR-M7FG9] 22.22.22.1 [FSM] BGP_Stop (Active->Idle), fd -1
2023-11-08 13:11:04.152 [DEBG] bgpd: [T91AW-FGMHW] bgp_fsm_change_status : vrf default(0), Status: Idle established_peers 0
2023-11-08 13:11:04.152 [DEBG] bgpd: [HKWM3-ZC5QP] 22.22.22.1 fd -1 went from Active to Idle
2023-11-08 13:11:05.154 [DEBG] bgpd: [ZQTB5-H8522] 22.22.22.1 [FSM] Timer (start timer expire).
2023-11-08 13:11:05.154 [DEBG] bgpd: [ZWCSR-M7FG9] 22.22.22.1 [FSM] BGP_Start (Idle->Connect), fd -1
2023-11-08 13:11:05.154 [DEBG] bgpd: [VKA2P-VGGNT] 22.22.22.1 [FSM] Connect error
2023-11-08 13:11:05.154 [DEBG] bgpd: [T91AW-FGMHW] bgp_fsm_change_status : vrf default(0), Status: Connect established_peers 0
2023-11-08 13:11:05.154 [DEBG] bgpd: [HKWM3-ZC5QP] 22.22.22.1 fd 27 went from Idle to Connect
2023-11-08 13:11:05.154 [DEBG] bgpd: [ZWCSR-M7FG9] 22.22.22.1 [FSM] TCP_connection_open_failed (Connect->Active), fd 27
2023-11-08 13:11:05.155 [DEBG] bgpd: [T91AW-FGMHW] bgp_fsm_change_status : vrf default(0), Status: Active established_peers 0
2023-11-08 13:11:05.155 [DEBG] bgpd: [HKWM3-ZC5QP] 22.22.22.1 fd -1 went from Connect to Active

show interface brief

madhuVM3(config-router)# do show interface brief
Interface       Status  VRF             Addresses
---------       ------  ---             ---------
enp0s3          up      default         10.0.2.15/24
enp0s8          up      default         192.168.62.222/24
enp0s9          up      default         192.168.131.254/24
lo              up      default         11.11.11.1/0

madhuVM3(config-router)# do show interface lo
Interface lo is up, line protocol is up
  Link ups:       0    last: (never)
  Link downs:     0    last: (never)
  vrf: default
  index 1 metric 0 mtu 65536 speed 0 txqlen 1000
  flags: <UP,LOOPBACK,RUNNING>
  Type: Loopback
  inet 11.11.11.1/0 lo:11
  Interface Type Other
  Interface Slave Type None
  protodown: off
madhuVM3(config-router)#

Interface       Status  VRF             Addresses
---------       ------  ---             ---------
enp0s3          up      default         10.0.2.15/24
enp0s8          up      default         192.168.62.220/24
enp0s9          up      default         192.168.131.250/24
lo              up      default         22.22.22.1/0

madhuVM4# show interface lo
Interface lo is up, line protocol is up
  Link ups:       0    last: (never)
  Link downs:     0    last: (never)
  vrf: default
  index 1 metric 0 mtu 65536 speed 0 txqlen 1000
  flags: <UP,LOOPBACK,RUNNING>
  Type: Loopback
  inet 22.22.22.1/0 lo:22
  Interface Type Other
  Interface Slave Type None
  protodown: off
madhuVM4#

FRR version

madhuVM3# show version
FRRouting 9.2-dev-MyOwnFRRVersion-gf6d708bc8 (madhuVM3) on Linux(5.4.0-166-generic).
Copyright 1996-2005 Kunihiro Ishiguro, et al.
This is a git build of frr-9.2-dev-74-gf6d708bc8
Associated branch(es):
        local:master
        github/frrouting/frr.git/master

configured with:
    '--prefix=/usr' '--includedir=${prefix}/include' '--bindir=${prefix}/bin' '--sbindir=${prefix}/lib/frr' '--libdir=${prefix}/lib/frr' '--libexecdir=${prefix}/lib/frr' '--localstatedir=/var/run/frr' '--sysconfdir=/etc/frr' '--with-moduledir=${prefix}/lib/frr/modules' '--enable-configfile-mask=0640' '--enable-logfile-mask=0640' '--enable-snmp=agentx' '--enable-multipath=64' '--enable-user=frr' '--enable-group=frr' '--enable-vty-group=frrvty' '--with-pkg-git-version' '--with-pkg-extra-version=-MyOwnFRRVersion'
madhuVM3#

Note: same config. with eBGP and disable connected-check is also not working.
depending on this issue's triage, I will raise a new issue if required.

[ton31337]

Please show the output of show ip route, show ip nht.

[Madhu-api]

VM1

madhuVM3# show ip route
Codes: K - kernel route, C - connected, S - static, R - RIP,
       O - OSPF, I - IS-IS, B - BGP, E - EIGRP, N - NHRP,
       T - Table, v - VNC, V - VNC-Direct, A - Babel, F - PBR,
       f - OpenFabric, t - Table-Direct,
       > - selected route, * - FIB route, q - queued, r - rejected, b - backup
       t - trapped, o - offload failure

K>* 0.0.0.0/0 [0/0] via 10.0.2.2, enp0s3, 00:29:26
C>* 10.0.2.0/24 is directly connected, enp0s3, 02:51:46
K>* 10.0.2.2/32 [0/100] is directly connected, enp0s3, 02:51:46
S>* 22.22.22.1/32 [1/0] via 192.168.62.220, enp0s8, weight 1, 00:53:10
C>* 192.168.62.0/24 is directly connected, enp0s8, 02:51:46
C>* 192.168.131.0/24 is directly connected, enp0s9, 02:51:46
madhuVM3#
madhuVM3#
madhuVM3# show ip nht
VRF default:
 Resolve via default: on
22.22.22.1
 resolved via static
 via 192.168.62.220, enp0s8 (vrf default), weight 1
 Client list: bgp(fd 17)
192.168.62.220
 resolved via connected
 is directly connected, enp0s8 (vrf default)
 Client list: static(fd 43)
madhuVM3#

VM2

madhuVM4# show ip route
Codes: K - kernel route, C - connected, S - static, R - RIP,
       O - OSPF, I - IS-IS, B - BGP, E - EIGRP, N - NHRP,
       T - Table, v - VNC, V - VNC-Direct, A - Babel, F - PBR,
       f - OpenFabric, t - Table-Direct,
       > - selected route, * - FIB route, q - queued, r - rejected, b - backup
       t - trapped, o - offload failure

K>* 0.0.0.0/0 [0/0] via 10.0.2.2, enp0s3, 00:30:26
C>* 10.0.2.0/24 is directly connected, enp0s3, 02:52:51
K>* 10.0.2.2/32 [0/100] is directly connected, enp0s3, 02:52:51
S>* 11.11.11.1/32 [1/0] via 192.168.62.222, enp0s8, weight 1, 00:54:41
C>* 192.168.62.0/24 is directly connected, enp0s8, 02:52:51
C>* 192.168.131.0/24 is directly connected, enp0s9, 02:52:51
madhuVM4#
madhuVM4#
madhuVM4# show ip nht
VRF default:
 Resolve via default: on
11.11.11.1
 resolved via static
 via 192.168.62.222, enp0s8 (vrf default), weight 1
 Client list: bgp(fd 17)
192.168.62.222
 resolved via connected
 is directly connected, enp0s8 (vrf default)
 Client list: static(fd 43)
madhuVM4#

[ton31337]

Also show the output of show bgp neighbors.

[Madhu-api]

VM1

madhuVM3# show ip bgp neighbors
BGP neighbor is 22.22.22.1, remote AS 100, local AS 100, internal link
  Local Role: undefined
  Remote Role: undefined
  BGP version 4, remote router ID 0.0.0.0, local router ID 11.11.11.1
  BGP state = Active
  Last read 00:17:36, Last write never
  Hold time is 180 seconds, keepalive interval is 60 seconds
  Configured hold time is 180 seconds, keepalive interval is 60 seconds
  Configured tcp-mss is 0, synced tcp-mss is 0
  Configured conditional advertisements interval is 60 seconds
  Graceful restart information:
    Local GR Mode: Helper*

    Remote GR Mode: NotApplicable

    R bit: False
    N bit: False
    Timers:
      Configured Restart Time(sec): 120
      Received Restart Time(sec): 0
      Configured LLGR Stale Path Time(sec): 0
  Message statistics:
    Inq depth is 0
    Outq depth is 0
                         Sent       Rcvd
    Opens:                  0          0
    Notifications:          0          0
    Updates:                0          0
    Keepalives:             0          0
    Route Refresh:          0          0
    Capability:             0          0
    Total:                  0          0
  Minimum time between advertisement runs is 0 seconds
  Update source is lo:11

 For address family: IPv4 Unicast
  Not part of any update group
  Inbound soft reconfiguration allowed
  Community attribute sent to this neighbor(all)
  0 accepted prefixes

  Connections established 0; dropped 0
  Last reset 00:17:36,   Socket Error (n/a)
  Internal BGP neighbor may be up to 255 hops away.
BGP Connect Retry Timer in Seconds: 120
Next connect timer due in 24 seconds
Read thread: off  Write thread: off  FD used: -1


madhuVM3#

VM2

madhuVM4# show ip bgp neighbors
BGP neighbor is 11.11.11.1, remote AS 100, local AS 100, internal link
  Local Role: undefined
  Remote Role: undefined
  BGP version 4, remote router ID 0.0.0.0, local router ID 22.22.22.1
  BGP state = Active
  Last read 00:18:31, Last write never
  Hold time is 180 seconds, keepalive interval is 60 seconds
  Configured hold time is 180 seconds, keepalive interval is 60 seconds
  Configured tcp-mss is 0, synced tcp-mss is 0
  Configured conditional advertisements interval is 60 seconds
  Graceful restart information:
    Local GR Mode: Helper*

    Remote GR Mode: NotApplicable

    R bit: False
    N bit: False
    Timers:
      Configured Restart Time(sec): 120
      Received Restart Time(sec): 0
      Configured LLGR Stale Path Time(sec): 0
  Message statistics:
    Inq depth is 0
    Outq depth is 0
                         Sent       Rcvd
    Opens:                  0          0
    Notifications:          0          0
    Updates:                0          0
    Keepalives:             0          0
    Route Refresh:          0          0
    Capability:             0          0
    Total:                  0          0
  Minimum time between advertisement runs is 0 seconds
  Update source is lo:22

 For address family: IPv4 Unicast
  Not part of any update group
  Inbound soft reconfiguration allowed
  Community attribute sent to this neighbor(all)
  0 accepted prefixes

  Connections established 0; dropped 0
  Last reset 00:18:31,   Socket Error (n/a)
  Internal BGP neighbor may be up to 255 hops away.
BGP Connect Retry Timer in Seconds: 120
Next connect timer due in 89 seconds
Read thread: off  Write thread: off  FD used: -1


madhuVM4#

[ton31337]

Socket errors means most likely you specified the wrong update source in neighbor 22.22.22.1 update-source lo:11. Try using an explicit IP address.

[Madhu-api]

after changing update-source to ip address instead of interface name, neighborship came up

5e520444ff6f# show running-config
Building configuration...

Current configuration:
!
frr version 9.1-dev-my-manual-build
frr defaults traditional
hostname 5e520444ff6f
no ipv6 forwarding
!
ip route 11.11.11.0/24 172.18.0.2
!
router bgp 1122
neighbor 11.11.11.1 remote-as 1122
neighbor 11.11.11.1 update-source 22.22.22.1
!
address-family ipv4 unicast
neighbor 11.11.11.1 soft-reconfiguration inbound
neighbor 11.11.11.1 route-map peer-routes in
exit-address-family
exit
!
route-map peer-routes permit 1
exit
!
end
5e520444ff6f# show ip bgp summary

IPv4 Unicast Summary (VRF default):
BGP router identifier 22.22.22.1, local AS number 1122 vrf-id 0
BGP table version 0
RIB entries 0, using 0 bytes of memory
Peers 1, using 20 KiB of memory

Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd PfxSnt Desc
11.11.11.1 4 1122 3 3 0 0 0 00:00:27 0 0 FRRouting/9.1-dev-my

Total number of neighbors 1
5e520444ff6f#

b5430a8673c2# show running-config
Building configuration...

Current configuration:
!
frr version 9.1-dev-my-manual-build
frr defaults traditional
hostname b5430a8673c2
no ipv6 forwarding
!
ip route 22.22.22.0/24 172.18.0.3
!
router bgp 1122
neighbor 22.22.22.1 remote-as 1122
neighbor 22.22.22.1 update-source 11.11.11.1
!
address-family ipv4 unicast
neighbor 22.22.22.1 soft-reconfiguration inbound
neighbor 22.22.22.1 route-map peer-routes in
exit-address-family
exit
!
route-map peer-routes permit 1
exit
!
end
b5430a8673c2# show ip bgp sum

IPv4 Unicast Summary (VRF default):
BGP router identifier 11.11.11.1, local AS number 1122 vrf-id 0
BGP table version 0
RIB entries 0, using 0 bytes of memory
Peers 1, using 20 KiB of memory

Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd PfxSnt Desc
22.22.22.1 4 1122 3 3 0 0 0 00:00:22 0 0 FRRouting/9.1-dev-my

Total number of neighbors 1
b5430a8673c2#