Duplicate packets - BGP EVPN MAC not inserted into fdb #14944

modzilla99 · 2023-12-05T10:12:52Z

modzilla99
Dec 5, 2023

Hey there,

I am working on an EVPN multihoming setup. The Hardware is connected with a LACP to 2 Cumulus switches that advertise the correct EVPN Routes into the fabric. On the Hypervisors the routes are received, but on one of them the MAC address of the baremetal host is not being set on the vxlan bridge member, so traffic is being flooded to both VTEPs, resulting in duplicated packets at the bmh.

Baremetal Host

root@bmh:~# ip -d a sh bond0
9: bond0: <BROADCAST,MULTICAST,MASTER,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 56:85:e6:1b:15:ca brd ff:ff:ff:ff:ff:ff promiscuity 0 
    bond mode 802.3ad miimon 100 updelay 0 downdelay 0 use_carrier 1 arp_interval 0 arp_validate none arp_all_targets any primary_reselect always fail_over_mac none xmit_hash_policy layer2 resend_igmp 1 num_grat_arp 1 all_slaves_active 0 min_links 0 lp_interval 1 packets_per_slave 1 lacp_rate slow ad_select stable ad_aggregator 1 ad_num_ports 2 ad_actor_key 15 ad_partner_key 15 ad_partner_mac 44:38:39:be:ef:aa ad_actor_sys_prio 65535 ad_user_port_key 0 ad_actor_system 00:00:00:00:00:00 tlb_dynamic_lb 1 numtxqueues 16 numrxqueues 16 gso_max_size 65536 gso_max_segs 65535 
    inet 10.1.2.35/24 scope global bond0
       valid_lft forever preferred_lft forever
root@bmh:~# ip nei sh 10.1.2.69
10.1.2.69 dev bond0 lladdr fa:16:3e:cc:6c:15 REACHABLE
root@bmh:~# ping 10.1.2.69                                     <--- hosted on HV1
PING 10.1.2.69 (10.1.2.69) 56(84) bytes of data.
64 bytes from 10.1.2.69: icmp_seq=1 ttl=64 time=3.21 ms
64 bytes from 10.1.2.69: icmp_seq=1 ttl=64 time=3.25 ms (DUP!)
64 bytes from 10.1.2.69: icmp_seq=2 ttl=64 time=1.40 ms
64 bytes from 10.1.2.69: icmp_seq=2 ttl=64 time=1.45 ms (DUP!)
^C
--- 10.1.2.69 ping statistics ---
2 packets transmitted, 2 received, +2 duplicates, 0% packet loss, time 1001ms
rtt min/avg/max/mdev = 1.405/2.330/3.252/0.903 ms
root@bmh:~# ip nei sh 10.1.2.34
10.1.2.34 dev bond0 lladdr 5a:6e:bd:67:4a:0d REACHABLE
root@bmh:~# ping 10.1.2.34                                    <--- hosted on HV2
PING 10.1.2.34 (10.1.2.34) 56(84) bytes of data.
64 bytes from 10.1.2.34: icmp_seq=1 ttl=64 time=0.209 ms
64 bytes from 10.1.2.34: icmp_seq=2 ttl=64 time=0.147 ms
^C
--- 10.1.2.34 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1026ms
rtt min/avg/max/mdev = 0.147/0.178/0.209/0.031 ms

Switches

leaf9# show evpn es
Type: B bypass, L local, R remote, N non-DF
ESI                            Type ES-IF                 VTEPs
03:44:38:39:be:ef:aa:00:00:01  LR   bond1                 10.77.1.10        
leaf9# show evpn mac vni 1111
Number of MACs (local and remote) known for this VNI: 4
Flags: B=bypass N=sync-neighs, I=local-inactive, P=peer-active, X=peer-proxy
MAC               Type   Flags Intf/Remote ES/VTEP            VLAN  Seq #'s
fa:17:9b:6a:58:17 remote       10.77.3.5                            0/0
fa:16:3e:cc:6c:15 remote       10.77.3.5                            0/0
5a:6e:bd:67:4a:0d remote       10.77.3.6                            0/0
56:85:e6:1b:15:ca local  X     bond1                          1111  0/0

leaf10# show evpn es
Type: B bypass, L local, R remote, N non-DF
ESI                            Type ES-IF                 VTEPs
03:44:38:39:be:ef:aa:00:00:01  LRN  bond1                 10.77.1.9
leaf10# show evpn mac vni 1111
Number of MACs (local and remote) known for this VNI: 4
Flags: B=bypass N=sync-neighs, I=local-inactive, P=peer-active, X=peer-proxy
MAC               Type   Flags Intf/Remote ES/VTEP            VLAN  Seq #'s
fa:17:9b:6a:58:17 remote       10.77.3.5                            0/0
fa:16:3e:cc:6c:15 remote       10.77.3.5                            0/0
5a:6e:bd:67:4a:0d remote       10.77.3.6                            0/0
56:85:e6:1b:15:ca local  PI    bond1                          1111  0/0

HVs

Same versions on both hvs:

root@hv1:~# lsb_release -a
No LSB modules are available.
Distributor ID:	Ubuntu
Description:	Ubuntu 20.04.6 LTS
Release:	20.04
Codename:	focal
root@hv1:~# dpkg -l | grep frr
ii  frr                                   9.1-0~ubuntu20.04.1                   amd64        FRRouting suite of internet protocols (BGP, OSPF, IS-IS, ...)
ii  frr-pythontools                       9.1-0~ubuntu20.04.1                   all          FRRouting suite - Python tools

hv1:~# bridge fdb show dev vxlan-1111 | sort
00:00:00:00:00:00 dst 10.77.1.10 self permanent
00:00:00:00:00:00 dst 10.77.1.9 self permanent
00:00:00:00:00:00 dst 10.77.3.6 self permanent
4a:eb:bb:ef:2a:15 master br-1111 permanent
4a:eb:bb:ef:2a:15 vlan 1 master br-1111 permanent
5a:6e:bd:67:4a:0d dst 10.77.3.6 self extern_learn 
5a:6e:bd:67:4a:0d extern_learn master br-1111 
5a:6e:bd:67:4a:0d vlan 1 extern_learn master br-1111

hv2:~# bridge fdb show dev vxlan-1111 | sort
00:00:00:00:00:00 dst 10.77.1.10 self permanent
00:00:00:00:00:00 dst 10.77.1.9 self permanent
00:00:00:00:00:00 dst 10.77.3.5 self permanent
56:85:e6:1b:15:ca dst 10.77.1.9 self                 <--- MAC of bmh missing on HV1
56:85:e6:1b:15:ca master br-1111 
5a:6e:bd:67:4a:0d master br-1111 permanent
5a:6e:bd:67:4a:0d vlan 1 master br-1111 permanent
fa:17:9b:6a:58:17 dst 10.77.3.5 self extern_learn 
fa:17:9b:6a:58:17 extern_learn master br-1111 
fa:17:9b:6a:58:17 vlan 1 extern_learn master br-1111

BGPD config (identical, except IP and AS):

router bgp 65273
 bgp router-id 10.77.3.5
 bgp bestpath as-path multipath-relax
 neighbor ens2f0np0 interface remote-as external
 neighbor ens2f1np1 interface remote-as external
 !
 address-family ipv4 unicast
  network 10.77.3.5/32
  redistribute connected route-map LOCAL_ROUTES
  neighbor ens2f0np0 filter-list HOST_ORIGINATED_ROUTES out
  neighbor ens2f1np1 filter-list HOST_ORIGINATED_ROUTES out
 exit-address-family
 !
 address-family l2vpn evpn
  neighbor ens2f0np0 activate
  neighbor ens2f1np1 activate
  advertise-all-vni
  advertise-svi-ip
 exit-address-family
exit

Debug

HV1

hv1# terminal monitor zebra 
hv1# do clear bgp l2vpn evpn *
2023-12-05 11:00:45.061 [DEBG] zebra: [JWJY6-CBZ1H] Recv MACIP DEL VNI 1111 MAC 56:85:e6:1b:15:ca Remote VTEP 10.77.1.9 from bgp
2023-12-05 11:00:45.061 [DEBG] zebra: [JCDA4-X4NXX] Recv VTEP DEL 10.77.1.9 VNI 1111 from bgp
2023-12-05 11:00:45.061 [DEBG] zebra: [JCDA4-X4NXX] Recv VTEP DEL 10.77.1.10 VNI 1111 from bgp
2023-12-05 11:00:45.061 [DEBG] zebra: [JWJY6-CBZ1H] Recv MACIP DEL VNI 1111 MAC 5a:6e:bd:67:4a:0d IP 10.1.2.34 Remote VTEP 10.77.3.6 from bgp
2023-12-05 11:00:45.061 [DEBG] zebra: [JWJY6-CBZ1H] Recv MACIP DEL VNI 1111 MAC 5a:6e:bd:67:4a:0d Remote VTEP 10.77.3.6 from bgp
2023-12-05 11:00:45.061 [DEBG] zebra: [JWJY6-CBZ1H] Recv MACIP DEL VNI 1111 MAC 5a:6e:bd:67:4a:0d IP fe80::586e:bdff:fe67:4a0d Remote VTEP 10.77.3.6 from bgp
2023-12-05 11:00:45.061 [DEBG] zebra: [JCDA4-X4NXX] Recv VTEP DEL 10.77.3.6 VNI 1111 from bgp
2023-12-05 11:00:45.071 [DEBG] zebra: [Q95WR-T7GTQ] es 03:44:38:39:be:ef:aa:00:00:01 free
2023-12-05 11:00:45.071 [DEBG] zebra: [VC5DR-N6AE8] Uninstall 10.77.1.9 from flood list for VNI 1111 intf vxlan-1111(1935)
2023-12-05 11:00:45.071 [DEBG] zebra: [VC5DR-N6AE8] Uninstall 10.77.1.10 from flood list for VNI 1111 intf vxlan-1111(1935)
2023-12-05 11:00:45.071 [DEBG] zebra: [JWQ3J-TKSAT] zebra_evpn_mac_del: MAC 5a:6e:bd:67:4a:0d flags AUTO 
2023-12-05 11:00:45.071 [DEBG] zebra: [VC5DR-N6AE8] Uninstall 10.77.3.6 from flood list for VNI 1111 intf vxlan-1111(1935)
2023-12-05 11:00:45.071 [DEBG] zebra: [HEW23-GKBKN] Del neighbor 10.1.2.34 intf br-1111(1934) -> L2-VNI 1111
2023-12-05 11:00:45.071 [DEBG] zebra: [HEW23-GKBKN] Del neighbor fe80::586e:bdff:fe67:4a0d intf br-1111(1934) -> L2-VNI 1111
2023-12-05 11:00:48.331 [DEBG] zebra: [QX4D7-WJPES] Recv VTEP ADD 10.77.1.9 VNI 1111 flood 0 from bgp
2023-12-05 11:00:48.331 [DEBG] zebra: [XAYAY-GEJ4Q] Recv MACIP ADD VNI 1111 MAC 56:85:e6:1b:15:ca flags 0x0 seq 0 VTEP 0.0.0.0 ESI 03:44:38:39:be:ef:aa:00:00:01 from bgp
2023-12-05 11:00:48.331 [DEBG] zebra: [QX4D7-WJPES] Recv VTEP ADD 10.77.1.9 VNI 1111 flood 0 from bgp
2023-12-05 11:00:48.331 [DEBG] zebra: [XAYAY-GEJ4Q] Recv MACIP ADD VNI 1111 MAC 56:85:e6:1b:15:ca flags 0x0 seq 0 VTEP 0.0.0.0 ESI 03:44:38:39:be:ef:aa:00:00:01 from bgp
2023-12-05 11:00:48.331 [DEBG] zebra: [QX4D7-WJPES] Recv VTEP ADD 10.77.1.10 VNI 1111 flood 0 from bgp
2023-12-05 11:00:48.331 [DEBG] zebra: [QX4D7-WJPES] Recv VTEP ADD 10.77.1.10 VNI 1111 flood 0 from bgp
2023-12-05 11:00:48.331 [DEBG] zebra: [QX4D7-WJPES] Recv VTEP ADD 10.77.3.6 VNI 1111 flood 0 from bgp
2023-12-05 11:00:48.331 [DEBG] zebra: [XAYAY-GEJ4Q] Recv MACIP ADD VNI 1111 MAC 5a:6e:bd:67:4a:0d flags 0x0 seq 0 VTEP 10.77.3.6 ESI - from bgp
2023-12-05 11:00:48.331 [DEBG] zebra: [XAYAY-GEJ4Q] Recv MACIP ADD VNI 1111 MAC 5a:6e:bd:67:4a:0d IP fe80::586e:bdff:fe67:4a0d flags 0x0 seq 0 VTEP 10.77.3.6 ESI - from bgp
2023-12-05 11:00:48.331 [DEBG] zebra: [XAYAY-GEJ4Q] Recv MACIP ADD VNI 1111 MAC 5a:6e:bd:67:4a:0d IP 10.1.2.34 flags 0x0 seq 0 VTEP 10.77.3.6 ESI - from bgp
2023-12-05 11:00:48.331 [DEBG] zebra: [QX4D7-WJPES] Recv VTEP ADD 10.77.3.6 VNI 1111 flood 0 from bgp
2023-12-05 11:00:48.331 [DEBG] zebra: [XAYAY-GEJ4Q] Recv MACIP ADD VNI 1111 MAC 5a:6e:bd:67:4a:0d flags 0x0 seq 0 VTEP 10.77.3.6 ESI - from bgp
2023-12-05 11:00:48.331 [DEBG] zebra: [XAYAY-GEJ4Q] Recv MACIP ADD VNI 1111 MAC 5a:6e:bd:67:4a:0d IP fe80::586e:bdff:fe67:4a0d flags 0x0 seq 0 VTEP 10.77.3.6 ESI - from bgp
2023-12-05 11:00:48.331 [DEBG] zebra: [XAYAY-GEJ4Q] Recv MACIP ADD VNI 1111 MAC 5a:6e:bd:67:4a:0d IP 10.1.2.34 flags 0x0 seq 0 VTEP 10.77.3.6 ESI - from bgp
2023-12-05 11:00:48.341 [DEBG] zebra: [XC8P3-66E56] Install 10.77.1.9 into flood list for VNI 1111 intf vxlan-1111(1935)
2023-12-05 11:00:48.341 [DEBG] zebra: [RYKQB-86J68] es 03:44:38:39:be:ef:aa:00:00:01 nhg 536870913 new
2023-12-05 11:00:48.341 [DEBG] zebra: [VPXR4-TZS7D] auto es 03:44:38:39:be:ef:aa:00:00:01 add on mac ref
2023-12-05 11:00:48.341 [DEBG] zebra: [S7Q3Q-N2C38] Processing neighbors on remote MAC 56:85:e6:1b:15:ca ADD, VNI 1111
2023-12-05 11:00:48.341 [DEBG] zebra: [XC8P3-66E56] Install 10.77.1.10 into flood list for VNI 1111 intf vxlan-1111(1935)
2023-12-05 11:00:48.341 [DEBG] zebra: [XC8P3-66E56] Install 10.77.3.6 into flood list for VNI 1111 intf vxlan-1111(1935)
2023-12-05 11:00:48.341 [DEBG] zebra: [JWQ3J-TKSAT] zebra_evpn_mac_add: MAC 5a:6e:bd:67:4a:0d flags None 
2023-12-05 11:00:48.341 [DEBG] zebra: [S7Q3Q-N2C38] Processing neighbors on remote MAC 5a:6e:bd:67:4a:0d ADD, VNI 1111
2023-12-05 11:00:48.341 [DEBG] zebra: [QEDXC-E5122] dpAdd remote MAC 5a:6e:bd:67:4a:0d VID 1

Thanks!

Answered by modzilla99

Dec 6, 2023

Dammit the leafs had an old FRR version that wouldn't share the Type 1 Routes with the HVs. Updating to a newer FRR version fixed it.

View full answer

modzilla99 · 2023-12-05T12:25:57Z

modzilla99
Dec 5, 2023
Author

Rookie mistake. I forgot to disable learning on the vxlan interface on HV2. That's why the entry's inserted without the extern_learn flag.

But the question still remains, why does FRR not add the MAC address of the multihomed node? I do receive and accept the Type 2 Route of the leafs. Or is it supposed to work that way? Should the switch that's not the DF just drop the packets?

0 replies

modzilla99 · 2023-12-06T08:12:48Z

modzilla99
Dec 6, 2023
Author

Dammit the leafs had an old FRR version that wouldn't share the Type 1 Routes with the HVs. Updating to a newer FRR version fixed it.

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Duplicate packets - BGP EVPN MAC not inserted into fdb #14944

{{title}}

Replies: 2 comments

{{title}}

{{title}}

Select a reply

Duplicate packets - BGP EVPN MAC not inserted into fdb #14944

modzilla99 Dec 5, 2023

Baremetal Host

Switches

HVs

Debug

HV1

Replies: 2 comments

modzilla99 Dec 5, 2023 Author

modzilla99 Dec 6, 2023 Author

modzilla99
Dec 5, 2023

modzilla99
Dec 5, 2023
Author

modzilla99
Dec 6, 2023
Author