NewValidation:NXOS Bug CSCwt59437: Overlay multicast traffic (native BUM) between PODS can get dropped

[rg5147]

(use upvote 👍 for attentions)

Validation Type

[ ] - Fault

[ ] - Config

[ X] - Bug

[ ] - Other

What needs to be validated

CSCwt59437: Overlay multicast traffic (native BUM) between PODS can get dropped
This is a NXOS bug but since it gets triggered when spines are rebooted (or upgraded) or changes in the IPN physical layer, it is important for us to highlight this issue. The issue takes a while to troubleshoot and it manifest as intermittent connectivity because of ARP aging and it could autofix based on how the ARP learning happens on the host.

The steps to validate the IPN version is below and it affects N9K running
version >=10.5(2) and < 10.5(6)

Get spine node IDs

Site1-apic1# moquery -c fabricNode | egrep -B 14 "role : spine" | grep ^dn
dn : topology/pod-1/node-201
dn : topology/pod-2/node-202

Check if this is a mpod setup where spines are different pods

if all spines are in the same pod then this validation will not apply

If multipod get interfaces facing the IPN using OSPF adjacency looksups

Site1-apic1# moquery -c ospfAdjEp | grep ^dn
dn : topology/pod-2/node-202/sys/ospf/inst-default/dom-overlay-1/if-[eth1/31.31]/adj-22.22.22.21
dn : topology/pod-2/node-202/sys/ospf/inst-default/dom-overlay-1/if-[eth1/32.32]/adj-22.22.22.21
dn : topology/pod-1/node-201/sys/ospf/inst-default/dom-overlay-1/if-[eth1/31.31]/adj-22.22.22.21
dn : topology/pod-1/node-201/sys/ospf/inst-default/dom-overlay-1/if-[eth1/32.32]/adj-22.22.22.21

Check lldp adjacency for these interfaces

sysDesc field mentions "Cisco Nexus Operating System (NX-OS) Software", once confirmed that we have a NXOS IPN, extract the version info example 10.3(1)

moquery -c lldpAdjEp

lldp.AdjEp
id : 1
capability : bridge,router
chassisIdT : mac
chassisIdV : 4c:5d:3c:cd:35:8f
childAction :
dn : sys/lldp/inst/if-[eth1/32]/adj-1
enCap : bridge,router
mgmtId : 83886080
mgmtIp : 10.122.186.53
mgmtPortMac : 4C:5D:3C:CD:35:8F
modTs : 2026-09-14T14:01:31.991-04:00
monPolDn : uni/fabric/monfab-default
name :
portDesc : Ethernet1/56
portIdT : if-name
portIdV : Ethernet1/56
portVlan : unspecified
rn : adj-1
stQual :
status :
sysDesc : Cisco Nexus Operating System (NX-OS) Software 10.3(1) TAC support: http://www.cisco.com/tac Copyright (c) 2002-2022, Cisco Systems, Inc. All rights reserved.
sysName : Site-1-FX2-ISN
ttl : 120

Also check cdp adjacency for these interfaces.

platId has the platform info. Look for PID matching N9K to confirm a nexus PID
"ver" field mentions "Cisco Nexus Operating System (NX-OS) Software" , once confirmed that we have a NXOS IPN, extract the version info example10.4(7)

cdpAdjEp
cap : router,stp-dispute,switch
childAction :
devId : SP-ACID-RTR-01.dev.boe.bankofengland.co.uk(FOX2243PDA2)
dn : sys/cdp/inst/if-[eth1/30]/adj-1
duplex : full
index : 1
instanceId : 2632:87
modTs : never
monPolDn :
mtu : 9216
name :
nativeVlan : unspecified
parentInstId: 2637:88059
platId : N9K-C9504
portId : Ethernet1/1
stQual :
status :
sysLoc :
sysName : SP-ACID-RTR-01
sysObjIdL : 12
sysObjIdV : 1,3,6,1,4,1,9,12,3,1,3,1507,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0
ver : Cisco Nexus Operating System (NX-OS) Software, Version 10.4(7)

Why it needs to be validated

Overlay multicast traffic (native BUM) between PODS can get dropped

Additional context