NewValidation: CSCwt53667 leaf stuck during upgrade due to presence of nxos image in ACI mode.

[rg5147]

(use upvote 👍 for attentions)

Validation Type

[ ] - Fault

[ ] - Config

[ X] - Bug

[ ] - Other

What needs to be validated

Post upgrade of switches we could see some switches stay inactive
The error may point to cert issue but the cause is due to CSCwt53667

How to avoid this bug

Check if switch is having =<32Gb of RAM, the "Total" field should have this info.
moquery -c procMemUsage

proc.MemUsage

Modname : sup
Free : 19429272
PercUsedMemory : 40
Total : 32569728 --> 32GB
Used : 13140456
childAction :
dn : topology/pod-1/node-301/sys/procmem/memusage-sup
memAlert : normal
modTs : never
monPolDn : uni/fabric/monfab-default
rn : memusage-sup
status :

Check bootflash to see if the switch has any NXOS image.
Not sure if there is a remote command from the apic to execute below but we need to check the contents of the bootflash and then check for any NXOS image

/bootflash

Site2-Leaf301# ls -la
total 9584028
drwxrwxr-x 5 root admin 4096 Aug 25 16:52 .
drwxr-xr-x 54 root root 1360 Aug 25 15:09 ..
drwxr-xr-x 7 root root 4096 Aug 24 19:04 .rpmstore
-rw-rw-rw- 1 root root 135 Aug 25 15:01 .user_config_port_profile.json
-rw-rw-rw- 1 root root 746745 Aug 25 15:07 CpuUsage.Log
-rw-rw-rw- 1 root admin 2824506024 Aug 25 15:19 aci-n9000-dk9.16.1.5e-cs_64.bin
-rw-rw-rw- 1 root root 2684028420 Aug 24 22:33 aci-n9000-dk9.16.1.5e.bin
-rw-r--r-- 1 root root 2329054724 Aug 24 23:48 auto-s
-rwxrwxrwx 1 root root 255 Jan 29 2024 clean-config-log
-rw-rw-rw- 1 root root 2 Aug 25 15:06 diag_bootup
-rw-r--r-- 1 root root 54 Aug 25 16:55 disk_log.txt
drwx------ 2 root root 49152 Aug 24 23:37 lost+found
drwxr-xr-x 4 root root 4096 Mar 22 2021 lxc
-rw-r--r-- 1 root root 4031942 Aug 25 16:50 mem_log.txt
-rw-r--r-- 1 root root 221910 Aug 25 16:50 mts_buffer_log.log
-rw-r--r-- 1 root root 1971323904 Mar 22 2021 nxos.9.3.5.bin --->
-rw-r--r-- 1 root admin 156 Aug 25 15:19 upgTimeline.txt
-rw-rw-rw- 1 root root 135 Aug 24 19:18 user_config_port_profile.json

Action: Delete the NXOS image manually on the flagged switch.

Why it needs to be validated

Describe why it needs to be validated and what may happen if it's not validated.

Additional context

Add any other context about the feature request here.