Upgrading Nexus 3548X-Cisco Nexus switches

I was working for a customer who needed to upgrade their Nexus switches, Yeah, you guessed it right! It was a nightmare after the network engineer had resigned and the consultant was hired to solve the issues. There were not supposed to be any issue, at least according to the customer’s description of the assignment. However, this was just a honeypot to get the consultant in. Not that I am complaining, hence I would never had been this specific experience richer.

So back to our beloved Nexus switches. Customer was planning to upgrade their Hypervisor environment, they had both Hyper-V and Vmware running. This was of course a couple of switches running vPC between them and were not stacked. vPC in my head is a bit like MLAG in Extreme switches. The diagram below is what is shown in Nexus documentation when you look up the definition of vPC, in case you are curious or are not familiar with the term virtual port channel. This makes the 2 switches seems as a single switch for the devices being connected to the switches in the vPC.

Cisco vpc nexus switches
Cisco vPC – source Cisco documentation

Some other vendors like Extreme which also are a favorite of mine use the term Mlag, which stand for multiple links aggregation group which sort of serves the same purpose. The diagram below depicts the scenarios

You can connect a switch or a server/host on the other end. In case of switch, you would utilize LACP or something similar while when using a server you can choose to team the Nics or just utilize LACP. This was just a degradation from the original topic, in case you were wondering about why vPC and what the heck is Mlag and why is this relevant in this post. The customer was running the scenario depicted in the picture above where, hosts were connected to two independent switches providing redundancy at all layers, and were utilizing LACP.

Coming back to the original topic, good was my initial reaction and thought, this would be an easy task, at least if I get a TFTP server in the management zone. After logging in, I checked the uptime of the switches in question… Ehmmmmm almost 1500 days, wow that is 4 years. So my question to the customer was, you have these central switches where almost all your production is running and these switches have not been restarted in over 4 years? Nope, the answer was very simple and may be somewhat typical. Everything was functioning so we did not want to disturb the Production. Thumbs up for Cisco, who make and produce these rock solid and stable switches, however they never intended or recommended to keep them running 24×7, 365 x 4 or 5 years. Neither does Cisco recommend not upgrading the firmware and especially over so many versions. You can just imagine how this story will fold out. If you are a network person, you might be feeling my pain already, if you are not, you still understand that systems running for years and outdated by many version does just not sound right, this applies both the operating systems and any device honestly speaking.

In this post we are going to talk about the process of upgrading our Nexus switches. So lets talk about the version and my wonderful journey through all major releases/updates. While performing the update or prior to performing the upgrade, I was on Google for quite some times, to see if I can find in a similar scenario. But alas, by luck or bad luck, however you consider it, I could not find a single post describing the entire process. That is when I decided to write this blogpost, if I successfully managed to upgrade this networks beasts. I guess, the upgrade process will be performed by experienced network engineers who do not need any guidance. But there is always a possibility that a case like this might surface after a resignation or unpredicted situation.

Software Version details

So our journey starts at version 6.0.2.A7.2, we will be upgrading the software to newest available version of Nexus OS i.e. NXOS 9.3.6. (at the time of writing)

The process starts by downloading all the required images. A big time saver at least for me was to download the images, verify file checksums and copying these to Usb drives that were connected directly to Nexus switches. There are a lot of good blog posts about the process or creating a the USB and other steps, but just for completeness I am including these as well. I started the process with formatting the USB using fat32 format. If you want to be 100% sure that the fomatting will function with Nexus switches you can format these on the switches, however, my personal experience is to just format on a normal windows 10 client. If you are reading this post and are trying to upgrade your nexus switches chances are you might not know about some of the issues that follow along with these switches.

The older versions of Cisco Nexus 3548 and Nexus 3548X did not have a lot of local storage. Or speaking frankly, storage was sufficient, but with time image size increased and this caused the local storage to become insufficient. This introduced the concept of compressed images. So, if you have one of these models you should download the compressed images. In the software download portal you will see these compact or compressed images, which is almost 1/3 of an uncompressed image size.

Nexus image downloads
Nexus image download – source cisco.com

The exact path you must take in order to upgrade your switches will depend on the current version you are running, however, Cisco has done a good job documenting this you can read more about the upgrade path here https://www.cisco.com/c/en/us/support/docs/ios-nx-os-software/nx-os-software/216492-upgrade-nexus-3524-and-3548-nx-os-softwa.html

Our upgrade path became

  • Started at version 6.0(2)A7(2) and upgraded to 6.0.2.A8.4a
  • Then upgraded to 6.0.2.A8.9
  • Then upgraded to 9.2.2
  • Then upgraded to 9.2.3
  • Then to 9.2.4
  • And finally to 9.3.6

There is not much pictures to share, just the commands. So I will just share the commands, so that you have these available. If you are running vPC, make sure to upgrade one and one switch to newer software, one switch following the other. Meaning, do not upgrade one from version 6.x to 9.x and then start on the other, instead upgrade one from 6.x to 6.x+1 and then upgrade the other switch to same version before proceeding with the next upgrade.

Take backup of the switches before proceeding, by using show run, then copy it to some place. Or just using your NMS, but then again, you would have not been on this blogpost reading how to upgrade Nexus switches from ancient versions to more current version software.

If the configuration is correct, the upgrade of one switch or the lacking of one switch should not cause any issues, as the partner should be able to completely handle all the traffic. However, if you have a standalone switch, you will cause downtime.

Upgrade Cisco Nexus to 6.0.2.A8.4a (Minor updates)

Make sure that you are connected either to console port or the management port, so that you do not loose any connectivity during the process. This is where there also is an advantage to have a Usb stick plugged into the switch.

The process is the same or almost the same for each release, I am skipping the step of checking checksum, as all was done before I started the process. It was done by using

show file bootflash:///filename.versionmajor.versionminor.versionminor.bin cksum
show file bootflash:///nxos.9.3.6.bin md5sum
show file usb1:n3500-uk9-kickstart.6.0.2.A8.4a.bin

The process will be copy the files locally, and then run the install command. On oldest version you will be using the kickstart files, while for the newer files you will be utilizing nxos.bin files.

copy usb1:n3500-uk9-kickstart.6.0.2.A8.4a.bin bootflash:
copy usb1:n3500-uk9.6.0.2.A8.4a.bin bootflash: 

install all kickstart bootflash:n3500-uk9-kickstart.6.0.2.A8.4a.bin system bootflash:n3500-uk9.6.0.2.A8.4a.bin

After running the commands for copying the image to local storage and running the install command, make sure to boot the switch. The first time, it will be rebooted automatically, but just to be on the safe side just reboot it.

Upgrade Cisco Nexus to 6.0.2.A8.9 (Minor updates)

The process will be same copy the files locally, and then run the install command. On oldest version you will be using the kickstart files, while for the newer files you will be utilizing nxos.bin files. After the install, delete old files.

copy usb1:n3500-uk9-kickstart.6.0.2.A8.9.bin bootflash:
copy usb1:n3500-uk9.6.0.2.A8.9.bin bootflash:  

install all kickstart bootflash:n3500-uk9-kickstart.6.0.2.A8.9.bin system bootflash:n3500-uk9.6.0.2.A8.9.bin

delete bootflash:/n3500-uk9-kickstart.6.0.2.A8.4a.bin
delete bootflash:/n3500-uk9.6.0.2.A8.4a.bin

After running the commands for copying the image to local storage and running the install command, make sure to boot the switch. The first time, it will be rebooted automatically, but just to be on the safe side just reboot it.

Upgrade Cisco Nexus to 9.2.2 (Major updates)

The process will be same copy the files locally, and then run the install command. You will be utilizing nxos.bin files. After the install, delete old files. Here is where you should use the compressed files, if you have not been able to get hold of a compressed file, you can compress the original file during download, however, it does require the image to be download by scp.

! If copying from SCP, and compressing the image  
copy scp://10.1.2.3/nxos.9.2.2.bin bootflash: compact vrf management
install 

! If just copying from USB
copy usb1:nxos.9.2.2.bin bootflash:


! Install the compact image file
install all nxos bootflash:nxos.9.2.2.bin

After running the commands for copying the image to local storage and running the install command, make sure to boot the switch. The first time, it will be rebooted automatically, but just to be on the safe side just reboot it.

Upgrade Cisco Nexus to 9.2.3 (Minor updates)

The process will be same copy the files locally, and then run the install command. You will be utilizing nxos.bin files. After the install, delete old files. Here is where you should use the compressed files, if you have not been able to get hold of a compressed file, you can compress the original file during download, however, it does require the image to be download by scp.

! Copy image locally
copy usb1:/compressedImages/nxos.9.2.3.bin bootflash:
 
!Check the command before running
show install all impact nxos bootflash:nxos.9.2.3.bin

!Install the new image
install all nxos bootflash:nxos.9.2.3.bin

! Delete old image after reboot to free up space 
delete nxos.9.2.2.bin

After running the commands for copying the image to local storage and running the install command, make sure to boot the switch. The first time, it will be rebooted automatically, but just to be on the safe side just reboot it. Delete the old image file to free up space.

Upgrade Cisco Nexus to 9.2.4 (Minor updates)

The process will be same copy the files locally, and then run the install command. You will be utilizing nxos.bin files. After the install, delete old files. Here is where you should use the compressed files, if you have not been able to get hold of a compressed file, you can compress the original file during download, however, it does require the image to be download by scp.

! Copy image locally
copy usb1:/pakket/nxos.9.2.4.bin bootflash:
 
!Check the command before running

show install all impact nxos bootflash:nxos.9.2.4.bin

!Install the new image
install all nxos bootflash:nxos.9.2.4.bin

! Delete old image after reboot to free up space 
delete nxos.9.2.3.bin

After running the commands for copying the image to local storage and running the install command, make sure to boot the switch. The first time, it will be rebooted automatically, but just to be on the safe side just reboot it. Delete the old image file to free up space.

Upgrade Cisco Nexus to 9.3.6 (Major updates)

The process will be same copy the files locally, and then run the install command. You will be utilizing nxos.bin files. After the install, delete old files. Here is where you should use the compressed files, if you have not been able to get hold of a compressed file, you can compress the original file during download, however, it does require the image to be download by scp.

! Copy image locally
copy usb1:/pakket/nxos.9.3.6.bin bootflash:
 
!Check the command before running
show install all impact nxos bootflash:nxos.9.3.6.bin

!Install the new image
install all nxos bootflash:nxos.9.3.6.bin

! Delete old image after reboot to free up space 
delete nxos.9.2.4.bin

After running the commands for copying the image to local storage and running the install command, make sure to boot the switch. The first time, it will be rebooted automatically, but just to be on the safe side just reboot it. Delete the old image file to free up space.

Additional resources

Other useful and handy resources you might want to study through before starting the upgrade are release notes. You can visit https://docs.paloaltonetworks.com/pan-os/9-1/pan-os-admin/monitoring/use-syslog-for-monitoring/syslog-field-descriptions/threat-log-fields.html

To read more about minor upgrades from version 6 to 6.x you can read https://www.cisco.com/c/en/us/support/docs/ios-nx-os-software/nx-os-software/216492-upgrade-nexus-3524-and-3548-nx-os-softwa.html#upgrade-6x-to-6x

And for general Nexus software upgrade instructions you can read https://www.cisco.com/c/en/us/td/docs/switches/datacenter/nexus3548/sw/92x/upgrade/guide/b-cisco-nexus-3500-software-upgrade-downgrade-guide-92x/b-cisco-nexus-3500-software-upgrade-downgrade-guide-92x_chapter_01.html

Leave a Reply

Your email address will not be published. Required fields are marked *