After reading a lot of horror stories of people on-line who tried to upgrade the firmware on the BladeSystem and had to roll back (due to not following the right upgrade procedure) and experienced hours of outages while trying to get the issue resolved by HP, I decided to write my own post on how to do it the right way.
Normally firmware updates are pretty straightforward however when dealing with Blades and especially the HP BladeSystem c7000 enclosure, there's a few things to check prior to doing any upgrades hence why I am writing this post as I have done all the reading, preparation work and the actual upgrade itself to save you a lot of hassle!
From time to time HP revises their line of servers and Blades are no different. While the HP BladeSystem c7000 chassis itself hasn't changed very much over the last decade, all the modular components that connect to it (think of all the removable modules, etc) are revised from time to time. Currently Blades are at Gen9. If you have free slots in the front of your chassis or the rear and you decide to add more components (Blades, Storage Blades, Switches, etc) and they are newer revisions, you will most likely need to upgrade your firmware if you aren't already performing routine quarterly updates.
Unlike other Blade solutions from the other vendors that have simple passthrough Fibre Channel uplinks, these Blade Enclosures come with HP VirtualConnect modules which are effectively switches and serve as an additional layer. Therefore upgrades on the chassis need to be done in a specific order and to specific firmware versions to make sure all the components in the chassis are compatible.
There are a few different variants of the Virtual Connect modules, this is one of them. Each Blade chassis has two for redundancy. The're also extremely expensive, (£15,000 last time I checked!) This is the specific model I worked with:
HP VC FlexFabric 10Gb/24-Port Module
There is a firmware compatibility matrix provided by HP:
This will tell you what the minimum and maximum firmware versions are required for each component to be compatible and work, for example mixing different Gen Blades.
The next thing you need to be aware is of the order to upgrade the firmware on the chassis and also to forget trying to update anything without downtime as even with properly designed redundancy, things can still go wrong and cause disruption/outages. If you have redundant enclosures, then migrate all your running vm's to the other chassis otherwise you'll have to shut the whole thing down to be safe.
This is the best order to upgrade your enclosure and everything in it:
- Blades (Firmware - BIOS, RAID Controller & Hard Disk firmware, NIC's, anything in the Mezzanine slots)
- HP Onboard Administrator modules (HP OA)
- HP Virtual Connect modules (HP VC)
Let me use my own situation as an example. I installed some HP blc460 Gen9 Blades in a c7000 chassis which already had Gen8 Blades installed for a client. However when I tried to access the iLo from the Onboard Administrator or configure the networking via the Virtual Connect Manager the new Blades appeared but were not configurable. The iLO was also not initialized. After checking the above Firmware Compatibility Matrix I realised this was due to the Gen9's requiring a new OA and VC firmware version.
In this case the Blades themselves were up to date, however the HP OA's were version 3.71 and the HP VC's were at version 3.75.
Now, according to the firmware compatibility matrix I should be between the minimum OA version of 4.11 and the maximum OA version of 4.50 and between the minimum VC version of 4.30 and the maximum VC version of 4.45.
However, you can't just jump to the latest even though the firmware compatibility matrix guide does not say otherwise (unlike most other firmware components). In fact you have to dig through large documents on the HP site in order to find this out. HP has a special Virtual Connect team (just for these modules!) as well that can advise you which firmware versions you need to upgrade to.
In my case this is how I did it without any issues:
To OA v4.01, then to VC v4.01
To OA v4.23, then to VC v4.20
To OA v4.30, then to VC v4.31
To OA v4.50, then to VC v4.45
In my case this took nearly 3 hours to complete. This was only due to the large jump in the version which had significantly new changes, features and improvements.
Also, don't forget to backup the OA and VC configs after each firmware update is successful! Running a health check to ensure the module is still in sync with the other is also a good idea.
The OA is simply upgraded via the OA web console. The VC, however needs a VCSU tool (Virtual Connect Support Utility tool) this can be downloaded from the HP Website:
For the VC, you need to specify the version of the firmware you are upgrading to each time. Firmware is downloadable from the HP website (you need a valid support contract/care pack).
To upgrade the VC to 4.01 for example:
Open CMD, CD to the VCSU program files path:
CMD > cd C:\Program Files (x86)\Hewlett-Packard Company\Virtual Connect Support Utility
Take a backup of the VC config:
vcsu –a configbackup –i 10.10.111.5 -u Administrator –p <OA password> -vcu Administrator –vcp <same as OA>
Run the update:
vcsu -a update -i <OA IP> -u <OA username> -p <OA Password> -l C:\vcfwall401.bin -vcu <VC username> -vcp <VC Password> -of parallel
Note: I downloaded v4.01 to my C:\ drive above, you will need to adjust this for your own.
After the upgrade is complete, then verify both modules are in sync and are up:
vcsu –a healthcheck –i 10.10.111.5 –u Administrator –p <OA password> -vcu Administrator –vcp <same as OA>
Once this is all done, the Gen9 Blades were detected properly in the Onboard Administrator, the iLO was functional and Virtual Connect Manager allowed me to assign Server profiles and were now usable.