Reverting an ESXi Upgrade

I was looking at the configuration of 3 VSphere servers at work today and determined that they were running an old BIOS. I upgraded their HP DL380 G9 BIOS from P89 v1.50 (07/20/2015) to P89 v2.72 (03/25/2019) using iLO.

The first two updates went out without a hitch. The last server? Not so good, as there was a cascading chain of errors.

After rebooting the server (which is necessary for the BIOS update to take effect), the ESXi cluster complained that the server was not reachable. I couldn’t ping it either.

I connected via. the Direct Console User Interface (DCUI) and found that the ESXi IP & DNS settings had reverted to old settings from several years ago. I manually reset them to the proper values and rebooted.

I then reconnected the server to the cluster, which failed because of a SSL certificate naming error. I had to rejoin it again, which resulted in an IO error:
Registration/Unregistration of third-party IO filter storage providers fails on a hosts (yes, the error message contains a subject/verb mismatch)

Dell has KB000058590: Dell EMC VxRail: Alarm for Registration/Unregistration of third-party IO filter storage providers fails. I followed the steps to replace the castore.pem file with one from a working server, and rebooted.

That resulted in "error loading /sb.v00 fatal error 33 (inconsistent data)" when trying to boot the ESXi OS.

At this point I reverted back to the prior hypervisor via. DCUI:

In the console screen of the ESXi host, press Ctrl+Alt+F2 to see the Direct Console User Interface (DCUI) screen.
Press F12 to view the shutdown options for the ESXi host.
Press F11 to reboot.
When the Hypervisor progress bar starts loading, press Shift+R (This must be done while the bar is loading, and not after). You will see the warning:
```
Current hypervisor will permanently be replaced  
with build: X.X.X-XXXXXX. Are you sure? [y/n]
```
Press Y to roll back the build.
Press Enter to boot.

ESXi came back online properly, but still complained about the IO filters. At this point, I noticed that while the IP was correct, DNS reverted to old entries. I set it back to the proper values.

The ESXi host is finally healthy in the cluster!… but now I have to upgrade it from 6.0 to 6.5 again.