What happens when the USB key or SD card I’ve installed VMware ESXi on fails

hardwarehphp-proliantvmware-esxi

An SD (SDHC) card installed in an HP ProLiant DL380p Gen8 server running VMware ESXi just failed 🙁

I encountered some ominous looking messages on the vCenter console and in the HP ProLiant ILO event log…

Lost connectivity to the device … backing the boot filesystem. As a
result, host configuration changes will not be saved to persistent
storage.

enter image description here

Embedded Flash/SD-CARD: Error writing media 0, physical block 848880:
Stack Exception.

enter image description here

VMware advocates the use of USB and SD (SDHC) boot devices for ESXi. It was one of the main reasons the smaller footprint ESXi was developed (versus the older ESX). I've spent much time highlighting the differences between ESXi's installable and embedded modes to coworkers and clients. However, these failures do seem to happen. In this case, this is my third instance.

Luckily, this is a vSphere cluster with SAN storage. What steps should be taken to remediate this failure?

Best Answer

Here's the process I used to resolve this:

VMware ESXi can be installed in an embedded mode or an installable mode. As outlined here, the installation mode is determined by the destination media and the size of the volume available to the ESXi installer.

  • USB, SDHC or any device less than 5GB in size: Embedded
  • Hard drives/volumes greater than or equal to 5GB in size: Installable

One of the unique attributes of running ESXi in embedded mode is that the OS is loaded into RAM and only touches the USB/SD device hourly during normal operation. In my situation, the system continued to operate, even with a failed SDHC device.

The error message I received in the vCenter interface indicated that configuration changes would not be saved, but the cluster was still usable. I left the system in this state for several days until I could get to the datacenter to replace the SD card.

With regard to steps to take following a failure of a USB or SD device, it is important to extract and save a copy of your host's settings!!

This is easily accomplished via PowerCLI or the vSphere CLI. I used PowerCLI running from the vCenter server:

Get-VMHostFirmware -VMHost 10.10.8.22 -BackupConfiguration -DestinationPath C:\Users\ewwhite\Downloads
  • Following that, I evacuated all virtual machines from the affected host and placed it in maintenance mode.
  • The host was then shut down, the SDHC card replaced with a new device, and I installed ESXi again.
  • Once the host was up again with a fresh ESXi install, I made the bare minimum configuration changes needed to make the host visible on the network; set IP information, vlan info and password.
  • I reloaded the saved configuration to the host via PowerCLI...

For this step, I used:

Set-VMHostFirmware -VMHost 10.10.8.22 -Restore -SourcePath c:\Users\ewwhite\configBundle-10.10.8.22.tgz -HostUser root -HostPassword YoMama!!

Restoring the configuration forces a host reboot. Once up again, I was able to issue a reconnect to rejoin the host in vCenter and exit maintenance mode.

If PowerCLI not available, the ESXi shell commands look like: 

vim-cmd hostsvc/firmware/backup_config

This produces a web link that you'll be able to browse to and download a tarball of the host's configuration.

You can SCP a configuration file to a host and use the following to restore the settings.

vim-cmd hostsvc/firmware/restore_config /tmp/configBundle.tgz

That's all!

Related Topic