ThisCalculator

Everyday

Test VMware ESXi Patches Before Production

Updated · Standard formula, shown below

Test VMware ESXi Patches Before Production

This guide walks you through a safe, repeatable process for testing VMware ESXi patches in a non-production environment before you apply them to your live hosts. You will learn how to stage the patch, create a test bed, apply the update, validate functionality, and roll back if something goes wrong. Following this workflow reduces the risk of downtime and unexpected behavior in your production cluster.

What You’ll Need

  • A non-production ESXi host or cluster that closely mirrors your production environment (same hardware family, drivers, and configuration).
  • VMware vCenter Server (or at least the ESXi CLI) to manage the test host.
  • Access to the VMware Customer Connect portal to download patches and the VMware Compatibility Guide.
  • A maintenance window for testing, even though it’s not production.
  • A backup or snapshot of the test host’s configuration (optional but recommended).
  • SSH access to the test host (if you plan to use the command line).

Step-by-Step

  1. Identify the patch you need. Log in to VMware Customer Connect and navigate to the Patches & Upgrades section. Find the specific patch (e.g., ESXi 7.0 U3f) and note its build number and release date. Read the release notes carefully—they list known issues, prerequisites, and any required firmware updates.
  2. Check hardware and driver compatibility. Use the VMware Compatibility Guide to confirm that your test host’s server model, storage controllers, and NICs are supported with the target ESXi version. Even if you’re just applying a patch (not a full upgrade), verify that the patch doesn’t include a driver update that could break your hardware.
  3. Set up a test environment. If you don’t have a dedicated test host, create a logical test environment within your existing cluster by placing a host in maintenance mode and isolating it from production traffic. Ideally, use a separate vCenter or a standalone host to avoid accidental cross-contamination.
  4. Back up the test host configuration. Before patching, export the ESXi configuration using the vSphere Client (Host > Configure > System > Backup Configuration) or via the command line with vim-cmd hostsvc/firmware/backup_config. This gives you a restore point if the patch fails.
  5. Download the patch. From the VMware Customer Connect portal, download the patch ZIP file to a location accessible by your vCenter or test host. Ensure the file integrity by checking the MD5 or SHA1 checksum provided on the download page.
  6. Upload the patch to a datastore. In the vSphere Client, go to your test host, navigate to Storage, and upload the ZIP file to a datastore (e.g., a local datastore or a shared datastore that only the test host can access). This step is required for both GUI and CLI methods.
  7. Apply the patch using the vSphere Client (GUI method). Right-click the test host, select Maintenance Mode, and confirm that all VMs are migrated or powered off. Then go to Host > Updates > Baselines, click Attach, and select the patch ZIP you uploaded. Click Stage and then Remediate. Follow the wizard and monitor the progress.
  8. Alternative: Apply the patch via CLI (ESXCLI). Enable SSH on the test host, then run the following command to install the patch:
    esxcli software vib update -d /vmfs/volumes/datastore1/ESXi-7.0U3f.zip
    Replace the path with your actual datastore and filename. This method is faster and gives you more detailed output.
  9. Reboot the host. After the patch is applied, reboot the test host as required (most patches require a reboot). Ensure the host comes back up and shows the new build number in the Summary tab.
  10. Validate the host health. Check the following after reboot:
    • Host status is green and not showing any alarms.
    • All drivers and firmware are loaded correctly (check with esxcli software vib list).
    • Storage and network connectivity are working (ping a few hosts, check datastore visibility).
    • vCenter can manage the host without errors.
  11. Test VM workloads. Power on a few test VMs that represent your typical production workload (e.g., a web server, a database VM). Perform basic operations: migrate a VM to another host (if you have a cluster), snapshot, and restore. This verifies that the patch didn’t break VM functionality.
  12. Monitor for a soak period. Leave the test host running for at least 24-48 hours, monitoring for any crashes, kernel panics, or performance degradation. Check the system logs (/var/log/vmkernel.log) for any new errors.
  13. Document the results. Record the patch version, test date, any issues found, and the resolution. This documentation is invaluable for future patches and for justifying the production rollout.
  14. Roll out to production. Once the test is successful, follow the same steps in production, but use a rolling upgrade strategy (one host at a time) to minimize impact. Always keep the test environment available for a few weeks after the production upgrade in case you need to compare behavior.

Common Mistakes

  • Skipping the compatibility check. Applying a patch that includes a driver update for hardware that isn’t supported can cause the host to fail to boot or lose connectivity.
  • Not using a test environment that mirrors production. If your test host has different NICs or storage controllers, you might miss driver conflicts that will occur in production.
  • Forgetting to back up the configuration. If the patch corrupts the host configuration, you may need to reinstall ESXi and reconfigure everything from scratch.
  • Patching without taking the host out of production. Even in a test environment, if the host is still serving VMs, the patching process can disrupt them.
  • Not checking the release notes for known issues. Some patches have specific prerequisites or known bugs that can be avoided by reading the documentation.
  • Skipping the soak period. A host that boots fine immediately after a patch might fail after a few hours due to a memory leak or driver issue.

Tips

  • Use the vSphere Lifecycle Manager (vLCM) if you’re on vSphere 7 or later—it automates the patching process and can include firmware updates, making the test process more streamlined.
  • Create a baseline in vCenter that includes the patch, and then attach that baseline to your test host. This makes it easy to apply the same patch to multiple hosts later.
  • If you have multiple test hosts, patch one at a time and compare the results. This helps isolate issues to a specific host configuration.
  • Keep a copy of the patch ZIP on a local datastore on the test host. This avoids network delays during the patch process.
  • Enable SSH only during testing and disable it afterwards for security.
  • Use the esxcli software vib list command to verify that the patch was applied correctly and that no other VIBs were inadvertently removed.

Troubleshooting

If the patch fails to apply or the host doesn’t come back up, do not panic. Here’s what to do:

  • Host doesn’t boot after patch. Try booting into the previous ESXi installation by selecting the previous boot bank in the GRUB menu (if available). If that fails, use the backup configuration you made earlier to restore the host.
  • Patch installation fails with an error. Check the exact error message. Common causes include insufficient space on the bootbank, missing dependencies, or a corrupt patch file. Re-download the patch and verify its checksum.
  • Network connectivity lost after patch. This is often due to a driver update. Check if the NIC is recognized using esxcli network nic list. If not, you may need to roll back the patch and install the correct driver manually.
  • Host shows as disconnected in vCenter. This can happen if the host’s management agent (hostd) fails to start. Try restarting the management agents via the console or SSH: services.sh restart.
  • VMs fail to power on after patch. Check the VM compatibility settings and ensure the host’s virtualization features are still enabled. Sometimes a patch resets BIOS settings.
  • If all else fails, roll back. Use esxcli software profile remove or reinstall ESXi from scratch and restore the configuration backup. Always have a plan for rollback before you start patching.

Frequently Asked Questions

Can I test a patch on a VM instead of a physical host?

No, ESXi patches are for the hypervisor itself, and you cannot install them on a VM. You need a physical host, even if it’s a low-end server, to test properly.

How long should I test a patch before deploying to production?

At minimum, 24-48 hours of soak time after a successful reboot. For critical patches, consider a week to ensure no delayed issues appear.

What if I don’t have a dedicated test host?

You can create a logical test environment by using a host in your cluster that is isolated with a separate vCenter, or by using a nested ESXi setup (ESXi running as a VM) for basic testing, but be aware that nested virtualization does not test hardware drivers.

Do I need to test every patch, even minor ones?

Yes, even minor patches can introduce regressions. However, you can prioritize testing based on the severity of the patch and the impact on your environment.

Can I use vSphere Update Manager (now vLCM) to test patches?

Yes, vLCM is the recommended tool. You can create a test baseline and apply it to a test host, then later apply the same baseline to production hosts.

What should I do if a patch fails in production after testing?

Immediately roll back the patch on the affected host using the same rollback procedure you practiced in testing. Keep the test environment running to investigate the cause without affecting production.

Related

Method: the standard formula for this calculation, rounded as each result states. See how the calculators work or report an error.