Mitigate Spectre and Meltdown impact with vSphere ESXi

As most of the folks in the IT, I’ve spent last couple of days researching about famous Spectre and Meltdown attacks and their possible impact to our infrastructure.

These security flaws are especially bad, not only because they’ve been here for more than 2 decades, but mitigation comes with significant performance impacts.

Where to start?

As these attacks have to be executed locally, I would recommend anyone to focus on systems which have internet access and systems which are shared among multiple users, especially:

• Multi tenant environments – if you are a hosting company
• Terminal servers
• Workstations/Desktops (Both physical and virtual)
• Mobile devices – Phone/tablets

Although most of the CPU vendors and their CPUs are affected to some extend, I will be focusing on Intel CPUs here, as those are most vulnerable.

OS patching is not enough!

Almost every OS vendor came out with their patches, however what may not be clear for an average user is, If you want to be fully protected against known exploits you have to also update your CPU microcode which is usually coming together with uEFI/BIOS updates from your hardware vendors.

If you have slow HW vendor or they do not support your motherboard anymore, it doesn’t mean you are entirely doomed yet. There is a way how to sideload CPU microcode during the OS boot. For Linux you can find instructions and microcode at Intel’s website directly. In case of Microsoft, occasionally in case of critical issues they do provide CPU microcode updates via Windows Update, although I don’t know for sure when or if they will decide to release it this time, anyway in such issue I would bet yes.

There are 3 types of attack:

• CVE-2017-5753 (variant #1/Spectre) Bounds-checking exploit during branching
• CVE-2017-5715 (variant #2/Spectre) Indirect branching poisoning attack
• CVE-2017-5754 (variant #3/Meltdown) Speculative cache loading

OS(kernel) patches alone will protect you against Variant #1 and Variant #3, however protection against Variant #2 requires CPU microcode update as well.

Protection in Virtualized environments

When it comes to virtualization, it is a bit trickier as there is an additional layer included.

By updating CPU microcode to fix Spectre you are presenting additional CPU features to your operating system, however this is not the case if you are running in virtual machine as those features may be masked by hypervisor, especially in this case as it is very fresh. You also need to patch your hypervisor, so it will present those features to the OS in virtual machines as well.

VMware released new security advisory VMSA-2018-0004 together with patches and  kb52085 yesterday regarding enabling this capability.

What I like especially, that they included also CPU microcode with the new patches, therefore you don’t have to wait for HW vendor to release uEFI updates. Although updating uEFI is still strongly recommended!

How to protect against Spectre #2  with minimizing performance hit

1. update your vCenter to following versions
   1. 6.5 U1e
   2. 6.0 U3d
   3. 5.5 U3g
2. update ESXi and CPU microcode
   1. ESXi650-201801401-BG hypervisor
   2. ESXi600-201801401-BG hypervisor
   3. ESXi550-201801401-BG hypervisor and microcode
   4. ESXi650-201801402-BG microcode
   5. ESXi600-201801402-BG microcode
3. Update VM Hardware version to at least version 9 (to be protected), ideally version 11 to minimize performance impact in case your CPU supports PCID and INVPCID (Process-Context Identifiers).
4. Remember you still have to patch your guest operating system

vCenter patches will, make sure EVC version in cluster will be updated to the latest version only after you update all hosts within the cluster

Note: ESXi will never override a newer version of the microcode patch provided by BIOS nor will the BIOS with an older version prevent ESXi from applying the newer version.

I recommend you following articles to read more about handling the issue in Red Hat and Microsoft’s Operating systems:

Controlling the Performance Impact of Microcode and Security Patches for CVE-2017-5754 CVE-2017-5715 and CVE-2017-5753 using Red Hat Enterprise Linux Tunables

Speculative Execution Exploit Performance Impacts 

Windows Client Guidance for IT Pros to protect against speculative execution side-channel vulnerabilities

Windows Server guidance to protect against speculative execution side-channel vulnerabilities

Understanding the performance impact of Spectre and Meltdown mitigations on Windows Systems

The following two tabs change content below.

• Bio
• Latest Posts

Dusan Tekeljak

With over 12 years of experience in the Virtualization field, currently working as a Senior Consultant for Evoila, contracted to VMware PSO, helping customers with Telco Cloud Platform bundle. Previous roles include VMware Architect for Public Cloud services at Etisalat and Senior Architect for the VMware platform at the largest retail bank in Slovakia. Background in closely related technologies includes server operating systems, networking, and storage. A former member of the VMware Center of Excellence at IBM and co-author of several Redpapers. The main scope of work involves designing and optimizing the performance of business-critical virtualized solutions on vSphere, including, but not limited to, Oracle WebLogic, MSSQL, and others. Holding several industry-leading IT certifications such as VCAP-DCD, VCAP-DCA, VCAP-NV, and MCITP. Honored with #vExpert2015-2019 awards by VMware for contributions to the community. Opinions are my own!

Latest posts by Dusan Tekeljak (see all)

• VM Latency Sensitivity set to High still fails with no (proper) warning - June 27, 2024
• ESXi 6.7 U1 fixes: APD and VMCP is not triggered even when no paths can service I/Os - November 30, 2018
• Update manager error: hosts could not enter maintenance mode - November 19, 2018