Understanding VBS Enclaves, Windows’ new security technology

Recently Microsoft announced efforts to improve security in its planned Recall AI assistant. Many of the details were not a surprise, as it built on familiar tools and services I’ve written about during the past few years, but the most important feature, VBS Enclaves, will be new to most of us.

VBS Enclaves are the latest piece of Microsoft’s push to use virtualization to secure Windows by isolating critical functions in encrypted virtual machines using its low-level Krypton hypervisor.

Windows’ always-on hypervisor, Krypton

Krypton is an important piece of the modern Windows, as it allows virtual machines and the host OS to share a scheduler, allowing features like Windows login to be isolated from the rest of the OS. This allows them to continue to run as Windows features while protecting passwords and biometric information. They stay isolated where malware running in the host Windows instance can’t access them.

Krypton is the foundation of much of Windows 11’s hardware security, working with your PC’s trusted platform modules (TPM) to manage encryption keys, digital signatures, and verification hashes. Think of it as a way of significantly reducing the risk of sensitive information leaking from your PC, as well as reducing the risk of malware replacing what would normally be trusted Windows functions.

Virtualization-based security has been on Microsoft’s radar for a long time, with a heritage that dates back to Windows Server 2016 and the introduction of Secure Windows containers. Although Windows’ support for Docker and other container-based application isolation tools allowed some form of process isolation, it wasn’t perfect.

Secure Windows containers mixed both containers and Hyper-V’s security tool to add more isolation by hosting secure containers in a virtual machine rather than on the host OS. This did add the overhead that comes with running a separate OS for your application containers, but features like Nano Server and Windows Server Core kept it to a minimum, with successive releases reducing server image sizes significantly.

Virtualization-based security in Windows 11

Microsoft mixed this approach with TPM capabilities to add enhanced security to Windows over time, enforcing it in Windows 11. That’s why Windows 11 requires hardware with TPM 2.0 support.

The TPM holds the keys and certificates to manage digital signatures so that tools like Windows memory integrity service can run in a hardened virtualized environment using Hyper-V-secured VMs running on the Krypton hypervisor. With memory integrity, kernel-mode drivers and binaries are checked for valid signatures before they run; unsigned code is blocked before it can compromise your PC.

Microsoft recently extended its virtualization-based security model to what it calls VBS Enclaves. If you’ve looked at implementing confidential computing on Windows Server or in Azure, you’ll be familiar with the concept of enclaves, using Intel’s SGX instruction set to lock down areas of memory, using them as a trusted execution environment. This approach requires specific processors, with the latest generation of SGX limited to enterprise Xeon hardware.

Using software to run trusted execution environments

VBS Enclaves offer a similar approach to securing memory but without requiring specific hardware. That allows Microsoft to provide secure enclaves on Intel, AMD, and Arm hardware. As a result, Recall will only run in trusted memory under the control of the Krypton hypervisor, with encryption keys managed by your PC’s TPM and with access controlled by Windows Hello to ensure user presence.

Putting a trusted execution environment on a PC is useful for more than securing AI. It protects sensitive data, adding a new level of protection beyond at rest and in motion: in use. While it does require more work to define and use a VBS Enclave, it’s worth it to have more security with only limited performance impact.

With Windows 11’s memory integrity tools, a VBS Enclave uses Windows’ integral hypervisor to create a new, isolated, high-privilege area of system memory: Virtual Trust Level 1. Most of your code, and Windows itself, continues to run at Virtual Trust Level 0. VTL 1 is used by a secure version of the Windows kernel, with its own isolated user mode. This is where your VBS Enclave runs, as part of an application that appears to cross the boundary between the two zones. In reality, you’re separating off the VTL 1 enclave and using secure channels to communicate with it from the rest of your application in VTL 0.

Using VBS Enclaves in your applications

So how do you build and use VBS Enclaves? First, you’ll need Windows 11 or Windows Server 2019 or later, with VBS enabled. You can do this from the Windows security tool, via a Group Policy, or with Intune to control it via MDM. It’s part of the Memory Integrity service, so you should really be enabling it on all supported devices to help reduce security risks, even if you don’t plan to use VBS Enclaves in your code.

The best way to think of it is as a way of using encrypted storage securely. So, for example, if you’re using a database to store sensitive data, you can use code running in an enclave to process and query that data, passing results to the rest of your application. You’re encapsulating data in a secure environment with only essential access allowed. No other parts of your system have access to the decryption keys, so on-disk data stays secure.

Code running in the VTL 1 environment must be signed by Microsoft, with an OS-level handoff between the two trust zones that resets CPU registers to reduce the risk of state transferring between relatively insecure user modes and your VBS Enclave. Using a VBS Enclave is, naturally, more computationally expensive, and operations take longer to run (though still only microseconds).

VBS Enclaves are DLLs and need a host application to run. You’re limited to a subset of the available Windows system-level C++ APIs, with a list of available Universal C Runtime APIs listed in the development documentation. Other APIs supported are in the VBS Enclave Runtime (Vertdll) and the Bcrypt cryptographic library.

Start with a sample enclave

Microsoft provides a useful sample application that illustrates the life cycle of a VBS Enclave application, showing how to call enclave functions from the host application. External functions need to be explicitly exported—and only those functions can be called by the host. Compiling an enclave needs specific configuration in the linker, ensuring that the right libraries are included and that the resulting DLL is instrumented correctly. Other features ensure that the VBS Enclave is protected from attacks via forged platform DLLs.

Once compiled, your code must be signed. A VBS Enclave signature requires three specific EKUs (Extended Key Usage): one for code signing, one for the enclave, and one for the author. In production you can use Microsoft’s own Trusted Signing service, which offers a profile for signing enclaves. This approach lets you automate signing with the Azure CLI.

There are some important points to consider when writing code that uses VBS Enclaves. They can be loaded by any application running on the host PC, so you need to write code in the enclave that explicitly treats anything outside the enclave as untrusted.

Essential security for sensitive data

The same technology is used for Azure SQL and SQL Server’s Always Encrypted feature. This approach ensures that only authorized users have access to sensitive data. T-SQL operations can cross the secure enclave boundary using confidential queries over an internal TLS channel. Operations take place inside the enclave, allowing the underlying data to always remain encrypted.

VBS Enclaves are necessarily complex, with significant restrictions over standard DLLs. Without those constraints, though, they’d only be as secure as a standard DLL. By locking-down code at a library and header level to run in a trusted execution environment, you’re significantly reducing the risk of data leaking, either deliberately or accidently.

Although it takes a fraction longer and requires some extra system resources, if you’re working with sensitive information, it’s a trade-off worth making. Using VBS Enclaves and encrypted storage should be essential wherever you’re using personally identifiable information, whether it’s a user’s vector-indexed Recall history, stored payment information, or health records.

If there’s any risk associated with data, then you really need a compelling reason not to use this technology. When it comes to sensitive data, VBS Enclaves should be the default choice.