Re: [PATCH] target/i386: KVM: add hack for Windows vCPU hotplug with SGX

[Sean Christopherson <seanjc@google.com>]

On Mon, Jun 09, 2025, Andrey Zhadchenko wrote:
> On 6/9/25 18:39, Sean Christopherson wrote:
> > On Mon, Jun 09, 2025, Denis V. Lunev wrote:
> > > > Does anything in edk2 run during the hotplug process (on real hardware
> > > > it does, because the whole hotplug is managed via SMM)? If so maybe that
> > > > could be a better place to write the value.
> > 
> > Yeah, I would expect firmware to write and lock IA32_FEATURE_CONTROL.
> > 
> > > > So many questions, but I'd really prefer to avoid this hack if the only
> > > > reason for it is SGX...
> > 
> > Does your setup actually support SGX?  I.e. expose EPC sections to the guest?
> > If not, can't you simply disable SGX in CPUID?
> 
> We do not have any TYPE_MEMORY_BACKEND_EPC objects in our default config,
> but have the following:
> sgx=on,sgx1=on,sgx-debug=on,sgx-mode64=on,sgx-provisionkey=on,sgx-tokenkey=on
> We found this during testing, and it can be disabled on our testing setup
> without any worries indeed.
> I have no data whether someone actually sets it properly in the wild, which
> may still be possible.

The reason I ask is because on bare metal, I'm pretty sure SGX is incompatible
with true CPU hotplug.  It can work for the virtualization case, but I wouldn't
be all that surprised if the answer here is "don't do that".

> > > Linux by itself handles this well and assigns MSRs properly (we observe
> > > corresponding set_msr on the hotplugged CPU).
> 
> I think Linux, at least old 4.4, does not write msr on hotplug.

Yeah, it's a newer thing.  5.6+ should initialize IA32_FEATURE_CONTROL if it's
left unlocked (commit 1db2a6e1e29f ("x86/intel: Initialize IA32_FEAT_CTL MSR at boot").

> Anyway it hotplugs fine and tolerates different value unlike Windows

Heh, probably only because the VM isn't actively using KVM at the time of hotplug.
In pre-5.6 kernels, i.e. without the aforementioned handling, KVM (in the guest)
would refuse to load (though the hotplug would still work).  But if the guest is
actively running (nested) VMs at the time of hotplug, the hotplugged vCPUs would
hit a #GP when attempting to do VMXON, and would likely crash the kernel.

> > Linux is much more tolerant of oddities, and quite a bit of effort went into
> > making sure that IA32_FEATURE_CONTROL was initialized if firmware left it unlocked.
> 
> Thanks everyone for the ideas. I focused on Windows too much and did not
> investigate into firmware, so perhaps this is rather a firmware problem?
> I think by default we are using seaBIOS, not ovmf/edk2. I will update after
> some testing with different configurations.

Generally speaking, firmware is expected to set and lock IA32_FEATURE_CONTROL.
But of course firmware doesn't always behave as expected, hence the hardening that
was added by commit 1db2a6e1e29f to avoid blowing up when running on weird/buggy
firmware.