Re: KVM default behavior change on module loading in kernel 6.12

Re: KVM default behavior change on module loading in kernel 6.12

[Sean Christopherson]

+lists and Paolo

On Mon, Oct 07, 2024, Vadim Galitsin wrote:
> Hi Sean,
> 
> My name is Vadim. I am from Oracle's VirtualBox team.
> 
> I noticed your commit b4886fab6fb6 (KVM: Add a module param to allow enabling
> virtualization when KVM is loaded) which is a part of 6.12-rc1 (and newer)
> kernel.
> 
> The issue I am observing on VBox side is that no VBox VMs can now be started
> by default. Historically, Qemu and VBox VMs cannot run in parallel (either of
> them should enable virtualization by its own). Previously, when
> virtualization was enabled at the event when Qemu VM starts, there was no
> such issue. I suspect VMware guys might have exactly the same problem now.
> 
> Commit has absolute sense for server virtualization and of course, feature
> can be disabled by specifying "kvm.enable_virt_at_load=0" in kernel command
> line (or by unloading kvmXXX module(s) manually), but it is probably rather
> inconvenient for desktop virtualization users who run other than Qemu VMs I
> think.
> 
> Would you consider to change the default behavior by having
> "kvm.enable_virt_at_load=0", so people who really need it, could explicitly
> enable it in kernel command line?

I'm not dead set against it, but my preference would be to force out-of-tree
hypervisor modules to adjust.  Leaving enable_virt_at_load off by default risks
performance regressions due to the CPU hotplug framework serially operating on
CPUs[1].  And, no offence to VirtualBox or VMware, I care much more about not
regressing KVM users than I care about inconveniencing out-of-tree hypervisors.

Long term, the right answer to this problem is to move virtualization enabling
to a separate module (*very* roughly sketeched out here[2]), which would allow
out-of-tree hypervisor modules to co-exist with KVM.  They would obviously need
to give up control of CR4.VMXE/VMXON/EFER.SVME, but I don't think that's an
unreasonable ask.

The multi-KVM idea aside, TDX support for trusted devices is coming down the pipe
and will need to enable VMX without KVM being involved in order to perform SEAMCALLs
from other subsystems.  I.e. sooner or later, I expect virtualization enabling to
be moved out of KVM.

Short term, one idea would be to have VirtualBox's module (and others) prepare
for that future by pinning kvm-{amd,intel}.ko, and then playing nice if VMX/SVM
is already enabled.

[1] https://lore.kernel.org/all/20240608000639.3295768-9-seanjc@google.com
[2] https://lore.kernel.org/all/20231107202002.667900-14-aghulati@google.com

Re: KVM default behavior change on module loading in kernel 6.12

[Paolo Bonzini]

On Mon, Oct 7, 2024 at 8:07 PM Sean Christopherson <seanjc@google.com> wrote:
> On Mon, Oct 07, 2024, Vadim Galitsin wrote:
> > Would you consider to change the default behavior by having
> > "kvm.enable_virt_at_load=0", so people who really need it, could explicitly
> > enable it in kernel command line?
>
> I'm not dead set against it, but my preference would be to force out-of-tree
> hypervisor modules to adjust.  Leaving enable_virt_at_load off by default risks
> performance regressions due to the CPU hotplug framework serially operating on
> CPUs[1].  And, no offence to VirtualBox or VMware, I care much more about not
> regressing KVM users than I care about inconveniencing out-of-tree hypervisors.
>
> Long term, the right answer to this problem is to move virtualization enabling
> to a separate module (*very* roughly sketeched out here[2]), which would allow
> out-of-tree hypervisor modules to co-exist with KVM.  They would obviously need
> to give up control of CR4.VMXE/VMXON/EFER.SVME, but I don't think that's an
> unreasonable ask.

I agree, VMXE/VMXON/SVME are mostly a one-shot thing.

I thought about adding a Kconfig for kvm.enable_virt_at_load, but it
is not really a good solution.  Distros would either leave it to the
default of y and inconvenience VirtualBox/VMware users, or set it to n
and have bad performance on machines with a large number of CPUs. The
problem is that "all distros set it to y" is an unstable equilibrium,
because it may take only one bug report to convince distros to flip
the value.

> Short term, one idea would be to have VirtualBox's module (and others) prepare
> for that future by pinning kvm-{amd,intel}.ko, and then playing nice if VMX/SVM
> is already enabled.

Yes, this is a good plan for allowing to work with both old and new KVM.

Alternatively if VirtualBox or VMware have an installer or rpm they
can write a file to /etc/modprobe.d to change the default, which
doesn't require code changes but is not a good idea.

Paolo