VMX ideas

VMX ideas

[Alexander Graf]

Hi list,

I really do like KVM and would love to make using it a complete no- 
brainer user experience. For the average user, loading a kernel module  
(and unloading it) isn't really one of the most common tasks.

So I was considering to automatically load kvm-intel and kvm-amd on  
bootup, when the CPU has a CPUID flag. Unfortunately it doesn't work  
that easily. VMX enters Root mode, setting a page to use for the VMCS.  
KVM sets up this page when the kvm-intel module gets loaded. This  
means, as soon as kvm-intel is modprobe'd, vmware, virtual box,  
parallels and the like are screwed.
 From a distribution perspective, this isn't exactly an ideal situation.

So I was thinking hard on what to do to circumvent this problem and  
came up with several approaches:

1. Export some functions that could be used by 3rd party hypervisors  
to "occupy" the VMCS region

This would mean, that the others' kernel modules have dependencies on  
KVM. I don't think that's really desirable. Furthermore it wouldn't  
work with kvm compiled as external module.


2. Create a root mode base framework

This is the Mac OS X approach. They have a small piece of code in the  
kernel, that sets up the root mode, handles suspend/resume and  
maintains a ref counter on its usage. Since the root mode is set on  
bootup, all hypervisors would need to make use of that framework.

While I like that approach for its simplicity, it does not allow for  
parallel execution of VMs with different hypervisors. Is this a real  
problem?


3. Create a root mode bloat framework

Taking approach 2. we could also export functions to handle VPID  
management and VMXPTRLD, which would make it possible for multiple  
hypervisors to run concurrently. This is the most elegant approach  
IMHO, but also needs the most maintenance, as new revisions of VMX  
could possibly break the whole concept.


I started working on 2. and wanted to put in some parts of 3.,  
realizing that it's more work and code change than I'd like. Any  
comments on these ideas? I'd really love to have KVM always loaded  
somehow.

Alex

Re: VMX ideas

[Joerg Roedel]

On Wed, Oct 29, 2008 at 03:52:21PM +0100, Alexander Graf wrote:
> Hi list,
> 
> I really do like KVM and would love to make using it a complete no- 
> brainer user experience. For the average user, loading a kernel module  
> (and unloading it) isn't really one of the most common tasks.
> 
> So I was considering to automatically load kvm-intel and kvm-amd on  
> bootup, when the CPU has a CPUID flag. Unfortunately it doesn't work  
> that easily. VMX enters Root mode, setting a page to use for the VMCS.  
> KVM sets up this page when the kvm-intel module gets loaded. This  
> means, as soon as kvm-intel is modprobe'd, vmware, virtual box,  
> parallels and the like are screwed.
> From a distribution perspective, this isn't exactly an ideal situation.
> 
> So I was thinking hard on what to do to circumvent this problem and  
> came up with several approaches:

If you want to allow parallel execution of different hypervisors at the
same time you have a similar problem with SVM because you don't know
which hypervisor uses which ASID.

Joerg

Re: VMX ideas

[Alexander Graf]

On 29.10.2008, at 16:01, Joerg Roedel wrote:

> On Wed, Oct 29, 2008 at 03:52:21PM +0100, Alexander Graf wrote:
>> Hi list,
>>
>> I really do like KVM and would love to make using it a complete no-
>> brainer user experience. For the average user, loading a kernel  
>> module
>> (and unloading it) isn't really one of the most common tasks.
>>
>> So I was considering to automatically load kvm-intel and kvm-amd on
>> bootup, when the CPU has a CPUID flag. Unfortunately it doesn't work
>> that easily. VMX enters Root mode, setting a page to use for the  
>> VMCS.
>> KVM sets up this page when the kvm-intel module gets loaded. This
>> means, as soon as kvm-intel is modprobe'd, vmware, virtual box,
>> parallels and the like are screwed.
>> From a distribution perspective, this isn't exactly an ideal  
>> situation.
>>
>> So I was thinking hard on what to do to circumvent this problem and
>> came up with several approaches:
>
> If you want to allow parallel execution of different hypervisors at  
> the
> same time you have a similar problem with SVM because you don't know
> which hypervisor uses which ASID.

Yes, while writing the mail I was thinking to bring that on too, but  
figured in the end that the mail's title is "VMX Ideas", so I  
didn't ;-).
Of course, if we want to go with 3., we'd need to write a common  
framework for the ASIDs in SVM that could handle activation of SVM for  
us too.

Right now the situation on VMX is a lot worse than on SVM though, as  
loading the module basically blocks every other hypervisor.

Alex

Re: VMX ideas

[Gerd Hoffmann]

Alexander Graf wrote:
> So I was thinking hard on what to do to circumvent this problem and came
> up with several approaches:

> 3. Create a root mode bloat framework

4. Enter vmx root mode when the first user opens /dev/kvm and leave it
when nobody uses it any more?

Also note that there has been some discussion on lkml to move vmx-root
on/off out of the kvm module into the kernel.  There are a few places
where it would be useful for the kernel to handle that without the kvm
module being involved, crashdump kexec for example.  With that and some
locking in place (so only one can use vmx-root at a time) and
hypervisors being friendly to each other and grab vmx-root only when
they have a guest to run we should be set?

cheers,
  Gerd

Re: VMX ideas

[Anthony Liguori]

Gerd Hoffmann wrote:
> Alexander Graf wrote:
>   
>> So I was thinking hard on what to do to circumvent this problem and came
>> up with several approaches:
>>     
>
>   
>> 3. Create a root mode bloat framework
>>     
>
> 4. Enter vmx root mode when the first user opens /dev/kvm and leave it
> when nobody uses it any more?
>
> Also note that there has been some discussion on lkml to move vmx-root
> on/off out of the kvm module into the kernel.  There are a few places
> where it would be useful for the kernel to handle that without the kvm
> module being involved, crashdump kexec for example.  With that and some
> locking in place (so only one can use vmx-root at a time) and
> hypervisors being friendly to each other and grab vmx-root only when
> they have a guest to run we should be set?
>   

There was some patch that went into KVM that would indicate in CPU state 
that VT root mode was enabled so that VMware could detect that someone 
else was using VT.  Look through the archives.

Regards,

Anthony Liguori

> cheers,
>   Gerd
> --
> To unsubscribe from this list: send the line "unsubscribe kvm" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>   

Re: VMX ideas

[Alexander Graf]

On 29.10.2008, at 16:45, Anthony Liguori wrote:

> Gerd Hoffmann wrote:
>> Alexander Graf wrote:
>>
>>> So I was thinking hard on what to do to circumvent this problem  
>>> and came
>>> up with several approaches:
>>>
>>
>>
>>> 3. Create a root mode bloat framework
>>>
>>
>> 4. Enter vmx root mode when the first user opens /dev/kvm and leave  
>> it
>> when nobody uses it any more?
>>
>> Also note that there has been some discussion on lkml to move vmx- 
>> root
>> on/off out of the kvm module into the kernel.  There are a few places
>> where it would be useful for the kernel to handle that without the  
>> kvm
>> module being involved, crashdump kexec for example.  With that and  
>> some
>> locking in place (so only one can use vmx-root at a time) and
>> hypervisors being friendly to each other and grab vmx-root only when
>> they have a guest to run we should be set?
>>
>
> There was some patch that went into KVM that would indicate in CPU  
> state that VT root mode was enabled so that VMware could detect that  
> someone else was using VT.  Look through the archives.

The only patch I've found is that KVM sets CR4.VMXE to 0 on unload  
(hardware_disable). This doesn't really help, as my goal is to have  
the module loaded at boot-time.

Alex

Re: VMX ideas

[Alexander Graf]

On 29.10.2008, at 16:36, Gerd Hoffmann wrote:

> Alexander Graf wrote:
>> So I was thinking hard on what to do to circumvent this problem and  
>> came
>> up with several approaches:
>
>> 3. Create a root mode bloat framework
>
> 4. Enter vmx root mode when the first user opens /dev/kvm and leave it
> when nobody uses it any more?
>
> Also note that there has been some discussion on lkml to move vmx-root
> on/off out of the kvm module into the kernel.  There are a few places
> where it would be useful for the kernel to handle that without the kvm
> module being involved, crashdump kexec for example.  With that and  
> some
> locking in place (so only one can use vmx-root at a time) and
> hypervisors being friendly to each other and grab vmx-root only when
> they have a guest to run we should be set?

If we simply enter/exit VT root mode on kvm starts/stops every single  
hypervisor would still have to maintain all the interaction bits  
itself, like "Is this CPU VT capable?", "May I run VT here?", hotplug  
events, suspend/resume, reboot, etc. It would help to have one place  
to keep all of that, so suspend doesn't magically break when using  
VMware vs. KVM.

That's basically the reason I didn't even propose 4. :-). If you think  
that's a good solution, feel free to convince me.

Alex

Re: VMX ideas

[Avi Kivity]

Alexander Graf wrote:
> Hi list,
>
> I really do like KVM and would love to make using it a complete 
> no-brainer user experience. For the average user, loading a kernel 
> module (and unloading it) isn't really one of the most common tasks.
>
> So I was considering to automatically load kvm-intel and kvm-amd on 
> bootup, when the CPU has a CPUID flag. Unfortunately it doesn't work 
> that easily. VMX enters Root mode, setting a page to use for the VMCS. 
> KVM sets up this page when the kvm-intel module gets loaded. This 
> means, as soon as kvm-intel is modprobe'd, vmware, virtual box, 
> parallels and the like are screwed.
> From a distribution perspective, this isn't exactly an ideal situation.
>
> So I was thinking hard on what to do to circumvent this problem and 
> came up with several approaches:
>
> 1. Export some functions that could be used by 3rd party hypervisors 
> to "occupy" the VMCS region
>
> This would mean, that the others' kernel modules have dependencies on 
> KVM. I don't think that's really desirable. Furthermore it wouldn't 
> work with kvm compiled as external module.
>
>
> 2. Create a root mode base framework
>
> This is the Mac OS X approach. They have a small piece of code in the 
> kernel, that sets up the root mode, handles suspend/resume and 
> maintains a ref counter on its usage. Since the root mode is set on 
> bootup, all hypervisors would need to make use of that framework.
>
> While I like that approach for its simplicity, it does not allow for 
> parallel execution of VMs with different hypervisors. Is this a real 
> problem?
>
>
> 3. Create a root mode bloat framework
>
> Taking approach 2. we could also export functions to handle VPID 
> management and VMXPTRLD, which would make it possible for multiple 
> hypervisors to run concurrently. This is the most elegant approach 
> IMHO, but also needs the most maintenance, as new revisions of VMX 
> could possibly break the whole concept.
>
>
> I started working on 2. and wanted to put in some parts of 3., 
> realizing that it's more work and code change than I'd like. Any 
> comments on these ideas? I'd really love to have KVM always loaded 
> somehow.

I like 2 as well.  We already started moving in this direction in order 
to get kexec working well with kvm.

Please copy virtualization@lists.linux-foundation.org and 
zach@vmware.com to get feedback.

(on the distro I use (F9), kvm is auto-loaded.  or did I change some 
config script and forget?)

-- 
error compiling committee.c: too many arguments to function

Re: VMX ideas

[Avi Kivity]

Gerd Hoffmann wrote:
>> 3. Create a root mode bloat framework
>>     
>
> 4. Enter vmx root mode when the first user opens /dev/kvm and leave it
> when nobody uses it any more?
>
> Also note that there has been some discussion on lkml to move vmx-root
> on/off out of the kvm module into the kernel.  There are a few places
> where it would be useful for the kernel to handle that without the kvm
> module being involved, crashdump kexec for example.  With that and some
> locking in place (so only one can use vmx-root at a time) and
> hypervisors being friendly to each other and grab vmx-root only when
> they have a guest to run we should be set

That means that although kvm is loaded it may be unable to service 
/dev/kvm...  something I'd like to avoid.

-- 
error compiling committee.c: too many arguments to function

Re: VMX ideas

[Daniel P. Berrange]

On Sun, Nov 02, 2008 at 12:26:00PM +0200, Avi Kivity wrote:
> 
> Please copy virtualization@lists.linux-foundation.org and 
> zach@vmware.com to get feedback.
> 
> (on the distro I use (F9), kvm is auto-loaded.  or did I change some 
> config script and forget?)

Yes, on Fedora, if you install the 'kvm' userspace package, we drop a
script into /etc/sysconfig/modules/kvm.modules that cause kvm-intel.ko
or kvm-amd.ko to be loaded at boot time.

Daniel
-- 
|: Red Hat, Engineering, London   -o-   http://people.redhat.com/berrange/ :|
|: http://libvirt.org  -o-  http://virt-manager.org  -o-  http://ovirt.org :|
|: http://autobuild.org       -o-         http://search.cpan.org/~danberr/ :|
|: GnuPG: 7D3B9505  -o-  F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 :|

Re: VMX ideas

[Avi Kivity]

Daniel P. Berrange wrote:
> Yes, on Fedora, if you install the 'kvm' userspace package, we drop a
> script into /etc/sysconfig/modules/kvm.modules that cause kvm-intel.ko
> or kvm-amd.ko to be loaded at boot time.
>   

Does the postinstall also load the modules?  If not, users who don't 
reboot are in for an unpleasant surprise.

-- 
error compiling committee.c: too many arguments to function

Re: VMX ideas

[Avi Kivity]

Avi Kivity wrote:
> Daniel P. Berrange wrote:
>> Yes, on Fedora, if you install the 'kvm' userspace package, we drop a
>> script into /etc/sysconfig/modules/kvm.modules that cause kvm-intel.ko
>> or kvm-amd.ko to be loaded at boot time.
>>   
>
> Does the postinstall also load the modules?  If not, users who don't 
> reboot are in for an unpleasant surprise.
>

I checked and it doesn't.

-- 
error compiling committee.c: too many arguments to function

Re: VMX ideas

[Glauber Costa]

On Sun, Nov 2, 2008 at 2:18 PM, Avi Kivity <avi@redhat.com> wrote:
> Avi Kivity wrote:
>>
>> Daniel P. Berrange wrote:
>>>
>>> Yes, on Fedora, if you install the 'kvm' userspace package, we drop a
>>> script into /etc/sysconfig/modules/kvm.modules that cause kvm-intel.ko
>>> or kvm-amd.ko to be loaded at boot time.
>>>
>>
>> Does the postinstall also load the modules?  If not, users who don't
>> reboot are in for an unpleasant surprise.
>>
>
> I checked and it doesn't.
Funny. Rawhide version does exactly that.
>
> --
> error compiling committee.c: too many arguments to function
>
> --
> To unsubscribe from this list: send the line "unsubscribe kvm" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>



-- 
Glauber  Costa.
"Free as in Freedom"
http://glommer.net

"The less confident you are, the more serious you have to act."

Re: VMX ideas

[Alexander Graf]

Am 03.11.2008 um 00:36 schrieb "Glauber Costa" <glommer@gmail.com>:

> On Sun, Nov 2, 2008 at 2:18 PM, Avi Kivity <avi@redhat.com> wrote:
>> Avi Kivity wrote:
>>>
>>> Daniel P. Berrange wrote:
>>>>
>>>> Yes, on Fedora, if you install the 'kvm' userspace package, we  
>>>> drop a
>>>> script into /etc/sysconfig/modules/kvm.modules that cause kvm- 
>>>> intel.ko
>>>> or kvm-amd.ko to be loaded at boot time.
>>>>
>>>
>>> Does the postinstall also load the modules?  If not, users who don't
>>> reboot are in for an unpleasant surprise.
>>>
>>
>> I checked and it doesn't.
> Funny. Rawhide version does exactly that.

So when a Fedora user installs the kvm userspace, he can't run vmware/ 
parallels/vbox and the like without fiddling?

That doesn't exactly sound like a good user experience to me. I'll try  
to post an RFC for 2. later today.

Alex

>
>>
>> --
>> error compiling committee.c: too many arguments to function
>>
>> --
>> To unsubscribe from this list: send the line "unsubscribe kvm" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>
>
>
>
> -- 
> Glauber  Costa.
> "Free as in Freedom"
> http://glommer.net
>
> "The less confident you are, the more serious you have to act."

Re: VMX ideas

[Alexander Graf]

Am 02.11.2008 um 11:28 schrieb Avi Kivity <avi@redhat.com>:

> Gerd Hoffmann wrote:
>>> 3. Create a root mode bloat framework
>>>
>>
>> 4. Enter vmx root mode when the first user opens /dev/kvm and leave  
>> it
>> when nobody uses it any more?
>>
>> Also note that there has been some discussion on lkml to move vmx- 
>> root
>> on/off out of the kvm module into the kernel.  There are a few places
>> where it would be useful for the kernel to handle that without the  
>> kvm
>> module being involved, crashdump kexec for example.  With that and  
>> some
>> locking in place (so only one can use vmx-root at a time) and
>> hypervisors being friendly to each other and grab vmx-root only when
>> they have a guest to run we should be set
>
> That means that although kvm is loaded it may be unable to service / 
> dev/kvm...  something I'd like to avoid.

This would happen even with a vmx framework in place - unless we  
really always activate root mode, which would break 32-bit vmware/vbox.

The way I have it implemented atm, root mode is activated on first  
usage. If kvm happens to be blocked out of using vmx due to various  
reasons, create_vcpu would fail. What exactly is bad about this?

And even if we were to always activate root mode, a vmx user that  
needs exclusive access (e.g. uses vpids) could still block kvm.

Alex

>
>
> -- 
> error compiling committee.c: too many arguments to function
>

Re: VMX ideas

[Daniel P. Berrange]

On Mon, Nov 03, 2008 at 08:24:49AM +0100, Alexander Graf wrote:
> 
> 
> Am 03.11.2008 um 00:36 schrieb "Glauber Costa" <glommer@gmail.com>:
> 
> >On Sun, Nov 2, 2008 at 2:18 PM, Avi Kivity <avi@redhat.com> wrote:
> >>Avi Kivity wrote:
> >>>
> >>>Daniel P. Berrange wrote:
> >>>>
> >>>>Yes, on Fedora, if you install the 'kvm' userspace package, we  
> >>>>drop a
> >>>>script into /etc/sysconfig/modules/kvm.modules that cause kvm- 
> >>>>intel.ko
> >>>>or kvm-amd.ko to be loaded at boot time.
> >>>>
> >>>
> >>>Does the postinstall also load the modules?  If not, users who don't
> >>>reboot are in for an unpleasant surprise.
> >>>
> >>
> >>I checked and it doesn't.
> >Funny. Rawhide version does exactly that.
> 
> So when a Fedora user installs the kvm userspace, he can't run vmware/ 
> parallels/vbox and the like without fiddling?

Fedora focuses on making the best possible free software experiance, 
without compromising it for sake of closed source software. If vendors
of out of tree kernel modules want a good experiance for their users,
they can work to get the neccessary support merged in upstream kernel
trees. Many vendors over the years have shipped out of tree network
or disk drivers - we don't avoid loading the in kernel driver just in
case in conflicts with a out of tree replacement nic/disk module. The
same applies for virtualization - one in tree kernel module should be
made to be sufficient for all userspace apps that want to use virt.
KVM is the only in tree virt module at this time, so we load it by
default. If the KVM module isn't sufficient to other userspace virt 
apps, then they need to work with the kernel community to address that,
not create many out of tree modules duplicating each other's functionality.


Daniel
-- 
|: Red Hat, Engineering, London   -o-   http://people.redhat.com/berrange/ :|
|: http://libvirt.org  -o-  http://virt-manager.org  -o-  http://ovirt.org :|
|: http://autobuild.org       -o-         http://search.cpan.org/~danberr/ :|
|: GnuPG: 7D3B9505  -o-  F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 :|