Linux with kvm-intel locks up VMplayer guest is started

Linux with kvm-intel locks up VMplayer guest is started

[Martin Michlmayr]

When I start a guest in vmplayer, my laptop locks up hard.  I found
out that this only happens when the kvm-intel module has been loaded.
When I unload this module, I can start vmware guests just fine.

I see this problem on a HP Compaq 2510p notebook with Intel Core2 Duo
U7600 @ 1.20GHz, but http://laptop006.livejournal.com/41314.html
describes exactly the same problem on a Lenovo T61 (custom build),
Core 2 T7500 on a Santa Rosa.  I've seen this with at least 2.6.24 and
2.6.25.

I don't have a serial console, but I started VMplayer and
immediately switched to the console but nothing is displayed
(i.e. no kernel panic; it simply locks up hard).

Any idea if this is a kernel bug, BIOS issue or a bug in VMplayer?

-- 
Martin Michlmayr
http://www.cyrius.com/

Re: Linux with kvm-intel locks up VMplayer guest is started

[Anthony Liguori]

Martin Michlmayr wrote:
> When I start a guest in vmplayer, my laptop locks up hard.  I found
> out that this only happens when the kvm-intel module has been loaded.
> When I unload this module, I can start vmware guests just fine.
>
> I see this problem on a HP Compaq 2510p notebook with Intel Core2 Duo
> U7600 @ 1.20GHz, but http://laptop006.livejournal.com/41314.html
> describes exactly the same problem on a Lenovo T61 (custom build),
> Core 2 T7500 on a Santa Rosa.  I've seen this with at least 2.6.24 and
> 2.6.25.
>
> I don't have a serial console, but I started VMplayer and
> immediately switched to the console but nothing is displayed
> (i.e. no kernel panic; it simply locks up hard).
>
> Any idea if this is a kernel bug, BIOS issue or a bug in VMplayer?
>   

This has been fixed in the latest KVM code base AFAIK.

Regards,

Anthony Liguori

Re: Linux with kvm-intel locks up VMplayer guest is started

[Martin Michlmayr]

* Anthony Liguori <anthony@codemonkey.ws> [2008-06-16 10:21]:
>> Any idea if this is a kernel bug, BIOS issue or a bug in VMplayer?
>
> This has been fixed in the latest KVM code base AFAIK.

Can you give some more information?  Which commit fixed the problem?
Was the patch submitted to the 2.6.25-stable tree?
-- 
Martin Michlmayr
http://www.cyrius.com/

Re: Linux with kvm-intel locks up VMplayer guest is started

[Anthony Liguori]

Martin Michlmayr wrote:
> * Anthony Liguori <anthony@codemonkey.ws> [2008-06-16 10:21]:
>   
>>> Any idea if this is a kernel bug, BIOS issue or a bug in VMplayer?
>>>       
>> This has been fixed in the latest KVM code base AFAIK.
>>     
>
> Can you give some more information?  Which commit fixed the problem?
>   

commit 1f4e1d64122369d65d8a9c616112780019fb6188
Author: Eli Collins <ecollins@vmware.com>
Date:   Sun Jun 1 20:24:40 2008 -0700

> Was the patch submitted to the 2.6.25-stable tree?
>   

No, it's not a regression.  I misread your first email btw, you have to 
unload the kvm-intel kernel module to use VMware.

VMware is a binary kernel module that's out of kernel.  KVM is not 
misbehaving and the fact that VMware breaks when the KVM module is 
loaded isn't our problem.  If they submitted their code for inclusion in 
mainline, we could possibly come up with solution for arbitrating who is 
using VT.

Regards,

Anthony Liguori

Re: Linux with kvm-intel locks up VMplayer guest is started

[Martin Michlmayr]

* Anthony Liguori <anthony@codemonkey.ws> [2008-06-17 08:16]:
> VMware is a binary kernel module that's out of kernel.  KVM is not
> misbehaving and the fact that VMware breaks when the KVM module is
> loaded isn't our problem.  If they submitted their code for
> inclusion in  mainline, we could possibly come up with solution for
> arbitrating who is  using VT.

I feared I'd get a response like this.  But unless this is a known
issue in VMware (which I don't think it is), you don't know whether
it's not a bug in kvm-intel.

-- 
Martin Michlmayr
http://www.cyrius.com/

Re: Linux with kvm-intel locks up VMplayer guest is started

[Martin Michlmayr]

* Martin Michlmayr <tbm@cyrius.com> [2008-06-17 15:32]:
> * Anthony Liguori <anthony@codemonkey.ws> [2008-06-17 08:16]:
> > VMware is a binary kernel module that's out of kernel.  KVM is not
> > misbehaving and the fact that VMware breaks when the KVM module is
> > loaded isn't our problem.  If they submitted their code for
> > inclusion in  mainline, we could possibly come up with solution for
> > arbitrating who is  using VT.
> 
> I feared I'd get a response like this.  But unless this is a known
> issue in VMware (which I don't think it is), you don't know whether
> it's not a bug in kvm-intel.

Sorry, Anthony, I just realized I misparsed your response.  So you're
saying that it's a known issue and that VMware is the problem.  Thanks
a lot!  I'll take it up with VMware.

-- 
Martin Michlmayr
http://www.cyrius.com/

Re: Linux with kvm-intel locks up VMplayer guest is started

[Javier Guerra]

On Tue, Jun 17, 2008 at 8:44 AM, Martin Michlmayr <tbm@cyrius.com> wrote:
> Sorry, Anthony, I just realized I misparsed your response.  So you're
> saying that it's a known issue and that VMware is the problem.  Thanks
> a lot!  I'll take it up with VMware.

i think what he's saying is that VMWare is a closed binary blob
executing in the kernel; so there's no way to certify anything with
this.

as soon as you put some unknown (and unknowable, unverifiable,
untrustable) code in the kernel, you can't know what will work and
what won't.

-- 
Javier

Re: Linux with kvm-intel locks up VMplayer guest is started

[Alan Cox]

On Tue, 17 Jun 2008 15:32:34 +0200
Martin Michlmayr <tbm@cyrius.com> wrote:

> * Anthony Liguori <anthony@codemonkey.ws> [2008-06-17 08:16]:
> > VMware is a binary kernel module that's out of kernel.  KVM is not
> > misbehaving and the fact that VMware breaks when the KVM module is
> > loaded isn't our problem.  If they submitted their code for
> > inclusion in  mainline, we could possibly come up with solution for
> > arbitrating who is  using VT.
> 
> I feared I'd get a response like this.  But unless this is a known
> issue in VMware (which I don't think it is), you don't know whether
> it's not a bug in kvm-intel.

So ask vmware. They have source to both parts we don't.

And its a bug in vmware by definition. If the emulator differs from the
real thing then the emulator is at fault. It might well be that the limit
is a genuine technical limit - but its still an emulator fault.

Alan

Re: Linux with kvm-intel locks up VMplayer guest is started

[Anthony Liguori]

Martin Michlmayr wrote:
> * Anthony Liguori <anthony@codemonkey.ws> [2008-06-17 08:16]:
>   
>> VMware is a binary kernel module that's out of kernel.  KVM is not
>> misbehaving and the fact that VMware breaks when the KVM module is
>> loaded isn't our problem.  If they submitted their code for
>> inclusion in  mainline, we could possibly come up with solution for
>> arbitrating who is  using VT.
>>     
>
> I feared I'd get a response like this.  But unless this is a known
> issue in VMware (which I don't think it is), you don't know whether
> it's not a bug in kvm-intel.
>   

We know exactly what the problem is.  KVM activates VT unconditionally.  
There's no hardware mechanism to arbitrate access to VT.  KVM is the 
only thing in the Linux kernel that uses VT so we don't have a software 
mechanism to arbitrate access to VT.

If the VMware code was upstream, then we could work together to make a 
software arbitration mechanism.  It's not, and worse yet, it's closed 
source so there's no chance it will be.  Even if someone wrote an 
arbitration mechanism and got VMware to use it, it still shouldn't be 
merged because KVM would be the only thing using that mechanism 
upstream.  I'm not interested in adding kernel infrastructure to support 
external binary kernel modules.

Regards,

Anthony Liguori

Re: Linux with kvm-intel locks up VMplayer guest is started

[Avi Kivity]

Anthony Liguori wrote:
>
> We know exactly what the problem is.  KVM activates VT 
> unconditionally.  There's no hardware mechanism to arbitrate access to 
> VT.  KVM is the only thing in the Linux kernel that uses VT so we 
> don't have a software mechanism to arbitrate access to VT.
>
> If the VMware code was upstream, then we could work together to make a 
> software arbitration mechanism.  It's not, and worse yet, it's closed 
> source so there's no chance it will be.  Even if someone wrote an 
> arbitration mechanism and got VMware to use it, it still shouldn't be 
> merged because KVM would be the only thing using that mechanism 
> upstream.  I'm not interested in adding kernel infrastructure to 
> support external binary kernel modules.

Well a recent patch from Eli Collins mentioned VMware are standardizing 
CR4.VMXE as an indicator of whether someone is using the VT hardware or 
not, and now kvm clears that bit when unloading.  We could check the bit 
and fail if is set, thus have working mutual exclusion.  Of course, it 
will only work with newer versions of kvm and vmware.

I don't see any reason to poke sticks into the wheels here.

-- 
I have a truly marvellous patch that fixes the bug which this
signature is too narrow to contain.

Re: Linux with kvm-intel locks up VMplayer guest is started

[devzero]

i`d like to give a comment about those claims in this thread (see below),  vmware kernel part would be closed source or binary module.

i think, they arenŽt. they just are not available via public/opensource code repository.

just go and download workstation 6.5 beta from:

http://download3.vmware.com/software/wkst/VMware-workstation-e.x.p-91182.i386.tar.gz
(you may need register for that)

unpack and take a look at vmware-distrib/lib/modules/source - there is source for 6 independent kernel
modules in those .tar files:

vmblock, vmci, vmmon, vmnet, vmppuser and vsock. (~3MB alltogether)

regards
roland


------------------------------------------------------------------------------------

>i think what he's saying is that VMWare is a closed binary blob
>executing in the kernel; so there's no way to certify anything with
>this.

>as soon as you put some unknown (and unknowable, unverifiable,
>untrustable) code in the kernel, you can't know what will work and
>what won't.

------------------------------------------------------------------------------------

>So ask vmware. They have source to both parts we don't."

------------------------------------------------------------------------------------

>If the VMware code was upstream, then we could work together to make a 
>software arbitration mechanism.  It's not, and worse yet, it's closed 
>source so there's no chance it will be.  Even if someone wrote an 
>arbitration mechanism and got VMware to use it, it still shouldn't be 
>merged because KVM would be the only thing using that mechanism 
>upstream.  I'm not interested in adding kernel infrastructure to support 
>external binary kernel modules.

------------------------------------------------------------------------------------

>VMware is a binary kernel module that's out of kernel.  KVM is not 
>misbehaving and the fact that VMware breaks when the KVM module is 
>loaded isn't our problem.  If they submitted their code for inclusion in 
>mainline, we could possibly come up with solution for arbitrating who is 
>using VT.

------------------------------------------------------------------------------------


_______________________________________________________________________
Jetzt neu! Schützen Sie Ihren PC mit McAfee und WEB.DE. 30 Tage
kostenlos testen. http://www.pc-sicherheit.web.de/startseite/?mc=022220

Re: Linux with kvm-intel locks up VMplayer guest is started

[Anthony Liguori]

devzero@web.de wrote:
> i`d like to give a comment about those claims in this thread (see below),  vmware kernel part would be closed source or binary module.
>
> i think, they arenŽt. they just are not available via public/opensource code repository.
>
> just go and download workstation 6.5 beta from:
>
> http://download3.vmware.com/software/wkst/VMware-workstation-e.x.p-91182.i386.tar.gz
> (you may need register for that)
>
> unpack and take a look at vmware-distrib/lib/modules/source - there is source for 6 independent kernel
> modules in those .tar files:
>
> vmblock, vmci, vmmon, vmnet, vmppuser and vsock. (~3MB alltogether)
>   

vmmon is sort of a stub that loads the actual monitor which is a binary 
blob.

Regards,

Anthony Liguori

> regards
> roland
>
>
> ------------------------------------------------------------------------------------
>
>   
>> i think what he's saying is that VMWare is a closed binary blob
>> executing in the kernel; so there's no way to certify anything with
>> this.
>>     
>
>   
>> as soon as you put some unknown (and unknowable, unverifiable,
>> untrustable) code in the kernel, you can't know what will work and
>> what won't.
>>     
>
> ------------------------------------------------------------------------------------
>
>   
>> So ask vmware. They have source to both parts we don't."
>>     
>
> ------------------------------------------------------------------------------------
>
>   
>> If the VMware code was upstream, then we could work together to make a 
>> software arbitration mechanism.  It's not, and worse yet, it's closed 
>> source so there's no chance it will be.  Even if someone wrote an 
>> arbitration mechanism and got VMware to use it, it still shouldn't be 
>> merged because KVM would be the only thing using that mechanism 
>> upstream.  I'm not interested in adding kernel infrastructure to support 
>> external binary kernel modules.
>>     
>
> ------------------------------------------------------------------------------------
>
>   
>> VMware is a binary kernel module that's out of kernel.  KVM is not 
>> misbehaving and the fact that VMware breaks when the KVM module is 
>> loaded isn't our problem.  If they submitted their code for inclusion in 
>> mainline, we could possibly come up with solution for arbitrating who is 
>> using VT.
>>     
>
> ------------------------------------------------------------------------------------
>
>
> _______________________________________________________________________
> Jetzt neu! Schützen Sie Ihren PC mit McAfee und WEB.DE. 30 Tage
> kostenlos testen. http://www.pc-sicherheit.web.de/startseite/?mc=022220
>
>   

Re: Linux with kvm-intel locks up VMplayer guest is started

[Anthony Liguori]

Avi Kivity wrote:
> Anthony Liguori wrote:
>>
>> We know exactly what the problem is.  KVM activates VT 
>> unconditionally.  There's no hardware mechanism to arbitrate access 
>> to VT.  KVM is the only thing in the Linux kernel that uses VT so we 
>> don't have a software mechanism to arbitrate access to VT.
>>
>> If the VMware code was upstream, then we could work together to make 
>> a software arbitration mechanism.  It's not, and worse yet, it's 
>> closed source so there's no chance it will be.  Even if someone wrote 
>> an arbitration mechanism and got VMware to use it, it still shouldn't 
>> be merged because KVM would be the only thing using that mechanism 
>> upstream.  I'm not interested in adding kernel infrastructure to 
>> support external binary kernel modules.
>
> Well a recent patch from Eli Collins mentioned VMware are 
> standardizing CR4.VMXE as an indicator of whether someone is using the 
> VT hardware or not, and now kvm clears that bit when unloading.  We 
> could check the bit and fail if is set, thus have working mutual 
> exclusion.  Of course, it will only work with newer versions of kvm 
> and vmware.

Yup, but the original bug reporter would not be helped by this.

> I don't see any reason to poke sticks into the wheels here.

An occasional poke at binary kernel modules is always a good thing IMHO :-)

Regards,

Anthony Liguori

Re: Linux with kvm-intel locks up VMplayer guest is started

[Eli Collins]

Hi Martin,

This lockup is fixed in the latest dot releases of Workstation, Player, 
and Server.

The vmmon module now checks if VT is already enabled (actually it just 
tests if CR4.VMXE is set since checking if the cpu is in VMX mode is a 
pain). The vmmon module enables VMX mode when world switching to our 
monitor and disables it when world switching back to the host. The 
kvm-intel module puts the cpu(s) in VMX mode and does not exit VMX mode 
until the module is unloaded, so if you want to run player and kvm you 
need to first unload the kvm-intel module. However, this only works with 
very recent kvm modules. If you're using a kvm-intel module that doesn't 
contain the patch I submitted a couple weeks ago then even unloading the 
kvm-intel module is not enough since it leaves CR4.VMXE set. So for "old" 
modules you need to not load the kvm-intel module in the first place.

Note that the above is only relevant to kvm-intel. The vmmon and kvm-amd 
modules do not need exclusive access to SVM mode. I've run VMs on kvm-69 
and Workstation 6.5 beta simultaneously on an AMD-V host without any 
issues.

Also note that this issue is relevant for any software that wants to use 
VT: QEMU, VirtualBox, Parallels, etc. won't be able to if the host (via 
the kvm-intel module) has already claimed it.

Thanks,
Eli