Linux in VirtualPC in KVM fails

Linux in VirtualPC in KVM fails

[Kevin Wolf]

Hi,

let's start with the scenario I tried to use: I have two levels of
virtualization. On the physical hardware I run a Linux with KVM. The KVM
guest is a Win2k3 VM which runs VirtualPC. In VirtualPC I try to run a
Linux again (openSUSE 11.1 to be specific, but that shouldn't matter).

The boot menu comes up nicely and so on, but early in the kernel boot it
crashes:

EIP is at kvm_deferred_mmu_op+0x46/0xbf
Call Trace:
 [<c0117f79>] kvm_mmu_write+0x59/0x61
 [<c011bad9>] set_pte_vaddr+0x95/0xec
 [<c011b3b2>] __native_set_fixmap+0x1d/0x24
 [<c054ae5b>] test_wp_bit+0x24/0x6c
 [<c054b6b1>] mem_init+0x295/0x2b8
 [<c053a8a3>] start_kernel+0x262/0x31f

Now obviously this is a KVM function where there should be none. The
problem seems to be that VirtualPC doesn't intercept cpuid and thus the
VirtualPC guest sees the KVM cpuid values where it better wouldn't.
Consequently, it turns on the paravirt support for KVM which is exactly
wrong and leads to the crash on the first hypercall.

The guest has no chance to detect correctly if it's running directly on
KVM or if there is another virtualization layer which can't emulate
cpuid. So the fix must involve the mechanism itself. Alex has suggested
to change the interface to use a KVM-specific MSR instead of cpuid as
these should be handled by any virtualization software. I'm copying him
so he can take over for the details, I just want to get the discussion
started.

So... Comments? Suggestions? Patches? ;-)

Kevin

Re: Linux in VirtualPC in KVM fails

[Avi Kivity]

Kevin Wolf wrote:
> Hi,
>
> let's start with the scenario I tried to use: I have two levels of
> virtualization. On the physical hardware I run a Linux with KVM. The KVM
> guest is a Win2k3 VM which runs VirtualPC. In VirtualPC I try to run a
> Linux again (openSUSE 11.1 to be specific, but that shouldn't matter).
>
> The boot menu comes up nicely and so on, but early in the kernel boot it
> crashes:
>
> EIP is at kvm_deferred_mmu_op+0x46/0xbf
> Call Trace:
>  [<c0117f79>] kvm_mmu_write+0x59/0x61
>  [<c011bad9>] set_pte_vaddr+0x95/0xec
>  [<c011b3b2>] __native_set_fixmap+0x1d/0x24
>  [<c054ae5b>] test_wp_bit+0x24/0x6c
>  [<c054b6b1>] mem_init+0x295/0x2b8
>  [<c053a8a3>] start_kernel+0x262/0x31f
>
> Now obviously this is a KVM function where there should be none. The
> problem seems to be that VirtualPC doesn't intercept cpuid and thus the
> VirtualPC guest sees the KVM cpuid values where it better wouldn't.
> Consequently, it turns on the paravirt support for KVM which is exactly
> wrong and leads to the crash on the first hypercall.
>
> The guest has no chance to detect correctly if it's running directly on
> KVM or if there is another virtualization layer which can't emulate
> cpuid. So the fix must involve the mechanism itself. Alex has suggested
> to change the interface to use a KVM-specific MSR instead of cpuid as
> these should be handled by any virtualization software. I'm copying him
> so he can take over for the details, I just want to get the discussion
> started.
>
> So... Comments? Suggestions? Patches? ;-)
>   

Gaa.  Looks like cpuid is totally broken by first-generation 
virtualization products.

We're in a nice compatibility mess.  We can't just switch paravirt 
detection methods since that will break older guests _and_ older hosts 
on non-nested virtualization (which is the common case).

So I suggest:
- add a new cpuid bit, for 'paravirt verification'
- if the new bit is present, the guest verifies that paravirt is truly 
enabled by reading an msr (and checking a signature).

Older guests or newer guests running on older hosts will still be 
broken, but there's a command line parameter for disabling paravirt on 
the guest, and we can add one for the host as well.

-- 
I have a truly marvellous patch that fixes the bug which this
signature is too narrow to contain.

Re: Linux in VirtualPC in KVM fails

[Alexander Graf]

On 09.01.2009, at 18:50, Avi Kivity wrote:

> Kevin Wolf wrote:
>> Hi,
>>
>> let's start with the scenario I tried to use: I have two levels of
>> virtualization. On the physical hardware I run a Linux with KVM.  
>> The KVM
>> guest is a Win2k3 VM which runs VirtualPC. In VirtualPC I try to  
>> run a
>> Linux again (openSUSE 11.1 to be specific, but that shouldn't  
>> matter).
>>
>> The boot menu comes up nicely and so on, but early in the kernel  
>> boot it
>> crashes:
>>
>> EIP is at kvm_deferred_mmu_op+0x46/0xbf
>> Call Trace:
>> [<c0117f79>] kvm_mmu_write+0x59/0x61
>> [<c011bad9>] set_pte_vaddr+0x95/0xec
>> [<c011b3b2>] __native_set_fixmap+0x1d/0x24
>> [<c054ae5b>] test_wp_bit+0x24/0x6c
>> [<c054b6b1>] mem_init+0x295/0x2b8
>> [<c053a8a3>] start_kernel+0x262/0x31f
>>
>> Now obviously this is a KVM function where there should be none. The
>> problem seems to be that VirtualPC doesn't intercept cpuid and thus  
>> the
>> VirtualPC guest sees the KVM cpuid values where it better wouldn't.
>> Consequently, it turns on the paravirt support for KVM which is  
>> exactly
>> wrong and leads to the crash on the first hypercall.
>>
>> The guest has no chance to detect correctly if it's running  
>> directly on
>> KVM or if there is another virtualization layer which can't emulate
>> cpuid. So the fix must involve the mechanism itself. Alex has  
>> suggested
>> to change the interface to use a KVM-specific MSR instead of cpuid as
>> these should be handled by any virtualization software. I'm copying  
>> him
>> so he can take over for the details, I just want to get the  
>> discussion
>> started.
>>
>> So... Comments? Suggestions? Patches? ;-)
>>
>
> Gaa.  Looks like cpuid is totally broken by first-generation  
> virtualization products.
>
> We're in a nice compatibility mess.  We can't just switch paravirt  
> detection methods since that will break older guests _and_ older  
> hosts on non-nested virtualization (which is the common case).

Shouldn't it be ok to push patches to linux-stable to use the CPUID  
and MSR information and simply not expose the CPUID identification in  
newer KVM versions? That way older guests on newer KVM don't use KVM  
paravirt (which should still be ok), but everything else runs as  
smoothly as possible.

Alex

> So I suggest:
> - add a new cpuid bit, for 'paravirt verification'
> - if the new bit is present, the guest verifies that paravirt is  
> truly enabled by reading an msr (and checking a signature).
>
> Older guests or newer guests running on older hosts will still be  
> broken, but there's a command line parameter for disabling paravirt  
> on the guest, and we can add one for the host as well.
>
> -- 
> I have a truly marvellous patch that fixes the bug which this
> signature is too narrow to contain.
>

Re: Linux in VirtualPC in KVM fails

[Anthony Liguori]

Alexander Graf wrote:
>
> Shouldn't it be ok to push patches to linux-stable to use the CPUID 
> and MSR information and simply not expose the CPUID identification in 
> newer KVM versions? That way older guests on newer KVM don't use KVM 
> paravirt (which should still be ok), but everything else runs as 
> smoothly as possible.

I'd say that VirtualPC is broken and leave it at that.

All modern hypervisors use CPUID to expose themselves to the guests.  
VirtualPC is totally capable of rewriting CPUID instructions.  Shame on 
it for not doing so.  I don't think we should change anything.

Regards,

Anthony Liguori

> Alex
>
>> So I suggest:
>> - add a new cpuid bit, for 'paravirt verification'
>> - if the new bit is present, the guest verifies that paravirt is 
>> truly enabled by reading an msr (and checking a signature).
>>
>> Older guests or newer guests running on older hosts will still be 
>> broken, but there's a command line parameter for disabling paravirt 
>> on the guest, and we can add one for the host as well.
>>
>> -- 
>> I have a truly marvellous patch that fixes the bug which this
>> signature is too narrow to contain.
>>
>
> -- 
> To unsubscribe from this list: send the line "unsubscribe kvm" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Linux in VirtualPC in KVM fails

[Anthony Liguori]

Avi Kivity wrote:
> Kevin Wolf wrote:
>> Hi,
>>
>> let's start with the scenario I tried to use: I have two levels of
>> virtualization. On the physical hardware I run a Linux with KVM. The KVM
>> guest is a Win2k3 VM which runs VirtualPC. In VirtualPC I try to run a
>> Linux again (openSUSE 11.1 to be specific, but that shouldn't matter).
>>
>> The boot menu comes up nicely and so on, but early in the kernel boot it
>> crashes:
>>
>> EIP is at kvm_deferred_mmu_op+0x46/0xbf
>> Call Trace:
>>  [<c0117f79>] kvm_mmu_write+0x59/0x61
>>  [<c011bad9>] set_pte_vaddr+0x95/0xec
>>  [<c011b3b2>] __native_set_fixmap+0x1d/0x24
>>  [<c054ae5b>] test_wp_bit+0x24/0x6c
>>  [<c054b6b1>] mem_init+0x295/0x2b8
>>  [<c053a8a3>] start_kernel+0x262/0x31f
>>
>> Now obviously this is a KVM function where there should be none. The
>> problem seems to be that VirtualPC doesn't intercept cpuid and thus the
>> VirtualPC guest sees the KVM cpuid values where it better wouldn't.
>> Consequently, it turns on the paravirt support for KVM which is exactly
>> wrong and leads to the crash on the first hypercall.
>>
>> The guest has no chance to detect correctly if it's running directly on
>> KVM or if there is another virtualization layer which can't emulate
>> cpuid. So the fix must involve the mechanism itself. Alex has suggested
>> to change the interface to use a KVM-specific MSR instead of cpuid as
>> these should be handled by any virtualization software. I'm copying him
>> so he can take over for the details, I just want to get the discussion
>> started.
>>
>> So... Comments? Suggestions? Patches? ;-)
>>   
>
> Gaa.  Looks like cpuid is totally broken by first-generation 
> virtualization products.

Not at all.  There's no reason that a JIT'ing virtualization product 
can't rewrite CPUID to a function call and then mask off unsupported 
bits.  It's a bug in the virtualization product if it doesn't do this.

Regards,

Anthony Liguori

Re: Linux in VirtualPC in KVM fails

[Alexander Graf]

On 09.01.2009, at 20:05, Anthony Liguori wrote:

> Avi Kivity wrote:
>> Kevin Wolf wrote:
>>> Hi,
>>>
>>> let's start with the scenario I tried to use: I have two levels of
>>> virtualization. On the physical hardware I run a Linux with KVM.  
>>> The KVM
>>> guest is a Win2k3 VM which runs VirtualPC. In VirtualPC I try to  
>>> run a
>>> Linux again (openSUSE 11.1 to be specific, but that shouldn't  
>>> matter).
>>>
>>> The boot menu comes up nicely and so on, but early in the kernel  
>>> boot it
>>> crashes:
>>>
>>> EIP is at kvm_deferred_mmu_op+0x46/0xbf
>>> Call Trace:
>>> [<c0117f79>] kvm_mmu_write+0x59/0x61
>>> [<c011bad9>] set_pte_vaddr+0x95/0xec
>>> [<c011b3b2>] __native_set_fixmap+0x1d/0x24
>>> [<c054ae5b>] test_wp_bit+0x24/0x6c
>>> [<c054b6b1>] mem_init+0x295/0x2b8
>>> [<c053a8a3>] start_kernel+0x262/0x31f
>>>
>>> Now obviously this is a KVM function where there should be none. The
>>> problem seems to be that VirtualPC doesn't intercept cpuid and  
>>> thus the
>>> VirtualPC guest sees the KVM cpuid values where it better wouldn't.
>>> Consequently, it turns on the paravirt support for KVM which is  
>>> exactly
>>> wrong and leads to the crash on the first hypercall.
>>>
>>> The guest has no chance to detect correctly if it's running  
>>> directly on
>>> KVM or if there is another virtualization layer which can't emulate
>>> cpuid. So the fix must involve the mechanism itself. Alex has  
>>> suggested
>>> to change the interface to use a KVM-specific MSR instead of cpuid  
>>> as
>>> these should be handled by any virtualization software. I'm  
>>> copying him
>>> so he can take over for the details, I just want to get the  
>>> discussion
>>> started.
>>>
>>> So... Comments? Suggestions? Patches? ;-)
>>>
>>
>> Gaa.  Looks like cpuid is totally broken by first-generation  
>> virtualization products.
>
> Not at all.  There's no reason that a JIT'ing virtualization product  
> can't rewrite CPUID to a function call and then mask off unsupported  
> bits.  It's a bug in the virtualization product if it doesn't do this.

Isn't one of the great things about virtualization the fact that you  
can do things you can on real hardware in the virtual machine? While  
I'm not exactly a fan of VirtualPC, I would still like it to work in  
KVM, as that's what real hardware is capable of.

Alex

Re: Linux in VirtualPC in KVM fails

[Avi Kivity]

Anthony Liguori wrote:
> Alexander Graf wrote:
>>
>> Shouldn't it be ok to push patches to linux-stable to use the CPUID 
>> and MSR information and simply not expose the CPUID identification in 
>> newer KVM versions? That way older guests on newer KVM don't use KVM 
>> paravirt (which should still be ok), but everything else runs as 
>> smoothly as possible.
>
> I'd say that VirtualPC is broken and leave it at that.
>
> All modern hypervisors use CPUID to expose themselves to the guests.  
> VirtualPC is totally capable of rewriting CPUID instructions.  

Apparently it isn't.  I know of one other fullvirt product that does not 
jit kernel code.  Or maybe they wanted to preserve consistency between 
kernel cpuid and host cpuid.

> Shame on it for not doing so.  I don't think we should change anything.

PC hardware and software are full of examples of such brokenness.  One 
of the most painful examples is Linux using the tsc for time of day 
despite the fact that the tsc was explicitly advertised not to be fit 
for that role.

If there's an easy and backwards compatible way to support Virtual PC, 
we should implement it.

-- 
I have a truly marvellous patch that fixes the bug which this
signature is too narrow to contain.

Re: Linux in VirtualPC in KVM fails

[Avi Kivity]

Alexander Graf wrote:
>> We're in a nice compatibility mess.  We can't just switch paravirt 
>> detection methods since that will break older guests _and_ older 
>> hosts on non-nested virtualization (which is the common case).
>
> Shouldn't it be ok to push patches to linux-stable to use the CPUID 
> and MSR information and simply not expose the CPUID identification in 
> newer KVM versions? That way older guests on newer KVM don't use KVM 
> paravirt (which should still be ok), but everything else runs as 
> smoothly as possible.

No.  It's a regression (esp. kvmclock).

It's perfectly legitimate to say we support Virtual PC beginning some 
kvm version (and to supply the command line workaround for older 
versions).  But we shouldn't knowingly break existing setups.

-- 
I have a truly marvellous patch that fixes the bug which this
signature is too narrow to contain.

Re: Linux in VirtualPC in KVM fails

[Anthony Liguori]

Alexander Graf wrote:
>
> Isn't one of the great things about virtualization the fact that you 
> can do things you can on real hardware in the virtual machine? While 
> I'm not exactly a fan of VirtualPC, I would still like it to work in 
> KVM, as that's what real hardware is capable of.

Paravirtualization support is advertised in userspace, just add a flag 
to QEMU to disable KVM paravirt support.

Regards,

Anthony Liguori

> Alex
>

Re: Linux in VirtualPC in KVM fails

[Anthony Liguori]

Avi Kivity wrote:
> Anthony Liguori wrote:
>> Alexander Graf wrote:
>>>
>>> Shouldn't it be ok to push patches to linux-stable to use the CPUID 
>>> and MSR information and simply not expose the CPUID identification 
>>> in newer KVM versions? That way older guests on newer KVM don't use 
>>> KVM paravirt (which should still be ok), but everything else runs as 
>>> smoothly as possible.
>>
>> I'd say that VirtualPC is broken and leave it at that.
>>
>> All modern hypervisors use CPUID to expose themselves to the guests.  
>> VirtualPC is totally capable of rewriting CPUID instructions.  
>
> Apparently it isn't.  I know of one other fullvirt product that does 
> not jit kernel code.  Or maybe they wanted to preserve consistency 
> between kernel cpuid and host cpuid.

How can you get away with not jitting kernel code?

Regards,

Anthony Liguori

Re: Linux in VirtualPC in KVM fails

[Avi Kivity]

Anthony Liguori wrote:
>>
>> Apparently it isn't.  I know of one other fullvirt product that does 
>> not jit kernel code.  Or maybe they wanted to preserve consistency 
>> between kernel cpuid and host cpuid.
>
> How can you get away with not jitting kernel code?
>

http://bellard.org/qemu/kqemu-tech.html#SEC12

-- 
error compiling committee.c: too many arguments to function

Re: Linux in VirtualPC in KVM fails

[Avi Kivity]

Anthony Liguori wrote:
> Alexander Graf wrote:
>>
>> Isn't one of the great things about virtualization the fact that you 
>> can do things you can on real hardware in the virtual machine? While 
>> I'm not exactly a fan of VirtualPC, I would still like it to work in 
>> KVM, as that's what real hardware is capable of.
>
> Paravirtualization support is advertised in userspace, just add a flag 
> to QEMU to disable KVM paravirt support.

That's helpful.  But I would like (in addition) a fix that does not 
require administrative involvement.

The host admin should not need to know anything about what the guest 
runs.  That's not achievable in practice, but where it is, we want it.

-- 
error compiling committee.c: too many arguments to function