win2k guest vm won't boot under Fedora 18 KVM

win2k guest vm won't boot under Fedora 18 KVM

[Earl Marwil]

Hi,

I'm looking for some guidance on how to get to the root cause of an
issue that I am observing with a win2k guest that won't boot under
Fedora 18 on one system but will boot on another. A few days ago I
posted on the fedora forum:

http://forums.fedoraforum.org/showthread.php?t=289401

I can repeat the details in this thread if requested. The issue is that,
with a fresh build of Fedora 18, updated to the most recent kernel and
packages on an external USB ssd, my win2k VM boots on my laptop (Core
i73720QM processor) but does not boot on my desktop system (Core i7-870
processor).

I'm not sure whether this is a kvm issue or a kernel issue. I'll be glad
to dig deeper, just let me what information is needed.

Thanks.

Re: win2k guest vm won't boot under Fedora 18 KVM

[Stefan Hajnoczi]

On Sat, Mar 09, 2013 at 12:43:32PM -0700, Earl Marwil wrote:
> Hi,
> 
> I'm looking for some guidance on how to get to the root cause of an
> issue that I am observing with a win2k guest that won't boot under
> Fedora 18 on one system but will boot on another. A few days ago I
> posted on the fedora forum:
> 
> http://forums.fedoraforum.org/showthread.php?t=289401
> 
> I can repeat the details in this thread if requested. The issue is that,
> with a fresh build of Fedora 18, updated to the most recent kernel and
> packages on an external USB ssd, my win2k VM boots on my laptop (Core
> i73720QM processor) but does not boot on my desktop system (Core i7-870
> processor).
> 
> I'm not sure whether this is a kvm issue or a kernel issue. I'll be glad
> to dig deeper, just let me what information is needed.

Hi Earl,
>From your forum post:

  KVM internal error. Suberror: 1
  emulation failure
  EAX=63700200 EBX=0000e6f5 ECX=0000000f EDX=00000936
  ...
  Code=74 1d b0 37 e6 70 eb 00 e4 71 eb 00 32 e4 c1 c0 04 c0 c8 04 <d5> 0a 3d 13 0
  0 75 04 b8 7a 15 c3 b8 00 00 c3 55 8b ec 1e 06 56 57 8b 46 04 8e d8 8b 76 06

Here is my guess:

Laptop has a CPU from 2012.  Desktop has a CPU from 2009.

Intel added "unrestricted guest" support to VMX.  This feature allows
the CPU to run real mode code in guest mode.

CPUs that do not support "unrestricted guest" (your desktop?) use an
emulator implemented in software inside the kvm.ko kernel module.

The emulator may be unable to handle the real mode instruction in the
particular kernel version you are running.

The laptop doesn't hit this issue because it supports "unrestricted
guest" while the desktop falls back to the emulator inside kvm.ko where
it hits the bug.

You may find that changing kernel versions on the desktop will make it
work.

The best would be to compile a vanilla Linux kernel for the desktop
machine to verify that this issue still happens.  If so, please post the
full "KVM internal error" output to this mailing list and hopefully
someone can fix the emulator.

Problem with my theory: I haven't figured out how to check which Intel
CPU models support "unrestricted guest", so I'm not 100% sure this is
the issue.

Stefan

Re: win2k guest vm won't boot under Fedora 18 KVM

[Gleb Natapov]

On Mon, Mar 11, 2013 at 09:09:13AM +0100, Stefan Hajnoczi wrote:
> On Sat, Mar 09, 2013 at 12:43:32PM -0700, Earl Marwil wrote:
> > Hi,
> > 
> > I'm looking for some guidance on how to get to the root cause of an
> > issue that I am observing with a win2k guest that won't boot under
> > Fedora 18 on one system but will boot on another. A few days ago I
> > posted on the fedora forum:
> > 
> > http://forums.fedoraforum.org/showthread.php?t=289401
> > 
> > I can repeat the details in this thread if requested. The issue is that,
> > with a fresh build of Fedora 18, updated to the most recent kernel and
> > packages on an external USB ssd, my win2k VM boots on my laptop (Core
> > i73720QM processor) but does not boot on my desktop system (Core i7-870
> > processor).
> > 
> > I'm not sure whether this is a kvm issue or a kernel issue. I'll be glad
> > to dig deeper, just let me what information is needed.
> 
> Hi Earl,
> >From your forum post:
> 
>   KVM internal error. Suberror: 1
>   emulation failure
>   EAX=63700200 EBX=0000e6f5 ECX=0000000f EDX=00000936
>   ...
>   Code=74 1d b0 37 e6 70 eb 00 e4 71 eb 00 32 e4 c1 c0 04 c0 c8 04 <d5> 0a 3d 13 0
>   0 75 04 b8 7a 15 c3 b8 00 00 c3 55 8b ec 1e 06 56 57 8b 46 04 8e d8 8b 76 06
> 
> Here is my guess:
> 
> Laptop has a CPU from 2012.  Desktop has a CPU from 2009.
> 
> Intel added "unrestricted guest" support to VMX.  This feature allows
> the CPU to run real mode code in guest mode.
> 
> CPUs that do not support "unrestricted guest" (your desktop?) use an
> emulator implemented in software inside the kvm.ko kernel module.
> 
> The emulator may be unable to handle the real mode instruction in the
> particular kernel version you are running.
> 
> The laptop doesn't hit this issue because it supports "unrestricted
> guest" while the desktop falls back to the emulator inside kvm.ko where
> it hits the bug.
> 
> You may find that changing kernel versions on the desktop will make it
> work.
> 
> The best would be to compile a vanilla Linux kernel for the desktop
> machine to verify that this issue still happens.  If so, please post the
> full "KVM internal error" output to this mailing list and hopefully
> someone can fix the emulator.
> 
> Problem with my theory: I haven't figured out how to check which Intel
> CPU models support "unrestricted guest", so I'm not 100% sure this is
> the issue.
> 
Your theory is correct. This is the same as [1] which is fixed in 3.9.
Depending on your kernel emulate_invalid_guest_state = 0 may help, or
may make things even worse :(

[1] https://bugzilla.kernel.org/show_bug.cgi?id=50921

--
			Gleb.

Re: win2k guest vm won't boot under Fedora 18 KVM

[Earl Marwil]

On 03/11/2013 02:27 AM, Gleb Natapov wrote:
> On Mon, Mar 11, 2013 at 09:09:13AM +0100, Stefan Hajnoczi wrote:
>> On Sat, Mar 09, 2013 at 12:43:32PM -0700, Earl Marwil wrote:
>>> Hi,
>>>
>>> I'm looking for some guidance on how to get to the root cause of an
>>> issue that I am observing with a win2k guest that won't boot under
>>> Fedora 18 on one system but will boot on another. A few days ago I
>>> posted on the fedora forum:
>>>
>>> http://forums.fedoraforum.org/showthread.php?t=289401
>>>
>>> I can repeat the details in this thread if requested. The issue is that,
>>> with a fresh build of Fedora 18, updated to the most recent kernel and
>>> packages on an external USB ssd, my win2k VM boots on my laptop (Core
>>> i73720QM processor) but does not boot on my desktop system (Core i7-870
>>> processor).
>>>
>>> I'm not sure whether this is a kvm issue or a kernel issue. I'll be glad
>>> to dig deeper, just let me what information is needed.
>> Hi Earl,
>> >From your forum post:
>>
>>   KVM internal error. Suberror: 1
>>   emulation failure
>>   EAX=63700200 EBX=0000e6f5 ECX=0000000f EDX=00000936
>>   ...
>>   Code=74 1d b0 37 e6 70 eb 00 e4 71 eb 00 32 e4 c1 c0 04 c0 c8 04 <d5> 0a 3d 13 0
>>   0 75 04 b8 7a 15 c3 b8 00 00 c3 55 8b ec 1e 06 56 57 8b 46 04 8e d8 8b 76 06
>>
>> Here is my guess:
>>
>> Laptop has a CPU from 2012.  Desktop has a CPU from 2009.
>>
>> Intel added "unrestricted guest" support to VMX.  This feature allows
>> the CPU to run real mode code in guest mode.
>>
>> CPUs that do not support "unrestricted guest" (your desktop?) use an
>> emulator implemented in software inside the kvm.ko kernel module.
>>
>> The emulator may be unable to handle the real mode instruction in the
>> particular kernel version you are running.
>>
>> The laptop doesn't hit this issue because it supports "unrestricted
>> guest" while the desktop falls back to the emulator inside kvm.ko where
>> it hits the bug.
>>
>> You may find that changing kernel versions on the desktop will make it
>> work.
>>
>> The best would be to compile a vanilla Linux kernel for the desktop
>> machine to verify that this issue still happens.  If so, please post the
>> full "KVM internal error" output to this mailing list and hopefully
>> someone can fix the emulator.
>>
>> Problem with my theory: I haven't figured out how to check which Intel
>> CPU models support "unrestricted guest", so I'm not 100% sure this is
>> the issue.
>>
> Your theory is correct. This is the same as [1] which is fixed in 3.9.
> Depending on your kernel emulate_invalid_guest_state = 0 may help, or
> may make things even worse :(
>
> [1] https://bugzilla.kernel.org/show_bug.cgi?id=50921
>
> --
> 			Gleb.
>
Thank you Stephan and Gleb. You are both right. I tried setting the
kvm_intel module parameter as Gleb suggested. Running under kernel
3.7.9-205.fc18.x86_64, my VM failed to start with the log message:

KVM: entry failed, hardware error 0x80000021

If you're running a guest on an Intel machine without unrestricted mode
support, the failure can be most likely due to the guest entering an invalid
state for Intel VT. For example, the guest maybe running in big real mode
which is not supported on less recent Intel processors.

EAX=00000011 EBX=18ae1000 ECX=00006a32 EDX=000fffa9
ESI=3ffeb0a1 EDI=00000000 EBP=000069f2 ESP=000069f2
EIP=0000c4cd EFL=00010006 [-----P-] CPL=0 II=0 A20=1 SMM=0 HLT=0
ES =fd39 000fd390 ffffffff 00809300 DPL=0 DS16 [-WA]
CS =f000 000f0000 0000ffff 00009b00 DPL=0 CS16 [-RA]
SS =0000 00000000 0000ffff 00009300 DPL=0 DS16 [-WA]
DS =0030 00000000 ffffffff 00809300 DPL=0 DS16 [-WA]
FS =0030 00000000 ffffffff 00809300 DPL=0 DS16 [-WA]
GS =ca00 000ca000 ffffffff 00809300 DPL=0 DS16 [-WA]
LDT=0000 00000000 0000ffff 00008200 DPL=0 LDT
TR =0000 00000000 0000ffff 00008b00 DPL=0 TSS32-busy
GDT=     000fd3a8 00000037
IDT=     000fd3e6 00000000
CR0=00000011 CR2=00000000 CR3=00000000 CR4=00000000
DR0=0000000000000000 DR1=0000000000000000 DR2=0000000000000000
DR3=0000000000000000
DR6=00000000ffff0ff0 DR7=0000000000000400
EFER=0000000000000000
Code=01 1e e0 d3 2e 0f 01 16 a0 d3 0f 20 c0 66 83 c8 01 0f 22 c0 <66> ea
d5 c4 0f 00 08 00 b8 10 00 00 00 8e d8 8e c0 8e d0 8e e0 8e e8 89 c8 ff
e2 89 c1 b8
qemu: terminating on signal 15 from pid 1589

However, under both kernel versions 3.8.1-201.fc18.x86_64 and
3.8.2-206.fc18.x86_64 I could successfully start my win2k VM.  I simply
reloaded the kvm_intel module:

# rmmod kvm_intel
# modprobe kvm_intel emulate_invalid_guest_state=0

I checked that the parameter was set,

#  cat /sys/module/kvm_intel/parameters/emulate_invalid_guest_state
N

and then started the VM. Everything seemed to work fine.

The kernels above are the stock kernels via the Fedora distribution. If
I can do anything to assist with some testing in the future, please let
me know. Thanks again for your help!

--Earl

Re: win2k guest vm won't boot under Fedora 18 KVM

[Gleb Natapov]

On Tue, Mar 12, 2013 at 01:09:44PM -0600, Earl Marwil wrote:
> 
> On 03/11/2013 02:27 AM, Gleb Natapov wrote:
> > On Mon, Mar 11, 2013 at 09:09:13AM +0100, Stefan Hajnoczi wrote:
> >> On Sat, Mar 09, 2013 at 12:43:32PM -0700, Earl Marwil wrote:
> >>> Hi,
> >>>
> >>> I'm looking for some guidance on how to get to the root cause of an
> >>> issue that I am observing with a win2k guest that won't boot under
> >>> Fedora 18 on one system but will boot on another. A few days ago I
> >>> posted on the fedora forum:
> >>>
> >>> http://forums.fedoraforum.org/showthread.php?t=289401
> >>>
> >>> I can repeat the details in this thread if requested. The issue is that,
> >>> with a fresh build of Fedora 18, updated to the most recent kernel and
> >>> packages on an external USB ssd, my win2k VM boots on my laptop (Core
> >>> i73720QM processor) but does not boot on my desktop system (Core i7-870
> >>> processor).
> >>>
> >>> I'm not sure whether this is a kvm issue or a kernel issue. I'll be glad
> >>> to dig deeper, just let me what information is needed.
> >> Hi Earl,
> >> >From your forum post:
> >>
> >>   KVM internal error. Suberror: 1
> >>   emulation failure
> >>   EAX=63700200 EBX=0000e6f5 ECX=0000000f EDX=00000936
> >>   ...
> >>   Code=74 1d b0 37 e6 70 eb 00 e4 71 eb 00 32 e4 c1 c0 04 c0 c8 04 <d5> 0a 3d 13 0
> >>   0 75 04 b8 7a 15 c3 b8 00 00 c3 55 8b ec 1e 06 56 57 8b 46 04 8e d8 8b 76 06
> >>
> >> Here is my guess:
> >>
> >> Laptop has a CPU from 2012.  Desktop has a CPU from 2009.
> >>
> >> Intel added "unrestricted guest" support to VMX.  This feature allows
> >> the CPU to run real mode code in guest mode.
> >>
> >> CPUs that do not support "unrestricted guest" (your desktop?) use an
> >> emulator implemented in software inside the kvm.ko kernel module.
> >>
> >> The emulator may be unable to handle the real mode instruction in the
> >> particular kernel version you are running.
> >>
> >> The laptop doesn't hit this issue because it supports "unrestricted
> >> guest" while the desktop falls back to the emulator inside kvm.ko where
> >> it hits the bug.
> >>
> >> You may find that changing kernel versions on the desktop will make it
> >> work.
> >>
> >> The best would be to compile a vanilla Linux kernel for the desktop
> >> machine to verify that this issue still happens.  If so, please post the
> >> full "KVM internal error" output to this mailing list and hopefully
> >> someone can fix the emulator.
> >>
> >> Problem with my theory: I haven't figured out how to check which Intel
> >> CPU models support "unrestricted guest", so I'm not 100% sure this is
> >> the issue.
> >>
> > Your theory is correct. This is the same as [1] which is fixed in 3.9.
> > Depending on your kernel emulate_invalid_guest_state = 0 may help, or
> > may make things even worse :(
> >
> > [1] https://bugzilla.kernel.org/show_bug.cgi?id=50921
> >
> > --
> > 			Gleb.
> >
> Thank you Stephan and Gleb. You are both right. I tried setting the
> kvm_intel module parameter as Gleb suggested. Running under kernel
> 3.7.9-205.fc18.x86_64, my VM failed to start with the log message:
> 
Yes, emulate_invalid_guest_state = 0 was broke in 3.7. 3.9, when
released, should work without it.

--
			Gleb.