KVM entry failed, hardware error

KVM entry failed, hardware error

[Johannes Bauer]

Hi list,

when trying to install Windows7 in a qemu-kvm 1.0.1 installation on
Gentoo on my host running

Linux joequad 3.3.7 #1 SMP PREEMPT Sun Jun 3 12:05:59 CEST 2012 x86_64
Intel(R) Core(TM)2 Quad CPU Q9550 @ 2.83GHz GenuineIntel GNU/Linux

with a host CPU of

processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 23
model name	: Intel(R) Core(TM)2 Quad CPU    Q9550  @ 2.83GHz
stepping	: 10
microcode	: 0xa07
cpu MHz		: 2003.000
cache size	: 6144 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
initial apicid	: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 13
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov
pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx lm
constant_tsc arch_perfmon pebs bts rep_good nopl aperfmperf pni dtes64
monitor ds_cpl vmx smx est tm2 ssse3 cx16 xtpr pdcm sse4_1 xsave lahf_lm
dts tpr_shadow vnmi flexpriority
bogomips	: 5680.90
clflush size	: 64
cache_alignment	: 64
address sizes	: 36 bits physical, 48 bits virtual
power management:

the installation first works fine but upon first reboot qemu terminates
with:

KVM: entry failed, hardware error 0x80000021

If you're runnning a guest on an Intel machine without unrestricted mode
support, the failure can be most likely due to the guest entering an invalid
state for Intel VT. For example, the guest maybe running in big real mode
which is not supported on less recent Intel processors.

EAX=00000010 EBX=00000080 ECX=00000000 EDX=00000080
ESI=0025da4a EDI=0007da4a EBP=00001f20 ESP=00000200
EIP=0000009b EFL=00000002 [-------] CPL=3 II=0 A20=1 SMM=0 HLT=0
ES =0020 00000200 0000ffff 00009300
CS =b000 002b0000 0000ffff 0000f300
SS =0020 00000200 0000ffff 0000f300
DS =0020 00000200 0000ffff 00009300
FS =0020 00000200 0000ffff 00009300
GS =0020 00000200 0000ffff 00009300
LDT=0000 00000000 0000ffff 00008200
TR =0000 00000000 0000ffff 00008b00
GDT=     002b0000 00000027
IDT=     00000000 000003ff
CR0=00000010 CR2=00000000 CR3=00000000 CR4=00000000
DR0=0000000000000000 DR1=0000000000000000 DR2=0000000000000000
DR3=0000000000000000
DR6=00000000ffff0ff0 DR7=0000000000000400
EFER=0000000000000000
Code=02 00 00 ea 91 00 00 00 18 00 0f 20 c0 66 83 e0 fe 0f 22 c0 <66> 31
c0 8e d8 8e c0 8e d0 66 bc 00 04 00 00 8e e0 8e e8 ea 00 00 00 20 00 00
00 20 4a da

which disassembles to

   0:   02 00                   add    (%eax),%al
   2:   00 ea                   add    %ch,%dl
   4:   91                      xchg   %eax,%ecx
   5:   00 00                   add    %al,(%eax)
   7:   00 18                   add    %bl,(%eax)
   9:   00 0f                   add    %cl,(%edi)
   b:   20 c0                   and    %al,%al
   d:   66 83 e0 fe             and    $0xfffffffe,%ax
  11:   0f 22 c0                mov    %eax,%cr0
  14:   66 31 c0                xor    %eax,%eax
  17:   8e d8                   mov    %ax,%ds
  19:   8e c0                   mov    %ax,%es
  1b:   8e d0                   mov    %ax,%ss
  1d:   66 bc 00 04 00 00       mov    $0x400,%esp
  23:   8e e0                   mov    %ax,%fs
  25:   8e e8                   mov    %ax,%gs
  27:   ea 00 00 00 20          ljmp   $0x2000,$0x0

Since the instruction clears PE of %cr0, I'm pretty sure it's trying to
enter real mode again (for a very brief period no doubt) before firing
up the OS. Since I believe my CPU is too old and does not support
"Unrestricted Guest", this causes qemu to barf.

Is this assessment correct? Is there some way of getting qemu to switch
to "soft" emulation in real mode and to use KVM in protected mode? Or is
it impossible to run Windows 7 with kvm on a Core Quad (which possibly
doesn't support "Unrestricted Guest")?

For reference, there's at least one more person having an identical
problem (I've also posted in that thread):
http://forums.gentoo.org/viewtopic-p-7054198.html

To find out if my CPU really does not suport "Unrestricted Guest" I've
also posted http://software.intel.com/en-us/forums/showthread.php?t=105687

Any help is greatly appreciated,
Best regards,
Joe

Re: KVM entry failed, hardware error

[Avi Kivity]

On 06/03/2012 03:18 PM, Johannes Bauer wrote:
> Hi list,
> 
> when trying to install Windows7 in a qemu-kvm 1.0.1 installation on
> Gentoo on my host running

Is that a 32-bit or 64-bit Windows 7?

> 
> Linux joequad 3.3.7 #1 SMP PREEMPT Sun Jun 3 12:05:59 CEST 2012 x86_64
> Intel(R) Core(TM)2 Quad CPU Q9550 @ 2.83GHz GenuineIntel GNU/Linux
> 
> with a host CPU of
> 
> processor	: 0
> vendor_id	: GenuineIntel
> cpu family	: 6
> model		: 23
> model name	: Intel(R) Core(TM)2 Quad CPU    Q9550  @ 2.83GHz
> stepping	: 10
> microcode	: 0xa07
> cpu MHz		: 2003.000
> cache size	: 6144 KB
> physical id	: 0
> siblings	: 4
> core id		: 0
> cpu cores	: 4
> apicid		: 0
> initial apicid	: 0
> fpu		: yes
> fpu_exception	: yes
> cpuid level	: 13
> wp		: yes
> flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov
> pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx lm
> constant_tsc arch_perfmon pebs bts rep_good nopl aperfmperf pni dtes64
> monitor ds_cpl vmx smx est tm2 ssse3 cx16 xtpr pdcm sse4_1 xsave lahf_lm
> dts tpr_shadow vnmi flexpriority
> bogomips	: 5680.90
> clflush size	: 64
> cache_alignment	: 64
> address sizes	: 36 bits physical, 48 bits virtual
> power management:
> 
> the installation first works fine but upon first reboot qemu terminates
> with:
> 
> KVM: entry failed, hardware error 0x80000021
> 
> If you're runnning a guest on an Intel machine without unrestricted mode
> support, the failure can be most likely due to the guest entering an invalid
> state for Intel VT. For example, the guest maybe running in big real mode
> which is not supported on less recent Intel processors.
> 
> EAX=00000010 EBX=00000080 ECX=00000000 EDX=00000080
> ESI=0025da4a EDI=0007da4a EBP=00001f20 ESP=00000200
> EIP=0000009b EFL=00000002 [-------] CPL=3 II=0 A20=1 SMM=0 HLT=0
> ES =0020 00000200 0000ffff 00009300
> CS =b000 002b0000 0000ffff 0000f300

Pre-unrestricted mode hosts require cs.selector (first number) * 16 ==
cs.base (second number).  This clearly isn't the case here.

> Since the instruction clears PE of %cr0, I'm pretty sure it's trying to
> enter real mode again (for a very brief period no doubt) before firing
> up the OS. Since I believe my CPU is too old and does not support
> "Unrestricted Guest", this causes qemu to barf.
> 
> Is this assessment correct? Is there some way of getting qemu to switch
> to "soft" emulation in real mode and to use KVM in protected mode?

# rmmod kvm_intel
# modprobe kvm_intel emulate_invalid_guest_state=1

Warning: experimental code.

> Or is
> it impossible to run Windows 7 with kvm on a Core Quad (which possibly
> doesn't support "Unrestricted Guest")?

It should be possible.  Did you do anything special with the guest
(install any software?) before rebooting?  Is this repeatable?

> For reference, there's at least one more person having an identical
> problem (I've also posted in that thread):
> http://forums.gentoo.org/viewtopic-p-7054198.html

Same cs.sel/cs.base as your report.

> To find out if my CPU really does not suport "Unrestricted Guest" I've
> also posted http://software.intel.com/en-us/forums/showthread.php?t=105687

You can run vmxcap [1] and see.


[1]
http://git.qemu.org/?p=qemu.git;a=blob_plain;f=scripts/kvm/vmxcap;hb=HEAD


-- 
error compiling committee.c: too many arguments to function

Re: KVM entry failed, hardware error

[Johannes Bauer]

On 03.06.2012 14:33, Avi Kivity wrote:

>> when trying to install Windows7 in a qemu-kvm 1.0.1 installation on
>> Gentoo on my host running
> 
> Is that a 32-bit or 64-bit Windows 7?

64 bit. But I've also (in despair) tried installing 32 Bit Windows 7,
with the exact same effect.

>> EAX=00000010 EBX=00000080 ECX=00000000 EDX=00000080
>> ESI=0025da4a EDI=0007da4a EBP=00001f20 ESP=00000200
>> EIP=0000009b EFL=00000002 [-------] CPL=3 II=0 A20=1 SMM=0 HLT=0
>> ES =0020 00000200 0000ffff 00009300
>> CS =b000 002b0000 0000ffff 0000f300
> 
> Pre-unrestricted mode hosts require cs.selector (first number) * 16 ==
> cs.base (second number).  This clearly isn't the case here.

I've checked with vmxcap and my PC indeed does not support unrestricted
mode. My Laptop (on which the identical image works perfectly) does.

> # rmmod kvm_intel
> # modprobe kvm_intel emulate_invalid_guest_state=1
> 
> Warning: experimental code.

Tried it out -- I don't get the error anymore, but qemu just hangs on
boot with 100% CPU. :-(

>> Or is
>> it impossible to run Windows 7 with kvm on a Core Quad (which possibly
>> doesn't support "Unrestricted Guest")?
> 
> It should be possible.  Did you do anything special with the guest
> (install any software?) before rebooting?  Is this repeatable?

Nothing at all, just a standard installation. Did not even get to the
point where I *could* install software (it reboots during the
installation and that's where it doesn't come up anymore).

I even tried to completing the installation on my laptop (which works),
then copy the image back on my desktop PC: Same error.

Any hints on how I could debug the hang with the emulation of invalid
guest state? Or is there a more recent version (git?) than a 3.3.7
kernel that I should try out?

Thanks for your help,
Best regards,
Joe

Re: KVM entry failed, hardware error

[Johannes Bauer]

On 03.06.2012 14:33, Avi Kivity wrote:

> You can run vmxcap [1] and see.

Ah, I forgot to attach the output of vmxcap, which might be of interest.
First the host where it does not work:

pin-based controls
  External interrupt exiting               yes
  NMI exiting                              yes
  Virtual NMIs                             yes
  Activate VMX-preemption timer            no
primary processor-based controls
  Interrupt window exiting                 yes
  Use TSC offsetting                       yes
  HLT exiting                              yes
  INVLPG exiting                           yes
  MWAIT exiting                            yes
  RDPMC exiting                            yes
  RDTSC exiting                            yes
  CR3-load exiting                         forced
  CR3-store exiting                        forced
  CR8-load exiting                         yes
  CR8-store exiting                        yes
  Use TPR shadow                           yes
  NMI-window exiting                       yes
  MOV-DR exiting                           yes
  Unconditional I/O exiting                yes
  Use I/O bitmaps                          yes
  Monitor trap flag                        no
  Use MSR bitmaps                          yes
  MONITOR exiting                          yes
  PAUSE exiting                            yes
  Activate secondary control               yes
secondary processor-based controls
  Virtualize APIC accesses                 yes
  Enable EPT                               no
  Descriptor-table exiting                 no
  Virtualize x2APIC mode                   no
  Enable VPID                              no
  WBINVD exiting                           yes
  Unrestricted guest                       no
  PAUSE-loop exiting                       no
VM-Exit controls
  Save debug controls                      forced
  Host address-space size                  yes
  Load IA32_PERF_GLOBAL_CTRL               yes
  Acknowledge interrupt on exit            yes
  Save IA32_PAT                            no
  Load IA32_PAT                            no
  Save IA32_EFER                           no
  Load IA32_EFER                           no
  Save VMX-preemption timer value          no
VM-Entry controls
  Load debug controls                      forced
  IA-64 mode guest                         yes
  Entry to SMM                             yes
  Deactivate dual-monitor treatment        yes
  Load IA32_PERF_GLOBAL_CTRL               yes
  Load IA32_PAT                            no
  Load IA32_EFER                           no
Miscellaneous data
  VMX-preemption timer scale (log2)        0
  Store EFER.LMA into IA-32e mode guest control no
  HLT activity state                       yes
  Shutdown activity state                  yes
  Wait-for-SIPI activity state             yes
  Number of CR3-target values              4
  MSR-load/store count recommenation       0
  MSEG revision identifier                 0
VPID and EPT capabilities
  Execute-only EPT translations            no
  Page-walk length 4                       no
  Paging-structure memory type UC          no
  Paging-structure memory type WB          no
  2MB EPT pages                            no
  1GB EPT pages                            no
  INVEPT supported                         no
  Single-context INVEPT                    no
  All-context INVEPT                       no
  INVVPID supported                        no
  Individual-address INVVPID               no
  Single-context INVVPID                   no
  All-context INVVPID                      no
  Single-context-retaining-globals INVVPID no


Then the one where it does:

pin-based controls
  External interrupt exiting               yes
  NMI exiting                              yes
  Virtual NMIs                             yes
  Activate VMX-preemption timer            yes
primary processor-based controls
  Interrupt window exiting                 yes
  Use TSC offsetting                       yes
  HLT exiting                              yes
  INVLPG exiting                           yes
  MWAIT exiting                            yes
  RDPMC exiting                            yes
  RDTSC exiting                            yes
  CR3-load exiting                         default
  CR3-store exiting                        default
  CR8-load exiting                         yes
  CR8-store exiting                        yes
  Use TPR shadow                           yes
  NMI-window exiting                       yes
  MOV-DR exiting                           yes
  Unconditional I/O exiting                yes
  Use I/O bitmaps                          yes
  Monitor trap flag                        yes
  Use MSR bitmaps                          yes
  MONITOR exiting                          yes
  PAUSE exiting                            yes
  Activate secondary control               yes
secondary processor-based controls
  Virtualize APIC accesses                 yes
  Enable EPT                               yes
  Descriptor-table exiting                 yes
  Virtualize x2APIC mode                   yes
  Enable VPID                              yes
  WBINVD exiting                           yes
  Unrestricted guest                       yes
  PAUSE-loop exiting                       no
VM-Exit controls
  Save debug controls                      default
  Host address-space size                  yes
  Load IA32_PERF_GLOBAL_CTRL               yes
  Acknowledge interrupt on exit            yes
  Save IA32_PAT                            yes
  Load IA32_PAT                            yes
  Save IA32_EFER                           yes
  Load IA32_EFER                           yes
  Save VMX-preemption timer value          yes
VM-Entry controls
  Load debug controls                      forced
  IA-64 mode guest                         yes
  Entry to SMM                             yes
  Deactivate dual-monitor treatment        yes
  Load IA32_PERF_GLOBAL_CTRL               yes
  Load IA32_PAT                            yes
  Load IA32_EFER                           yes
Miscellaneous data
  VMX-preemption timer scale (log2)        5
  Store EFER.LMA into IA-32e mode guest control yes
  HLT activity state                       yes
  Shutdown activity state                  yes
  Wait-for-SIPI activity state             yes
  Number of CR3-target values              4
  MSR-load/store count recommenation       0
  MSEG revision identifier                 0
VPID and EPT capabilities
  Execute-only EPT translations            yes
  Page-walk length 4                       yes
  Paging-structure memory type UC          yes
  Paging-structure memory type WB          yes
  2MB EPT pages                            yes
  1GB EPT pages                            no
  INVEPT supported                         yes
  Single-context INVEPT                    yes
  All-context INVEPT                       yes
  INVVPID supported                        yes
  Individual-address INVVPID               yes
  Single-context INVVPID                   yes
  All-context INVVPID                      yes
  Single-context-retaining-globals INVVPID yes

Best regards,
Joe

Re: KVM entry failed, hardware error

[Avi Kivity]

On 06/03/2012 04:01 PM, Johannes Bauer wrote:
> On 03.06.2012 14:33, Avi Kivity wrote:
> 
>>> when trying to install Windows7 in a qemu-kvm 1.0.1 installation on
>>> Gentoo on my host running
>> 
>> Is that a 32-bit or 64-bit Windows 7?
> 
> 64 bit. But I've also (in despair) tried installing 32 Bit Windows 7,
> with the exact same effect.
> 
>>> EAX=00000010 EBX=00000080 ECX=00000000 EDX=00000080
>>> ESI=0025da4a EDI=0007da4a EBP=00001f20 ESP=00000200
>>> EIP=0000009b EFL=00000002 [-------] CPL=3 II=0 A20=1 SMM=0 HLT=0
>>> ES =0020 00000200 0000ffff 00009300
>>> CS =b000 002b0000 0000ffff 0000f300
>> 
>> Pre-unrestricted mode hosts require cs.selector (first number) * 16 ==
>> cs.base (second number).  This clearly isn't the case here.


Gleb points out that cs.base == gdt.base, which doesn't make a lot of
sense.  So some corruption has happened earlier.  Please follow
http://www.linux-kvm.org/page/Tracing and post the last few thousand
lines somewhere.  Trace a uniprocessor guest if the problem reproduces
that way.


> 
> I've checked with vmxcap and my PC indeed does not support unrestricted
> mode. My Laptop (on which the identical image works perfectly) does.
> 
>> # rmmod kvm_intel
>> # modprobe kvm_intel emulate_invalid_guest_state=1
>> 
>> Warning: experimental code.
> 
> Tried it out -- I don't get the error anymore, but qemu just hangs on
> boot with 100% CPU. :-(

Not too surprising, this hasn't been tested in quite a while.

> 
>>> Or is
>>> it impossible to run Windows 7 with kvm on a Core Quad (which possibly
>>> doesn't support "Unrestricted Guest")?
>> 
>> It should be possible.  Did you do anything special with the guest
>> (install any software?) before rebooting?  Is this repeatable?
> 
> Nothing at all, just a standard installation. Did not even get to the
> point where I *could* install software (it reboots during the
> installation and that's where it doesn't come up anymore).
> 
> I even tried to completing the installation on my laptop (which works),
> then copy the image back on my desktop PC: Same error.


That should help the tracing effort.

> 
> Any hints on how I could debug the hang with the emulation of invalid
> guest state? Or is there a more recent version (git?) than a 3.3.7
> kernel that I should try out?

3.3.7 should definitely work.  Let's see what the traces tell us.


-- 
error compiling committee.c: too many arguments to function

Re: KVM entry failed, hardware error

[Johannes Bauer]

On 03.06.2012 15:43, Avi Kivity wrote:
> On 06/03/2012 04:01 PM, Johannes Bauer wrote:
>> On 03.06.2012 14:33, Avi Kivity wrote:
>>
>>>> when trying to install Windows7 in a qemu-kvm 1.0.1 installation on
>>>> Gentoo on my host running
>>>
>>> Is that a 32-bit or 64-bit Windows 7?
>>
>> 64 bit. But I've also (in despair) tried installing 32 Bit Windows 7,
>> with the exact same effect.
>>
>>>> EAX=00000010 EBX=00000080 ECX=00000000 EDX=00000080
>>>> ESI=0025da4a EDI=0007da4a EBP=00001f20 ESP=00000200
>>>> EIP=0000009b EFL=00000002 [-------] CPL=3 II=0 A20=1 SMM=0 HLT=0
>>>> ES =0020 00000200 0000ffff 00009300
>>>> CS =b000 002b0000 0000ffff 0000f300
>>>
>>> Pre-unrestricted mode hosts require cs.selector (first number) * 16 ==
>>> cs.base (second number).  This clearly isn't the case here.
> 
> Gleb points out that cs.base == gdt.base, which doesn't make a lot of
> sense.  So some corruption has happened earlier.  Please follow
> http://www.linux-kvm.org/page/Tracing and post the last few thousand
> lines somewhere.  Trace a uniprocessor guest if the problem reproduces
> that way.

Tried that -- and failed miserably.

Something in the trace format appears to have changed. When using
trace-cmd report, I first get a parser warning:

trace-cmd: No such file or directory
  bad op token {
  failed to read event print fmt for kvm_emulate_insn
version = 6
CPU 0 is empty
cpus=4

The ENOENT errno doesn't correspond to that error, strace showed that
occuring much earlier in something unrelated. The warning is generated by:

#1  0x0000000000412c66 in process_op (event=0x6537c0, arg=0x6564f0,
tok=0x7fffffffd508) at /home/joe/qemu/trace/trace-cmd/parse-events.c:1686
#2  0x0000000000414eef in process_arg_token (event=0x6537c0,
arg=0x6564f0, tok=0x7fffffffd598, type=EVENT_OP) at
/home/joe/qemu/trace/trace-cmd/parse-events.c:2604
#3  0x0000000000412727 in process_arg (event=0x6537c0, arg=0x6564f0,
tok=0x7fffffffd598) at /home/joe/qemu/trace/trace-cmd/parse-events.c:1514
#4  0x00000000004146b4 in process_paren (event=0x6537c0, arg=0x6564f0,
tok=0x7fffffffd5f8) at /home/joe/qemu/trace/trace-cmd/parse-events.c:2352
#5  0x0000000000414eb0 in process_arg_token (event=0x6537c0,
arg=0x6564f0, tok=0x7fffffffd678, type=EVENT_DELIM) at
/home/joe/qemu/trace/trace-cmd/parse-events.c:2596
#6  0x0000000000412727 in process_arg (event=0x6537c0, arg=0x6564f0,
tok=0x7fffffffd678) at /home/joe/qemu/trace/trace-cmd/parse-events.c:1514
#7  0x0000000000414f87 in event_read_print_args (event=0x6537c0,
list=0x656490) at /home/joe/qemu/trace/trace-cmd/parse-events.c:2637
#8  0x000000000041524f in event_read_print (event=0x6537c0) at
/home/joe/qemu/trace/trace-cmd/parse-events.c:2723
#9  0x0000000000419040 in pevent_parse_event (pevent=0x63a0d0,
    buf=0x653390 "name: kvm_emulate_insn\nID:
29\nformat:\n\tfield:unsigned short
common_type;\toffset:0;\tsize:2;\tsigned:0;\n\tfield:unsigned char
common_flags;\toffset:2;\tsize:1;\tsigned:0;\n\tfield:unsigned char
common_preempt_"..., size=1053, sys=0x640790 "kvm") at
/home/joe/qemu/trace/trace-cmd/parse-events.c:4626
#10 0x000000000042190a in read_event_file (handle=0x63a030,
system=0x640790 "kvm", size=1053, print=0) at
/home/joe/qemu/trace/trace-cmd/trace-input.c:386
#11 0x0000000000421a90 in read_event_files (handle=0x63a030, print=0) at
/home/joe/qemu/trace/trace-cmd/trace-input.c:450
#12 0x0000000000421cb3 in tracecmd_read_headers (handle=0x63a030) at
/home/joe/qemu/trace/trace-cmd/trace-input.c:539
#13 0x000000000040c6a5 in trace_report (argc=2, argv=0x7fffffffd9d8) at
/home/joe/qemu/trace/trace-cmd/trace-read.c:1025
#14 0x00000000004057e7 in main (argc=2, argv=0x7fffffffd9d8) at
/home/joe/qemu/trace/trace-cmd/trace-cmd.c:147

The decoding that follows is complete garbage, which is why I think it's
of no use to you (i.e. the insn= part just displays some random
undefined memory from the looks).

Therefore, I've uploaded the compressed trace.dat file, so you can maybe
have a look why the "report" tool barfs and interpret it correctly. I
can't figure it out. The trace is here:

http://spornkuller.de/trace.dat.bz2

The trace was generated with a buffer of 40000 (double of what was given
in the example), when it was lower I had overruns. This way I had none.
Also, to simulate a single cpu, I've run qemu with "-smp 1", hopefully
that's correct.

>> I even tried to completing the installation on my laptop (which works),
>> then copy the image back on my desktop PC: Same error.
> 
> That should help the tracing effort.

Should I also generate a trace there and look for the corresponding
opcodes that fail on my main machine?

Best regards,
Joe

Re: KVM entry failed, hardware error

[Gleb Natapov]

On Sun, Jun 03, 2012 at 06:25:33PM +0200, Johannes Bauer wrote:
> Therefore, I've uploaded the compressed trace.dat file, so you can maybe
> have a look why the "report" tool barfs and interpret it correctly. I
> can't figure it out. The trace is here:
> 
> http://spornkuller.de/trace.dat.bz2
> 
I can read this trace. Can you do "info pci" in qemu's monitor
after failure? What is your command line?

--
			Gleb.

Re: KVM entry failed, hardware error

[Avi Kivity]

On 06/04/2012 11:53 AM, Gleb Natapov wrote:
> On Sun, Jun 03, 2012 at 06:25:33PM +0200, Johannes Bauer wrote:
>> Therefore, I've uploaded the compressed trace.dat file, so you can maybe
>> have a look why the "report" tool barfs and interpret it correctly. I
>> can't figure it out. The trace is here:
>> 
>> http://spornkuller.de/trace.dat.bz2
>> 
> I can read this trace. Can you do "info pci" in qemu's monitor
> after failure? What is your command line?

Also after the failure:

 x/256b 0x2b0000



-- 
error compiling committee.c: too many arguments to function

Re: KVM entry failed, hardware error

[Johannes Bauer]

On 04.06.2012 10:53, Gleb Natapov wrote:
> On Sun, Jun 03, 2012 at 06:25:33PM +0200, Johannes Bauer wrote:
>> Therefore, I've uploaded the compressed trace.dat file, so you can maybe
>> have a look why the "report" tool barfs and interpret it correctly. I
>> can't figure it out. The trace is here:
>>
>> http://spornkuller.de/trace.dat.bz2
>>
> I can read this trace.

Hm, weird. But good that it works on your side. I get a lot of:

trace-cmd: No such file or directory
  bad op token {
  failed to read event print fmt for kvm_emulate_insn
version = 6
CPU 0 is empty
cpus=4
 qemu-system-x86-10775 [001]  2512.220779: kvm_fpu:              load
 qemu-system-x86-10775 [001]  2512.220782: kvm_entry:            vcpu 0
 qemu-system-x86-10775 [001]  2512.220785: kvm_exit:             reason
EXCEPTION_NMI rip 0xfff0 info ffff0 80000b0e
 qemu-system-x86-10775 [001]  2512.220787: kvm_page_fault:       address
ffff0 error_code 14
 qemu-system-x86-10775 [001]  2512.220796: kvm_entry:            vcpu 0
 qemu-system-x86-10775 [001]  2512.220798: kvm_exit:             reason
EXCEPTION_NMI rip 0xc81e info 0 80000b0d
 qemu-system-x86-10775 [001]  2512.220803: kvm_emulate_insn:     [FAILED
TO PARSE] rip=51230 csbase=983040 len=3 insn= �f%����"���f flags=0 failed=0
 qemu-system-x86-10775 [001]  2512.220806: kvm_entry:            vcpu 0
 qemu-system-x86-10775 [001]  2512.220807: kvm_exit:             reason
EXCEPTION_NMI rip 0xc827 info 0 80000b0d
 qemu-system-x86-10775 [001]  2512.220808: kvm_emulate_insn:     [FAILED
TO PARSE] rip=51239 csbase=983040 len=3 insn="���f�flags=0 failed=0
[...]

> Can you do "info pci" in qemu's monitor
> after failure? 

(qemu) info pci
  Bus  0, device   0, function 0:
    Host bridge: PCI device 8086:1237
      id ""
  Bus  0, device   1, function 0:
    ISA bridge: PCI device 8086:7000
      id ""
  Bus  0, device   1, function 1:
    IDE controller: PCI device 8086:7010
      BAR4: I/O at 0xc000 [0xc00f].
      id ""
  Bus  0, device   1, function 3:
    Bridge: PCI device 8086:7113
      IRQ 9.
      id ""
  Bus  0, device   2, function 0:
    VGA controller: PCI device 1013:00b8
      BAR0: 32 bit prefetchable memory at 0xf0000000 [0xf1ffffff].
      BAR1: 32 bit memory at 0xf2000000 [0xf2000fff].
      BAR6: 32 bit memory at 0xffffffffffffffff [0x0000fffe].
      id ""
  Bus  0, device   3, function 0:
    Ethernet controller: PCI device 8086:100e
      IRQ 11.
      BAR0: 32 bit memory at 0xf2020000 [0xf203ffff].
      BAR1: I/O at 0xc040 [0xc07f].
      BAR6: 32 bit memory at 0xffffffffffffffff [0x0001fffe].
      id ""
  Bus  0, device   4, function 0:
    SCSI controller: PCI device 1af4:1001
      IRQ 11.
      BAR0: I/O at 0xc080 [0xc0bf].
      BAR1: 32 bit memory at 0xf2060000 [0xf2060fff].
      id ""


>What is your command line?

bin/qemu-system-x86_64 -cpu host -enable-kvm -net nic -net
user,smb=Share,restrict=on -drive
media=disk,file=Windows7_x32.qcow2,if=virtio -m 2048 -smp 1 -nographic

(added "-nographic" to be able to enter the console)

Also, as per Avi's request:

(qemu) x/256b 0x2b0000
00000000002b0000: 0xeb 0x26 0x27 0x00 0x00 0x00 0x2b 0x00
00000000002b0008: 0xff 0xff 0x00 0x00 0x00 0x9a 0xcf 0x00
00000000002b0010: 0xff 0xff 0x00 0x00 0x00 0x92 0xcf 0x00
00000000002b0018: 0xff 0xff 0x00 0x00 0x2b 0x9f 0x00 0x00
00000000002b0020: 0xff 0xff 0x00 0x02 0x00 0x93 0x00 0x00
00000000002b0028: 0x8a 0x15 0x68 0xbc 0x00 0x00 0xa1 0xbf
00000000002b0030: 0x00 0x2b 0x00 0x85 0xc0 0x74 0x06 0x8b
00000000002b0038: 0x1d 0xbb 0x00 0x2b 0x00 0xa1 0xc7 0x00
00000000002b0040: 0x2b 0x00 0x85 0xc0 0x74 0x06 0x8b 0x15
00000000002b0048: 0xc3 0x00 0x2b 0x00 0xbe 0x00 0x00 0x20
00000000002b0050: 0x00 0x31 0xc0 0x31 0xff 0x66 0x8b 0x3d
00000000002b0058: 0xb5 0x00 0x2b 0x00 0xc1 0xe7 0x04 0x66
00000000002b0060: 0xa1 0xb3 0x00 0x2b 0x00 0x01 0xc7 0x8b
00000000002b0068: 0x0d 0xb7 0x00 0x2b 0x00 0xfc 0xf3 0xa4
00000000002b0070: 0x0f 0x01 0x15 0x02 0x00 0x2b 0x00 0x66
00000000002b0078: 0xb8 0x20 0x00 0x8e 0xd8 0x8e 0xc0 0x8e
00000000002b0080: 0xe0 0x8e 0xe8 0x8e 0xd0 0xbc 0x00 0x02
00000000002b0088: 0x00 0x00 0xea 0x91 0x00 0x00 0x00 0x18
00000000002b0090: 0x00 0x0f 0x20 0xc0 0x66 0x83 0xe0 0xfe
00000000002b0098: 0x0f 0x22 0xc0 0x66 0x31 0xc0 0x8e 0xd8
00000000002b00a0: 0x8e 0xc0 0x8e 0xd0 0x66 0xbc 0x00 0x04
00000000002b00a8: 0x00 0x00 0x8e 0xe0 0x8e 0xe8 0xea 0x00
00000000002b00b0: 0x00 0x00 0x20 0x00 0x00 0x00 0x20 0x4a
00000000002b00b8: 0xda 0x05 0x00 0x80 0x00 0x00 0x00 0x01
00000000002b00c0: 0x00 0x00 0x00 0x80 0x00 0x00 0x00 0x01
00000000002b00c8: 0x00 0x00 0x00 0x01 0x00 0x00 0x00 0x90
00000000002b00d0: 0x90 0x90 0x55 0x89 0xe5 0x57 0x8b 0x7d
00000000002b00d8: 0x0c 0x56 0x53 0x8b 0x5d 0x08 0x6a 0x23
00000000002b00e0: 0x6a 0x00 0x68 0x80 0x05 0x00 0x00 0xe8
00000000002b00e8: 0xf3 0xcf 0x00 0x00 0x83 0xc4 0x0c 0x83
00000000002b00f0: 0x3d 0xe0 0x04 0x03 0x00 0x00 0xc6 0x05
00000000002b00f8: 0x80 0x05 0x00 0x00 0x13 0x74 0x0d 0x53

Best regards,
Joe

Re: KVM entry failed, hardware error

[Johannes Bauer]

On 04.06.2012 20:28, Johannes Bauer wrote:
>> What is your command line?
> 
> bin/qemu-system-x86_64 -cpu host -enable-kvm -net nic -net
> user,smb=Share,restrict=on -drive
> media=disk,file=Windows7_x32.qcow2,if=virtio -m 2048 -smp 1 -nographic

Just noticed that the output I just provided was for the 32 Bit version
of Windows 7. Did the same (info pci and the memdump) for the 64 Bit
version again and diffed them with meld -- they're *perfectly*
identical. Just so there isn't any confusion.

Best regards,
Joe

Re: KVM entry failed, hardware error

[David Ahern]

On 6/4/12 12:28 PM, Johannes Bauer wrote:
> On 04.06.2012 10:53, Gleb Natapov wrote:
>> On Sun, Jun 03, 2012 at 06:25:33PM +0200, Johannes Bauer wrote:
>>> Therefore, I've uploaded the compressed trace.dat file, so you can maybe
>>> have a look why the "report" tool barfs and interpret it correctly. I
>>> can't figure it out. The trace is here:
>>>
>>> http://spornkuller.de/trace.dat.bz2
>>>
>> I can read this trace.
> Hm, weird. But good that it works on your side. I get a lot of:
>
> trace-cmd: No such file or directory
>    bad op token {
>    failed to read event print fmt for kvm_emulate_insn
> version = 6
> CPU 0 is empty
>

You need to install the plugins for trace-cmd:
make install_plugins

you could also try setting:
export TRACE_CMD_PLUGIN_DIR=/path/to/plugins

David

Re: KVM entry failed, hardware error

[Avi Kivity]

On 06/03/2012 04:01 PM, Johannes Bauer wrote:

>> # rmmod kvm_intel
>> # modprobe kvm_intel emulate_invalid_guest_state=1
>> 
>> Warning: experimental code.
> 
> Tried it out -- I don't get the error anymore, but qemu just hangs on
> boot with 100% CPU. :-(

Please retry with

  git://git.kernel.org/pub/scm/virt/kvm/kvm.git big-real-mode

-- 
error compiling committee.c: too many arguments to function

Re: KVM entry failed, hardware error

[Johannes Bauer]

On 06.06.2012 17:53, Avi Kivity wrote:

>>> # rmmod kvm_intel
>>> # modprobe kvm_intel emulate_invalid_guest_state=1
>>>
>>> Warning: experimental code.
>>
>> Tried it out -- I don't get the error anymore, but qemu just hangs on
>> boot with 100% CPU. :-(
> 
> Please retry with
> 
>   git://git.kernel.org/pub/scm/virt/kvm/kvm.git big-real-mode

Same thing:

$ uname -a
Linux joequad 3.5.0-rc1-46053-gf2ebd42 #1 SMP PREEMPT Wed Jun 6 19:49:21
CEST 2012 x86_64 Intel(R) Core(TM)2 Quad CPU Q9550 @ 2.83GHz
GenuineIntel GNU/Linux

$ cat /proc/cmdline
BOOT_IMAGE=/boot/vmlinuz-3.5.0-rc1-46053-gf2ebd42 root=/dev/sda1 ro
kvm-intel.emulate_invalid_guest_state=1

QEmu hangs with 100% CPU. Is there anything I can try to be able to tell
you *where* it hangs?

Best regards,
Johannes

Re: KVM entry failed, hardware error

[Johannes Bauer]

On 05.06.2012 15:59, David Ahern wrote:

> You need to install the plugins for trace-cmd:
> make install_plugins
> 
> you could also try setting:
> export TRACE_CMD_PLUGIN_DIR=/path/to/plugins

Aha, with that I get more messages:

trace-cmd: No such file or directory
  cound not load plugin '/home/joe/qEmu/trace/trace-cmd/plugin_kvm.so'
/home/joe/qemu/trace/trace-cmd/plugin_kvm.so: undefined symbol:
ud_translate_att

However, during the build process, udis86 is linked in like it should be:

[...]
gcc -g -Wall  -I.  '-DPLUGIN_DIR="/usr/local/share/trace-cmd/plugins"'
-DHAVE_UDIS86 -ludis86  -DHAVE_BLK_TC_FLUSH  -shared -nostartfiles -o
plugin_blk.so plugin_blk.o

But when looking, I just noticed that it does not appear in the imports

$ ldd plugin_kvm.so
	linux-vdso.so.1 =>  (0x00007fff3cfff000)
	libc.so.6 => /lib64/libc.so.6 (0x00007f6429e1c000)
	/lib64/ld-linux-x86-64.so.2 (0x00007f642a3e4000)

Nor do I actually *have* the .so of udis86 in my /usr/lib directory.
Gentoo by default just built the static library. Weird! I'll recompile
myself and then it'll work for sure.

Best regards,
Joe

Re: KVM entry failed, hardware error

[Avi Kivity]

On 06/06/2012 09:07 PM, Johannes Bauer wrote:
> On 06.06.2012 17:53, Avi Kivity wrote:
> 
>>>> # rmmod kvm_intel
>>>> # modprobe kvm_intel emulate_invalid_guest_state=1
>>>>
>>>> Warning: experimental code.
>>>
>>> Tried it out -- I don't get the error anymore, but qemu just hangs on
>>> boot with 100% CPU. :-(
>> 
>> Please retry with
>> 
>>   git://git.kernel.org/pub/scm/virt/kvm/kvm.git big-real-mode
> 
> Same thing:
> 
> $ uname -a
> Linux joequad 3.5.0-rc1-46053-gf2ebd42 #1 SMP PREEMPT Wed Jun 6 19:49:21
> CEST 2012 x86_64 Intel(R) Core(TM)2 Quad CPU Q9550 @ 2.83GHz
> GenuineIntel GNU/Linux
> 
> $ cat /proc/cmdline
> BOOT_IMAGE=/boot/vmlinuz-3.5.0-rc1-46053-gf2ebd42 root=/dev/sda1 ro
> kvm-intel.emulate_invalid_guest_state=1
> 
> QEmu hangs with 100% CPU. Is there anything I can try to be able to tell
> you *where* it hangs?
> 


add -monitor stdio to the command line and then:

(qemu) info registers
(qemu) x/20i 0xcsbase + $eip


(for csbase substitute the second value for CS in 'info registers')

Run info registers a few times and note whether eip changes or not.

-- 
error compiling committee.c: too many arguments to function

Re: KVM entry failed, hardware error

[Johannes Bauer]

On 07.06.2012 09:12, Avi Kivity wrote:

> add -monitor stdio to the command line and then:
> 
> (qemu) info registers
> (qemu) x/20i 0xcsbase + $eip
>
> Run info registers a few times and note whether eip changes or not.

It does not. Here's where it hangs:

(qemu) info registers
EAX=000023de EBX=00000b70 ECX=00000b90 EDX=00000002
ESI=002523de EDI=00000b84 EBP=0000146e ESP=0000146e
EIP=000008d7 EFL=00000202 [-------] CPL=3 II=0 A20=1 SMM=0 HLT=0
ES =23de 00023de0 0000ffff 0000f300
CS =2000 00020000 0000ffff 0000f300
SS =23de 00023de0 0000ffff 0000f300
DS =23de 00023de0 0000ffff 0000f300
FS =0060 00023de0 0000ffff 00009300
GS =0060 00023de0 0000ffff 00009300
LDT=0000 00000000 ffffffff 00c00000
TR =0040 feffd000 00002088 00008b00
GDT=     0001f000 0000007f
IDT=     00000000 0000ffff
CR0=00000010 CR2=00000000 CR3=00000000 CR4=00000000
DR0=0000000000000000 DR1=0000000000000000 DR2=0000000000000000
DR3=0000000000000000
DR6=00000000ffff0ff0 DR7=0000000000000400
EFER=0000000000000000
FCW=037f FSW=0000 [ST=0] FTW=00 MXCSR=00001f80
FPR0=0000000000000000 0000 FPR1=0000000000000000 0000
FPR2=0000000000000000 0000 FPR3=0000000000000000 0000
FPR4=0000000000000000 0000 FPR5=0000000000000000 0000
FPR6=0000000000000000 0000 FPR7=0000000000000000 0000
XMM00=00000000000000000000000000000000
XMM01=00000000000000000000000000000000
XMM02=00000000000000000000000000000000
XMM03=00000000000000000000000000000000
XMM04=00000000000000000000000000000000
XMM05=00000000000000000000000000000000
XMM06=00000000000000000000000000000000
XMM07=00000000000000000000000000000000

(qemu) x/20i 0x20000 + $eip
0x00000000000208d7:  leave
0x00000000000208d8:  ret
0x00000000000208d9:  enter  $0x0,$0x0
0x00000000000208dd:  push   %ebp
0x00000000000208df:  push   %ebx
0x00000000000208e1:  push   %esi
0x00000000000208e3:  push   %edi
0x00000000000208e5:  mov    %esp,%ebx
0x00000000000208e8:  mov    %ebx,%edi
0x00000000000208eb:  add    $0x14,%edi
0x00000000000208ef:  addr32 mov (%edi),%eax
0x00000000000208f3:  mov    $0x1480,%sp
0x00000000000208f6:  xor    %bp,%bp
0x00000000000208f8:  movzwl %bp,%ebp
0x00000000000208fc:  movzwl %sp,%esp
0x0000000000020900:  push   %ebx
0x0000000000020902:  push   %eax
0x0000000000020904:  call   0x20919
0x0000000000020907:  add    $0x4,%sp
0x000000000002090a:  pop    %ebx

And this is where it came from and tries to return to:

(qemu) x /8hx 0x23de0 + $esp
000000000002524e: 0x1474 0x092a 0x0001 0x0000 0x0907 0x4970 0x0002 0x0b70

(qemu) x/20i 0x20000 + 0x92a - 0x15
0x0000000000020915:  pop    %ebp
0x0000000000020917:  leave
0x0000000000020918:  ret
0x0000000000020919:  enter  $0x0,$0x0
0x000000000002091d:  mov    0x1510,%ax
0x0000000000020920:  push   %ax
0x0000000000020921:  and    %ax,%ax
0x0000000000020923:  je     0x2092a
0x0000000000020927:  call   0x20871
0x000000000002092a:  push   %bx
0x000000000002092b:  push   %di
0x000000000002092c:  push   %si
0x000000000002092d:  push   %ds
0x000000000002092e:  push   %es
0x000000000002092f:  push   %bp
0x0000000000020930:  mov    0x4(%bp),%eax
0x0000000000020934:  mov    %ax,%bp
0x0000000000020936:  and    $0xf,%bp
0x0000000000020939:  shr    $0x4,%eax
0x000000000002093d:  mov    %ax,%ds

Here's the whole function that causes the hangup:

(qemu) x/39i 0x20000 + 0x871
0x0000000000020871:  enter  $0x0,$0x0
0x0000000000020875:  push   %ebx
0x0000000000020877:  mov    0x1510,%ax
0x000000000002087a:  and    %ax,%ax
0x000000000002087c:  je     0x208d5
0x0000000000020880:  sgdtw  0x1500
0x0000000000020885:  sidtw  0x1508
0x000000000002088a:  movw   $0x0,0x1510
0x0000000000020890:  mov    %cr0,%eax
0x0000000000020893:  mov    %eax,0x1514
0x0000000000020897:  and    $0x7ffffffe,%eax
0x000000000002089d:  mov    %eax,%cr0
0x00000000000208a0:  jmp    0x208a5
0x00000000000208a2:  nop
0x00000000000208a3:  nop
0x00000000000208a4:  nop
0x00000000000208a5:  mov    %cr3,%eax
0x00000000000208a8:  nop
0x00000000000208a9:  nop
0x00000000000208aa:  nop
0x00000000000208ab:  nop
0x00000000000208ac:  mov    %eax,%cr3
0x00000000000208af:  pushw  0x1536
0x00000000000208b3:  pop    %es
0x00000000000208b4:  mov    $0x8c6,%bx
0x00000000000208b7:  mov    0x1536,%ax
0x00000000000208ba:  mov    %ax,%es:-0x2(%bx)
0x00000000000208be:  ljmp   *%es:-0x4(%bx)
0x00000000000208c2:  (bad)
0x00000000000208c3:  or     %al,(%bx,%si)
0x00000000000208c5:  and    %ah,0x1534(%bx,%di)
0x00000000000208c9:  mov    %ax,%ds
0x00000000000208cb:  mov    %ax,%ss
0x00000000000208cd:  mov    %ax,%es
0x00000000000208cf:  lidtw  0x14f8
0x00000000000208d4:  sti
0x00000000000208d5:  pop    %ebx
0x00000000000208d7:  leave
0x00000000000208d8:  ret

Best regards,
Joe

Re: KVM entry failed, hardware error

[Avi Kivity]

On 06/07/2012 01:03 PM, Johannes Bauer wrote:
> On 07.06.2012 09:12, Avi Kivity wrote:
> 
>> add -monitor stdio to the command line and then:
>> 
>> (qemu) info registers
>> (qemu) x/20i 0xcsbase + $eip
>>
>> Run info registers a few times and note whether eip changes or not.
> 
> It does not. Here's where it hangs:
> 
> (qemu) x/20i 0x20000 + $eip
> 0x00000000000208d7:  leave

Pretty straightforward, we need to emulate the leave instruction.  I'll
update the branch and notify you.

-- 
error compiling committee.c: too many arguments to function

Re: KVM entry failed, hardware error

[Avi Kivity]

On 06/07/2012 01:54 PM, Avi Kivity wrote:
> On 06/07/2012 01:03 PM, Johannes Bauer wrote:
>> On 07.06.2012 09:12, Avi Kivity wrote:
>> 
>>> add -monitor stdio to the command line and then:
>>> 
>>> (qemu) info registers
>>> (qemu) x/20i 0xcsbase + $eip
>>>
>>> Run info registers a few times and note whether eip changes or not.
>> 
>> It does not. Here's where it hangs:
>> 
>> (qemu) x/20i 0x20000 + $eip
>> 0x00000000000208d7:  leave
> 
> Pretty straightforward, we need to emulate the leave instruction.  I'll
> update the branch and notify you.
> 

Please try the big-real-mode branch again.  It contains emulation for
the missing instruction, plus a bunch of tweaks which allowed it to boot
Fedora 17 smp with emulate_invalid_guest_state=1.

-- 
error compiling committee.c: too many arguments to function

Re: KVM entry failed, hardware error

[Johannes Bauer]

On 07.06.2012 16:52, Avi Kivity wrote:

> Please try the big-real-mode branch again.  It contains emulation for
> the missing instruction, plus a bunch of tweaks which allowed it to boot
> Fedora 17 smp with emulate_invalid_guest_state=1.

Progress!

So now I'm on

Linux joequad 3.5.0-rc1-46078-g54d5c7c #2 SMP PREEMPT Thu Jun 7 17:11:38
CEST 2012 x86_64 Intel(R) Core(TM)2 Quad CPU Q9550 @ 2.83GHz
GenuineIntel GNU/Linux

And see the following (kind of weird) behavior: I can boot up now. But
it's bound to some condition:

When not using "-monitor stdio", I have the same behavior as before.

When using "-monitor stdio", it locks up occasionaly, but appears to
resume operation whenever I type "info registers". That way I was able
to boot up (by entering about 50 times "info registers", then it had
left real mode and booted up normally).

I was first a bit confused, but this is definitely what I'm seeing:
Because of shutting down the VM I had put the guest in a "Windows
Recovery" mode where a text-mode progress bar appears with the text
"Windows is loading files...". The bar locks up -- when I type "info
registers", it progresses about 2-4 characters and locks up again (until
it's fully loaded, then it just works).

Note that "c" does NOT cause the VM to resume, only "info registers"
does. dmesg shows nothing out of the ordinary.

But I think we're getting somewhere! Thank you very much for your
support already!

Best regards,
Joe

Re: KVM entry failed, hardware error

[Avi Kivity]

On 06/07/2012 06:39 PM, Johannes Bauer wrote:
> On 07.06.2012 16:52, Avi Kivity wrote:
>
> > Please try the big-real-mode branch again.  It contains emulation for
> > the missing instruction, plus a bunch of tweaks which allowed it to boot
> > Fedora 17 smp with emulate_invalid_guest_state=1.
>
> Progress!

Great!

> So now I'm on
>
> Linux joequad 3.5.0-rc1-46078-g54d5c7c #2 SMP PREEMPT Thu Jun 7 17:11:38
> CEST 2012 x86_64 Intel(R) Core(TM)2 Quad CPU Q9550 @ 2.83GHz
> GenuineIntel GNU/Linux
>
> And see the following (kind of weird) behavior: I can boot up now. But
> it's bound to some condition:
>
> When not using "-monitor stdio", I have the same behavior as before.
>
> When using "-monitor stdio", it locks up occasionaly, but appears to
> resume operation whenever I type "info registers". That way I was able
> to boot up (by entering about 50 times "info registers", then it had
> left real mode and booted up normally).
>
> I was first a bit confused, but this is definitely what I'm seeing:
> Because of shutting down the VM I had put the guest in a "Windows
> Recovery" mode where a text-mode progress bar appears with the text
> "Windows is loading files...". The bar locks up -- when I type "info
> registers", it progresses about 2-4 characters and locks up again (until
> it's fully loaded, then it just works).
>
> Note that "c" does NOT cause the VM to resume, only "info registers"
> does. dmesg shows nothing out of the ordinary.

I'm guessing this is 5152902652.  Try bumping 'unsigned count = 130' (by
adding zeros at the end, don't bother with anything less).  If you
increase it too much qemu may hang; but kill -9 should unfreeze it.


-- 
I have a truly marvellous patch that fixes the bug which this
signature is too narrow to contain.

Re: KVM entry failed, hardware error

[Johannes Bauer]

On 07.06.2012 19:25, Avi Kivity wrote:

>> Note that "c" does NOT cause the VM to resume, only "info registers"
>> does. dmesg shows nothing out of the ordinary.
> 
> I'm guessing this is 5152902652.  Try bumping 'unsigned count = 130' (by
> adding zeros at the end, don't bother with anything less).  If you
> increase it too much qemu may hang; but kill -9 should unfreeze it.

Doesn't seem to be right -- still got the same problem. I first bumped
it up to 1300 and inserted debugging output to see how many cycles are
actually spent in the loop. It enters the emulation mode so frequently
(and leaves it again) that the dmesg buffer ran over (128kB). So I
changed the debugging to give me the lowest cycle count that it ever has
after the loop:

handle_invalid_guest_state: emulation left, new low count 1295
handle_invalid_guest_state: emulation left, new low count 1292
handle_invalid_guest_state: emulation left, new low count 1291
handle_invalid_guest_state: emulation left, new low count 1245

Which means that it spends a maximum of 55 cycles in the loop (well
below the original 130 even). So my change had no effect. Any other
ideas maybe?

Best regards,
Joe

Re: KVM entry failed, hardware error

[Johannes Bauer]

On 07.06.2012 19:25, Avi Kivity wrote:

>> Note that "c" does NOT cause the VM to resume, only "info registers"
>> does. dmesg shows nothing out of the ordinary.
> 
> I'm guessing this is 5152902652.  Try bumping 'unsigned count = 130' (by
> adding zeros at the end, don't bother with anything less).  If you
> increase it too much qemu may hang; but kill -9 should unfreeze it.

Okay, here's more clues. They're not exactly meaningful for me, but
probably of interest to you.

Besides the "low count value after loop exit" value, I've introduced two
counters which both are incremented inside the loop. Also I've modified
the loop to be one return only (i.e. ret = foobar; goto out instead of
return).

Then when I have a script which continuously echos "info registers" and
pipe that output together with qemu, I get this startup:

2012-06-07 21:39:34 handle_invalid_guest_state: emulation left, new loop
low count 1295, total of 5 emulated insns
2012-06-07 21:39:34 handle_invalid_guest_state: emulation left, new loop
low count 1292, total of 13 emulated insns
2012-06-07 21:39:34 handle_invalid_guest_state: emulation left, 1293
iterations left, 100005 emulated insn, loop low count 1292
2012-06-07 21:39:34 handle_invalid_guest_state: emulation left, new loop
low count 1291, total of 131214 emulated insns
2012-06-07 21:39:34 handle_invalid_guest_state: emulation left, new loop
low count 1245, total of 131269 emulated insns
2012-06-07 21:39:35 handle_invalid_guest_state: emulation left, 1299
iterations left, 200005 emulated insn, loop low count 1245
2012-06-07 21:39:35 handle_invalid_guest_state: emulation left, 1245
iterations left, 300012 emulated insn, loop low count 1245
2012-06-07 21:39:35 handle_invalid_guest_state: emulation left, 1291
iterations left, 400013 emulated insn, loop low count 1245
2012-06-07 21:39:35 handle_invalid_guest_state: emulation left, 1245
iterations left, 500050 emulated insn, loop low count 1245
2012-06-07 21:39:35 handle_invalid_guest_state: emulation left, 1245
iterations left, 600059 emulated insn, loop low count 1245
2012-06-07 21:39:35 handle_invalid_guest_state: emulation left, 1299
iterations left, 700059 emulated insn, loop low count 1245
2012-06-07 21:39:35 handle_invalid_guest_state: emulation left, 1299
iterations left, 800059 emulated insn, loop low count 1245
2012-06-07 21:39:36 handle_invalid_guest_state: emulation left, 1299
iterations left, 900059 emulated insn, loop low count 1245
2012-06-07 21:39:36 handle_invalid_guest_state: emulation left, 1299
iterations left, 1000059 emulated insn, loop low count 1245
2012-06-07 21:39:36 handle_invalid_guest_state: emulation left, new loop
low count 1228, total of 1030349 emulated insns
2012-06-07 21:39:36 handle_invalid_guest_state: emulation left, 1291
iterations left, 1100063 emulated insn, loop low count 1228
2012-06-07 21:39:36 handle_invalid_guest_state: emulation left, 1299
iterations left, 1200063 emulated insn, loop low count 1228
2012-06-07 21:39:36 handle_invalid_guest_state: emulation left, 1299
iterations left, 1300063 emulated insn, loop low count 1228
2012-06-07 21:39:36 handle_invalid_guest_state: emulation left, 1245
iterations left, 1400113 emulated insn, loop low count 1228
2012-06-07 21:39:37 handle_invalid_guest_state: emulation left, 1245
iterations left, 1500145 emulated insn, loop low count 1228

After which it has booted up and does not emulate anymore. Note the
maximum time spent in the loop is 72 iterations.

But when I just start up and do not issue ANY "info registers", I get
the following output:

2012-06-07 21:41:44 handle_invalid_guest_state: emulation left, new loop
low count 1295, total of 5 emulated insns
2012-06-07 21:41:44 handle_invalid_guest_state: emulation left, new loop
low count 1292, total of 13 emulated insns
2012-06-07 21:41:44 handle_invalid_guest_state: emulation left, 1293
iterations left, 100004 emulated insn, loop low count 1292
2012-06-07 21:41:44 handle_invalid_guest_state: emulation left, new loop
low count 1291, total of 131955 emulated insns
2012-06-07 21:41:44 handle_invalid_guest_state: emulation left, new loop
low count 1245, total of 132010 emulated insns
2012-06-07 21:41:44 handle_invalid_guest_state: emulation left, 1299
iterations left, 200004 emulated insn, loop low count 1245
2012-06-07 21:41:45 handle_invalid_guest_state: emulation left, 1299
iterations left, 300004 emulated insn, loop low count 1245
2012-06-07 21:41:45 handle_invalid_guest_state: emulation left, 1299
iterations left, 400004 emulated insn, loop low count 1245
2012-06-07 21:41:45 handle_invalid_guest_state: emulation left, 1299
iterations left, 500004 emulated insn, loop low count 1245
2012-06-07 21:41:45 handle_invalid_guest_state: emulation left, 1299
iterations left, 600004 emulated insn, loop low count 1245
2012-06-07 21:41:45 handle_invalid_guest_state: emulation left, 1299
iterations left, 700004 emulated insn, loop low count 1245
2012-06-07 21:41:45 handle_invalid_guest_state: emulation left, 1299
iterations left, 800004 emulated insn, loop low count 1245
2012-06-07 21:41:45 handle_invalid_guest_state: emulation left, 1299
iterations left, 900004 emulated insn, loop low count 1245
2012-06-07 21:41:45 handle_invalid_guest_state: emulation left, 1299
iterations left, 1000004 emulated insn, loop low count 1245
[...]
2012-06-07 21:42:12 handle_invalid_guest_state: emulation left, 1299
iterations left, 22800004 emulated insn, loop low count 1245
2012-06-07 21:42:12 handle_invalid_guest_state: emulation left, 1299
iterations left, 22900004 emulated insn, loop low count 1245
2012-06-07 21:42:12 handle_invalid_guest_state: emulation left, 1299
iterations left, 23000004 emulated insn, loop low count 1245

In an infinite loop. Which looks to be as if it is continuously exiting
after just one iteration (count at leave is 1299). Maybe I'll fiddle
some more and am able to provide some insight (probably you already know
what's going on, but it won't hurt I guess).

Best regards,
Joe

Re: KVM entry failed, hardware error

[Johannes Bauer]

On 07.06.2012 21:46, Johannes Bauer wrote:

> In an infinite loop. Which looks to be as if it is continuously exiting
> after just one iteration (count at leave is 1299). Maybe I'll fiddle
> some more and am able to provide some insight (probably you already know
> what's going on, but it won't hurt I guess).

Followup on that: I've shorted the debug output (so it doesn't get
wrapped in mail and can be more easily read) and I've enumerated all
five possible exists from the loop:

1. intr_window_requested && (kvm_get_rflags(&vmx->vcpu) & X86_EFLAGS_IF)
2. err == EMULATE_DO_MMIO
3. err != EMULATE_DONE
4. signal_pending(current)
5. while loop exit

And added that to the loop. When it hangs, it always exists by #1:

higs: new low cnt 1295, 5 emu insns
higs: new low cnt 1292, 13 emu insns
higs: left count=1293, 100002 emu insns, llc=1292, rsn=5
higs: new low cnt 1291, 135348 emu insns
higs: new low cnt 1245, 135403 emu insns
higs: new low cnt 1228, 183120 emu insns
higs: left count=1291, 200007 emu insns, llc=1228, rsn=5
higs: left count=1299, 300007 emu insns, llc=1228, rsn=1
higs: left count=1299, 400007 emu insns, llc=1228, rsn=1
higs: left count=1299, 500007 emu insns, llc=1228, rsn=1
higs: left count=1299, 600007 emu insns, llc=1228, rsn=1
higs: left count=1299, 700007 emu insns, llc=1228, rsn=1
higs: left count=1299, 800007 emu insns, llc=1228, rsn=1
higs: left count=1299, 900007 emu insns, llc=1228, rsn=1
higs: left count=1299, 1000007 emu insns, llc=1228, rsn=1
higs: left count=1299, 1100007 emu insns, llc=1228, rsn=1
higs: left count=1299, 1200007 emu insns, llc=1228, rsn=1
higs: left count=1299, 1300007 emu insns, llc=1228, rsn=1
higs: left count=1299, 1400007 emu insns, llc=1228, rsn=1
higs: left count=1299, 1500007 emu insns, llc=1228, rsn=1
higs: left count=1299, 1600007 emu insns, llc=1228, rsn=1
higs: left count=1299, 1700007 emu insns, llc=1228, rsn=1
higs: left count=1299, 1800007 emu insns, llc=1228, rsn=1
higs: left count=1299, 1900007 emu insns, llc=1228, rsn=1
higs: left count=1299, 2000007 emu insns, llc=1228, rsn=1
higs: left count=1299, 2100007 emu insns, llc=1228, rsn=1
[...]

If there's any more output I can provide to help you track down the
problem at hand, please let me know.

Best regards,
Joe

Re: KVM entry failed, hardware error

[Avi Kivity]

On 06/07/2012 10:14 PM, Johannes Bauer wrote:
> On 07.06.2012 19:25, Avi Kivity wrote:
> 
>>> Note that "c" does NOT cause the VM to resume, only "info registers"
>>> does. dmesg shows nothing out of the ordinary.
>> 
>> I'm guessing this is 5152902652.  Try bumping 'unsigned count = 130' (by
>> adding zeros at the end, don't bother with anything less).  If you
>> increase it too much qemu may hang; but kill -9 should unfreeze it.
> 
> Doesn't seem to be right -- still got the same problem. I first bumped
> it up to 1300 and inserted debugging output to see how many cycles are
> actually spent in the loop. It enters the emulation mode so frequently
> (and leaves it again) that the dmesg buffer ran over (128kB). So I
> changed the debugging to give me the lowest cycle count that it ever has
> after the loop:
> 
> handle_invalid_guest_state: emulation left, new low count 1295
> handle_invalid_guest_state: emulation left, new low count 1292
> handle_invalid_guest_state: emulation left, new low count 1291
> handle_invalid_guest_state: emulation left, new low count 1245
> 
> Which means that it spends a maximum of 55 cycles in the loop (well
> below the original 130 even). So my change had no effect. Any other
> ideas maybe?
> 

Looks like we weren't dealing with interrupts correctly.  I pushed some
patches, please pull again and retry.


-- 
error compiling committee.c: too many arguments to function

Re: KVM entry failed, hardware error

[Johannes Bauer]

On 10.06.2012 17:19, Avi Kivity wrote:

> Looks like we weren't dealing with interrupts correctly.  I pushed some
> patches, please pull again and retry.

Updated to cf3d9372065470403e0780599ca612553211a10b and it works
perfectly for me!

Thank you very much for the support! Very cool!
Best regards,
Joe

Re: KVM entry failed, hardware error

[Avi Kivity]

On 06/12/2012 01:23 PM, Johannes Bauer wrote:
> On 10.06.2012 17:19, Avi Kivity wrote:
> 
>> Looks like we weren't dealing with interrupts correctly.  I pushed some
>> patches, please pull again and retry.
> 
> Updated to cf3d9372065470403e0780599ca612553211a10b and it works
> perfectly for me!

Good to hear.  We'll try to get emulate_invalid_guest_state=1 to be the
default for 3.6.


-- 
error compiling committee.c: too many arguments to function