how to debug unhandled vm exit: 0x11?

how to debug unhandled vm exit: 0x11?

[Neo Jia]

hi,

I am seeing an unhandled vm exit: 0x11 on Win7 with KVM-88 release and
wondering if I am still able to dump the code from guest OS when this
happens. But it looks that all instructions are 0s after adding one
more print code after dumping the guest registers.

And it is very likely that this problem is fixed in the latest qemu
code base but I still would like to know how to debug and investigate
this kind of problem. BTW, I am using 32-bit qemu + 64-bit KVM kernel
module.

unhandled vm exit: 0x11
rax 0000000000000000 rbx 000000006e4ad180 rcx 0000000000000010 rdx
0000000000000300
rsi 000000006e4ad080 rdi 0000000003a7d000 rsp 000000000553fdf0 rbp
0000000000000010
r8  0000000000000000 r9  0000000000000000 r10 0000000000000000 r11
0000000000000000
r12 0000000000000000 r13 0000000000000000 r14 0000000000000000 r15
0000000000000000
rip 000000006e381339 rflags 00010246
cs 001b (00000000/ffffffff p 1 dpl 3 db 1 s 1 type b l 0 g 1 avl 0)
ds 0023 (00000000/ffffffff p 1 dpl 3 db 1 s 1 type 3 l 0 g 1 avl 0)
es 0023 (00000000/ffffffff p 1 dpl 3 db 1 s 1 type 3 l 0 g 1 avl 0)
ss 0023 (00000000/ffffffff p 1 dpl 3 db 1 s 1 type 3 l 0 g 1 avl 0)
fs 003b (7ffd6000/00000fff p 1 dpl 3 db 1 s 1 type 3 l 0 g 0 avl 0)
gs 0000 (00000000/ffffffff p 0 dpl 0 db 0 s 0 type 0 l 0 g 0 avl 0)
tr 0028 (80198000/000020ab p 1 dpl 0 db 0 s 0 type b l 0 g 0 avl 0)
ldt 0000 (00000000/ffffffff p 0 dpl 0 db 0 s 0 type 0 l 0 g 0 avl 0)
gdt 81423000/3ff
idt 81423400/7ff
cr0 80010031 cr2 30d7000 cr3 126d0000 cr4 6d8 cr8 0 efer 0
code: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 -->
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00

Thanks,
Neo
-- 
I would remember that if researchers were not ambitious
probably today we haven't the technology we are using!

Re: how to debug unhandled vm exit: 0x11?

[Avi Kivity]

  On 07/26/2010 07:01 PM, Neo Jia wrote:
> hi,
>
> I am seeing an unhandled vm exit: 0x11 on Win7 with KVM-88 release and
> wondering if I am still able to dump the code from guest OS when this
> happens. But it looks that all instructions are 0s after adding one
> more print code after dumping the guest registers.
>
> And it is very likely that this problem is fixed in the latest qemu
> code base but I still would like to know how to debug and investigate
> this kind of problem. BTW, I am using 32-bit qemu + 64-bit KVM kernel
> module.
>
> unhandled vm exit: 0x11

The first step is to look up exit reason 0x11 in the manual and see what 
it means.

-- 
error compiling committee.c: too many arguments to function

Re: how to debug unhandled vm exit: 0x11?

[ewheeler]

> O
> n 07/26/2010 07:01 PM, Neo Jia wrote:
> > hi,
> >
> > I am seeing an unhandled vm exit: 0x11 on Win7 with KVM-88 release and
> > wondering if I am still able to dump the code from guest OS when this
> > happens. But it looks that all instructions are 0s after adding one
> > more print code after dumping the guest registers.
> >
> > And it is very likely that this problem is fixed in the latest qemu
> > code base but I still would like to know how to debug and investigate
> > this kind of problem. BTW, I am using 32-bit qemu + 64-bit KVM kernel
> > module.
> >
> > unhandled vm exit: 0x11

I happened to be in the intel docs today:
  0x11 (17) RSM. Guest software attempted to execute RSM in SMM.

What is RSM and SMM?

> 
> The first step is to look up exit reason 0x11 in the manual and see what 
> it means.
> 

Re: how to debug unhandled vm exit: 0x11?

[Avi Kivity]

  On 07/26/2010 08:58 PM, ewheeler wrote:
>> O
>> n 07/26/2010 07:01 PM, Neo Jia wrote:
>>> hi,
>>>
>>> I am seeing an unhandled vm exit: 0x11 on Win7 with KVM-88 release and
>>> wondering if I am still able to dump the code from guest OS when this
>>> happens. But it looks that all instructions are 0s after adding one
>>> more print code after dumping the guest registers.
>>>
>>> And it is very likely that this problem is fixed in the latest qemu
>>> code base but I still would like to know how to debug and investigate
>>> this kind of problem. BTW, I am using 32-bit qemu + 64-bit KVM kernel
>>> module.
>>>
>>> unhandled vm exit: 0x11
> I happened to be in the intel docs today:
>    0x11 (17) RSM. Guest software attempted to execute RSM in SMM.

As it happens, the 0x11 should be looked up as a KVM_EXIT_REASON 
(kvm.h), not in the manuals as I said.  It's a kvm internal error.

> What is RSM and SMM?

These are documented in the manuals.

-- 
error compiling committee.c: too many arguments to function

Re: how to debug unhandled vm exit: 0x11?

[Neo Jia]

On Tue, Jul 27, 2010 at 3:04 AM, Avi Kivity <avi@redhat.com> wrote:
>  On 07/26/2010 08:58 PM, ewheeler wrote:
>>>
>>> O
>>> n 07/26/2010 07:01 PM, Neo Jia wrote:
>>>>
>>>> hi,
>>>>
>>>> I am seeing an unhandled vm exit: 0x11 on Win7 with KVM-88 release and
>>>> wondering if I am still able to dump the code from guest OS when this
>>>> happens. But it looks that all instructions are 0s after adding one
>>>> more print code after dumping the guest registers.
>>>>
>>>> And it is very likely that this problem is fixed in the latest qemu
>>>> code base but I still would like to know how to debug and investigate
>>>> this kind of problem. BTW, I am using 32-bit qemu + 64-bit KVM kernel
>>>> module.
>>>>
>>>> unhandled vm exit: 0x11

Avi,

I found the instruction that caused this problem:

emulation failed (failure) rip 71f14651 66 0f 7f 07

And according to Intel, this is a MOVDQA. So, do we already have this
instruction emulated as I am using a pretty old version of KVM
(release 88)? If yes, could you point me to the file I need to look at
for that specific patch?

Currently, I am trying to use coalesced_mmio as you suggested in
another thread:
http://www.mail-archive.com/kvm@vger.kernel.org/msg25695.html

Thanks,
Neo

>>
>> I happened to be in the intel docs today:
>>   0x11 (17) RSM. Guest software attempted to execute RSM in SMM.
>
> As it happens, the 0x11 should be looked up as a KVM_EXIT_REASON (kvm.h),
> not in the manuals as I said.  It's a kvm internal error.
>
>> What is RSM and SMM?
>
> These are documented in the manuals.
>
> --
> error compiling committee.c: too many arguments to function
>
>



-- 
I would remember that if researchers were not ambitious
probably today we haven't the technology we are using!

Re: how to debug unhandled vm exit: 0x11?

[Neo Jia]

On Tue, Sep 28, 2010 at 11:40 AM, Neo Jia <neojia@gmail.com> wrote:
> On Tue, Jul 27, 2010 at 3:04 AM, Avi Kivity <avi@redhat.com> wrote:
>>  On 07/26/2010 08:58 PM, ewheeler wrote:
>>>>
>>>> O
>>>> n 07/26/2010 07:01 PM, Neo Jia wrote:
>>>>>
>>>>> hi,
>>>>>
>>>>> I am seeing an unhandled vm exit: 0x11 on Win7 with KVM-88 release and
>>>>> wondering if I am still able to dump the code from guest OS when this
>>>>> happens. But it looks that all instructions are 0s after adding one
>>>>> more print code after dumping the guest registers.
>>>>>
>>>>> And it is very likely that this problem is fixed in the latest qemu
>>>>> code base but I still would like to know how to debug and investigate
>>>>> this kind of problem. BTW, I am using 32-bit qemu + 64-bit KVM kernel
>>>>> module.
>>>>>
>>>>> unhandled vm exit: 0x11
>
> Avi,
>
> I found the instruction that caused this problem:
>
> emulation failed (failure) rip 71f14651 66 0f 7f 07
>
> And according to Intel, this is a MOVDQA. So, do we already have this
> instruction emulated as I am using a pretty old version of KVM
> (release 88)? If yes, could you point me to the file I need to look at
> for that specific patch?
>
> Currently, I am trying to use coalesced_mmio as you suggested in
> another thread:
> http://www.mail-archive.com/kvm@vger.kernel.org/msg25695.html

Just found out that coalesced_mmio doesn't help in my case.

Thanks,
Neo

>
> Thanks,
> Neo
>
>>>
>>> I happened to be in the intel docs today:
>>>   0x11 (17) RSM. Guest software attempted to execute RSM in SMM.
>>
>> As it happens, the 0x11 should be looked up as a KVM_EXIT_REASON (kvm.h),
>> not in the manuals as I said.  It's a kvm internal error.
>>
>>> What is RSM and SMM?
>>
>> These are documented in the manuals.
>>
>> --
>> error compiling committee.c: too many arguments to function
>>
>>
>
>
>
> --
> I would remember that if researchers were not ambitious
> probably today we haven't the technology we are using!
>



-- 
I would remember that if researchers were not ambitious
probably today we haven't the technology we are using!

Re: how to debug unhandled vm exit: 0x11?

[Avi Kivity]

  On 09/28/2010 08:40 PM, Neo Jia wrote:
> I found the instruction that caused this problem:
>
> emulation failed (failure) rip 71f14651 66 0f 7f 07
>
> And according to Intel, this is a MOVDQA. So, do we already have this
> instruction emulated as I am using a pretty old version of KVM
> (release 88)? If yes, could you point me to the file I need to look at
> for that specific patch?
>

movdqa is not emulated.

-- 
I have a truly marvellous patch that fixes the bug which this
signature is too narrow to contain.

Re: how to debug unhandled vm exit: 0x11?

[Neo Jia]

On Wed, Sep 29, 2010 at 1:38 AM, Avi Kivity <avi@redhat.com> wrote:
>  On 09/28/2010 08:40 PM, Neo Jia wrote:
>>
>> I found the instruction that caused this problem:
>>
>> emulation failed (failure) rip 71f14651 66 0f 7f 07
>>
>> And according to Intel, this is a MOVDQA. So, do we already have this
>> instruction emulated as I am using a pretty old version of KVM
>> (release 88)? If yes, could you point me to the file I need to look at
>> for that specific patch?
>>
>
> movdqa is not emulated.

I am going to give a try to emulate this instruction. BTW, do we have
any unit test for those emulated x86 instruction sets?

Thanks,
Neo

>
> --
> I have a truly marvellous patch that fixes the bug which this
> signature is too narrow to contain.
>
>



-- 
I would remember that if researchers were not ambitious
probably today we haven't the technology we are using!

Re: how to debug unhandled vm exit: 0x11?

[Avi Kivity]

  On 10/01/2010 08:16 AM, Neo Jia wrote:
> On Wed, Sep 29, 2010 at 1:38 AM, Avi Kivity<avi@redhat.com>  wrote:
> >    On 09/28/2010 08:40 PM, Neo Jia wrote:
> >>
> >>  I found the instruction that caused this problem:
> >>
> >>  emulation failed (failure) rip 71f14651 66 0f 7f 07
> >>
> >>  And according to Intel, this is a MOVDQA. So, do we already have this
> >>  instruction emulated as I am using a pretty old version of KVM
> >>  (release 88)? If yes, could you point me to the file I need to look at
> >>  for that specific patch?
> >>
> >
> >  movdqa is not emulated.
>
> I am going to give a try to emulate this instruction. BTW, do we have
> any unit test for those emulated x86 instruction sets?

Indeed we do:

http://git.kernel.org/?p=virt/kvm/kvm-unit-tests.git;a=summary

Check out x86/emulator.c.

-- 
I have a truly marvellous patch that fixes the bug which this
signature is too narrow to contain.

Re: how to debug unhandled vm exit: 0x11?

[Neo Jia]

On Fri, Oct 1, 2010 at 7:29 AM, Avi Kivity <avi@redhat.com> wrote:
>  On 10/01/2010 08:16 AM, Neo Jia wrote:
>>
>> On Wed, Sep 29, 2010 at 1:38 AM, Avi Kivity<avi@redhat.com>  wrote:
>> >    On 09/28/2010 08:40 PM, Neo Jia wrote:
>> >>
>> >>  I found the instruction that caused this problem:
>> >>
>> >>  emulation failed (failure) rip 71f14651 66 0f 7f 07
>> >>
>> >>  And according to Intel, this is a MOVDQA. So, do we already have this
>> >>  instruction emulated as I am using a pretty old version of KVM
>> >>  (release 88)? If yes, could you point me to the file I need to look at
>> >>  for that specific patch?
>> >>
>> >
>> >  movdqa is not emulated.
>>
>> I am going to give a try to emulate this instruction. BTW, do we have
>> any unit test for those emulated x86 instruction sets?
>
> Indeed we do:
>
> http://git.kernel.org/?p=virt/kvm/kvm-unit-tests.git;a=summary
>
> Check out x86/emulator.c.

BTW, I have a question about saving FPU, especially those XMM
registers. I don't see an explicit save FPU after exiting guest due to
an exception (MMIO writes). The only thing I saw about fpu operation
is fpu restore right before loading guest.

Is there anything I missed here?

Thanks,
Neo

>
> --
> I have a truly marvellous patch that fixes the bug which this
> signature is too narrow to contain.
>
>



-- 
I would remember that if researchers were not ambitious
probably today we haven't the technology we are using!

Re: how to debug unhandled vm exit: 0x11?

[Avi Kivity]

  On 10/11/2010 07:46 AM, Neo Jia wrote:
> BTW, I have a question about saving FPU, especially those XMM
> registers. I don't see an explicit save FPU after exiting guest due to
> an exception (MMIO writes). The only thing I saw about fpu operation
> is fpu restore right before loading guest.
>
> Is there anything I missed here?

kvm_put_guest_fpu.

-- 
I have a truly marvellous patch that fixes the bug which this
signature is too narrow to contain.

Re: how to debug unhandled vm exit: 0x11?

[Neo Jia]

On Sun, Oct 10, 2010 at 11:30 PM, Avi Kivity <avi@redhat.com> wrote:
>  On 10/11/2010 07:46 AM, Neo Jia wrote:
>>
>> BTW, I have a question about saving FPU, especially those XMM
>> registers. I don't see an explicit save FPU after exiting guest due to
>> an exception (MMIO writes). The only thing I saw about fpu operation
>> is fpu restore right before loading guest.
>>
>> Is there anything I missed here?
>
> kvm_put_guest_fpu.

I found that function and it will be called by vcpu_put eventually
inside kvm_arch_vcpu_ioctl_run, but kvm_mmu_page_fault is called much
earlier than that inside kvm exit exception handler. so, the fxsave
data for the guest image might not be saved at that moment, when I am
going to emulate this instruction?

Thanks,
Neo

>
> --
> I have a truly marvellous patch that fixes the bug which this
> signature is too narrow to contain.
>
>



-- 
I would remember that if researchers were not ambitious
probably today we haven't the technology we are using!

Re: how to debug unhandled vm exit: 0x11?

[Avi Kivity]

  On 10/11/2010 08:49 AM, Neo Jia wrote:
> On Sun, Oct 10, 2010 at 11:30 PM, Avi Kivity<avi@redhat.com>  wrote:
> >    On 10/11/2010 07:46 AM, Neo Jia wrote:
> >>
> >>  BTW, I have a question about saving FPU, especially those XMM
> >>  registers. I don't see an explicit save FPU after exiting guest due to
> >>  an exception (MMIO writes). The only thing I saw about fpu operation
> >>  is fpu restore right before loading guest.
> >>
> >>  Is there anything I missed here?
> >
> >  kvm_put_guest_fpu.
>
> I found that function and it will be called by vcpu_put eventually
> inside kvm_arch_vcpu_ioctl_run, but kvm_mmu_page_fault is called much
> earlier than that inside kvm exit exception handler. so, the fxsave
> data for the guest image might not be saved at that moment, when I am
> going to emulate this instruction?

Just call it when you want to be sure it is in memory.

-- 
error compiling committee.c: too many arguments to function

Re: how to debug unhandled vm exit: 0x11?

[Avi Kivity]

  On 10/11/2010 04:27 PM, Avi Kivity wrote:
>  On 10/11/2010 08:49 AM, Neo Jia wrote:
>> On Sun, Oct 10, 2010 at 11:30 PM, Avi Kivity<avi@redhat.com>  wrote:
>> >    On 10/11/2010 07:46 AM, Neo Jia wrote:
>> >>
>> >>  BTW, I have a question about saving FPU, especially those XMM
>> >>  registers. I don't see an explicit save FPU after exiting guest 
>> due to
>> >>  an exception (MMIO writes). The only thing I saw about fpu operation
>> >>  is fpu restore right before loading guest.
>> >>
>> >>  Is there anything I missed here?
>> >
>> >  kvm_put_guest_fpu.
>>
>> I found that function and it will be called by vcpu_put eventually
>> inside kvm_arch_vcpu_ioctl_run, but kvm_mmu_page_fault is called much
>> earlier than that inside kvm exit exception handler. so, the fxsave
>> data for the guest image might not be saved at that moment, when I am
>> going to emulate this instruction?
>
> Just call it when you want to be sure it is in memory.
>

btw, I just found an old branch of mine that implements movdqa, I'll see 
if I can refresh and repost it.

-- 
error compiling committee.c: too many arguments to function

Re: how to debug unhandled vm exit: 0x11?

[Avi Kivity]

  On 10/11/2010 04:31 PM, Avi Kivity wrote:
>
> btw, I just found an old branch of mine that implements movdqa, I'll 
> see if I can refresh and repost it.
>

It's pretty hard to rebase; it's in kvm.git sse-mmio if you're interested.

-- 
error compiling committee.c: too many arguments to function

Re: how to debug unhandled vm exit: 0x11?

[Neo Jia]

On Mon, Oct 11, 2010 at 7:27 AM, Avi Kivity <avi@redhat.com> wrote:
>  On 10/11/2010 08:49 AM, Neo Jia wrote:
>>
>> On Sun, Oct 10, 2010 at 11:30 PM, Avi Kivity<avi@redhat.com>  wrote:
>> >    On 10/11/2010 07:46 AM, Neo Jia wrote:
>> >>
>> >>  BTW, I have a question about saving FPU, especially those XMM
>> >>  registers. I don't see an explicit save FPU after exiting guest due to
>> >>  an exception (MMIO writes). The only thing I saw about fpu operation
>> >>  is fpu restore right before loading guest.
>> >>
>> >>  Is there anything I missed here?
>> >
>> >  kvm_put_guest_fpu.
>>
>> I found that function and it will be called by vcpu_put eventually
>> inside kvm_arch_vcpu_ioctl_run, but kvm_mmu_page_fault is called much
>> earlier than that inside kvm exit exception handler. so, the fxsave
>> data for the guest image might not be saved at that moment, when I am
>> going to emulate this instruction?
>
> Just call it when you want to be sure it is in memory.

that helps and looks like my implementation is working. As I am
debugging with kvm-88 version still, I need to integrate back to HEAD
and add unit test.

after that I will send out a patch for review.

Thanks,
Neo

>
> --
> error compiling committee.c: too many arguments to function
>
>



-- 
I would remember that if researchers were not ambitious
probably today we haven't the technology we are using!