kvm: emulation failure

kvm: emulation failure

[Ram Pai]

I see this problem with a x86 sles10 guest running on x86_64 intel host.
If the guest is reset abruptly and rebooted, some where 
before grub sequence it hangs and the following message is seen in the
logs

emulation failed (pagetable) rip 7ed5 66 60 ac 20.

I located this instruction sequence in isolinux.bin on the iso ;if that
is relevant.


I did some analysis and find that there is an ept violation, which is
handled and then the next instruction '66 60' is attempted to decode 
and emulate. But decode fails. kvm continues loops in the kernel
in __vcpu_run().

the code path is 

kvm_run() -> __vcpu_run() -> vcpu_enter_guest() -> kvm_handle_exit() ->
handle_ept_violation() -> kvm_mmu_page_fault() -> emulate_instruction() ->
x86_decode_insn()


Any insights here on how to fix the problem is appreciated. And if a fix already exists even better :)

thanks,
RP

Re: kvm: emulation failure

[Sheng Yang]

On Saturday 20 June 2009 03:23:40 Ram Pai wrote:
> I see this problem with a x86 sles10 guest running on x86_64 intel host.
> If the guest is reset abruptly and rebooted, some where
> before grub sequence it hangs and the following message is seen in the
> logs
>
> emulation failed (pagetable) rip 7ed5 66 60 ac 20.
>
> I located this instruction sequence in isolinux.bin on the iso ;if that
> is relevant.
>
>
> I did some analysis and find that there is an ept violation, which is
> handled and then the next instruction '66 60' is attempted to decode
> and emulate. But decode fails. kvm continues loops in the kernel
> in __vcpu_run().
>
> the code path is
>
> kvm_run() -> __vcpu_run() -> vcpu_enter_guest() -> kvm_handle_exit() ->
> handle_ept_violation() -> kvm_mmu_page_fault() -> emulate_instruction() ->
> x86_decode_insn()
>

Hi Ram

Seems KVM failed to emulate a unknown instruction. 

00000000  6660              pushad
00000002  AC                lodsb

And PUSHAD have not implemented in x86_emulate.c.

But I am a little curious about why this code path was only triggered when 
reset. Maybe other issue exists.

-- 
regards
Yang, Sheng

>
> Any insights here on how to fix the problem is appreciated. And if a fix
> already exists even better :)
>
> thanks,
> RP
>
> --
> To unsubscribe from this list: send the line "unsubscribe kvm" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: kvm: emulation failure

[Ram Pai]

On Mon, 2009-06-22 at 13:12 +0800, Sheng Yang wrote:
> On Saturday 20 June 2009 03:23:40 Ram Pai wrote:
> > I see this problem with a x86 sles10 guest running on x86_64 intel host.
> > If the guest is reset abruptly and rebooted, some where
> > before grub sequence it hangs and the following message is seen in the
> > logs
> >
> > emulation failed (pagetable) rip 7ed5 66 60 ac 20.
> >
> > I located this instruction sequence in isolinux.bin on the iso ;if that
> > is relevant.
> >
> >
> > I did some analysis and find that there is an ept violation, which is
> > handled and then the next instruction '66 60' is attempted to decode
> > and emulate. But decode fails. kvm continues loops in the kernel
> > in __vcpu_run().
> >
> > the code path is
> >
> > kvm_run() -> __vcpu_run() -> vcpu_enter_guest() -> kvm_handle_exit() ->
> > handle_ept_violation() -> kvm_mmu_page_fault() -> emulate_instruction() ->
> > x86_decode_insn()
> >
> 
> Hi Ram
> 
> Seems KVM failed to emulate a unknown instruction. 
> 
> 00000000  6660              pushad
> 00000002  AC                lodsb
> 
> And PUSHAD have not implemented in x86_emulate.c.

Thanks Sheng for your response,

Good. that was the conclusion i had reached reading the code. However
was not sure whether the (a) the code path should have never reached
there or (b) the code must have learnt to emulate pushad.

Sounds like (b) is the case.

> 
> But I am a little curious about why this code path was only triggered when 
> reset. Maybe other issue exists.

What do you want me to check? I have seen ept violation code getting
triggered a few number of times at various stages. But the one reported
above is the only case where the instruction being emulated is 66 60. 

one more observation:
  seen only if the /boot partition is reiserfs. I have been unable to
  reproduce this with /boot being ext3.


thanks and let me know,
RP

Re: kvm: emulation failure

[Sheng Yang]

On Monday 22 June 2009 14:55:46 Ram Pai wrote:
> On Mon, 2009-06-22 at 13:12 +0800, Sheng Yang wrote:
> > On Saturday 20 June 2009 03:23:40 Ram Pai wrote:
> > > I see this problem with a x86 sles10 guest running on x86_64 intel
> > > host. If the guest is reset abruptly and rebooted, some where
> > > before grub sequence it hangs and the following message is seen in the
> > > logs
> > >
> > > emulation failed (pagetable) rip 7ed5 66 60 ac 20.
> > >
> > > I located this instruction sequence in isolinux.bin on the iso ;if that
> > > is relevant.
> > >
> > >
> > > I did some analysis and find that there is an ept violation, which is
> > > handled and then the next instruction '66 60' is attempted to decode
> > > and emulate. But decode fails. kvm continues loops in the kernel
> > > in __vcpu_run().
> > >
> > > the code path is
> > >
> > > kvm_run() -> __vcpu_run() -> vcpu_enter_guest() -> kvm_handle_exit() ->
> > > handle_ept_violation() -> kvm_mmu_page_fault() -> emulate_instruction()
> > > -> x86_decode_insn()
> >
> > Hi Ram
> >
> > Seems KVM failed to emulate a unknown instruction.
> >
> > 00000000  6660              pushad
> > 00000002  AC                lodsb
> >
> > And PUSHAD have not implemented in x86_emulate.c.
>
> Thanks Sheng for your response,
>
> Good. that was the conclusion i had reached reading the code. However
> was not sure whether the (a) the code path should have never reached
> there or (b) the code must have learnt to emulate pushad.
>
> Sounds like (b) is the case.
>
> > But I am a little curious about why this code path was only triggered
> > when reset. Maybe other issue exists.
>
> What do you want me to check? I have seen ept violation code getting
> triggered a few number of times at various stages. But the one reported
> above is the only case where the instruction being emulated is 66 60.

Don't have clue now. I think the only thing we can do now is to have 
instruction emulated, then wait to see what would happen next.

EPT violation is common, for set up EPT pagetable and handle MMIO.

> one more observation:
>   seen only if the /boot partition is reiserfs. I have been unable to
>   reproduce this with /boot being ext3.

Maybe reiserfs did extra thing to do the cleanup after a abruptly reset? 
Sounds reasonable. :)

-- 
regards
Yang, Sheng


> thanks and let me know,
> RP

Re: kvm: emulation failure

[Avi Kivity]

On 06/22/2009 09:55 AM, Ram Pai wrote:
> On Mon, 2009-06-22 at 13:12 +0800, Sheng Yang wrote:
>    
>> On Saturday 20 June 2009 03:23:40 Ram Pai wrote:
>>      
>>> I see this problem with a x86 sles10 guest running on x86_64 intel host.
>>> If the guest is reset abruptly and rebooted, some where
>>> before grub sequence it hangs and the following message is seen in the
>>> logs
>>>
>>> emulation failed (pagetable) rip 7ed5 66 60 ac 20.
>>>
>>> I located this instruction sequence in isolinux.bin on the iso ;if that
>>> is relevant.
>>>
>>>
>>> I did some analysis and find that there is an ept violation, which is
>>> handled and then the next instruction '66 60' is attempted to decode
>>> and emulate. But decode fails. kvm continues loops in the kernel
>>> in __vcpu_run().
>>>
>>> the code path is
>>>
>>> kvm_run() ->  __vcpu_run() ->  vcpu_enter_guest() ->  kvm_handle_exit() ->
>>> handle_ept_violation() ->  kvm_mmu_page_fault() ->  emulate_instruction() ->
>>> x86_decode_insn()
>>>
>>>        
>> Hi Ram
>>
>> Seems KVM failed to emulate a unknown instruction.
>>
>> 00000000  6660              pushad
>> 00000002  AC                lodsb
>>
>> And PUSHAD have not implemented in x86_emulate.c.
>>      
>
> Thanks Sheng for your response,
>
> Good. that was the conclusion i had reached reading the code. However
> was not sure whether the (a) the code path should have never reached
> there or (b) the code must have learnt to emulate pushad.
>
> Sounds like (b) is the case.
>
>    

With ept, the only reason to emulate is mmio.  It's very unlikely that 
the guest is using the pusha instruction for mmio, so the guest is 
probably confused here.

Current kvm.git will return an error here, and current qemu-kvm.git will 
stop the guest on error so we can debug.  But the real problem likely 
started much earlier, I'm not sure we'll get much useful information.

Is this problem reproducible?

>> But I am a little curious about why this code path was only triggered when
>> reset. Maybe other issue exists.
>>      
>
> What do you want me to check? I have seen ept violation code getting
> triggered a few number of times at various stages. But the one reported
> above is the only case where the instruction being emulated is 66 60.
>
> one more observation:
>    seen only if the /boot partition is reiserfs. I have been unable to
>    reproduce this with /boot being ext3.
>    

Please try it with current sources and post the output of 'info 
registers' in the monitor.

-- 
error compiling committee.c: too many arguments to function

Re: kvm: emulation failure

[Ram Pai]

On Mon, 2009-06-22 at 11:26 +0300, Avi Kivity wrote:
> On 06/22/2009 09:55 AM, Ram Pai wrote:
> > On Mon, 2009-06-22 at 13:12 +0800, Sheng Yang wrote:
> >    
> >> On Saturday 20 June 2009 03:23:40 Ram Pai wrote:
> >>      
> >>> I see this problem with a x86 sles10 guest running on x86_64 intel host.
> >>> If the guest is reset abruptly and rebooted, some where
> >>> before grub sequence it hangs and the following message is seen in the
> >>> logs
> >>>
> >>> emulation failed (pagetable) rip 7ed5 66 60 ac 20.
> >>>
> >>> I located this instruction sequence in isolinux.bin on the iso ;if that
> >>> is relevant.
> >>>
> >>>
> >>> I did some analysis and find that there is an ept violation, which is
> >>> handled and then the next instruction '66 60' is attempted to decode
> >>> and emulate. But decode fails. kvm continues loops in the kernel
> >>> in __vcpu_run().
> >>>
> >>> the code path is
> >>>
> >>> kvm_run() ->  __vcpu_run() ->  vcpu_enter_guest() ->  kvm_handle_exit() ->
> >>> handle_ept_violation() ->  kvm_mmu_page_fault() ->  emulate_instruction() ->
> >>> x86_decode_insn()
> >>>
> >>>        
> >> Hi Ram
> >>
> >> Seems KVM failed to emulate a unknown instruction.
> >>
> >> 00000000  6660              pushad
> >> 00000002  AC                lodsb
> >>
> >> And PUSHAD have not implemented in x86_emulate.c.
> >>      
> >
> > Thanks Sheng for your response,
> >
> > Good. that was the conclusion i had reached reading the code. However
> > was not sure whether the (a) the code path should have never reached
> > there or (b) the code must have learnt to emulate pushad.
> >
> > Sounds like (b) is the case.
> >
> >    
> 
> With ept, the only reason to emulate is mmio.  It's very unlikely that 
> the guest is using the pusha instruction for mmio, so the guest is 
> probably confused here.
> 
> Current kvm.git will return an error here, and current qemu-kvm.git will 
> stop the guest on error so we can debug.  But the real problem likely 
> started much earlier, I'm not sure we'll get much useful information.
> 
> Is this problem reproducible?

I can reproduce this at will. the latest qem-kvm.git hangs looping in
the kernel, spewing out regularly the following message
"emulation failed (pagetable) rip 7ed5 66 60 ac 20"

the way to reproduce it -- install sles10sp2 i386; I suppose 
one could probably reproduce using a opensuse guest too(though I have
not tried myself). After installation, reset the
guest through the monitor, and the guest will hang just before
grub takes control.  Key is to have the --cdrom <iso> on the command
line.


> 
> >> But I am a little curious about why this code path was only triggered when
> >> reset. Maybe other issue exists.
> >>      
> >
> > What do you want me to check? I have seen ept violation code getting
> > triggered a few number of times at various stages. But the one reported
> > above is the only case where the instruction being emulated is 66 60.
> >
> > one more observation:
> >    seen only if the /boot partition is reiserfs. I have been unable to
> >    reproduce this with /boot being ext3.
> >    
> 
> Please try it with current sources and post the output of 'info 
> registers' in the monitor.

EAX=00000080 EBX=00000000 ECX=00000000 EDX=00000000
ESI=0000aa1a EDI=0004cd0c EBP=00000000 ESP=0000d562
EIP=00009e67 EFL=00033282 [--S----] CPL=3 II=0 A20=1 SMM=0 HLT=0
ES =0000 00000000 0000ffff 0000f300
CS =0000 00000000 0000ffff 0000f300
SS =99e9 00099e90 0000ffff 0000f300
DS =0000 00000000 0000ffff 0000f300
FS =0000 00000000 0000ffff 0000f300
GS =0000 00000000 0000ffff 0000f300
LDT=0000 00000000 0000ffff 00008200
TR =0080 fffbd000 00002088 00008b00
GDT=     00040620 0000000f
IDT=     00000000 000003ff
CR0=00000010 CR2=00000000 CR3=00000000 CR4=00000000
DR0=00000000 DR1=00000000 DR2=00000000 DR3=00000000 
DR6=ffff0ff0 DR7=00000400
FCW=037f FSW=0000 [ST=0] FTW=00 MXCSR=00000000
FPR0=0000000000000000 0000 FPR1=0000000000000000 0000
FPR2=0000000000000000 0000 FPR3=0000000000000000 0000
FPR4=0000000000000000 0000 FPR5=0000000000000000 0000
FPR6=0000000000000000 0000 FPR7=0000000000000000 0000
XMM00=00000000000000000000000000000000 XMM01=00000000000000000000000000000000
XMM02=00000000000000000000000000000000 XMM03=00000000000000000000000000000000
XMM04=00000000000000000000000000000000 XMM05=00000000000000000000000000000000
XMM06=00000000000000000000000000000000 XMM07=00000000000000000000000000000000



RP