nSVM: Booting L2 results in L1 hang and a "skip_emulated_instruction"

nSVM: Booting L2 results in L1 hang and a "skip_emulated_instruction"

[Kashyap Chamarthy]

Hi, 

This was tested with kernel-3.19.0-1.fc22) and QEMU (qemu-2.2.0-5.fc22)
on L0 & L1.


Description
-----------

Inside L1, boot a nested KVM guest (L2) . Instead of a full blown
guest, let's use `qemu-sanity-check` with KVM:

    $ qemu-sanity-check --accel=kvm

Wwich gives you this CLI (run from a different shell), that confirms
that the L2 guest is indeed running on KVM (and not TCG):

  $ ps -ef | grep -i qemu
  root       763   762 35 11:49 ttyS0    00:00:00 qemu-system-x86_64 -nographic -nodefconfig -nodefaults -machine accel=kvm -no-reboot -serial file:/tmp/tmp.rl3naPaCkZ.out -kernel /boot/vmlinuz-3.19.0-1.fc21.x86_64 -initrd /usr/lib64/qemu-sanity-check/initrd -append console=ttyS0 oops=panic panic=-1


Which results in:

  (a) L1 (guest hypervisor) completely hangs and is unresponsive. But
      when I query libvirt, (`virsh list`) the guest is still reported
      as 'running'

  (b) On L0, I notice a ton of these messages:

        skip_emulated_instruction: ip 0xffec next 0xffffffff8105e964


I can get `dmesg`, `dmidecode` , `x86info -a` on L0 and L1 if it helps
in narrowing down the issue.


Related bug and reproducer details
----------------------------------


https://bugzilla.redhat.com/show_bug.cgi?id=1191665 --  Nested KVM with
AMD: L2 (nested guest) fails with "divide error: 0000 [#1] SMP"




-- 
/kashyap

Re: nSVM: Booting L2 results in L1 hang and a "skip_emulated_instruction"

[Jan Kiszka]

On 2015-02-11 19:12, Kashyap Chamarthy wrote:
> Hi, 
> 
> This was tested with kernel-3.19.0-1.fc22) and QEMU (qemu-2.2.0-5.fc22)
> on L0 & L1.
> 
> 
> Description
> -----------
> 
> Inside L1, boot a nested KVM guest (L2) . Instead of a full blown
> guest, let's use `qemu-sanity-check` with KVM:
> 
>     $ qemu-sanity-check --accel=kvm
> 
> Wwich gives you this CLI (run from a different shell), that confirms
> that the L2 guest is indeed running on KVM (and not TCG):
> 
>   $ ps -ef | grep -i qemu
>   root       763   762 35 11:49 ttyS0    00:00:00 qemu-system-x86_64 -nographic -nodefconfig -nodefaults -machine accel=kvm -no-reboot -serial file:/tmp/tmp.rl3naPaCkZ.out -kernel /boot/vmlinuz-3.19.0-1.fc21.x86_64 -initrd /usr/lib64/qemu-sanity-check/initrd -append console=ttyS0 oops=panic panic=-1
> 
> 
> Which results in:
> 
>   (a) L1 (guest hypervisor) completely hangs and is unresponsive. But
>       when I query libvirt, (`virsh list`) the guest is still reported
>       as 'running'
> 
>   (b) On L0, I notice a ton of these messages:
> 
>         skip_emulated_instruction: ip 0xffec next 0xffffffff8105e964
> 
> 
> I can get `dmesg`, `dmidecode` , `x86info -a` on L0 and L1 if it helps
> in narrowing down the issue.
> 
> 
> Related bug and reproducer details
> ----------------------------------
> 
> 
> https://bugzilla.redhat.com/show_bug.cgi?id=1191665 --  Nested KVM with
> AMD: L2 (nested guest) fails with "divide error: 0000 [#1] SMP"
> 
> 

Is this a regression (of the kernel)? If so, can you bisect to the
commit that introduced it?

Jan

-- 
Siemens AG, Corporate Technology, CT RTC ITP SES-DE
Corporate Competence Center Embedded Linux

Re: nSVM: Booting L2 results in L1 hang and a "skip_emulated_instruction"

[Kashyap Chamarthy]

On Thu, Feb 12, 2015 at 07:12:06AM +0100, Jan Kiszka wrote:
> On 2015-02-11 19:12, Kashyap Chamarthy wrote:
> > Hi, 
> > 
> > This was tested with kernel-3.19.0-1.fc22) and QEMU (qemu-2.2.0-5.fc22)
> > on L0 & L1.
> > 
> > 
> > Description
> > -----------
> > 
> > Inside L1, boot a nested KVM guest (L2) . Instead of a full blown
> > guest, let's use `qemu-sanity-check` with KVM:
> > 
> >     $ qemu-sanity-check --accel=kvm
> > 
> > Wwich gives you this CLI (run from a different shell), that confirms
> > that the L2 guest is indeed running on KVM (and not TCG):
> > 
> >   $ ps -ef | grep -i qemu
> >   root       763   762 35 11:49 ttyS0    00:00:00 qemu-system-x86_64 -nographic -nodefconfig -nodefaults -machine accel=kvm -no-reboot -serial file:/tmp/tmp.rl3naPaCkZ.out -kernel /boot/vmlinuz-3.19.0-1.fc21.x86_64 -initrd /usr/lib64/qemu-sanity-check/initrd -append console=ttyS0 oops=panic panic=-1
> > 
> > 
> > Which results in:
> > 
> >   (a) L1 (guest hypervisor) completely hangs and is unresponsive. But
> >       when I query libvirt, (`virsh list`) the guest is still reported
> >       as 'running'
> > 
> >   (b) On L0, I notice a ton of these messages:
> > 
> >         skip_emulated_instruction: ip 0xffec next 0xffffffff8105e964
> > 
> > 
> > I can get `dmesg`, `dmidecode` , `x86info -a` on L0 and L1 if it helps
> > in narrowing down the issue.
> > 
> > 
> > Related bug and reproducer details
> > ----------------------------------
> > 
> > 
> > https://bugzilla.redhat.com/show_bug.cgi?id=1191665 --  Nested KVM with
> > AMD: L2 (nested guest) fails with "divide error: 0000 [#1] SMP"
> > 
> > 
> 
> Is this a regression (of the kernel)? If so, can you bisect to the
> commit that introduced it?

[Sorry, I missed this reply and just noticed it.]

I can't certainly say that it's a regression of Kernel. I also heard
from Dan Berrange (CCed), that when an booting L2 guest it caused L1 to
panic and reboot on AMD. I don't have the AMD physical machine that I
tested this on, will try to find one this week and see if I can bisect.

Thanks.

-- 
/kashyap