BUG: Guest lockup and 100% CPU load on host

BUG: Guest lockup and 100% CPU load on host

[Jan C. Bernauer]

Hi,
 
 while I was trying to get pci passthrough to work without vt-d using 
the 1to1 mapping (it works ;),
I stumbled upon a solid lock up of the guest. 
(Unpatched source shows the same behaviour)

 I see a strange look up of the guest OS with the following "features":
 - guest doesn't respond anymore
 - triggered by load in the guest. 
 - it seems it is easier to trigger with -smp 2
 - host shows qemu process with 100% cpu
 - seems to be cured by  --no-kvm-irqchip 
 - --no-kvm-pit does not help
 - qmeu console command system_reset works
 
Right now, guest OS is linux 2.6.27.1
Latest KVM from git, host is 2.6.28.
Older kvms seem to exhibit the same problem.


short gdb session:

Program received signal SIGINT, Interrupt.
[Switching to Thread 0x7f9dfbb7c6f0 (LWP 13898)]
0x00007f9dfa4e1a92 in select () from /lib/libc.so.6
(gdb) info threads
  2 Thread 0x7f9dd7d54950 (LWP 13907)  0x00007f9dfa4e1027 in ioctl () 
from /lib/libc.so.6
* 1 Thread 0x7f9dfbb7c6f0 (LWP 13898)  0x00007f9dfa4e1a92 in select () 
from /lib/libc.so.6
(gdb) bt
#0  0x00007f9dfa4e1a92 in select () from /lib/libc.so.6
#1  0x0000000000409a2b in main_loop_wait (timeout=0) at 
/usr/src/kvm-userspace/qemu/vl.c:3639
#2  0x0000000000518c6a in kvm_main_loop () at 
/usr/src/kvm-userspace/qemu/qemu-kvm.c:600
#3  0x000000000040db93 in main (argc=<value optimized out>, 
argv=0x7fff03badc08, envp=<value optimized out>)
    at /usr/src/kvm-userspace/qemu/vl.c:3801
(gdb) thread 2
[Switching to thread 2 (Thread 0x7f9dd7d54950 (LWP 13907))]#0  
0x00007f9dfa4e1027 in ioctl () from /lib/libc.so.6
(gdb) bt
#0  0x00007f9dfa4e1027 in ioctl () from /lib/libc.so.6
#1  0x0000000000545d8e in kvm_run (kvm=0xde1040, vcpu=0, env=0xe5b070) 
at libkvm.c:887
#2  0x0000000000518d89 in kvm_cpu_exec (env=<value optimized out>) at 
/usr/src/kvm-userspace/qemu/qemu-kvm.c:210
#3  0x0000000000519078 in ap_main_loop (_env=<value optimized out>) at 
/usr/src/kvm-userspace/qemu/qemu-kvm.c:417
#4  0x00007f9dfb032007 in start_thread () from /lib/libpthread.so.0
#5  0x00007f9dfa4e838d in clone () from /lib/libc.so.6
#6  0x0000000000000000 in ?? ()


kvm is started by:

/usr/local/bin/qemu-system-x86_64   -vnc :2 -drive 
file=/dev/mapper/mykvmparition,if=virtio,boot=on -net 
nic,model=virtio,macaddr=52:54:00:12:34:a0 -net 
tap,script=/etc/kvm/kvm-ifup,downscript=/etc/kvm/kvm-ifdown  -m 512

Host is an AMD 4850e, all OS are 64bit.

The syslog  on the host seems clean. 

If you have anything for me to try or need some more information, let me 
know.

Best regards,
Jan

Re: BUG: Guest lockup and 100% CPU load on host

[James Thomason]

Jan, 

I spent some time with the same problem over the last two days.  Avi Kivity
suggested adding "clocksource=acpi_pm" to the guest kernel boot line, and
that seems to have resolved the issue. Would you give that a try and let me
know the result?  Also I believe the following bugs might be related:

[ 2351676 ] Guests hang periodically on Ubuntu-8.10
[ 2353811 ] Solaris 10 guest unstable
[ 2494730 ] Guests "stalling" on kvm-82
[ 2138079 ] kvm locks up system
[ 2113643 ] guests AND host still getting stuck under CPU load

Regards, 
James


On 1/15/09 2:10 PM, "Jan C. Bernauer" <jan.bernauer@t-online.de> wrote:

> Hi,
>  
>  while I was trying to get pci passthrough to work without vt-d using
> the 1to1 mapping (it works ;),
> I stumbled upon a solid lock up of the guest.
> (Unpatched source shows the same behaviour)
> 
>  I see a strange look up of the guest OS with the following "features":
>  - guest doesn't respond anymore
>  - triggered by load in the guest.
>  - it seems it is easier to trigger with -smp 2
>  - host shows qemu process with 100% cpu
>  - seems to be cured by  --no-kvm-irqchip
>  - --no-kvm-pit does not help
>  - qmeu console command system_reset works
>  
> Right now, guest OS is linux 2.6.27.1
> Latest KVM from git, host is 2.6.28.
> Older kvms seem to exhibit the same problem.

Re: BUG: Guest lockup and 100% CPU load on host

[Jan Bernauer]

Hi,

 thanks for the info! It seems quite stable now, but I haven't tried
-smp 2 yet.

 Timing problems seem plausible: I have a dvb-card passed through to the
guest. With normal setup, the frontend detection of the driver module
breaks every other try with a i2c timeout.
I changed the cpufreq governor of the host from ondemand to performance,
and the guest seemed stable, until suddenly the timeouts filled the logs
again.

Until now, with clocksource=acpi_pm, I saw only occasional irq timeouts
of the card (like, under 10 in 30 minutes), under heavy load of the guest.


James Thomason wrote:
> I spent some time with the same problem over the last two days.  Avi Kivity
> suggested adding "clocksource=acpi_pm" to the guest kernel boot line, and
> that seems to have resolved the issue. Would you give that a try and let me
> know the result?  Also I believe the following bugs might be related:
> 
> [ 2351676 ] Guests hang periodically on Ubuntu-8.10
> [ 2353811 ] Solaris 10 guest unstable
> [ 2494730 ] Guests "stalling" on kvm-82
> [ 2138079 ] kvm locks up system
> [ 2113643 ] guests AND host still getting stuck under CPU load
> 



Best regards,
Jan


-- 
Jan C. Bernauer                       email: bernauer@kph.uni-mainz.de
Institut für Kernphysik               tel  : 06131 - 39 25826
Johannes-Gutenberg-Universität Mainz  fax  : 06131 - 39 22964
Johann-Joachim-Becher-Weg 45
55099 Mainz