Gests periodically stuck for ~ 10-15 seconds - what to do?

Gests periodically stuck for ~ 10-15 seconds - what to do?

[Hermann Himmelbauer]

Hi,
I'm quite new to virtualization and KVM, I have a 2*4-core Intel machine here 
with 16GB RAM and Debian Lenny as host. I further installed two Debian Lenny 
guests, one with 2 CPUs, the other with one. The guests were installed 
similar to this:

virt-install --connect qemu:///system -n MyVMName -r 1024 --vcpus=2 -f 
path/to/qcow2_file -s 12 -c ~/debian-503-amd64-netinst.iso --vnc -k 
de --noautoconsole --os-type linux --os-variant 
debianLenny --accelerate --network=bridge:br0 --hvm

The guests have the virtio_blk and virtio_net devices installed to improve 
disk/networking performance.

After a while I experienced strange guest lockups, whereas I got the 
famous "CPU #0 stuck for ...seconds". I then upgraded both the host and the 
guest kernel from 2.6.26 (Lenny default) to 2.6.32-bpo.5-amd64, moreover I 
upgraded kvm to qemu-kvm-0.12.5

The "CPU stuck" messages are gone now, but I still experience stucks in the 
guest. They come from time to time, sometimes not for an hour, sometimes 
every 5 seconds. These "stucks" last something between 3 seconds to 30 
seconds.

I am not 100% sure but it seems that these stucks come from the disk I/O. I 
wrote two little scripts, one that outputs the continuity of disk reads, the 
other the continuity of CPU calculations. It's interesting to see that the 
calculations continue while the disk-read script is stuck, which leads me to 
the conclusion that the problem is disk-I/O bound.

It's moreover interesting to see that these stucks happen synchronously at 
both guest systems, while the host seems not to be affected.

Moreover, it's interesting that the read speed is sometimes stable at ~ 
80MB/s, while it is at other times only at ~ 40 MB/s.

During these tests the system is fully idle (apart from the tests, of course). 
The program "iotop" shows the I/O fully idle during these stucks.

Rebooting the guests seem to make the problem go away - but only for some 
time.

The whole issue is quite bad as the guests are virtually unusable with these 
stucks, working is horrific, a simple "ls" takes sometimes up to 30 seconds 
although the whole system (host + other guests) are fully idle and I have no  
clue how to fix this.

Any ideas?

Best Regards,
Hermann

-- 
hermann@qwer.tk
GPG key ID: 299893C7 (on keyservers)
FP: 0124 2584 8809 EF2A DBF9  4902 64B4 D16B 2998 93C7

Re: Gests periodically stuck for ~ 10-15 seconds - what to do?

[Hermann Himmelbauer]

Am Samstag 23 Oktober 2010 22:37:41 schrieb Hermann Himmelbauer:
> Hi,
> I'm quite new to virtualization and KVM, I have a 2*4-core Intel machine
> here with 16GB RAM and Debian Lenny as host. I further installed two Debian
> Lenny guests, one with 2 CPUs, the other with one. The guests were
> installed similar to this:
>
> virt-install --connect qemu:///system -n MyVMName -r 1024 --vcpus=2 -f
> path/to/qcow2_file -s 12 -c ~/debian-503-amd64-netinst.iso --vnc -k
> de --noautoconsole --os-type linux --os-variant
> debianLenny --accelerate --network=bridge:br0 --hvm

Regarding a suggestion on #IRC that the above does not tell enough about the 
KVM configuration, here are the command lines for both guests:

/usr/bin/kvm -S -M pc-0.12 -enable-kvm -m 1024 -smp 
2,sockets=2,cores=1,threads=1 -name vmUranos -uuid 
8e5139ce-c561-c52f-35e1-07db9bc5045b -nodefaults -chardev 
socket,id=monitor,path=/var/lib/libvirt/qemu/vmUranos.monitor,server,nowait -mon 
chardev=monitor,mode=readline -rtc base=utc -boot c -drive 
if=none,media=cdrom,id=drive-ide0-1-0,readonly=on -device 
ide-drive,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0 -drive 
file=/dev/capella_raptor/UranosBase,if=none,id=drive-virtio-disk0,boot=on,cache=none -device 
virtio-blk-pci,bus=pci.0,addr=0x4,drive=drive-virtio-disk0,id=virtio-disk0 -device 
virtio-net-pci,vlan=0,id=net0,mac=54:52:00:03:f4:ca,bus=pci.0,addr=0x5 -net 
tap,fd=17,vlan=0,name=hostnet0 -chardev pty,id=serial0 -device 
isa-serial,chardev=serial0 -usb -vnc 127.0.0.1:0 -k de -vga cirrus -device 
virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x3

/usr/bin/kvm -S -M pc -enable-kvm -m 1024 -smp 
1,sockets=1,cores=1,threads=1 -name galemo -uuid 
171b4536-84ea-041d-d318-16b8fb20f855 -nodefaults -chardev 
socket,id=monitor,path=/var/lib/libvirt/qemu/galemo.monitor,server,nowait -mon 
chardev=monitor,mode=readline -rtc base=utc -boot c -drive 
if=none,media=cdrom,id=drive-ide0-1-0,readonly=on -device 
ide-drive,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0 -drive 
file=/dev/capella_data1/galemo,if=none,id=drive-virtio-disk0,boot=on -device 
virtio-blk-pci,bus=pci.0,addr=0x4,drive=drive-virtio-disk0,id=virtio-disk0 -device 
virtio-net-pci,vlan=0,id=net0,mac=54:52:00:45:9c:d9,bus=pci.0,addr=0x5 -net 
tap,fd=18,vlan=0,name=hostnet0 -chardev pty,id=serial0 -device 
isa-serial,chardev=serial0 -usb -vnc 127.0.0.1:1 -k de -vga cirrus -device 
virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x3

Best Regards,
Hermann

-- 
hermann@qwer.tk
GPG key ID: 299893C7 (on keyservers)
FP: 0124 2584 8809 EF2A DBF9  4902 64B4 D16B 2998 93C7