From mboxrd@z Thu Jan 1 00:00:00 1970 From: Hermann Himmelbauer Subject: Gests periodically stuck for ~ 10-15 seconds - what to do? Date: Sat, 23 Oct 2010 22:37:41 +0200 Message-ID: <201010232237.42005.dusty@qwer.tk> Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit To: kvm@vger.kernel.org Return-path: Received: from mailbackup.inode.at ([213.229.60.24]:49382 "EHLO mailbackup.inode.at" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1753406Ab0JWUz3 (ORCPT ); Sat, 23 Oct 2010 16:55:29 -0400 Received: from [62.99.145.8] (port=44429 helo=mx.inode.at) by mailbackup.inode.at with esmtp (Exim 4.67) (envelope-from ) id 1P9krX-0003sC-Ix for kvm@vger.kernel.org; Sat, 23 Oct 2010 22:38:47 +0200 Received: from [83.64.146.242] (port=14130 helo=horn.qwer.tk) by smartmx-08.inode.at with esmtp (Exim 4.69) (envelope-from ) id 1P9kqY-00045p-1b for kvm@vger.kernel.org; Sat, 23 Oct 2010 22:37:46 +0200 Received: from localhost (localhost [127.0.0.1]) by horn.qwer.tk (Postfix) with ESMTP id 76EAA202AD52 for ; Sat, 23 Oct 2010 22:37:45 +0200 (CEST) Received: from horn.qwer.tk ([127.0.0.1]) by localhost (horn [127.0.0.1]) (amavisd-new, port 10024) with LMTP id 01023-01 for ; Sat, 23 Oct 2010 22:37:42 +0200 (CEST) Received: from piano.qwer.tk (piano.qwer.tk [192.168.50.71]) by horn.qwer.tk (Postfix) with ESMTP id 48224202AD41 for ; Sat, 23 Oct 2010 22:37:42 +0200 (CEST) Content-Disposition: inline Sender: kvm-owner@vger.kernel.org List-ID: Hi, I'm quite new to virtualization and KVM, I have a 2*4-core Intel machine here with 16GB RAM and Debian Lenny as host. I further installed two Debian Lenny guests, one with 2 CPUs, the other with one. The guests were installed similar to this: virt-install --connect qemu:///system -n MyVMName -r 1024 --vcpus=2 -f path/to/qcow2_file -s 12 -c ~/debian-503-amd64-netinst.iso --vnc -k de --noautoconsole --os-type linux --os-variant debianLenny --accelerate --network=bridge:br0 --hvm The guests have the virtio_blk and virtio_net devices installed to improve disk/networking performance. After a while I experienced strange guest lockups, whereas I got the famous "CPU #0 stuck for ...seconds". I then upgraded both the host and the guest kernel from 2.6.26 (Lenny default) to 2.6.32-bpo.5-amd64, moreover I upgraded kvm to qemu-kvm-0.12.5 The "CPU stuck" messages are gone now, but I still experience stucks in the guest. They come from time to time, sometimes not for an hour, sometimes every 5 seconds. These "stucks" last something between 3 seconds to 30 seconds. I am not 100% sure but it seems that these stucks come from the disk I/O. I wrote two little scripts, one that outputs the continuity of disk reads, the other the continuity of CPU calculations. It's interesting to see that the calculations continue while the disk-read script is stuck, which leads me to the conclusion that the problem is disk-I/O bound. It's moreover interesting to see that these stucks happen synchronously at both guest systems, while the host seems not to be affected. Moreover, it's interesting that the read speed is sometimes stable at ~ 80MB/s, while it is at other times only at ~ 40 MB/s. During these tests the system is fully idle (apart from the tests, of course). The program "iotop" shows the I/O fully idle during these stucks. Rebooting the guests seem to make the problem go away - but only for some time. The whole issue is quite bad as the guests are virtually unusable with these stucks, working is horrific, a simple "ls" takes sometimes up to 30 seconds although the whole system (host + other guests) are fully idle and I have no clue how to fix this. Any ideas? Best Regards, Hermann -- hermann@qwer.tk GPG key ID: 299893C7 (on keyservers) FP: 0124 2584 8809 EF2A DBF9 4902 64B4 D16B 2998 93C7