From mboxrd@z Thu Jan 1 00:00:00 1970 From: Erik Brakkee Subject: Re: Workload spikes on KVM host when doing IO on a guest... Date: Wed, 03 Apr 2013 22:19:54 +0200 Message-ID: <515C8EEA.7040204@brakkee.org> References: <4FB84114.1090600@brakkee.org> <4FB8F7A8.5050504@redhat.com> <4FB923BB.9070306@brakkee.org> <4FB924D0.1020704@redhat.com> <4FB929DE.8000500@brakkee.org> <4FB92EA3.7070706@redhat.com> Mime-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: KVM list To: Avi Kivity Return-path: Received: from smtp-vbr14.xs4all.nl ([194.109.24.34]:4411 "EHLO smtp-vbr14.xs4all.nl" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1762609Ab3DCUUD (ORCPT ); Wed, 3 Apr 2013 16:20:03 -0400 Received: from brakkee.org (a83-163-128-75.adsl.xs4all.nl [83.163.128.75]) by smtp-vbr14.xs4all.nl (8.13.8/8.13.8) with SMTP id r33KJx8M087322 for ; Wed, 3 Apr 2013 22:20:00 +0200 (CEST) (envelope-from erik@brakkee.org) In-Reply-To: <4FB92EA3.7070706@redhat.com> Sender: kvm-owner@vger.kernel.org List-ID: Avi Kivity wrote: > On 05/20/2012 08:29 PM, Erik Brakkee wrote: >> Avi Kivity wrote: >>> On 05/20/2012 08:02 PM, Erik Brakkee wrote: >>>> [...] >>>> Thanks for this information. Unfortunately, io="native" in domain.xml >>>> is not supported by opensuse 11.3. It is supported in 12.1 so it >>>> appears that the version of KVM I have on the server is too old. I >>>> tried it on a system running the newer version and indeed, as you say >>>> the load disappears completely when using io="native". >>>> >>>> I am going to update the host now (probably to centos 6.2) to get rid >>>> of this problem. >>> To be clear: it's not a problem. It's completely normal, and doesn't >>> affect anything. >> The only problem with it is that it leads to high workload spikes, >> which is normally a reason to have a good look at what is going on. In >> this case, the newer version of KVM should help eliminate these >> spikes, so that the next time I see a spike in the workload I know >> that I have to look into something. In the mean time I have migrated the host machine to run centos 6.2 and later 6.3. There I could use the aio=native option together with cache=none and I saw a significant drop in thw workload while doing a lot of IO on the guest. However, after upgrading to centos 6.4, I now see huge workload spikes again, but the virtual machine configuration is the same. Also, I am seeing a high usage of swap on the host machine. It almost looks as if the aio=native option is being ignored or could it be a different problem? Here is one of the processes: qemu 3026 4.6 17.9 4751536 4419208 ? Sl Mar27 136:21 /usr/libexec/qemu-kvm -name sparrow -S -M rhel6.2.0 -enable-kvm -m 4096 -smp 2,sockets=2,cores=1,threads=1 -uuid 1389eb3f-8f26-686e-93c4-21267a66ec52 -nodefconfig -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/sparrow.monitor,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc -no-shutdown -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -drive file=/dev/bootdisks/sparrow,if=none,id=drive-virtio-disk0,format=raw,cache=none,aio=native -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x3,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 -drive file=/dev/sparrow/disk,if=none,id=drive-virtio-disk1,format=raw,cache=none,aio=native -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x4,drive=drive-virtio-disk1,id=virtio-disk1 -netdev tap,fd=23,id=hostnet0,vhost=on,vhostfd=26 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:01:01:03,bus=pci.0,addr=0x5 -netdev tap,fd=27,id=hostnet1,vhost=on,vhostfd=28 -device virtio-net-pci,netdev=hostnet1,id=net1,mac=52:54:00:01:01:04,bus=pci.0,addr=0x7 -chardev pty,id=charserial0 -device isa-serial,chardev=charserial0,id=serial0 -vnc 127.0.0.1:2 -vga cirrus -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x6 I can also see that the swap space is exclusively used by qemu-kvm: [root@falcon bin]# ./examineswap ... Overall swap used: 1264924 kB ======================================== kB pid name ======================================== 575092 3026 qemu-kvm 391468 2882 qemu-kvm 298364 2953 qemu-kvm (see at the end of the examineswap script). This snapshot of the swap space is taken shortly after enabling swap and it continues to fill up the entire swap space, with still a lot of memory free [root@falcon bin]# free total used free shared buffers cached Mem: 24662972 24460920 202052 0 10904048 40712 -/+ buffers/cache: 13516160 11146812 Swap: 2097144 1508516 588628 Also, I have set the swappiness to 0 but it does not help: [root@falcon bin]# sysctl vm.swappiness vm.swappiness = 0 Details on the OS and KVM: [root@falcon ~]# uname -a Linux falcon.fritz.box 2.6.32-358.2.1.el6.x86_64 #1 SMP Wed Mar 13 00:26:49 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux [root@falcon ~]# cat /etc/issue CentOS release 6.4 (Final) Kernel \r on an \m [root@falcon ~]# rpm -qa | grep kvm qemu-kvm-0.12.1.2-2.355.0.1.el6.centos.2.x86_64 I am also getting a workload on the host of approximately 20 now whereas the workload on the VM doing the IO is just 6. What I am doing at this time is creating a backup over iSCSI with a target on the physical host and an initiator on the VM. The VM has created snapshot logical volumes and is copying data to the iSCSI target. More spcifically, I am using this script that I developed myself: http://wamblee.org/snapshot.html Do you have any idea what this could be? Cheers Erik PS. The examineswap script I downloaded #!/bin/bash # find-out-what-is-using-your-swap.sh # -- Get current swap usage for all running processes # -- # -- rev.0.3, 2012-09-03, Jan Smid - alignment and intendation, sorting # -- rev.0.2, 2012-08-09, Mikko Rantalainen - pipe the output to "sort -nk3" to get sorted output # -- rev.0.1, 2011-05-27, Erik Ljungstrom - initial version SCRIPT_NAME=`basename $0`; SORT="kb"; # {pid|kB|name} as first parameter, [default: kb] [ "$1" != "" ] && { SORT="$1"; } [ ! -x `which mktemp` ] && { echo "ERROR: mktemp is not available!"; exit; } MKTEMP=`which mktemp`; TMP=`${MKTEMP} -d`; [ ! -d "${TMP}" ] && { echo "ERROR: unable to create temp dir!"; exit; } >${TMP}/${SCRIPT_NAME}.pid; >${TMP}/${SCRIPT_NAME}.kb; >${TMP}/${SCRIPT_NAME}.name; SUM=0; OVERALL=0; echo "${OVERALL}" > ${TMP}/${SCRIPT_NAME}.overal; for DIR in `find /proc/ -maxdepth 1 -type d -regex "^/proc/[0-9]+"`; do PID=`echo $DIR | cut -d / -f 3` PROGNAME=`ps -p $PID -o comm --no-headers` for SWAP in `grep Swap $DIR/smaps 2>/dev/null| awk '{ print $2 }'` do let SUM=$SUM+$SWAP done if (( $SUM > 0 )); then echo -n "."; echo -e "${PID}\t${SUM}\t${PROGNAME}" >> ${TMP}/${SCRIPT_NAME}.pid; echo -e "${SUM}\t${PID}\t${PROGNAME}" >> ${TMP}/${SCRIPT_NAME}.kb; echo -e "${PROGNAME}\t${SUM}\t${PID}" >> ${TMP}/${SCRIPT_NAME}.name; fi let OVERALL=$OVERALL+$SUM SUM=0 done echo "${OVERALL}" > ${TMP}/${SCRIPT_NAME}.overal; echo; echo "Overall swap used: ${OVERALL} kB"; echo "========================================"; case "${SORT}" in name ) echo -e "name\tkB\tpid"; echo "========================================"; cat ${TMP}/${SCRIPT_NAME}.name|sort -r; ;; kb ) echo -e "kB\tpid\tname"; echo "========================================"; cat ${TMP}/${SCRIPT_NAME}.kb|sort -rh; ;; pid | * ) echo -e "pid\tkB\tname"; echo "========================================"; cat ${TMP}/${SCRIPT_NAME}.pid|sort -rh; ;; esac rm -fR "${TMP}/";