From mboxrd@z Thu Jan 1 00:00:00 1970 From: john cooper Subject: Re: bad virtio disk performance Date: Mon, 27 Apr 2009 19:40:11 -0400 Message-ID: <49F6425B.6010000@redhat.com> References: <20090427151204.GA16943@xanadu.blop.info> <49F5ED05.6020703@third-harmonic.com> <20090427182830.GA32574@xanadu.blop.info> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit Cc: john cooper , kvm@vger.kernel.org, john.cooper@redhat.com To: Lucas Nussbaum Return-path: Received: from mx2.redhat.com ([66.187.237.31]:56570 "EHLO mx2.redhat.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1760091AbZD0XpX (ORCPT ); Mon, 27 Apr 2009 19:45:23 -0400 In-Reply-To: <20090427182830.GA32574@xanadu.blop.info> Sender: kvm-owner@vger.kernel.org List-ID: Lucas Nussbaum wrote: > On 27/04/09 at 13:36 -0400, john cooper wrote: >> Lucas Nussbaum wrote: > > non-virtio: > kvm -drive file=/tmp/debian-amd64.img,if=scsi,cache=writethrough -net > nic,macaddr=00:16:3e:5a:28:1,model=e1000 -net tap -nographic -kernel > /boot/vmlinuz-2.6.29 -initrd /boot/initrd.img-2.6.29 -append > root=/dev/sda1 ro console=tty0 console=ttyS0,9600,8n1 > > virtio: > kvm -drive file=/tmp/debian-amd64.img,if=virtio,cache=writethrough -net > nic,macaddr=00:16:3e:5a:28:1,model=e1000 -net tap -nographic -kernel > /boot/vmlinuz-2.6.29 -initrd /boot/initrd.img-2.6.29 -append > root=/dev/vda1 ro console=tty0 console=ttyS0,9600,8n1 > One suggestion would be to use a separate drive for the virtio vs. non-virtio comparison to avoid a Heisenberg effect. > > So, apparently, with virtio, there's a lot more data being written to > disk. The underlying filesystem is ext3, and is monted as /tmp. It only > contains the VM image file. Another difference is that, with virtio, the > data was shared equally over all 4 CPUs, with without virt-io, CPU0 and > CPU1 did all the work. > In the virtio log, I also see a (null) process doing a lot of writes. Can't say what is causing that -- only took a look at the short logs. However the isolation suggested above may help factor that out if you need to pursue this path. > > I uploaded the logs to http://blop.info/bazaar/virtio/, if you want to > take a look. In the virtio case i/o is being issued from multiple threads. You could be hitting the cfq close-cooperator bug we saw as late as 2.6.28. A quick test to rule this in/out would be to change the block scheduler to other than cfq for the host device where the backing image resides -- in your case the host device containing /tmp/debian-amd64.img. Eg for /dev/sda1: # cat /sys/block/sda/queue/scheduler noop anticipatory deadline [cfq] # echo deadline > /sys/block/sda/queue/scheduler # cat /sys/block/sda/queue/scheduler noop anticipatory [deadline] cfq And re-run your test to see if this brings virtio vs. non-virtio closer to the expected performance. -john -- john.cooper@redhat.com