From mboxrd@z Thu Jan 1 00:00:00 1970 From: John Leach Subject: Re: bad O_DIRECT read and write performance with small block sizes with virtio Date: Tue, 03 Aug 2010 15:52:53 +0100 Message-ID: <1280847173.13790.51.camel@dogen> References: <1280769301.11871.10.camel@dogen> <4C570470.9000803@codemonkey.ws> <4C57B8AB.5030703@redhat.com> Mime-Version: 1.0 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: 7bit Cc: Stefan Hajnoczi , kvm@vger.kernel.org, Anthony Liguori To: dlaor@redhat.com Return-path: Received: from smtp118.iad.emailsrvr.com ([207.97.245.118]:52118 "EHLO smtp118.iad.emailsrvr.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S932070Ab0HCOwz (ORCPT ); Tue, 3 Aug 2010 10:52:55 -0400 In-Reply-To: <4C57B8AB.5030703@redhat.com> Sender: kvm-owner@vger.kernel.org List-ID: On Tue, 2010-08-03 at 09:35 +0300, Dor Laor wrote: > On 08/02/2010 11:50 PM, Stefan Hajnoczi wrote: > > On Mon, Aug 2, 2010 at 6:46 PM, Anthony Liguori wrote: > >> On 08/02/2010 12:15 PM, John Leach wrote: > >>> > >>> Hi, > >>> > >>> I've come across a problem with read and write disk IO performance when > >>> using O_DIRECT from within a kvm guest. With O_DIRECT, reads and writes > >>> are much slower with smaller block sizes. Depending on the block size > >>> used, I've seen 10 times slower. > >>> > >>> For example, with an 8k block size, reading directly from /dev/vdb > >>> without O_DIRECT I see 750 MB/s, but with O_DIRECT I see 79 MB/s. > >>> > >>> As a comparison, reading in O_DIRECT mode in 8k blocks directly from the > >>> backend device on the host gives 2.3 GB/s. Reading in O_DIRECT mode > >>> from a xen guest on the same hardware manages 263 MB/s. > >>> > >> > >> Stefan has a few fixes for this behavior that help a lot. One of them > >> (avoiding memset) is already upstream but not in 0.12.x. > >> > >> The other two are not done yet but should be on the ML in the next couple > >> weeks. They involve using ioeventfd for notification and unlocking the > >> block queue lock while doing a kick notification. > > > > Thanks for mentioning those patches. The ioeventfd patch will be sent > > this week, I'm checking that migration works correctly and then need > > to check that vhost-net still works. > > > >>> Writing is affected in the same way, and exhibits the same behaviour > >>> with O_SYNC too. > >>> > >>> Watching with vmstat on the host, I see the same number of blocks being > >>> read, but about 14 times the number of context switches in O_DIRECT mode > >>> (4500 cs vs. 63000 cs) and a little more cpu usage. > >>> > >>> The device I'm writing to is a device-mapper zero device that generates > >>> zeros on read and throws away writes, you can set it up > >>> at /dev/mapper/zero like this: > >>> > >>> echo "0 21474836480 zero" | dmsetup create zero > >>> > >>> My libvirt config for the disk is: > >>> > >>> > >>> > >>> > >>> > >>>
>>> function='0x0'/> > >>> > >>> > >>> which translates to the kvm arg: > >>> > >>> -device > >>> virtio-blk-pci,bus=pci.0,addr=0x4,drive=drive-virtio-disk0,id=virtio-disk0 > >>> -drive file=/dev/mapper/zero,if=none,id=drive-virtio-disk1,cache=none > > aio=native and change the io scheduler on the host to deadline should > help as well. No improvement in this case (I was already using deadline on the host, and just tested with aio=native). Tried with a real disk backend too, still no improvement. I'll try with and without once I get Stefan's other patches too though. Thanks, John.