Re: bad O_DIRECT read and write performance with small block sizes with virtio

[John Leach <john@brightbox.co.uk>]

On Tue, 2010-08-03 at 09:35 +0300, Dor Laor wrote:
> On 08/02/2010 11:50 PM, Stefan Hajnoczi wrote:
> > On Mon, Aug 2, 2010 at 6:46 PM, Anthony Liguori<anthony@codemonkey.ws>  wrote:
> >> On 08/02/2010 12:15 PM, John Leach wrote:
> >>>
> >>> Hi,
> >>>
> >>> I've come across a problem with read and write disk IO performance when
> >>> using O_DIRECT from within a kvm guest.  With O_DIRECT, reads and writes
> >>> are much slower with smaller block sizes.  Depending on the block size
> >>> used, I've seen 10 times slower.
> >>>
> >>> For example, with an 8k block size, reading directly from /dev/vdb
> >>> without O_DIRECT I see 750 MB/s, but with O_DIRECT I see 79 MB/s.
> >>>
> >>> As a comparison, reading in O_DIRECT mode in 8k blocks directly from the
> >>> backend device on the host gives 2.3 GB/s.  Reading in O_DIRECT mode
> >>> from a xen guest on the same hardware manages 263 MB/s.
> >>>
> >>
> >> Stefan has a few fixes for this behavior that help a lot.  One of them
> >> (avoiding memset) is already upstream but not in 0.12.x.
> >>
> >> The other two are not done yet but should be on the ML in the next couple
> >> weeks.  They involve using ioeventfd for notification and unlocking the
> >> block queue lock while doing a kick notification.
> >
> > Thanks for mentioning those patches.  The ioeventfd patch will be sent
> > this week, I'm checking that migration works correctly and then need
> > to check that vhost-net still works.
> >
> >>> Writing is affected in the same way, and exhibits the same behaviour
> >>> with O_SYNC too.
> >>>
> >>> Watching with vmstat on the host, I see the same number of blocks being
> >>> read, but about 14 times the number of context switches in O_DIRECT mode
> >>> (4500 cs vs. 63000 cs) and a little more cpu usage.
> >>>
> >>> The device I'm writing to is a device-mapper zero device that generates
> >>> zeros on read and throws away writes, you can set it up
> >>> at /dev/mapper/zero like this:
> >>>
> >>> echo "0 21474836480 zero" | dmsetup create zero
> >>>
> >>> My libvirt config for the disk is:
> >>>
> >>> <disk type='block' device='disk'>
> >>>    <driver cache='none'/>
> >>>    <source dev='/dev/mapper/zero'/>
> >>>    <target dev='vdb' bus='virtio'/>
> >>>    <address type='pci' domain='0x0000' bus='0x00' slot='0x06'
> >>> function='0x0'/>
> >>> </disk>
> >>>
> >>> which translates to the kvm arg:
> >>>
> >>> -device
> >>> virtio-blk-pci,bus=pci.0,addr=0x4,drive=drive-virtio-disk0,id=virtio-disk0
> >>> -drive file=/dev/mapper/zero,if=none,id=drive-virtio-disk1,cache=none
> 
> aio=native and change the io scheduler on the host to deadline should 
> help as well.

No improvement in this case (I was already using deadline on the host,
and just tested with aio=native). Tried with a real disk backend too,
still no improvement.

I'll try with and without once I get Stefan's other patches too though.

Thanks,

John.