bad O_DIRECT read and write performance with small block sizes with virtio

bad O_DIRECT read and write performance with small block sizes with virtio

[John Leach]

Hi,

I've come across a problem with read and write disk IO performance when
using O_DIRECT from within a kvm guest.  With O_DIRECT, reads and writes
are much slower with smaller block sizes.  Depending on the block size
used, I've seen 10 times slower.

For example, with an 8k block size, reading directly from /dev/vdb
without O_DIRECT I see 750 MB/s, but with O_DIRECT I see 79 MB/s.

As a comparison, reading in O_DIRECT mode in 8k blocks directly from the
backend device on the host gives 2.3 GB/s.  Reading in O_DIRECT mode
from a xen guest on the same hardware manages 263 MB/s.

Writing is affected in the same way, and exhibits the same behaviour
with O_SYNC too.

Watching with vmstat on the host, I see the same number of blocks being
read, but about 14 times the number of context switches in O_DIRECT mode
(4500 cs vs. 63000 cs) and a little more cpu usage.

The device I'm writing to is a device-mapper zero device that generates
zeros on read and throws away writes, you can set it up
at /dev/mapper/zero like this:

echo "0 21474836480 zero" | dmsetup create zero

My libvirt config for the disk is:

<disk type='block' device='disk'>
  <driver cache='none'/>
  <source dev='/dev/mapper/zero'/>
  <target dev='vdb' bus='virtio'/>
  <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/>
</disk>

which translates to the kvm arg:

-device
virtio-blk-pci,bus=pci.0,addr=0x4,drive=drive-virtio-disk0,id=virtio-disk0 -drive file=/dev/mapper/zero,if=none,id=drive-virtio-disk1,cache=none

I'm testing with dd:

dd if=/dev/vdb of=/dev/null bs=8k iflag=direct

As a side note, as you increase the block size read performance in
O_DIRECT mode starts to overtake non O_DIRECT mode reads (from about
150k block size). By 550k block size I'm seeing 1 GB/s reads with
O_DIRECT and 770 MB/s without.

Of course I see this performance situation with real disks too, I just
wanted to rule out the variables of moving metal around.

I get the same situation on Centos 5.5 and the latest RHEL 6 beta (which
is kvm 0.12 and kernel 2.6.32).  Hardware is a Dell i510 with 64GB RAM
and 12 Intel(R) Xeon(R) L5640 2.27GHz CPUs (running only one kvm guest
with 1G ram).  Host disk scheduler is deadline, guest disk scheduler is
noop.

Guest distro is Ubuntu Lucid, 2.6.32-22-server. I've tried with both
32bit pae and 64bit guest kernels.

Anyone got any thoughts on this?

Thanks,

John.

Re: bad O_DIRECT read and write performance with small block sizes with virtio

[Anthony Liguori]

On 08/02/2010 12:15 PM, John Leach wrote:
> Hi,
>
> I've come across a problem with read and write disk IO performance when
> using O_DIRECT from within a kvm guest.  With O_DIRECT, reads and writes
> are much slower with smaller block sizes.  Depending on the block size
> used, I've seen 10 times slower.
>
> For example, with an 8k block size, reading directly from /dev/vdb
> without O_DIRECT I see 750 MB/s, but with O_DIRECT I see 79 MB/s.
>
> As a comparison, reading in O_DIRECT mode in 8k blocks directly from the
> backend device on the host gives 2.3 GB/s.  Reading in O_DIRECT mode
> from a xen guest on the same hardware manages 263 MB/s.
>    

Stefan has a few fixes for this behavior that help a lot.  One of them 
(avoiding memset) is already upstream but not in 0.12.x.

The other two are not done yet but should be on the ML in the next 
couple weeks.  They involve using ioeventfd for notification and 
unlocking the block queue lock while doing a kick notification.

Regards,

Anthony Liguori

> Writing is affected in the same way, and exhibits the same behaviour
> with O_SYNC too.
>
> Watching with vmstat on the host, I see the same number of blocks being
> read, but about 14 times the number of context switches in O_DIRECT mode
> (4500 cs vs. 63000 cs) and a little more cpu usage.
>
> The device I'm writing to is a device-mapper zero device that generates
> zeros on read and throws away writes, you can set it up
> at /dev/mapper/zero like this:
>
> echo "0 21474836480 zero" | dmsetup create zero
>
> My libvirt config for the disk is:
>
> <disk type='block' device='disk'>
>    <driver cache='none'/>
>    <source dev='/dev/mapper/zero'/>
>    <target dev='vdb' bus='virtio'/>
>    <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/>
> </disk>
>
> which translates to the kvm arg:
>
> -device
> virtio-blk-pci,bus=pci.0,addr=0x4,drive=drive-virtio-disk0,id=virtio-disk0 -drive file=/dev/mapper/zero,if=none,id=drive-virtio-disk1,cache=none
>
> I'm testing with dd:
>
> dd if=/dev/vdb of=/dev/null bs=8k iflag=direct
>
> As a side note, as you increase the block size read performance in
> O_DIRECT mode starts to overtake non O_DIRECT mode reads (from about
> 150k block size). By 550k block size I'm seeing 1 GB/s reads with
> O_DIRECT and 770 MB/s without.
>
> Of course I see this performance situation with real disks too, I just
> wanted to rule out the variables of moving metal around.
>
> I get the same situation on Centos 5.5 and the latest RHEL 6 beta (which
> is kvm 0.12 and kernel 2.6.32).  Hardware is a Dell i510 with 64GB RAM
> and 12 Intel(R) Xeon(R) L5640 2.27GHz CPUs (running only one kvm guest
> with 1G ram).  Host disk scheduler is deadline, guest disk scheduler is
> noop.
>
> Guest distro is Ubuntu Lucid, 2.6.32-22-server. I've tried with both
> 32bit pae and 64bit guest kernels.
>
> Anyone got any thoughts on this?
>
> Thanks,
>
> John.
>
>
> --
> To unsubscribe from this list: send the line "unsubscribe kvm" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>    

Re: bad O_DIRECT read and write performance with small block sizes with virtio

[Stefan Hajnoczi]

On Mon, Aug 2, 2010 at 6:46 PM, Anthony Liguori <anthony@codemonkey.ws> wrote:
> On 08/02/2010 12:15 PM, John Leach wrote:
>>
>> Hi,
>>
>> I've come across a problem with read and write disk IO performance when
>> using O_DIRECT from within a kvm guest.  With O_DIRECT, reads and writes
>> are much slower with smaller block sizes.  Depending on the block size
>> used, I've seen 10 times slower.
>>
>> For example, with an 8k block size, reading directly from /dev/vdb
>> without O_DIRECT I see 750 MB/s, but with O_DIRECT I see 79 MB/s.
>>
>> As a comparison, reading in O_DIRECT mode in 8k blocks directly from the
>> backend device on the host gives 2.3 GB/s.  Reading in O_DIRECT mode
>> from a xen guest on the same hardware manages 263 MB/s.
>>
>
> Stefan has a few fixes for this behavior that help a lot.  One of them
> (avoiding memset) is already upstream but not in 0.12.x.
>
> The other two are not done yet but should be on the ML in the next couple
> weeks.  They involve using ioeventfd for notification and unlocking the
> block queue lock while doing a kick notification.

Thanks for mentioning those patches.  The ioeventfd patch will be sent
this week, I'm checking that migration works correctly and then need
to check that vhost-net still works.

>> Writing is affected in the same way, and exhibits the same behaviour
>> with O_SYNC too.
>>
>> Watching with vmstat on the host, I see the same number of blocks being
>> read, but about 14 times the number of context switches in O_DIRECT mode
>> (4500 cs vs. 63000 cs) and a little more cpu usage.
>>
>> The device I'm writing to is a device-mapper zero device that generates
>> zeros on read and throws away writes, you can set it up
>> at /dev/mapper/zero like this:
>>
>> echo "0 21474836480 zero" | dmsetup create zero
>>
>> My libvirt config for the disk is:
>>
>> <disk type='block' device='disk'>
>>   <driver cache='none'/>
>>   <source dev='/dev/mapper/zero'/>
>>   <target dev='vdb' bus='virtio'/>
>>   <address type='pci' domain='0x0000' bus='0x00' slot='0x06'
>> function='0x0'/>
>> </disk>
>>
>> which translates to the kvm arg:
>>
>> -device
>> virtio-blk-pci,bus=pci.0,addr=0x4,drive=drive-virtio-disk0,id=virtio-disk0
>> -drive file=/dev/mapper/zero,if=none,id=drive-virtio-disk1,cache=none
>>
>> I'm testing with dd:
>>
>> dd if=/dev/vdb of=/dev/null bs=8k iflag=direct
>>
>> As a side note, as you increase the block size read performance in
>> O_DIRECT mode starts to overtake non O_DIRECT mode reads (from about
>> 150k block size). By 550k block size I'm seeing 1 GB/s reads with
>> O_DIRECT and 770 MB/s without.

Can you take QEMU out of the picture and run the same test on the host:

dd if=/dev/vdb of=/dev/null bs=8k iflag=direct
vs
dd if=/dev/vdb of=/dev/null bs=8k

This isn't quite the same because QEMU will use a helper thread doing
preadv.  I'm not sure what syscall dd will use.

It should be close enough to determine whether QEMU and device
emulation are involved at all though, or whether these differences are
due to the host kernel code path down to the device mapper zero device
being different for normal vs O_DIRECT.

Stefan

Re: bad O_DIRECT read and write performance with small block sizes with virtio

[Dor Laor]

On 08/02/2010 11:50 PM, Stefan Hajnoczi wrote:
> On Mon, Aug 2, 2010 at 6:46 PM, Anthony Liguori<anthony@codemonkey.ws>  wrote:
>> On 08/02/2010 12:15 PM, John Leach wrote:
>>>
>>> Hi,
>>>
>>> I've come across a problem with read and write disk IO performance when
>>> using O_DIRECT from within a kvm guest.  With O_DIRECT, reads and writes
>>> are much slower with smaller block sizes.  Depending on the block size
>>> used, I've seen 10 times slower.
>>>
>>> For example, with an 8k block size, reading directly from /dev/vdb
>>> without O_DIRECT I see 750 MB/s, but with O_DIRECT I see 79 MB/s.
>>>
>>> As a comparison, reading in O_DIRECT mode in 8k blocks directly from the
>>> backend device on the host gives 2.3 GB/s.  Reading in O_DIRECT mode
>>> from a xen guest on the same hardware manages 263 MB/s.
>>>
>>
>> Stefan has a few fixes for this behavior that help a lot.  One of them
>> (avoiding memset) is already upstream but not in 0.12.x.
>>
>> The other two are not done yet but should be on the ML in the next couple
>> weeks.  They involve using ioeventfd for notification and unlocking the
>> block queue lock while doing a kick notification.
>
> Thanks for mentioning those patches.  The ioeventfd patch will be sent
> this week, I'm checking that migration works correctly and then need
> to check that vhost-net still works.
>
>>> Writing is affected in the same way, and exhibits the same behaviour
>>> with O_SYNC too.
>>>
>>> Watching with vmstat on the host, I see the same number of blocks being
>>> read, but about 14 times the number of context switches in O_DIRECT mode
>>> (4500 cs vs. 63000 cs) and a little more cpu usage.
>>>
>>> The device I'm writing to is a device-mapper zero device that generates
>>> zeros on read and throws away writes, you can set it up
>>> at /dev/mapper/zero like this:
>>>
>>> echo "0 21474836480 zero" | dmsetup create zero
>>>
>>> My libvirt config for the disk is:
>>>
>>> <disk type='block' device='disk'>
>>>    <driver cache='none'/>
>>>    <source dev='/dev/mapper/zero'/>
>>>    <target dev='vdb' bus='virtio'/>
>>>    <address type='pci' domain='0x0000' bus='0x00' slot='0x06'
>>> function='0x0'/>
>>> </disk>
>>>
>>> which translates to the kvm arg:
>>>
>>> -device
>>> virtio-blk-pci,bus=pci.0,addr=0x4,drive=drive-virtio-disk0,id=virtio-disk0
>>> -drive file=/dev/mapper/zero,if=none,id=drive-virtio-disk1,cache=none

aio=native and change the io scheduler on the host to deadline should 
help as well.

>>>
>>> I'm testing with dd:
>>>
>>> dd if=/dev/vdb of=/dev/null bs=8k iflag=direct
>>>
>>> As a side note, as you increase the block size read performance in
>>> O_DIRECT mode starts to overtake non O_DIRECT mode reads (from about
>>> 150k block size). By 550k block size I'm seeing 1 GB/s reads with
>>> O_DIRECT and 770 MB/s without.
>
> Can you take QEMU out of the picture and run the same test on the host:
>
> dd if=/dev/vdb of=/dev/null bs=8k iflag=direct
> vs
> dd if=/dev/vdb of=/dev/null bs=8k
>
> This isn't quite the same because QEMU will use a helper thread doing
> preadv.  I'm not sure what syscall dd will use.
>
> It should be close enough to determine whether QEMU and device
> emulation are involved at all though, or whether these differences are
> due to the host kernel code path down to the device mapper zero device
> being different for normal vs O_DIRECT.
>
> Stefan
> --
> To unsubscribe from this list: send the line "unsubscribe kvm" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: bad O_DIRECT read and write performance with small block sizes with virtio

[John Leach]

On Mon, 2010-08-02 at 21:50 +0100, Stefan Hajnoczi wrote:
> On Mon, Aug 2, 2010 at 6:46 PM, Anthony Liguori <anthony@codemonkey.ws> wrote:
> > On 08/02/2010 12:15 PM, John Leach wrote:
> >>
> >> Hi,
> >>
> >> I've come across a problem with read and write disk IO performance when
> >> using O_DIRECT from within a kvm guest.  With O_DIRECT, reads and writes
> >> are much slower with smaller block sizes.  Depending on the block size
> >> used, I've seen 10 times slower.
> >>
> >> For example, with an 8k block size, reading directly from /dev/vdb
> >> without O_DIRECT I see 750 MB/s, but with O_DIRECT I see 79 MB/s.
> >>
> >> As a comparison, reading in O_DIRECT mode in 8k blocks directly from the
> >> backend device on the host gives 2.3 GB/s.  Reading in O_DIRECT mode
> >> from a xen guest on the same hardware manages 263 MB/s.
> >>
> >
> > Stefan has a few fixes for this behavior that help a lot.  One of them
> > (avoiding memset) is already upstream but not in 0.12.x.

Anthony, that patch is already applied in the RHEL6 package I'm been
testing with - I've just manually confirmed that.  Thanks though.

> >
> > The other two are not done yet but should be on the ML in the next couple
> > weeks.  They involve using ioeventfd for notification and unlocking the
> > block queue lock while doing a kick notification.
> 
> Thanks for mentioning those patches.  The ioeventfd patch will be sent
> this week, I'm checking that migration works correctly and then need
> to check that vhost-net still works.

I'll give them a test as soon as I can get hold of them, thanks Stefan!

> >> Writing is affected in the same way, and exhibits the same behaviour
> >> with O_SYNC too.
> >>
> >> Watching with vmstat on the host, I see the same number of blocks being
> >> read, but about 14 times the number of context switches in O_DIRECT mode
> >> (4500 cs vs. 63000 cs) and a little more cpu usage.
> >>
> >> The device I'm writing to is a device-mapper zero device that generates
> >> zeros on read and throws away writes, you can set it up
> >> at /dev/mapper/zero like this:
> >>
> >> echo "0 21474836480 zero" | dmsetup create zero
> >>
> >> My libvirt config for the disk is:
> >>
> >> <disk type='block' device='disk'>
> >>   <driver cache='none'/>
> >>   <source dev='/dev/mapper/zero'/>
> >>   <target dev='vdb' bus='virtio'/>
> >>   <address type='pci' domain='0x0000' bus='0x00' slot='0x06'
> >> function='0x0'/>
> >> </disk>
> >>
> >> which translates to the kvm arg:
> >>
> >> -device
> >> virtio-blk-pci,bus=pci.0,addr=0x4,drive=drive-virtio-disk0,id=virtio-disk0
> >> -drive file=/dev/mapper/zero,if=none,id=drive-virtio-disk1,cache=none
> >>
> >> I'm testing with dd:
> >>
> >> dd if=/dev/vdb of=/dev/null bs=8k iflag=direct
> >>
> >> As a side note, as you increase the block size read performance in
> >> O_DIRECT mode starts to overtake non O_DIRECT mode reads (from about
> >> 150k block size). By 550k block size I'm seeing 1 GB/s reads with
> >> O_DIRECT and 770 MB/s without.
> 
> Can you take QEMU out of the picture and run the same test on the host:
> 
> dd if=/dev/vdb of=/dev/null bs=8k iflag=direct
> vs
> dd if=/dev/vdb of=/dev/null bs=8k
> 
> This isn't quite the same because QEMU will use a helper thread doing
> preadv.  I'm not sure what syscall dd will use.
> 
> It should be close enough to determine whether QEMU and device
> emulation are involved at all though, or whether these differences are
> due to the host kernel code path down to the device mapper zero device
> being different for normal vs O_DIRECT.


dd if=/dev/mapper/zero of=/dev/null bs=8k count=1000000 iflag=direct
8192000000 bytes (8.2 GB) copied, 3.46529 s, 2.4 GB/s

dd if=/dev/mapper/zero of=/dev/null bs=8k count=1000000
8192000000 bytes (8.2 GB) copied, 5.5741 s, 1.5 GB/s

dd is just using read.

Thanks,

John.

Re: bad O_DIRECT read and write performance with small block sizes with virtio

[Avi Kivity]

  On 08/03/2010 05:40 PM, John Leach wrote:
>
> dd if=/dev/mapper/zero of=/dev/null bs=8k count=1000000 iflag=direct
> 8192000000 bytes (8.2 GB) copied, 3.46529 s, 2.4 GB/s
>
> dd if=/dev/mapper/zero of=/dev/null bs=8k count=1000000
> 8192000000 bytes (8.2 GB) copied, 5.5741 s, 1.5 GB/s
>
> dd is just using read.
>

What's /dev/mapper/zero?  A real volume or a zero target?

-- 
error compiling committee.c: too many arguments to function

Re: bad O_DIRECT read and write performance with small block sizes with virtio

[John Leach]

On Tue, 2010-08-03 at 09:35 +0300, Dor Laor wrote:
> On 08/02/2010 11:50 PM, Stefan Hajnoczi wrote:
> > On Mon, Aug 2, 2010 at 6:46 PM, Anthony Liguori<anthony@codemonkey.ws>  wrote:
> >> On 08/02/2010 12:15 PM, John Leach wrote:
> >>>
> >>> Hi,
> >>>
> >>> I've come across a problem with read and write disk IO performance when
> >>> using O_DIRECT from within a kvm guest.  With O_DIRECT, reads and writes
> >>> are much slower with smaller block sizes.  Depending on the block size
> >>> used, I've seen 10 times slower.
> >>>
> >>> For example, with an 8k block size, reading directly from /dev/vdb
> >>> without O_DIRECT I see 750 MB/s, but with O_DIRECT I see 79 MB/s.
> >>>
> >>> As a comparison, reading in O_DIRECT mode in 8k blocks directly from the
> >>> backend device on the host gives 2.3 GB/s.  Reading in O_DIRECT mode
> >>> from a xen guest on the same hardware manages 263 MB/s.
> >>>
> >>
> >> Stefan has a few fixes for this behavior that help a lot.  One of them
> >> (avoiding memset) is already upstream but not in 0.12.x.
> >>
> >> The other two are not done yet but should be on the ML in the next couple
> >> weeks.  They involve using ioeventfd for notification and unlocking the
> >> block queue lock while doing a kick notification.
> >
> > Thanks for mentioning those patches.  The ioeventfd patch will be sent
> > this week, I'm checking that migration works correctly and then need
> > to check that vhost-net still works.
> >
> >>> Writing is affected in the same way, and exhibits the same behaviour
> >>> with O_SYNC too.
> >>>
> >>> Watching with vmstat on the host, I see the same number of blocks being
> >>> read, but about 14 times the number of context switches in O_DIRECT mode
> >>> (4500 cs vs. 63000 cs) and a little more cpu usage.
> >>>
> >>> The device I'm writing to is a device-mapper zero device that generates
> >>> zeros on read and throws away writes, you can set it up
> >>> at /dev/mapper/zero like this:
> >>>
> >>> echo "0 21474836480 zero" | dmsetup create zero
> >>>
> >>> My libvirt config for the disk is:
> >>>
> >>> <disk type='block' device='disk'>
> >>>    <driver cache='none'/>
> >>>    <source dev='/dev/mapper/zero'/>
> >>>    <target dev='vdb' bus='virtio'/>
> >>>    <address type='pci' domain='0x0000' bus='0x00' slot='0x06'
> >>> function='0x0'/>
> >>> </disk>
> >>>
> >>> which translates to the kvm arg:
> >>>
> >>> -device
> >>> virtio-blk-pci,bus=pci.0,addr=0x4,drive=drive-virtio-disk0,id=virtio-disk0
> >>> -drive file=/dev/mapper/zero,if=none,id=drive-virtio-disk1,cache=none
> 
> aio=native and change the io scheduler on the host to deadline should 
> help as well.

No improvement in this case (I was already using deadline on the host,
and just tested with aio=native). Tried with a real disk backend too,
still no improvement.

I'll try with and without once I get Stefan's other patches too though.

Thanks,

John.

Re: bad O_DIRECT read and write performance with small block sizes with virtio

[John Leach]

On Tue, 2010-08-03 at 17:44 +0300, Avi Kivity wrote:
> On 08/03/2010 05:40 PM, John Leach wrote:
> >
> > dd if=/dev/mapper/zero of=/dev/null bs=8k count=1000000 iflag=direct
> > 8192000000 bytes (8.2 GB) copied, 3.46529 s, 2.4 GB/s
> >
> > dd if=/dev/mapper/zero of=/dev/null bs=8k count=1000000
> > 8192000000 bytes (8.2 GB) copied, 5.5741 s, 1.5 GB/s
> >
> > dd is just using read.
> >
> 
> What's /dev/mapper/zero?  A real volume or a zero target?
> 

zero target:

echo "0 21474836480 zero" | dmsetup create zero

The same performance penalty occurs when using real disks though, I just
moved to a zero target to rule out the variables of spinning metal and
raid controller caches.

John.

Re: bad O_DIRECT read and write performance with small block sizes with virtio

[Avi Kivity]

  On 08/03/2010 05:57 PM, John Leach wrote:
> On Tue, 2010-08-03 at 17:44 +0300, Avi Kivity wrote:
>> On 08/03/2010 05:40 PM, John Leach wrote:
>>> dd if=/dev/mapper/zero of=/dev/null bs=8k count=1000000 iflag=direct
>>> 8192000000 bytes (8.2 GB) copied, 3.46529 s, 2.4 GB/s
>>>
>>> dd if=/dev/mapper/zero of=/dev/null bs=8k count=1000000
>>> 8192000000 bytes (8.2 GB) copied, 5.5741 s, 1.5 GB/s
>>>
>>> dd is just using read.
>>>
>> What's /dev/mapper/zero?  A real volume or a zero target?
>>
> zero target:
>
> echo "0 21474836480 zero" | dmsetup create zero
>
> The same performance penalty occurs when using real disks though, I just
> moved to a zero target to rule out the variables of spinning metal and
> raid controller caches.

Don't, it's confusing things.  I'd expect dd to be slower with 
iflag=direct since the kernel can't do readahead an instead must 
roundtrip to the controller.  With a zero target it's faster since it 
doesn't have to roundtrip and instead avoids a copy.

-- 
error compiling committee.c: too many arguments to function

Re: bad O_DIRECT read and write performance with small block sizes with virtio

[Stefan Hajnoczi]

On Tue, Aug 3, 2010 at 5:14 PM, Avi Kivity <avi@redhat.com> wrote:
>  On 08/03/2010 05:57 PM, John Leach wrote:
>>
>> On Tue, 2010-08-03 at 17:44 +0300, Avi Kivity wrote:
>>>
>>> On 08/03/2010 05:40 PM, John Leach wrote:
>>>>
>>>> dd if=/dev/mapper/zero of=/dev/null bs=8k count=1000000 iflag=direct
>>>> 8192000000 bytes (8.2 GB) copied, 3.46529 s, 2.4 GB/s
>>>>
>>>> dd if=/dev/mapper/zero of=/dev/null bs=8k count=1000000
>>>> 8192000000 bytes (8.2 GB) copied, 5.5741 s, 1.5 GB/s
>>>>
>>>> dd is just using read.
>>>>
>>> What's /dev/mapper/zero?  A real volume or a zero target?
>>>
>> zero target:
>>
>> echo "0 21474836480 zero" | dmsetup create zero
>>
>> The same performance penalty occurs when using real disks though, I just
>> moved to a zero target to rule out the variables of spinning metal and
>> raid controller caches.
>
> Don't, it's confusing things.  I'd expect dd to be slower with iflag=direct
> since the kernel can't do readahead an instead must roundtrip to the
> controller.  With a zero target it's faster since it doesn't have to
> roundtrip and instead avoids a copy.

I agree that the devicemapper zero target acts differently from a
physical disk and you may wish to drop it in favor of realistic disk
characteristics.

Here is the virtio-ioeventfd patch which improves performance
especially for multi-threaded workloads:
http://repo.or.cz/w/qemu/stefanha.git/commitdiff/889460ccc86a0121df9c77a698f50bf007005b7f

(The patch is against qemu.git not qemu-kvm.git.)

A single-threaded dd will not benefit; it may see a small degradation
due to additional latency.  But I want to share this patch with you
while I'm still testing and preparing it for upstream so you can take
it on board for any benchmarking.

Stefan