Can you run the same performance tests with the following patches (using 
sync=on instead of cache=off)?

You'll need my aio_init fix too.  I suspect this will give equally good 
performance to your patch set.  That's not saying your patch set isn't 
useful, but I would like to get performance to be better for the case 
that we're going through the page cache.

Regards,

Anthony Liguori

Ryan Harper wrote:
> The patchset adds additional AIO driver abstraction to the block raw driver to
> support multiple aio implementations for each device.  The first patch pulls
> the posix aio implementation out of the block-raw device using a generic call
> to the newly created AIO Driver structure.  The posix aio implementation that
> was contained in block-raw-posix.c has been refactored in to aio-posix.c.  The
> next patch adds a linux aio implementation for raw devices being opened
> O_DIRECT via cache=off drive option.  We only use linux aio when cache=off as
> linux aio falls back to synchronous ops if not opened with O_DIRECT flag.
>
> Addtional work has been done on top of QEMU for KVM and virtio-blk devices.
> While virtio-blk is not yet upstream in QEMU, the AIO changes here provide a
> tremendous performance improvement (from 7.6% of native, to 100% of randwrite,
> and 3.9% of native, to 101.4% of native for seq write) for virtio
> devices with cache=off.
>
> Storage subsystem:
>   IBM EXP300 - 14 Disk Fiber Expansion, 17G - 15K RPMS
>   Host: AMD Barcelona, 2 socket, 8G RAM
>   HBA: QLogic Corp. ISP2312-based 2Gb Fibre Channel to PCI-X HBA (rev 02)
>
> Benchmark[1]:
>            fio --name=guestrun --filename=/dev/mapper/volumes-fibre \
>                --rw=randwrite --bs=16k --ioengine=libaio --direct=1 \
>                --norandommap --runtime=120 --time_based --numjobs=1 \
>                --group_reporting --thread --size=25g --write_lat_log \
>                --write_bw_log --iodepth=74
>
> Qemu parameters:
>  -m 1024 \
>  -drive file=/images/npt2-guest-virtio.qcow2,if=ide,boot=on,snapshot=off \
>  -drive file=/dev/mapper/volumes-fibre,if=virtio,cache=(on|off) \
>  -drive file=/dev/mapper/volumes-npt2--dom1,if=virtio,cache=off
>  -net nic,macaddr=00:FF:FF:00:00:01,model=rtl8139 -net tap -vnc :123 \
>  -monitor stdio
>
> Guest io scheduler: noop
>
> Results:
>  These results are with the patch series applied to KVM (plus a small KVM only
>  change -- KVM patches forthcoming).
>
>
>  16k randwrite 1 thread, 74 iodepth | MB/s | avg sub lat (us) | avg comp lat (ms)
>  ---------------------------------------+---------------------+------------------
>  baremetal (O_DIRECT, aka cache=off)| 61.2 |   13.07          |  19.59
>  kvm: cache=off posix-aio w/o patch |  4.7 | 3467.44          | 254.08
>  kvm: cache=off linux-aio           | 61.1 |   75.35          |  19.57
>  kvm: cache=on  posix-aio w/o patch |127.0 |  115.78          |   9.19
>  kvm: cache=on  posix-aio w/ patch  |126.0 |   67.35          |   9.30
>
>
>  16k write 1 thread, 74 iodepth     | MB/s | avg sub lat (us) | avg comp lat (ms)
>  ---------------------------------------+---------------------+------------------
>  baremetal (O_DIRECT, aka cache=off)|128.1 |   10.90          |   9.45
>  kvm: cache=off posix-aio w/o patch |  5.1 | 3152.00          | 231.06 
>  kvm: cache=off linux-aio           |130.0 |   83.83          |   8.99
>  kvm: cache=on  posix-aio w/o patch |184.0 |   80.46          |   6.35
>  kvm: cache=on  posix-aio w/ patch  |165.0 |   70.90          |   7.09
>
>
> 1. http://brick.kernel.dk/snaps/fio-1.21.tar.bz2
>