[Qemu-devel] [PATCH 0/3] Refactor AIO to allow multiple AIO implementations

[Ryan Harper <ryanh@us.ibm.com>]

The patchset adds additional AIO driver abstraction to the block raw driver to
support multiple aio implementations for each device.  The first patch pulls
the posix aio implementation out of the block-raw device using a generic call
to the newly created AIO Driver structure.  The posix aio implementation that
was contained in block-raw-posix.c has been refactored in to aio-posix.c.  The
next patch adds a linux aio implementation for raw devices being opened
O_DIRECT via cache=off drive option.  We only use linux aio when cache=off as
linux aio falls back to synchronous ops if not opened with O_DIRECT flag.

Addtional work has been done on top of QEMU for KVM and virtio-blk devices.
While virtio-blk is not yet upstream in QEMU, the AIO changes here provide a
tremendous performance improvement (from 7.6% of native, to 100% of randwrite,
and 3.9% of native, to 101.4% of native for seq write) for virtio
devices with cache=off.

Storage subsystem:
  IBM EXP300 - 14 Disk Fiber Expansion, 17G - 15K RPMS
  Host: AMD Barcelona, 2 socket, 8G RAM
  HBA: QLogic Corp. ISP2312-based 2Gb Fibre Channel to PCI-X HBA (rev 02)

Benchmark[1]:
           fio --name=guestrun --filename=/dev/mapper/volumes-fibre \
               --rw=randwrite --bs=16k --ioengine=libaio --direct=1 \
               --norandommap --runtime=120 --time_based --numjobs=1 \
               --group_reporting --thread --size=25g --write_lat_log \
               --write_bw_log --iodepth=74

Qemu parameters:
 -m 1024 \
 -drive file=/images/npt2-guest-virtio.qcow2,if=ide,boot=on,snapshot=off \
 -drive file=/dev/mapper/volumes-fibre,if=virtio,cache=(on|off) \
 -drive file=/dev/mapper/volumes-npt2--dom1,if=virtio,cache=off
 -net nic,macaddr=00:FF:FF:00:00:01,model=rtl8139 -net tap -vnc :123 \
 -monitor stdio

Guest io scheduler: noop

Results:
 These results are with the patch series applied to KVM (plus a small KVM only
 change -- KVM patches forthcoming).


 16k randwrite 1 thread, 74 iodepth | MB/s | avg sub lat (us) | avg comp lat (ms)
 ---------------------------------------+---------------------+------------------
 baremetal (O_DIRECT, aka cache=off)| 61.2 |   13.07          |  19.59
 kvm: cache=off posix-aio w/o patch |  4.7 | 3467.44          | 254.08
 kvm: cache=off linux-aio           | 61.1 |   75.35          |  19.57
 kvm: cache=on  posix-aio w/o patch |127.0 |  115.78          |   9.19
 kvm: cache=on  posix-aio w/ patch  |126.0 |   67.35          |   9.30


 16k write 1 thread, 74 iodepth     | MB/s | avg sub lat (us) | avg comp lat (ms)
 ---------------------------------------+---------------------+------------------
 baremetal (O_DIRECT, aka cache=off)|128.1 |   10.90          |   9.45
 kvm: cache=off posix-aio w/o patch |  5.1 | 3152.00          | 231.06 
 kvm: cache=off linux-aio           |130.0 |   83.83          |   8.99
 kvm: cache=on  posix-aio w/o patch |184.0 |   80.46          |   6.35
 kvm: cache=on  posix-aio w/ patch  |165.0 |   70.90          |   7.09


1. http://brick.kernel.dk/snaps/fio-1.21.tar.bz2