From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mailman by lists.gnu.org with tmda-scanned (Exim 4.43) id 1KhufL-0001sc-BU for qemu-devel@nongnu.org; Mon, 22 Sep 2008 19:18:03 -0400 Received: from exim by lists.gnu.org with spam-scanned (Exim 4.43) id 1KhufI-0001px-Cz for qemu-devel@nongnu.org; Mon, 22 Sep 2008 19:18:03 -0400 Received: from [199.232.76.173] (port=45217 helo=monty-python.gnu.org) by lists.gnu.org with esmtp (Exim 4.43) id 1KhufI-0001ps-AF for qemu-devel@nongnu.org; Mon, 22 Sep 2008 19:18:00 -0400 Received: from e2.ny.us.ibm.com ([32.97.182.142]:53609) by monty-python.gnu.org with esmtps (TLS-1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.60) (envelope-from ) id 1KhufH-00056t-Up for qemu-devel@nongnu.org; Mon, 22 Sep 2008 19:18:00 -0400 Received: from d01relay02.pok.ibm.com (d01relay02.pok.ibm.com [9.56.227.234]) by e2.ny.us.ibm.com (8.13.8/8.13.8) with ESMTP id m8MNHeuB032737 for ; Mon, 22 Sep 2008 19:17:40 -0400 Received: from d01av04.pok.ibm.com (d01av04.pok.ibm.com [9.56.224.64]) by d01relay02.pok.ibm.com (8.13.8/8.13.8/NCO v9.1) with ESMTP id m8MNHZDU241086 for ; Mon, 22 Sep 2008 19:17:40 -0400 Received: from d01av04.pok.ibm.com (loopback [127.0.0.1]) by d01av04.pok.ibm.com (8.12.11.20060308/8.13.3) with ESMTP id m8MNHZtV031594 for ; Mon, 22 Sep 2008 19:17:35 -0400 From: Ryan Harper Date: Mon, 22 Sep 2008 18:17:31 -0500 Message-Id: <1222125454-21744-1-git-send-email-ryanh@us.ibm.com> Subject: [Qemu-devel] [PATCH 0/3] Refactor AIO to allow multiple AIO implementations Reply-To: qemu-devel@nongnu.org List-Id: qemu-devel.nongnu.org List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: qemu-devel@nongnu.org Cc: aliguori@us.ibm.com, kvm@vger.kernel.org The patchset adds additional AIO driver abstraction to the block raw driver to support multiple aio implementations for each device. The first patch pulls the posix aio implementation out of the block-raw device using a generic call to the newly created AIO Driver structure. The posix aio implementation that was contained in block-raw-posix.c has been refactored in to aio-posix.c. The next patch adds a linux aio implementation for raw devices being opened O_DIRECT via cache=off drive option. We only use linux aio when cache=off as linux aio falls back to synchronous ops if not opened with O_DIRECT flag. Addtional work has been done on top of QEMU for KVM and virtio-blk devices. While virtio-blk is not yet upstream in QEMU, the AIO changes here provide a tremendous performance improvement (from 7.6% of native, to 100% of randwrite, and 3.9% of native, to 101.4% of native for seq write) for virtio devices with cache=off. Storage subsystem: IBM EXP300 - 14 Disk Fiber Expansion, 17G - 15K RPMS Host: AMD Barcelona, 2 socket, 8G RAM HBA: QLogic Corp. ISP2312-based 2Gb Fibre Channel to PCI-X HBA (rev 02) Benchmark[1]: fio --name=guestrun --filename=/dev/mapper/volumes-fibre \ --rw=randwrite --bs=16k --ioengine=libaio --direct=1 \ --norandommap --runtime=120 --time_based --numjobs=1 \ --group_reporting --thread --size=25g --write_lat_log \ --write_bw_log --iodepth=74 Qemu parameters: -m 1024 \ -drive file=/images/npt2-guest-virtio.qcow2,if=ide,boot=on,snapshot=off \ -drive file=/dev/mapper/volumes-fibre,if=virtio,cache=(on|off) \ -drive file=/dev/mapper/volumes-npt2--dom1,if=virtio,cache=off -net nic,macaddr=00:FF:FF:00:00:01,model=rtl8139 -net tap -vnc :123 \ -monitor stdio Guest io scheduler: noop Results: These results are with the patch series applied to KVM (plus a small KVM only change -- KVM patches forthcoming). 16k randwrite 1 thread, 74 iodepth | MB/s | avg sub lat (us) | avg comp lat (ms) ---------------------------------------+---------------------+------------------ baremetal (O_DIRECT, aka cache=off)| 61.2 | 13.07 | 19.59 kvm: cache=off posix-aio w/o patch | 4.7 | 3467.44 | 254.08 kvm: cache=off linux-aio | 61.1 | 75.35 | 19.57 kvm: cache=on posix-aio w/o patch |127.0 | 115.78 | 9.19 kvm: cache=on posix-aio w/ patch |126.0 | 67.35 | 9.30 16k write 1 thread, 74 iodepth | MB/s | avg sub lat (us) | avg comp lat (ms) ---------------------------------------+---------------------+------------------ baremetal (O_DIRECT, aka cache=off)|128.1 | 10.90 | 9.45 kvm: cache=off posix-aio w/o patch | 5.1 | 3152.00 | 231.06 kvm: cache=off linux-aio |130.0 | 83.83 | 8.99 kvm: cache=on posix-aio w/o patch |184.0 | 80.46 | 6.35 kvm: cache=on posix-aio w/ patch |165.0 | 70.90 | 7.09 1. http://brick.kernel.dk/snaps/fio-1.21.tar.bz2