Re: [Qemu-devel] [5323] Implement an fd pool to get real AIO with posix-aio

[Ryan Harper <ryanh@us.ibm.com>]

* Anthony Liguori <anthony@codemonkey.ws> [2008-09-26 11:03]:
> Revision: 5323
>           http://svn.sv.gnu.org/viewvc/?view=rev&root=qemu&revision=5323
> Author:   aliguori
> Date:     2008-09-26 15:59:29 +0000 (Fri, 26 Sep 2008)
> 
> Log Message:
> -----------
> Implement an fd pool to get real AIO with posix-aio
> 
> This patch implements a simple fd pool to allow many AIO requests with
> posix-aio.  The result is significantly improved performance (identical to that
> reported for linux-aio) for both cache=on and cache=off.
> 
> The fundamental problem with posix-aio is that it limits itself to one thread
> per-file descriptor.  I don't know why this is, but this patch provides a simple
> mechanism to work around this (duplicating the file descriptor).
> 
> This isn't a great solution, but it seems like a reasonable intermediate step
> between posix-aio and a custom thread-pool to replace it.
> 
> Ryan Harper will be posting some performance analysis he did comparing posix-aio
> with fd pooling against linux-aio.  The size of the posix-aio thread pool and
> the fd pool were largely determined by him based on this analysis.

I'll have some more data to post in a bit, but for now, bumping the fd
pool up to 64 and ensuring we init aio to support a thread per fd, we
mostly match linux aio performance with a simpler implementation.  For
randomwrites, fd_pool lags a bit, but I've got other data that shows in
most scenarios, fd_pool matches linux aio performance and does so with
less CPU consumption.

Results:

16k randwrite 1 thread, 74 iodepth | MB/s | avg sub lat (us) | avg comp lat (ms)
-----------------------------------+------+------------------+------------------
baremetal (O_DIRECT, aka cache=off)| 61.2 |   13.07          |  19.59
kvm: cache=off posix-aio w/o patch |  4.7 | 3467.44          | 254.08
kvm: cache=off linux-aio           | 61.1 |   75.35          |  19.57
kvm: cache=on  posix-aio w/o patch |127.0 |  115.78          |   9.19
kvm: cache=on  posix-aio w/ patch  |126.0 |   67.35          |   9.30
------------ new results ----------+------+------------------+------------------
kvm:cache=off posix-aio fd_pool[16]| 33.5 |   14.28          |  49.19
kvm:cache=off posix-aio fd_pool[64]| 51.1 |   14.86          |  23.66


16k write 1 thread, 74 iodepth     | MB/s | avg sub lat (us) | avg comp lat (ms)
-----------------------------------+------+------------------+------------------
baremetal (O_DIRECT, aka cache=off)|128.1 |   10.90          |   9.45
kvm: cache=off posix-aio w/o patch |  5.1 | 3152.00          | 231.06 
kvm: cache=off linux-aio           |130.0 |   83.83          |   8.99
kvm: cache=on  posix-aio w/o patch |184.0 |   80.46          |   6.35
kvm: cache=on  posix-aio w/ patch  |165.0 |   70.90          |   7.09
------------ new results ----------+------+------------------+------------------
kvm:cache=off posix-aio fd_pool[16]| 78.2 |   58.24          |  15.43
kvm:cache=off posix-aio fd_pool[64]|129.0 |   71.62          |   9.11


-- 
Ryan Harper
Software Engineer; Linux Technology Center
IBM Corp., Austin, Tx
(512) 838-9253   T/L: 678-9253
ryanh@us.ibm.com