Re: [Qemu-devel] [5323] Implement an fd pool to get real AIO with posix-aio

[Ryan Harper <ryanh@us.ibm.com>]

* Anthony Liguori <aliguori@us.ibm.com> [2008-09-26 13:37]:
> Ryan Harper wrote:
> >* Anthony Liguori <anthony@codemonkey.ws> [2008-09-26 11:03]:

> >kvm: cache=on  posix-aio w/o patch |127.0 |  115.78          |   9.19
> >kvm: cache=on  posix-aio w/ patch  |126.0 |   67.35          |   9.30
> >  
> 
> It looks like 127mb/s is pretty close to the optimal cached write time.  
> When using caching, writes can complete almost immediately so it's not 
> surprising that submission latency is so low (even though it's blocking 
> during submission).
> 
> I am surprised that w/patch has a latency that's so high.  I think that 
> suggests that requests are queuing up.  I bet increasing the aio_num 
> field would reduce this number.

Yeah, there is plenty of room to twiddle with the threads and number of
outstanding ios, but that'll take quite a bit of time to generate the
data and compare.

> >------------ new results 
> >----------+------+------------------+------------------
> >kvm:cache=off posix-aio fd_pool[16]| 33.5 |   14.28          |  49.19
> >kvm:cache=off posix-aio fd_pool[64]| 51.1 |   14.86          |  23.66
> >  
> 
> I assume you tried to bump from 64 to something higher and couldn't make 
> up the lost bandwidth?

Very slightly, switching to 128 threads/fds gave another 1MB/s. 

> >16k write 1 thread, 74 iodepth     | MB/s | avg sub lat (us) | avg comp 
> >lat (ms)
> >-----------------------------------+------+------------------+------------------
> >baremetal (O_DIRECT, aka cache=off)|128.1 |   10.90          |   9.45
> >kvm: cache=off posix-aio w/o patch |  5.1 | 3152.00          | 231.06 
> >kvm: cache=off linux-aio           |130.0 |   83.83          |   8.99
> >kvm: cache=on  posix-aio w/o patch |184.0 |   80.46          |   6.35
> >kvm: cache=on  posix-aio w/ patch  |165.0 |   70.90          |   7.09
> >------------ new results 
> >----------+------+------------------+------------------
> >kvm:cache=off posix-aio fd_pool[16]| 78.2 |   58.24          |  15.43
> >kvm:cache=off posix-aio fd_pool[64]|129.0 |   71.62          |   9.11
> >  
> 
> That's a nice result.  We could probably improve the latency by tweaking 
> the queue sizes.

Yeah, I was quite pleased to see a simpler solution perform so well.
> 
> Very nice work!  Thanks for doing the thorough analysis.

Thanks, very happy to see a signficant improvement in IO here.

-- 
Ryan Harper
Software Engineer; Linux Technology Center
IBM Corp., Austin, Tx
(512) 838-9253   T/L: 678-9253
ryanh@us.ibm.com