Re: [Qemu-devel] Re: [PATCH 1/3] Refactor AIO interface to allow other AIO implementations

[Avi Kivity <avi@qumranet.com>]

Jamie Lokier wrote:
> Avi Kivity wrote:
>   
>>> At such a tiny difference, I'm wondering why Linux-AIO exists at all,
>>> as it complicates the kernel rather a lot.  I can see the theoretical
>>> appeal, but if performance is so marginal, I'm surprised it's in
>>> there.
>>>       
>> Linux aio exists, but that's all that can be said for it.  It works 
>> mostly for raw disks, doesn't integrate with networking, and doesn't 
>> advance at the same pace as the rest of the kernel.  I believe only 
>> databases use it (and a userspace filesystem I wrote some time ago).
>>     
>
> And video streaming on some embedded devices with no MMU!  (Due to the
> page cache heuristics working poorly with no MMU, sustained reliable
> streaming is managed with O_DIRECT and the app managing cache itself
> (like a database), and that needs AIO to keep the request queue busy.
> At least, that's the theory.)
>
>   

Could use threads as well, no?

>>> I'm also surprised the Glibc implementation of AIO using ordinary
>>> threads is so close to it.  
>>>       
>> Why are you surprised?
>>     
>
> Because I've read that Glibc AIO (which uses a thread pool) is a
> relatively poor performer as AIO implementations go, and is only there
> for API compatibility, not suggested for performance.
>
> But I read that quite a while ago, perhaps it's changed.
>
>   

It's me at fault here.  I just assumed that because it's easy to do aio 
in a thread pool efficiently, that's what glibc does.

Unfortunately the code does some ridiculous things like not service 
multiple requests on a single fd in parallel.  I see absolutely no 
reason for it (the code says "fight for resources").

So my comments only apply to linux-aio vs a sane thread pool.  Sorry for 
spreading confusion.

>> Actually the glibc implementation could be improved from what I've 
>> heard.  My estimates are for a thread pool implementation, but there is 
>> not reason why glibc couldn't achieve exactly the same performance.
>>     
>
> Erm...  I thought you said it _does_ achieve nearly the same
> performance, not that it _could_.
>
> Do you mean it could achieve exactly the same performance by using
> Linux AIO when possible?
>
>   

It could and should.  It probably doesn't.

A simple thread pool implementation could come within 10% of Linux aio 
for most workloads.  It will never be "exactly", but for small numbers 
of disks, close enough.

>>> And then, I'm wondering why use AIO it
>>> all: it suggests QEMU would run about as fast doing synchronous I/O in
>>> a few dedicated I/O threads.
>>>       
>> Posix aio is the unix API for this, why not use it?
>>     
>
> Because far more host platforms have threads than have POSIX AIO.  (I
> suspect both options will end up supported in the end, as dedicated
> I/O threads were already suggested for other things.)
>   

Agree.

>   
>>>> Also, I'd presume that those that need 10K IOPS and above will not place 
>>>> their high throughput images on a filesystem; rather on a separate SAN 
>>>> LUN.
>>>>         
>>> Does the separate LUN make any difference?  I thought O_DIRECT on a
>>> filesystem was meant to be pretty close to block device performance.
>>>       
>> On a good extent-based filesystem like XFS you will get good performance 
>> (though more cpu overhead due to needing to go through additional 
>> mapping layers.  Old clunkers like ext3 will require additional seeks or 
>> a ton of cache (1 GB per 1 TB).
>>     
>
> Hmm.  Thanks.  I may consider switching to XFS now....
>
>   

I'm rooting for btrfs myself.

-- 
error compiling committee.c: too many arguments to function


-------------------------------------------------------------------------
This SF.net email is sponsored by the 2008 JavaOne(SM) Conference 
Don't miss this year's exciting event. There's still time to save $100. 
Use priority code J8TL2D2. 
http://ad.doubleclick.net/clk;198757673;13503038;p?http://java.sun.com/javaone