Re: [Qemu-devel] Re: [PATCH 1/3] Refactor AIO interface to allow other AIO implementations

Avi Kivity
2008-04-22T08:10:28Z

Jamie Lokier wrote:
> Avi Kivity wrote:
>
>>> At such a tiny difference, I'm wondering why Linux-AIO exists at all,
>>> as it complicates the kernel rather a lot. I can see the theoretical
>>> appeal, but if performance is so marginal, I'm surprised it's in
>>> there.
>>>
>> Linux aio exists, but that's all that can be said for it. It works
>> mostly for raw disks, doesn't integrate with networking, and doesn't
>> advance at the same pace as the rest of the kernel. I believe only
>> databases use it (and a userspace filesystem I wrote some time ago).
>>
>
> And video streaming on some embedded devices with no MMU! (Due to the
> page cache heuristics working poorly with no MMU, sustained reliable
> streaming is managed with O_DIRECT and the app managing cache itself
> (like a database), and that needs AIO to keep the request queue busy.
> At least, that's the theory.)
>
>

Could use threads as well, no?

>>> I'm also surprised the Glibc implementation of AIO using ordinary
>>> threads is so close to it.
>>>
>> Why are you surprised?
>>
>
> Because I've read that Glibc AIO (which uses a thread pool) is a
> relatively poor performer as AIO implementations go, and is only there
> for API compatibility, not suggested for performance.
>
> But I read that quite a while ago, perhaps it's changed.
>
>

It's me at fault here. I just assumed that because it's easy to do aio
in a thread pool efficiently, that's what glibc does.

Unfortunately the code does some ridiculous things like not service
multiple requests on a single fd in parallel. I see absolutely no
reason for it (the code says "fight for resources").

So my comments only apply to linux-aio vs a sane thread pool. Sorry for
spreading confusion.

>> Actually the glibc implementation could be improved from what I've
>> heard. My estimates are for a thread pool implementation, but there is
>> not reason why glibc couldn't achieve exactly the same performance.
>>
>
> Erm... I thought you said it _does_ achieve nearly the same
> performance, not that it _could_.
>
> Do you mean it could achieve exactly the same performance by using
> Linux AIO when possible?
>
>

It could and should. It probably doesn't.

A simple thread pool implementation could come within 10% of Linux aio
for most workloads. It will never be "exactly", but for small numbers
of disks, close enough.

>>> And then, I'm wondering why use AIO it
>>> all: it suggests QEMU would run about as fast doing synchronous I/O in
>>> a few dedicated I/O threads.
>>>
>> Posix aio is the unix API for this, why not use it?
>>
>
> Because far more host platforms have threads than have POSIX AIO. (I
> suspect both options will end up supported in the end, as dedicated
> I/O threads were already suggested for other things.)
>

Agree.

>
>>>> Also, I'd presume that those that need 10K IOPS and above will not place
>>>> their high throughput images on a filesystem; rather on a separate SAN
>>>> LUN.
>>>>
>>> Does the separate LUN make any difference? I thought O_DIRECT on a
>>> filesystem was meant to be pretty close to block device performance.
>>>
>> On a good extent-based filesystem like XFS you will get good performance
>> (though more cpu overhead due to needing to go through additional
>> mapping layers. Old clunkers like ext3 will require additional seeks or
>> a ton of cache (1 GB per 1 TB).
>>
>
> Hmm. Thanks. I may consider switching to XFS now....
>
>

I'm rooting for btrfs myself.

--
error compiling committee.c: too many arguments to function