Re: Approaches to making io_submit not block

[Andrew Morton <akpm@linux-foundation.org>]

On Tue, 30 Aug 2011 16:03:42 -0700
Jeremy Allison <jra@samba.org> wrote:

> On Tue, Aug 30, 2011 at 03:54:38PM -0700, Andrew Morton wrote:
> > 
> > Also, glibc has userspace for POSIX AIO.  A successful kernel-based
> > implementation would result in glibc migrating away from its current
> > implementation.  So we should work with the glibc developers on ensuring
> > that the migration can happen.
> 
> Unfortunately the glibc userspace POSIX AIO limits asynchronicity to
> one outstanding request per file descriptor. From aio_misc.c in glibc:
> 
>   if (runp != NULL
>       && runp->aiocbp->aiocb.aio_fildes == aiocbp->aiocb.aio_fildes)
>     {
>       /* The current file descriptor is worked on.  It makes no sense
>          to start another thread since this new thread would fight
>          with the running thread for the resources.  But we also cannot
>          say that the thread processing this desriptor shall immediately
>          after finishing the current job process this request if there
>          are other threads in the running queue which have a higher
>          priority.  */
> 
>       /* Simply enqueue it after the running one according to the
>          priority.  */
> 
> I have often wondered if this is actually the case ? I created
> my own glibc with a patches AIO that removed this restriction
> (thus had multiple outstanding threads on a single fd). In testing
> I saw a dramatic increase in performance (2x speedup) but then
> testing with use in actual code (Samba smbd) it made the client
> throughput *worse*. I never got to the bottom of this and so
> didn't submit my fixes to glibc.
> 
> Any ideas if this is still the case ? Or comments on why glibc
> insists on only one outstanding request per fd ? Is this really
> needed for kernel performance ?
> 

I don't know.  Uli cc'ed.