Re: Approaches to making io_submit not block

[Jeremy Allison <jra@samba.org>]

On Wed, Aug 31, 2011 at 07:04:10AM -0400, Ulrich Drepper wrote:
> On 08/30/2011 07:11 PM, Andrew Morton wrote:
> > I don't know.  Uli cc'ed.
> 
> glibc has to create the parallelism of the operations through threads.
> More threads mean more overhead.  There are insane people out there who
> wrote code which pushed the number of helper threads into the hundreds
> (at that time a high number, today this would be in the thousands).
> Anyway, any scheme which is worth changing the code for would be in the
> kernel.  Only the kernel knows which requests can actually be handled
> concurrently.

I get that, but isn't that what the aio_init(const struct aioinit *init) call is meant to
solve ?

After all:

struct aioinit
  {
    int aio_threads;            /* Maximal number of threads.  */
    int aio_num;                /* Number of expected simultanious requests. */
    int aio_locks;              /* Not used.  */
    int aio_usedba;             /* Not used.  */
    int aio_debug;              /* Not used.  */
    int aio_numusers;           /* Not used.  */
    int aio_idle_time;          /* Number of seconds before idle thread
                                   terminates.  */
    int aio_reserved;
  };

Would seem to be the existing way to limit this. What I don't understand
is why you restrict the pthread aio implementation to only allow
*one* outstanding request per file descriptor ?

Surely it doesn't matter what fd the requests are outstanding on, isn't
it the total number of outstanding threads that really matter ?

By limiting this to one outstanding request per fd, this prevents
any parallelization for requests on a single file for a file server
written to use the aio interface in glibc. Windows SMB2 client issue
multiple simultaneous asynchronous requests on a single file, and
the current glibc implementation throttles them to be linear.

Volker had to create a vfs_aio_fork Samba VFS module that uses
forked processes and shared memory communication to get around
this exact problem. It does improve performance for SMB2 clients,
but means we have many more processes than we'd otherwise need
if we didn't have the restriction.

Do you have benchmark data that shows that limiting outstanding requests
to one per fd is a performance win ?

Do you want to see my patch to glibc that removes this restriction
so we can test this on any benchmarks you used to decide on the
one-request-per-fd restriction ?

Jeremy.