linux io_submit syscall duration

linux io_submit syscall duration

[Raz Ben-Jehuda(caro)]

Bnejamin hello.

My name is raz and i have encountered a problem with  io_submit.
The maximum duration of a single io_submit operation when
heavily stressing the system with large number of big (1MB) ios,
reaches several hundereds ms.

I have been profiling it and it seems that the problem is the
file->f_op->aio_read operation,
a call that is made in fs/aio.c when coming from:
 sys_io_submit -->
   io_submit_one -->
	  aio_run_iocb -->
   		 *retry


The test is initiating several hundered 1MB IOs over a single block device.

I understand that the assumption made was aio_read is asynchronous and no
delay will occure, but isn't possible to do it in the workqueue context ?

would appreciate your help.
thank you

raz

Re: linux io_submit syscall duration

[Jens Axboe]

On Wed, Nov 15 2006, Raz Ben-Jehuda(caro) wrote:
> Bnejamin hello.
> 
> My name is raz and i have encountered a problem with  io_submit.
> The maximum duration of a single io_submit operation when
> heavily stressing the system with large number of big (1MB) ios,
> reaches several hundereds ms.
> 
> I have been profiling it and it seems that the problem is the
> file->f_op->aio_read operation,
> a call that is made in fs/aio.c when coming from:
> sys_io_submit -->
>   io_submit_one -->
> 	  aio_run_iocb -->
>   		 *retry
> 
> 
> The test is initiating several hundered 1MB IOs over a single block device.
> 
> I understand that the assumption made was aio_read is asynchronous and no
> delay will occure, but isn't possible to do it in the workqueue context ?

This was discussed no more than a week ago on this list, please search the
archives for the thread.

-- 
Jens Axboe

Re: linux io_submit syscall duration

[Jens Axboe]

(don't top post and don't drop the cc list!)

On Wed, Nov 15 2006, Raz Ben-Jehuda(caro) wrote:
> first thank you from your quick reply.
> 
> I looked and what I found is :
> "io_submit() taking a long time to return ?" from 2006-07-14.

Subject is "async I/O seems to be blocking on 2.6.15" and the posting is
from 2006-11-3. I said about a week ago, not 4 months.

> 1.  It is NOT the first IO. already took care for that in my test.

That was a different problem.

> 2.  The test uses block device and does not walk through a file system.

So would not apply.

> 3.  Taking a look at the code, it seems that the main problem is the fact
>     that in io_submit_one all the pending requests are being rerun.
> 
>    io_submit_one()
>   {
>    ...
>        if (!list_empty(&ctx->run_list)) {
>                /* drain the run list */
>              while (__aio_run_iocbs(ctx))
>                      ;
>      }
>   ...
>   }
> 
>  This code was put in remarks.
> the second is in:
>  aio_run_iocb()
>   ...
>    current->io_wait = &iocb->ki_wait;
>    if ( iocb->ki_retried >  1)
>        ret = retry(iocb);
>    else{
>        ret = -EIOCBRETRY;
>        kiocbSetKicked(iocb);
>    }
>     current->io_wait = NULL;
>  ....
> 
>  With these fixes io_submit delay reduced to zero ms.
> 
> I would appreciate your review on these fixes.

Post a patch so it can be reviewed.

-- 
Jens Axboe