Re: async I/O seems to be blocking on 2.6.15

[Jens Axboe <jens.axboe@oracle.com>]

On Mon, Nov 06 2006, Brent Baccala wrote:
> On Mon, 6 Nov 2006, Chen, Kenneth W wrote:
> 
> >I've tried that myself too and see similar result.  One thing to note is
> >that I/O being submitted are pretty big at 1MB, so the vector list inside
> >bio is going to be pretty long and it will take a while to construct that.
> >Drop the size for each I/O to something like 4KB will significantly reduce
> >the time.  I haven't done the measurement whether the time to submit I/O
> >grows linearly with respect to I/O size.  Most likely it will.  If it is
> >not, then we might have a scaling problem (though I don't believe we have
> >this problem).
> >
> >- Ken
> >
> >
> 
> I'm basically an end user here (as far as the kernel is concerned), so
> let me ask the basic "dumb user" question here:
> 
> How should I do my async I/O if I just want to read or write
> sequentially through a file, using O_DIRECT, and letting the CPU get
> some work done in the meantime?  What about more random access?

For sequential io, you'll pretty quickly hit the transfer size sweet
spot where growing the io larger wont yield any benefits. In this case
you don't want a huge queue depth, 100 ios is an insane amount for
sequential io. If you have a properly sized io unit and a depth of 4 or
so, I doubt you'll see any improvement beyond that on most hardware.

For random io, you want a bigger queue depth. If the hardware can do
queueing, you want to make sure that you can fill that queue. 100 ios is
still a lot in this case, though.

> I've already concluded that I should try to keep my read and write
> files on seperate disks and hopefully on seperate controllers, but I
> still seem to be fighting this thing to keep it from blocking.

Shrink your queue size and/or io size :-)

-- 
Jens Axboe