Re: [PATCH] AIO: Don't plug the I/O queue in do_io_submit()

[Chris Mason <chris.mason@oracle.com>]

On Tue, Dec 13, 2011 at 05:26:07PM -0600, Dave Kleikamp wrote:
> On 12/13/2011 04:18 PM, Jeff Moyer wrote:
> > Dave Kleikamp <dave.kleikamp@oracle.com> writes:
> > 
> >> Asynchronous I/O latency to a solid-state disk greatly increased
> >> between the 2.6.32 and 3.0 kernels. By removing the plug from
> >> do_io_submit(), we observed a 34% improvement in the I/O latency.
> >>
> >> Unfortunately, at this level, we don't know if the request is to
> >> a rotating disk or not.
> > 
> > I'm guessing I know the answer to this, but what workload were you
> > testing, and can you provide more concrete evidence than "latency
> > greatly increased?"
> 
> It is a piece of a larger industry-standard benchmark and you're
> probably guessing correctly. The "greatly increased" latency was
> actually slightly higher the improvement I get with this patch. So the
> patch brought the latency nearly down to where it was before.
> 
>  I will try a microbenchmark to see if I get similar behavior, but I
> wanted to throw this out here to get input.

The better IO latency did bump the overall benchmark score by 3%, and it
did end up bringing our latencies on par with solaris runs on similar
hardware.

We didn't find this one through exhaustive tracing...instead we used a more
traditional approach involving a list of Jens' commits and a dart board.
So, we don't have a lot of data yet on exactly why the plug is hurting.

But, I'm starting to wonder if the plug makes sense here at all.  We're
queueing up IO in the main submit loop, and the aio submit might be
spanning any number of devices on a large variety of filesystems.  The
actual direct IO call may be pretty expensive.

My guess for why this helps is contention on the aio context lock
between the submission code and the end_io softirq code.  We bash on
that lock a number of times during the IO submit, and the whole time
we're holding on to our list of plugged IOs instead of giving the
hardware the chance to process them.

-chris