From: Kent Overstreet <koverstreet@google.com>
To: Zach Brown <zab@redhat.com>
Cc: linux-kernel@vger.kernel.org, linux-aio@kvack.org,
linux-fsdevel@vger.kernel.org, bcrl@kvack.org, jmoyer@redhat.com,
axboe@kernel.dk, viro@zeniv.linux.org.uk
Subject: Re: [PATCH 00/25] AIO performance improvements/cleanups
Date: Thu, 29 Nov 2012 11:01:33 -0800 [thread overview]
Message-ID: <20121129190133.GF15094@google.com> (raw)
In-Reply-To: <20121129000303.GF18574@lenny.home.zabbo.net>
On Wed, Nov 28, 2012 at 04:03:03PM -0800, Zach Brown wrote:
> On Wed, Nov 28, 2012 at 08:43:24AM -0800, Kent Overstreet wrote:
> > Bunch of performance improvements and cleanups Zach Brown and I have
> > been working on. The code should be pretty solid at this point, though
> > it could of course use more review and testing.
>
> Thanks for sending these out. I have some initial review comments
> that'll follow, but I'm running out of steam today. I'll continue
> tomorrow.
>
> > The results in my testing are pretty impressive, particularly when an
> > ioctx is being shared between multiple threads. In my crappy synthetic
> > benchmark, with 4 threads submitting and one thread reaping completions,
> > I saw overhead in the aio code go from ~50% (mostly ioctx lock
> > contention) to low single digits. Performance with ioctx per thread
> > improved too, but I'd have to rerun those benchmarks.
>
> You should probably mention that those four threads were *spinning* on
> io_submit() :). I'm still guessing that this unreasonably inflated the
> contention amongst submitters and that without this inflation we might
> not find the per-cpu ioctx refcounts worth the trouble.
Yeah, should've mentioned that :) It was intentionally a worst case
scenario for aio.
> > Performance wise, the end result of this patch series is that submitting
> > a kiocb writes to _no_ shared cachelines - the penalty for sharing an
> > ioctx is gone there. There's still going to be some cacheline contention
> > when we deliver the completions to the aio ringbuffer (at least if you
> > have interrupts being delivered on multiple cores, which for high end
> > stuff you do) but I have a couple more patches not in this series that
> > implement coalescing for that (by taking advantage of interrupt
> > coalescing). With that, there's basically no bottlenecks or performance
> > issues to speak of in the aio code.
>
> Yeah, this is good stuff. Thanks for pushing it.
>
> We should mention Jens' omnibus patch that also took on these problems:
>
> http://git.kernel.dk/?p=linux-block.git;a=commit;h=6b6723fc3e4f24dbd80526df935ca115ead578c6
Oh yeah. I think this patch series solves everything Jens was working on
in the aio code, but there's still dio stuff in that patch that's worth
looking at.
--
To unsubscribe, send a message with 'unsubscribe linux-aio' in
the body to majordomo@kvack.org. For more info on Linux AIO,
see: http://www.kvack.org/aio/
Don't email: <a href=mailto:"aart@kvack.org">aart@kvack.org</a>
next prev parent reply other threads:[~2012-11-29 19:01 UTC|newest]
Thread overview: 53+ messages / expand[flat|nested] mbox.gz Atom feed top
2012-11-28 16:43 [PATCH 00/25] AIO performance improvements/cleanups Kent Overstreet
2012-11-28 16:43 ` [PATCH 01/25] mm: remove old aio use_mm() comment Kent Overstreet
2012-11-28 16:43 ` [PATCH 02/25] aio: remove dead code from aio.h Kent Overstreet
2012-11-28 16:43 ` [PATCH 03/25] gadget: remove only user of aio retry Kent Overstreet
2012-11-28 16:43 ` [PATCH 04/25] aio: remove retry-based AIO Kent Overstreet
2012-11-28 16:43 ` [PATCH 05/25] char: add aio_{read,write} to /dev/{null,zero} Kent Overstreet
2012-11-28 16:43 ` [PATCH 06/25] aio: Kill return value of aio_complete() Kent Overstreet
2012-11-28 16:43 ` [PATCH 07/25] aio: kiocb_cancel() Kent Overstreet
2012-11-29 0:07 ` Zach Brown
2012-11-29 0:58 ` Kent Overstreet
2012-11-28 16:43 ` [PATCH 08/25] aio: Move private stuff out of aio.h Kent Overstreet
2012-11-28 16:43 ` [PATCH 09/25] aio: dprintk() -> pr_debug() Kent Overstreet
2012-11-28 16:43 ` [PATCH 10/25] aio: do fget() after aio_get_req() Kent Overstreet
2012-11-28 16:43 ` [PATCH 11/25] aio: Make aio_put_req() lockless Kent Overstreet
2012-11-28 16:43 ` [PATCH 12/25] aio: Refcounting cleanup Kent Overstreet
2012-11-29 0:17 ` Zach Brown
2012-11-29 1:12 ` Kent Overstreet
2012-11-29 0:46 ` Benjamin LaHaise
2012-11-29 1:38 ` Kent Overstreet
2012-11-28 16:43 ` [PATCH 13/25] aio: Convert read_events() to hrtimers Kent Overstreet
2012-11-29 0:24 ` Zach Brown
2012-11-29 1:05 ` Kent Overstreet
2012-11-28 16:43 ` [PATCH 14/25] aio: Make aio_read_evt() more efficient Kent Overstreet
2012-11-29 0:38 ` Zach Brown
2012-11-29 19:31 ` Kent Overstreet
2012-11-30 0:20 ` Kent Overstreet
2012-11-28 16:43 ` [PATCH 15/25] aio: Use cancellation list lazily Kent Overstreet
2012-11-28 16:43 ` [PATCH 16/25] aio: Change reqs_active to include unreaped completions Kent Overstreet
2012-11-28 16:43 ` [PATCH 17/25] aio: Kill batch allocation Kent Overstreet
2012-11-28 16:43 ` [PATCH 18/25] aio: Kill struct aio_ring_info Kent Overstreet
2012-11-28 16:43 ` [PATCH 19/25] aio: Give shared kioctx fields their own cachelines Kent Overstreet
2012-11-28 16:43 ` [PATCH 20/25] aio: reqs_active -> reqs_available Kent Overstreet
2012-11-28 16:43 ` [PATCH 21/25] aio: percpu reqs_available Kent Overstreet
2012-11-28 16:43 ` [PATCH 22/25] Generic dynamic per cpu refcounting Kent Overstreet
2012-11-29 18:45 ` Andi Kleen
2012-11-29 18:57 ` Kent Overstreet
2012-11-29 18:59 ` Andi Kleen
2012-11-29 19:12 ` Kent Overstreet
2012-11-29 19:20 ` Andi Kleen
2012-11-29 19:29 ` Kent Overstreet
2012-11-29 19:34 ` Benjamin LaHaise
2012-11-29 20:22 ` Kent Overstreet
2012-11-29 20:42 ` Andi Kleen
2012-11-29 20:45 ` Kent Overstreet
2012-11-29 20:54 ` Andi Kleen
2012-11-29 20:59 ` Kent Overstreet
2012-11-29 21:57 ` Jamie Lokier
2012-11-28 16:43 ` [PATCH 23/25] aio: Percpu ioctx refcount Kent Overstreet
2012-11-28 16:43 ` [PATCH 24/25] aio: use xchg() instead of completion_lock Kent Overstreet
2012-11-28 16:43 ` [PATCH 25/25] aio: Don't include aio.h in sched.h Kent Overstreet
2012-11-29 0:03 ` [PATCH 00/25] AIO performance improvements/cleanups Zach Brown
2012-11-29 19:01 ` Kent Overstreet [this message]
-- strict thread matches above, loose matches on Subject: below --
2012-11-28 3:19 Kent Overstreet
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20121129190133.GF15094@google.com \
--to=koverstreet@google.com \
--cc=axboe@kernel.dk \
--cc=bcrl@kvack.org \
--cc=jmoyer@redhat.com \
--cc=linux-aio@kvack.org \
--cc=linux-fsdevel@vger.kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=viro@zeniv.linux.org.uk \
--cc=zab@redhat.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).