Re: [PATCH] xfs: reduce lock traffic on incore sb lock

From: Dave Chinner <david@fromorbit.com>
To: Christoph Hellwig <hch@infradead.org>
Cc: xfs@oss.sgi.com
Subject: Re: [PATCH] xfs: reduce lock traffic on incore sb lock
Date: Wed, 29 Sep 2010 15:57:48 +1000	[thread overview]
Message-ID: <20100929055748.GK5665@dastard> (raw)
In-Reply-To: <20100929040425.GA29691@infradead.org>

On Wed, Sep 29, 2010 at 12:04:25AM -0400, Christoph Hellwig wrote:
> On Wed, Sep 29, 2010 at 10:51:40AM +1000, Dave Chinner wrote:
> > From: Dave Chinner <dchinner@redhat.com>
> > 
> > Under heavy parallel unlink workloads, the incore superblock lock is
> > heavily trafficed in xfs_mod_incore_sb_batch(). This is despite the
> > fact that the counters being modified are typically the counters
> > that are per-cpu and do not require the lock. IOWs, we are locking
> > and unlocking the superblock lock needlessly, and the result is that
> > it is third most heavily contended lock in the system under these
> > workloads.
> > 
> > Fix this by only locking the superblock lock when we are modifying a
> > counter protected by it. This completely removes the m_sb_lock from
> > lock_stat traces during create/remove workloads.
> 
> God spot of the idiocy there, but I really don't like the patch.

Fair enough - it is a rather quick hack. ;)

> I've started writing a small patches series solving the issue slightly
> better by cleaning up this area a bit.

Oh, cool. That code is quite a tangle.

> After this we will never use xfs_mod_incore_sb/xfs_mod_incore_sb_batch
> for the percpu counters but rather make those always go through
> xfs_icsb_modify_counters.  I'll need to quickly finish it up and will
> send it out soon.

FWIW, I've got a prototype that converts the per-cpu counters to the
generic per-cpu counter infrastructure. It chops out almost all the
xfs_icsb_* stuff (including xfs_icsb_modify_counters()) and has a
diffstat of:

6 files changed, 317 insertions(+), 709 deletions(-)

It needs a significant cleanup of xfs_mod_incore_sb() before/after
the conversion which I haven't done yet because I haven't quite got
my new percpu_counter_test_and_add_delta() function working
correctly yet. I spotted this locking problem when testing the
patch...

That said, there's no reason why my percpu counter code needs to run
through xfs_mod_incore_sb() at all. If we have a separate path for
per-cpu counters then I can rework my code on top of that....

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs