Re: [PATCH] vfs: Avoid IPI storm due to bh LRU invalidation

[Jan Kara <jack@suse.cz>]

On Tue 07-02-12 18:25:18, Gilad Ben-Yossef wrote:
> On Tue, Feb 7, 2012 at 12:25 AM, Jan Kara <jack@suse.cz> wrote:
> > On Mon 06-02-12 13:17:17, Andrew Morton wrote:
> >> On Mon, 6 Feb 2012 17:47:32 +0100
> >> Jan Kara <jack@suse.cz> wrote:
> >>
> >> > On Mon 06-02-12 21:12:36, Srivatsa S. Bhat wrote:
> >> > > On 02/06/2012 07:25 PM, Jan Kara wrote:
> >> > >
> >> > > > When discovery of lots of disks happen in parallel, we call
> >> > > > invalidate_bh_lrus() once for each disk from partitioning code resulting in a
> >> > > > storm of IPIs and causing a softlockup detection to fire (it takes several
> >> > > > *minutes* for a machine to execute all the invalidate_bh_lrus() calls).
> >>
> >> Gad.  How many disks are we talking about here?
> >  I think something around hundred scsi disks in this case (number of
> > physical drives is actually lower but multipathing blows it up). I actually
> > saw machines with close to thousand scsi disks (yes, they had names like
> > sdabc ;).
> 
> LOL. Is that a huge SCSI disk array in your server or your are just
> happy to see me... ? :-)
> >
> ...
> >> > >
> >> > > Something related that you might be interested in:
> >> > > https://lkml.org/lkml/2012/2/5/109
> >> > >
> >> > > (This is part of Gilad's patchset that tries to reduce cross-CPU IPI
> >> > > interference.)
> >> >   Thanks for the pointer. I didn't know about it. As Hannes wrote, this
> >> > need not be enough for our use case as there might indeed be some bhs in
> >> > the LRU. But I'd be interested how well the patchset works anyway. Maybe it
> >> > would be enough because after all when we invalidate LRUs subsequent
> >> > callers will see them empty and not issue IPI? Hannes, can you give a try
> >> > to the patches?
> 
> I think its worth a shot since the mutex just delays the IPIs instead
> of canceling them
> altogether.
  Well, mutex will just delay callers but the sequence logic behind the
mutex will reduce number of IPIs a lot - all waiters for mutex will be
satisfied by a single signalling of all CPUs while previously they would
each do the signalling.

> A somewhat similar issue in the direct reclaim path of the buddy
> allocator trying to reclaim per cpu pages was causing a massive storm of
> IPIs during OOM with concurrent work loads and the IPI noise patches
> mitigate 85% of the IPIs sent just by checking to see if there are any
> per cpu pages on the CPU you are about to IPI, so maybe the same kind of
> logic applies here as well.

								Honza
-- 
Jan Kara <jack@suse.cz>
SUSE Labs, CR