From mboxrd@z Thu Jan 1 00:00:00 1970 From: Jan Kara Subject: Re: [PATCH] vfs: Avoid IPI storm due to bh LRU invalidation Date: Tue, 7 Feb 2012 19:29:26 +0100 Message-ID: <20120207182926.GD1043@quack.suse.cz> References: <1328536531-19034-1-git-send-email-jack@suse.cz> <4F2FF4EC.1000104@linux.vnet.ibm.com> <20120206164732.GH6890@quack.suse.cz> <20120206131717.c4346f72.akpm@linux-foundation.org> <20120206222517.GD24840@quack.suse.cz> Mime-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Transfer-Encoding: QUOTED-PRINTABLE Cc: Jan Kara , Andrew Morton , "Srivatsa S. Bhat" , linux-fsdevel@vger.kernel.org, LKML , hare@suse.de, Al Viro , Christoph Hellwig To: Gilad Ben-Yossef Return-path: Content-Disposition: inline In-Reply-To: Sender: linux-kernel-owner@vger.kernel.org List-Id: linux-fsdevel.vger.kernel.org On Tue 07-02-12 18:25:18, Gilad Ben-Yossef wrote: > On Tue, Feb 7, 2012 at 12:25 AM, Jan Kara wrote: > > On Mon 06-02-12 13:17:17, Andrew Morton wrote: > >> On Mon, 6 Feb 2012 17:47:32 +0100 > >> Jan Kara wrote: > >> > >> > On Mon 06-02-12 21:12:36, Srivatsa S. Bhat wrote: > >> > > On 02/06/2012 07:25 PM, Jan Kara wrote: > >> > > > >> > > > When discovery of lots of disks happen in parallel, we call > >> > > > invalidate_bh_lrus() once for each disk from partitioning co= de resulting in a > >> > > > storm of IPIs and causing a softlockup detection to fire (it= takes several > >> > > > *minutes* for a machine to execute all the invalidate_bh_lru= s() calls). > >> > >> Gad. =A0How many disks are we talking about here? > > =A0I think something around hundred scsi disks in this case (number= of > > physical drives is actually lower but multipathing blows it up). I = actually > > saw machines with close to thousand scsi disks (yes, they had names= like > > sdabc ;). >=20 > LOL. Is that a huge SCSI disk array in your server or your are just > happy to see me... ? :-) > > > ... > >> > > > >> > > Something related that you might be interested in: > >> > > https://lkml.org/lkml/2012/2/5/109 > >> > > > >> > > (This is part of Gilad's patchset that tries to reduce cross-C= PU IPI > >> > > interference.) > >> > =A0 Thanks for the pointer. I didn't know about it. As Hannes wr= ote, this > >> > need not be enough for our use case as there might indeed be som= e bhs in > >> > the LRU. But I'd be interested how well the patchset works anywa= y. Maybe it > >> > would be enough because after all when we invalidate LRUs subseq= uent > >> > callers will see them empty and not issue IPI? Hannes, can you g= ive a try > >> > to the patches? >=20 > I think its worth a shot since the mutex just delays the IPIs instead > of canceling them > altogether. Well, mutex will just delay callers but the sequence logic behind the mutex will reduce number of IPIs a lot - all waiters for mutex will be satisfied by a single signalling of all CPUs while previously they woul= d each do the signalling. > A somewhat similar issue in the direct reclaim path of the buddy > allocator trying to reclaim per cpu pages was causing a massive storm= of > IPIs during OOM with concurrent work loads and the IPI noise patches > mitigate 85% of the IPIs sent just by checking to see if there are an= y > per cpu pages on the CPU you are about to IPI, so maybe the same kind= of > logic applies here as well. Honza --=20 Jan Kara SUSE Labs, CR