From mboxrd@z Thu Jan 1 00:00:00 1970 From: "Paul E. McKenney" Subject: Re: 2.6.39-rc4+: Kernel leaking memory during FS scanning, regression? Date: Mon, 25 Apr 2011 11:28:40 -0700 Message-ID: <20110425182840.GK2468@linux.vnet.ibm.com> References: <20110424235928.71af51e0@neptune.home> <20110425114429.266A.A69D9226@jp.fujitsu.com> <20110425111705.786ef0c5@neptune.home> <20110425180450.1ede0845@neptune.home> <20110425190032.7904c95d@neptune.home> <20110425172914.GB2468@linux.vnet.ibm.com> Reply-To: paulmck@linux.vnet.ibm.com Mime-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Transfer-Encoding: quoted-printable Cc: Bruno =?iso-8859-1?Q?Pr=E9mont?= , Linus Torvalds , Mike Frysinger , KOSAKI Motohiro , linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-fsdevel@vger.kernel.org, "Paul E. McKenney" , Pekka Enberg To: sedat.dilek@gmail.com Return-path: Content-Disposition: inline In-Reply-To: Sender: owner-linux-mm@kvack.org List-Id: linux-fsdevel.vger.kernel.org On Mon, Apr 25, 2011 at 08:13:27PM +0200, Sedat Dilek wrote: > On Mon, Apr 25, 2011 at 7:29 PM, Paul E. McKenney > wrote: > > On Mon, Apr 25, 2011 at 07:00:32PM +0200, Bruno Pr=E9mont wrote: > >> On Mon, 25 April 2011 Linus Torvalds wrote: > >> > 2011/4/25 Bruno Pr=E9mont : > >> > > > >> > > kmemleak reports 86681 new leaks between shortly after boot and = -2 state. > >> > > (and 2348 additional ones between -2 and -4). > >> > > >> > I wouldn't necessarily trust kmemleak with the whole RCU-freeing > >> > thing. In your slubinfo reports, the kmemleak data itself also ten= ds > >> > to overwhelm everything else - none of it looks unreasonable per s= e. > >> > > >> > That said, you clearly have a *lot* of filp entries. I wouldn't > >> > consider it unreasonable, though, because depending on load those = may > >> > well be fine. Perhaps you really do have some application(s) that = hold > >> > thousands of files open. The default file limit is 1024 (I think),= but > >> > you can raise it, and some programs do end up opening tens of > >> > thousands of files for filesystem scanning purposes. > >> > > >> > That said, I would suggest simply trying a saner kernel configurat= ion, > >> > and seeing if that makes a difference: > >> > > >> > > Yes, it's uni-processor system, so SMP=3Dn. > >> > > TINY_RCU=3Dy, PREEMPT_VOLUNTARY=3Dy (whole /proc/config.gz attac= hed keeping > >> > > compression) > >> > > >> > I'm not at all certain that TINY_RCU is appropriate for > >> > general-purpose loads. I'd call it more of a "embedded low-perform= ance > >> > option". > >> > >> Well, TINY_RCU is the only option when doing PREEMPT_VOLUNTARY on > >> SMP=3Dn... > > > > You can either set SMP=3Dy and NR_CPUS=3D1 or you can handed-edit > > init/Kconfig to remove the dependency on SMP. =A0Just change the > > > > =A0 =A0 =A0 =A0depends on !PREEMPT && SMP > > > > to: > > > > =A0 =A0 =A0 =A0depends on !PREEMPT > > > > This will work fine, especially for experimental purposes. > > > >> > The _real_ RCU implementation ("tree rcu") forces quiescent states > >> > every few jiffies and has logic to handle "I've got tons of RCU > >> > events, I really need to start handling them now". All of which I > >> > think tiny-rcu lacks. > >> > >> Going to try it out (will take some time to compile), kmemleak disab= led. > >> > >> > So right now I suspect that you have a situation where you just ha= ve a > >> > simple load that just ends up never triggering any RCU cleanup, an= d > >> > the tiny-rcu thing just keeps on gathering events and delays freei= ng > >> > stuff almost arbitrarily long. > >> > >> I hope tiny-rcu is not that broken... as it would mean driving any > >> PREEMPT_NONE or PREEMPT_VOLUNTARY system out of memory when compilin= g > >> packages (and probably also just unpacking larger tarballs or runnin= g > >> things like du). > > > > If it is broken, I will fix it. =A0;-) > > > > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0= =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0Thanx, Paul > > > >> And with system doing nothing (except monitoring itself) memory usag= e > >> goes increasing all the time until it starves (well it seems to keep > >> ~20M free, pushing processes it can to swap). Config is just being > >> make oldconfig from working 2.6.38 kernel (answering default for new > >> options) > >> > >> Memory usage evolution graph in first message of this thread: > >> http://thread.gmane.org/gmane.linux.kernel.mm/61909/focus=3D1130480 > >> > >> Attached graph matching numbers of previous mail. (dropping caches w= as at > >> 17:55, system idle since then) > >> > >> Bruno > >> > >> > >> > So try CONFIG_PREEMPT and CONFIG_TREE_PREEMPT_RCU to see if the > >> > behavior goes away. That would confirm the "it's just tinyrcu bein= g > >> > too dang stupid" hypothesis. > >> > > >> > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0Linus > > > > > > -- > > To unsubscribe from this list: send the line "unsubscribe linux-fsdev= el" in > > the body of a message to majordomo@vger.kernel.org > > More majordomo info at =A0http://vger.kernel.org/majordomo-info.html > > >=20 > Hi, >=20 > I was playing with Debian's kernel-buildsystem for -rc4 with a > self-defined '686-up' so-called flavour. >=20 > Here I have a Banias Pentium-M (UP, *no* PAE) and still experimenting > with kernel-config options. >=20 > CONFIG_X86_UP_APIC=3Dy > CONFIG_X86_UP_IOAPIC=3Dy >=20 > ...is not possible with CONFIG_SMP=3Dy Right, hence my advice to hand-edit init/Kconfig for experimental purposes. Once that is done, you can select CONFIG_TREE_RCU with CONFIG_SMP=3Dn. Thanx, Paul > These settings are possible by not hacking existing Kconfigs: >=20 > $ egrep 'M486|M686|X86_UP|CONFIG_SMP|NR_CPUS|PREEMPT|_RCU|_HIGHMEM|PAE' > debian/build/build_i386_none_686-up/.config > CONFIG_TREE_PREEMPT_RCU=3Dy > # CONFIG_TINY_RCU is not set > # CONFIG_TINY_PREEMPT_RCU is not set > CONFIG_PREEMPT_RCU=3Dy > # CONFIG_RCU_TRACE is not set > CONFIG_RCU_FANOUT=3D32 > # CONFIG_RCU_FANOUT_EXACT is not set > # CONFIG_TREE_RCU_TRACE is not set > CONFIG_PREEMPT_NOTIFIERS=3Dy > # CONFIG_SMP is not set > # CONFIG_M486 is not set > CONFIG_M686=3Dy > CONFIG_NR_CPUS=3D1 > # CONFIG_PREEMPT_NONE is not set > # CONFIG_PREEMPT_VOLUNTARY is not set > CONFIG_PREEMPT=3Dy > CONFIG_X86_UP_APIC=3Dy > CONFIG_X86_UP_IOAPIC=3Dy > CONFIG_HIGHMEM4G=3Dy > # CONFIG_HIGHMEM64G is not set > CONFIG_HIGHMEM=3Dy > CONFIG_DEBUG_PREEMPT=3Dy > # CONFIG_SPARSE_RCU_POINTER is not set > # CONFIG_DEBUG_HIGHMEM is not set > # CONFIG_RCU_TORTURE_TEST is not set > # CONFIG_RCU_CPU_STALL_DETECTOR is not set > # CONFIG_PREEMPT_TRACER is not set >=20 > But I also see these warnings: >=20 > .config:2106:warning: override: TREE_PREEMPT_RCU changes choice state > .config:2182:warning: override: PREEMPT changes choice state >=20 > Not sure how to interprete them, so I am a bit careful :-). >=20 > ( Untested - not compiled yet! ) >=20 > - Sedat - -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Fight unfair telecom internet charges in Canada: sign http://stopthemeter= .ca/ Don't email: email@kvack.org