From mboxrd@z Thu Jan 1 00:00:00 1970 From: "Paul E. McKenney" Subject: Re: 2.6.39-rc4+: Kernel leaking memory during FS scanning, regression? Date: Wed, 27 Apr 2011 15:59:49 -0700 Message-ID: <20110427225949.GB2135@linux.vnet.ibm.com> References: <20110426112756.GF4308@linux.vnet.ibm.com> <20110426183859.6ff6279b@neptune.home> <20110426190918.01660ccf@neptune.home> <20110427081501.5ba28155@pluto.restena.lu> <20110427204139.1b0ea23b@neptune.home> <20110427222727.GU2135@linux.vnet.ibm.com> Reply-To: paulmck@linux.vnet.ibm.com Mime-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Transfer-Encoding: quoted-printable Cc: Bruno =?iso-8859-1?Q?Pr=E9mont?= , Linus Torvalds , Ingo Molnar , Peter Zijlstra , Mike Frysinger , KOSAKI Motohiro , linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-fsdevel@vger.kernel.org, "Paul E. McKenney" , Pekka Enberg To: Thomas Gleixner Return-path: Content-Disposition: inline In-Reply-To: Sender: owner-linux-mm@kvack.org List-Id: linux-fsdevel.vger.kernel.org On Thu, Apr 28, 2011 at 12:32:50AM +0200, Thomas Gleixner wrote: > On Wed, 27 Apr 2011, Paul E. McKenney wrote: > > On Thu, Apr 28, 2011 at 12:06:11AM +0200, Thomas Gleixner wrote: > > > On Wed, 27 Apr 2011, Bruno Pr=E9mont wrote: > > > > On Wed, 27 April 2011 Bruno Pr=E9mont wrote: > > > > Voluntary context switches stay constant from the time on SLABs p= ile up. > > > > (which makes sense as it doesn't run get CPU slices anymore) > > > >=20 > > > > > > Can you please enable CONFIG_SCHED_DEBUG and provide the outp= ut of > > > > > > /proc/sched_stat when the problem surfaces and a minute after= the > > > > > > first snapshot? > > > >=20 > > > > hm, did you mean CONFIG_SCHEDSTAT or /proc/sched_debug? > > > >=20 > > > > I did use CONFIG_SCHED_DEBUG (and there is no /proc/sched_stat) s= o I took > > > > /proc/sched_debug which exists... (attached, taken about 7min and= +1min > > > > after SLABs started piling up), though build processes were SIGST= OPped > > > > during first minute. > > >=20 > > > Oops. /proc/sched_debug is the right thing. > > >=20 > > > > printk wrote (in case its timestamp is useful, more below): > > > > [ 518.480103] sched: RT throttling activated > > >=20 > > > Ok. Aside of the fact that the CPU time accounting is completely ho= sed > > > this is pointing to the root cause of the problem. > > >=20 > > > kthread_rcu seems to run in circles for whatever reason and the RT > > > throttler catches it. After that things go down the drain completel= y > > > as it should get on the CPU again after that 50ms throttling break. > >=20 > > Ah. This could happen if there was a huge number of callbacks, in > > which case blimit would be set very large and kthread_rcu could then > > go CPU-bound. And this workload was generating large numbers of > > callbacks due to filesystem operations, right? > >=20 > > So, perhaps I should kick kthread_rcu back to SCHED_NORMAL if blimit > > has been set high. Or have some throttling of my own. I must confes= s > > that throttling kthread_rcu for two hours seems a bit harsh. ;-) >=20 > That's not the intended thing. See below. >=20 > > If this was just throttling kthread_rcu for a few hundred millisecond= s, > > or even for a second or two, things would be just fine. > >=20 > > Left to myself, I will put together a patch that puts callback proces= sing > > down to SCHED_NORMAL in the case where there are huge numbers of > > callbacks to be processed. >=20 > Well that's going to paper over the problem at hand possibly. I really > don't see why that thing would run for more than 950ms in a row even > if there is a large number of callbacks pending. True enough, it would probably take millions of callbacks to keep rcu_do_batch() busy for 950 milliseconds. Possible, but hopefully unlikely. Hmmm... If this is happening, I should see it in the debug stuff that Sedat sent me. And the biggest change I see in a 15-second interval is 50,000 RCU callbacks, which is large, but should not be problematic. Even if they all showed up at once, I would hope that they could be invoked within a few hundred milliseconds. > And then I don't have an explanation for the hosed CPU accounting and > why that thing does not get another 950ms RT time when the 50ms > throttling break is over. Would problems in the CPU accounting result in spurious throttles, or are we talking different types of accounting here? Thanx, Paul -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Fight unfair telecom internet charges in Canada: sign http://stopthemeter= .ca/ Don't email: email@kvack.org