From mboxrd@z Thu Jan 1 00:00:00 1970 From: "Paul E. McKenney" Subject: Re: 2.6.39-rc4+: Kernel leaking memory during FS scanning, regression? Date: Tue, 26 Apr 2011 04:27:56 -0700 Message-ID: <20110426112756.GF4308@linux.vnet.ibm.com> References: <20110425180450.1ede0845@neptune.home> <20110425190032.7904c95d@neptune.home> <20110425203606.4e78246c@neptune.home> <20110425191607.GL2468@linux.vnet.ibm.com> <20110425231016.34b4293e@neptune.home> <20110425214933.GO2468@linux.vnet.ibm.com> <20110426081904.0d2b1494@pluto.restena.lu> Reply-To: paulmck@linux.vnet.ibm.com Mime-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Transfer-Encoding: quoted-printable Cc: Mike Frysinger , Linus Torvalds , KOSAKI Motohiro , linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-fsdevel@vger.kernel.org, "Paul E. McKenney" , Pekka Enberg To: Bruno =?iso-8859-1?Q?Pr=E9mont?= Return-path: Content-Disposition: inline In-Reply-To: <20110426081904.0d2b1494@pluto.restena.lu> Sender: owner-linux-mm@kvack.org List-Id: linux-fsdevel.vger.kernel.org On Tue, Apr 26, 2011 at 08:19:04AM +0200, Bruno Pr=E9mont wrote: > On Mon, 25 Apr 2011 14:49:33 "Paul E. McKenney" wrote: > > On Mon, Apr 25, 2011 at 02:30:02PM -0700, Linus Torvalds wrote: > > > 2011/4/25 Bruno Pr=E9mont : > > > > > > > > Between 1-slabinfo and 2-slabinfo some values increased (a lot) w= hile a few > > > > ones did decrease. Don't know which ones are RCU-affected and whi= ch ones are > > > > not. > > >=20 > > > It really sounds as if the tiny-rcu kthread somehow just stops > > > handling callbacks. The ones that keep increasing do seem to be all > > > rcu-free'd (but I didn't really check). > > >=20 > > > The thing is shown as running: > > >=20 > > > root 6 0.0 0.0 0 0 ? R 22:14 0:00 \= _ > > > [rcu_kthread] > > >=20 > > > but nothing seems to happen and the CPU time hasn't increased at al= l. > > >=20 > > > I dunno. Makes no sense to me, but yeah, I'm definitely blaming > > > tiny-rcu. Paul, any ideas? > >=20 > > So the only ways I know for something to be runnable but not run on > > a uniprocessor are: > >=20 > > 1. The CPU is continually busy with higher-priority work. > > This doesn't make sense in this case because the system > > is idle much of the time. > >=20 > > 2. The system is hibernating. This doesn't make sense, otherwise > > "ps" wouldn't run either. > >=20 > > Any others ideas on how the heck a process can get into this state? > > (I have thus far been completely unable to reproduce it.) > >=20 > > The process in question has a loop in rcu_kthread() in kernel/rcutiny= .c. > > This loop contains a wait_event_interruptible(), waits for a global f= lag > > to become non-zero. > >=20 > > It is awakened by invoke_rcu_kthread() in that same file, which > > simply sets the flag to 1 and does a wake_up(), all with hardirqs > > disabled. > >=20 > > Hmmm... One "hail mary" patch below. What it does is make rcu_kthre= ad > > run at normal priority rather than at real-time priority. This is > > not for inclusion -- it breaks RCU priority boosting. But well worth > > trying. > >=20 > > Thanx, Paul > >=20 > > ---------------------------------------------------------------------= --- > >=20 > > diff --git a/kernel/rcutiny.c b/kernel/rcutiny.c > > index 0c343b9..4551824 100644 > > --- a/kernel/rcutiny.c > > +++ b/kernel/rcutiny.c > > @@ -314,11 +314,15 @@ EXPORT_SYMBOL_GPL(rcu_barrier_sched); > > */ > > static int __init rcu_spawn_kthreads(void) > > { > > +#if 0 > > struct sched_param sp; > > +#endif > > =20 > > rcu_kthread_task =3D kthread_run(rcu_kthread, NULL, "rcu_kthread"); > > +#if 0 > > sp.sched_priority =3D RCU_BOOST_PRIO; > > sched_setscheduler_nocheck(rcu_kthread_task, SCHED_FIFO, &sp); > > +#endif > > return 0; > > } > > early_initcall(rcu_spawn_kthreads); >=20 > I will give that patch a shot on Wednesday evening (European time) as I > wont have enough time in front of the affected box until then to do any > deeper testing. (same for trying to out with the other -rc kernels as > suggested by Mike) Thank you for both of these!!! > Though I will use the few minutes I have this evening to try to fetch > kernel traces of running tasks with sysrq+t which may eventually give > us a hint at where rcu_thread is stuck/waiting. This would be very helpful to me! For my part, I will use some plane time today to stare at my code some more and see what bugs I can find. Linus, in the meantime, please feel free to revert 687d7a960 (rcu: restrict TREE_RCU to SMP builds with !PREEMPT), which would allow anyone not wanting to help chase this down to get on with their lives. Thanx, Paul -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Fight unfair telecom internet charges in Canada: sign http://stopthemeter= .ca/ Don't email: email@kvack.org