From: "Bruno Prémont" <bonbons@linux-vserver.org>
To: paulmck@linux.vnet.ibm.com
Cc: Mike Frysinger <vapier.adi@gmail.com>,
Linus Torvalds <torvalds@linux-foundation.org>,
KOSAKI Motohiro <kosaki.motohiro@jp.fujitsu.com>,
linux-kernel@vger.kernel.org, linux-mm@kvack.org,
linux-fsdevel@vger.kernel.org,
"Paul E. McKenney" <paul.mckenney@linaro.org>,
Pekka Enberg <penberg@kernel.org>
Subject: Re: 2.6.39-rc4+: Kernel leaking memory during FS scanning, regression?
Date: Tue, 26 Apr 2011 18:38:59 +0200 [thread overview]
Message-ID: <20110426183859.6ff6279b@neptune.home> (raw)
In-Reply-To: <20110426112756.GF4308@linux.vnet.ibm.com>
[-- Attachment #1: Type: text/plain, Size: 3683 bytes --]
On Tue, 26 April 2011 "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> wrote:
> On Tue, Apr 26, 2011 at 08:19:04AM +0200, Bruno Prémont wrote:
> > Though I will use the few minutes I have this evening to try to fetch
> > kernel traces of running tasks with sysrq+t which may eventually give
> > us a hint at where rcu_thread is stuck/waiting.
>
> This would be very helpful to me!
Here it comes:
rcu_kthread (when build processes are STOPped):
[ 836.050003] rcu_kthread R running 7324 6 2 0x00000000
[ 836.050003] dd473f28 00000046 5a000240 dd65207c dd407360 dd651d40 0000035c dd473ed8
[ 836.050003] c10bf8a2 c14d63d8 dd65207c dd473f28 dd445040 dd445040 dd473eec c10be848
[ 836.050003] dd651d40 dd407360 ddfdca00 dd473f14 c10bfde2 00000000 00000001 000007b6
[ 836.050003] Call Trace:
[ 836.050003] [<c10bf8a2>] ? check_object+0x92/0x210
[ 836.050003] [<c10be848>] ? init_object+0x38/0x70
[ 836.050003] [<c10bfde2>] ? free_debug_processing+0x112/0x1f0
[ 836.050003] [<c103d9fd>] ? lock_timer_base+0x2d/0x70
[ 836.050003] [<c13c8ec7>] schedule_timeout+0x137/0x280
[ 836.050003] [<c10c02b8>] ? kmem_cache_free+0xe8/0x140
[ 836.050003] [<c103db60>] ? sys_gettid+0x20/0x20
[ 836.050003] [<c13c9064>] schedule_timeout_interruptible+0x14/0x20
[ 836.050003] [<c10736e0>] rcu_kthread+0xa0/0xc0
[ 836.050003] [<c104de00>] ? wake_up_bit+0x70/0x70
[ 836.050003] [<c1073640>] ? rcu_process_callbacks+0x60/0x60
[ 836.050003] [<c104d874>] kthread+0x74/0x80
[ 836.050003] [<c104d800>] ? flush_kthread_worker+0x90/0x90
[ 836.050003] [<c13caeb6>] kernel_thread_helper+0x6/0xd
a few minutes later when build processes have been killed:
[ 966.930008] rcu_kthread R running 7324 6 2 0x00000000
[ 966.930008] dd473f28 00000046 5a000240 dd65207c dd407360 dd651d40 0000035c dd473ed8
[ 966.930008] c10bf8a2 c14d63d8 dd65207c dd473f28 dd445040 dd445040 dd473eec c10be848
[ 966.930008] dd651d40 dd407360 ddfdca00 dd473f14 c10bfde2 00000000 00000001 000007b6
[ 966.930008] Call Trace:
[ 966.930008] [<c10bf8a2>] ? check_object+0x92/0x210
[ 966.930008] [<c10be848>] ? init_object+0x38/0x70
[ 966.930008] [<c10bfde2>] ? free_debug_processing+0x112/0x1f0
[ 966.930008] [<c103d9fd>] ? lock_timer_base+0x2d/0x70
[ 966.930008] [<c13c8ec7>] schedule_timeout+0x137/0x280
[ 966.930008] [<c10c02b8>] ? kmem_cache_free+0xe8/0x140
[ 966.930008] [<c103db60>] ? sys_gettid+0x20/0x20
[ 966.930008] [<c13c9064>] schedule_timeout_interruptible+0x14/0x20
[ 966.930008] [<c10736e0>] rcu_kthread+0xa0/0xc0
[ 966.930008] [<c104de00>] ? wake_up_bit+0x70/0x70
[ 966.930008] [<c1073640>] ? rcu_process_callbacks+0x60/0x60
[ 966.930008] [<c104d874>] kthread+0x74/0x80
[ 966.930008] [<c104d800>] ? flush_kthread_worker+0x90/0x90
[ 966.930008] [<c13caeb6>] kernel_thread_helper+0x6/0xd
Attached (gzipped) the complete dmesg log (dmesg-t1 contains dmesg from boot until
after first sysrq+t -- dmesg-t2 the output of sysrq+t 2 minutes later
after having killed build processes).
Just in case, I joined slabinfo.
Ten minutes later rcu_kthread trace has not changed at all.
>
> For my part, I will use some plane time today to stare at my code some
> more and see what bugs I can find.
Possibly useful detail, it's somewhere during my compile that rcu_kthread
seems to stop doing its job as after booting things look fine (no slabs
piling up). Some part of the whole emerge -> configure and/or emerge -> make -> gcc
process tree must be confusing kernel as it's only then that things
start piling up (don't know if other kinds of work trigger it as well)
Bruno
[-- Attachment #2: dmesg-t1.gz --]
[-- Type: application/x-gzip, Size: 24299 bytes --]
[-- Attachment #3: dmesg-t2.gz --]
[-- Type: application/x-gzip, Size: 11534 bytes --]
[-- Attachment #4: slabinfo.gz --]
[-- Type: application/x-gzip, Size: 2283 bytes --]
next prev parent reply other threads:[~2011-04-26 16:38 UTC|newest]
Thread overview: 88+ messages / expand[flat|nested] mbox.gz Atom feed top
2011-04-24 18:21 2.6.39-rc4+: Kernel leaking memory during FS scanning, regression? Bruno Prémont
2011-04-24 21:59 ` Bruno Prémont
2011-04-25 2:42 ` KOSAKI Motohiro
2011-04-25 7:47 ` Mike Frysinger
2011-04-25 9:17 ` Bruno Prémont
2011-04-25 9:25 ` Pekka Enberg
2011-04-25 10:34 ` Bruno Prémont
2011-04-25 11:41 ` Bruno Prémont
2011-04-25 11:47 ` Pekka Enberg
2011-04-25 12:11 ` Bruno Prémont
2011-04-25 12:14 ` Tetsuo Handa
2011-04-25 12:21 ` Tetsuo Handa
2011-04-25 15:22 ` Linus Torvalds
2011-04-25 16:04 ` Bruno Prémont
2011-04-25 16:31 ` Linus Torvalds
2011-04-25 17:00 ` Bruno Prémont
2011-04-25 17:10 ` Linus Torvalds
2011-04-25 17:20 ` Linus Torvalds
2011-04-25 18:36 ` Bruno Prémont
2011-04-25 19:16 ` Paul E. McKenney
2011-04-25 21:10 ` Bruno Prémont
2011-04-25 21:26 ` Paul E. McKenney
2011-04-25 21:30 ` Linus Torvalds
2011-04-25 21:49 ` Paul E. McKenney
2011-04-26 6:19 ` Bruno Prémont
2011-04-26 11:27 ` Paul E. McKenney
2011-04-26 16:38 ` Bruno Prémont [this message]
2011-04-26 17:09 ` Bruno Prémont
2011-04-26 17:18 ` Linus Torvalds
2011-04-26 22:28 ` Thomas Gleixner
2011-04-27 6:15 ` Bruno Prémont
2011-04-27 18:41 ` Bruno Prémont
2011-04-27 19:16 ` Pádraig Brady
2011-04-27 19:34 ` Bruno Prémont
2011-04-27 22:05 ` Paul E. McKenney
2011-04-27 20:40 ` Bruno Prémont
2011-04-27 22:07 ` Paul E. McKenney
2011-04-28 6:10 ` Bruno Prémont
2011-04-27 22:06 ` Thomas Gleixner
2011-04-27 22:27 ` Paul E. McKenney
2011-04-27 22:32 ` Thomas Gleixner
2011-04-27 22:59 ` Paul E. McKenney
2011-04-27 23:28 ` Linus Torvalds
2011-04-27 23:46 ` Linus Torvalds
2011-04-28 9:09 ` Thomas Gleixner
2011-04-28 9:17 ` Sedat Dilek
2011-04-28 9:40 ` Thomas Gleixner
2011-04-28 10:12 ` Mike Galbraith
2011-04-28 9:45 ` Sedat Dilek
2011-04-28 10:26 ` Paul E. McKenney
2011-04-28 13:30 ` Mike Galbraith
2011-04-28 15:28 ` Sedat Dilek
2011-04-28 15:44 ` Sedat Dilek
2011-04-28 15:48 ` Linus Torvalds
2011-04-28 18:49 ` Thomas Gleixner
2011-04-28 20:23 ` Bruno Prémont
2011-04-28 20:29 ` Thomas Gleixner
2011-04-28 20:44 ` Bruno Prémont
2011-04-28 21:04 ` Thomas Gleixner
2011-04-28 21:51 ` john stultz
2011-04-28 22:02 ` Thomas Gleixner
2011-04-28 23:06 ` Sedat Dilek
2011-04-28 23:35 ` Sedat Dilek
2011-04-29 0:42 ` Paul E. McKenney
2011-04-29 9:34 ` Thomas Gleixner
2011-04-29 7:55 ` Sedat Dilek
2011-04-29 18:09 ` Mike Frysinger
2011-04-29 18:26 ` Thomas Gleixner
2011-04-29 19:31 ` Bruno Prémont
2011-04-29 20:10 ` Thomas Gleixner
2011-04-29 20:14 ` Bruno Prémont
2011-04-30 9:14 ` Sedat Dilek
2011-04-28 20:41 ` Sedat Dilek
2011-04-28 19:22 ` Mike Galbraith
2011-04-27 21:55 ` Paul E. McKenney
2011-04-28 6:22 ` Bruno Prémont
2011-04-28 10:26 ` Paul E. McKenney
2011-04-26 17:12 ` Linus Torvalds
2011-04-26 18:50 ` Paul E. McKenney
2011-04-26 19:17 ` Sedat Dilek
2011-04-27 22:02 ` Paul E. McKenney
2011-04-25 22:08 ` Mike Frysinger
2011-04-25 17:29 ` Paul E. McKenney
2011-04-25 18:13 ` Sedat Dilek
2011-04-25 18:28 ` Paul E. McKenney
2011-04-25 17:26 ` Paul E. McKenney
2011-04-27 10:28 ` Catalin Marinas
2011-04-25 17:51 ` Pekka Enberg
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20110426183859.6ff6279b@neptune.home \
--to=bonbons@linux-vserver.org \
--cc=kosaki.motohiro@jp.fujitsu.com \
--cc=linux-fsdevel@vger.kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-mm@kvack.org \
--cc=paul.mckenney@linaro.org \
--cc=paulmck@linux.vnet.ibm.com \
--cc=penberg@kernel.org \
--cc=torvalds@linux-foundation.org \
--cc=vapier.adi@gmail.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).