From: "Paul E. McKenney" <paulmck@linux.vnet.ibm.com>
To: Jim Schutt <jaschut@sandia.gov>
Cc: linux-kernel@vger.kernel.org
Subject: Re: mcelog stalls on 2.6.39-rc5
Date: Thu, 28 Apr 2011 17:31:10 -0700 [thread overview]
Message-ID: <20110429003110.GC2191@linux.vnet.ibm.com> (raw)
In-Reply-To: <4DB990A1.8090205@sandia.gov>
On Thu, Apr 28, 2011 at 10:06:57AM -0600, Jim Schutt wrote:
> Jim Schutt wrote:
> >Paul E. McKenney wrote:
> >>On Wed, Apr 27, 2011 at 01:59:01PM -0600, Jim Schutt wrote:
> >>>Hi,
> >>>
> >>>Testing 2.6.39-rc5 is giving me the following stall:
> >>>
> >>> [ 5767.731001] INFO: rcu_sched_state detected stall on CPU
> >>>1 (t=60001 jiffies)
> >>> [ 5767.732001] INFO: rcu_sched_state detected stalls on
> >>>CPUs/tasks: { 1} (detected by 0, t=60002 jiffies)
> >>> [ 5947.763001] INFO: rcu_sched_state detected stall on CPU
> >>>1 (t=240032 jiffies)
> >>> [ 5947.764001] INFO: rcu_sched_state detected stalls on
> >>>CPUs/tasks: { 1} (detected by 0, t=240034 jiffies)
> >>> [ 6024.489362] libceph: mon0 172.17.40.34:6789 socket closed
> >>> [ 6121.281139] INFO: task mcelog:6513 blocked for more
> >>>than 120 seconds.
> >>> [ 6121.287575] "echo 0 >
> >>>/proc/sys/kernel/hung_task_timeout_secs" disables this
> >>>message.
> >>> [ 6121.295397] ffff880177aefce8 0000000000000082
> >>>ffffffff810339b6 ffff880226d35a40
> >>> [ 6121.302840] ffff88018226c3b0 ffff88018226c3b0
> >>>0000000000011e80 ffff880226d35a40
> >>> [ 6121.310284] ffff88018226c760 ffff880177aefe80
> >>>ffff880177aefd18 ffffffff813af047
> >>> [ 6121.317725] Call Trace:
> >>> [ 6121.320176] [<ffffffff810339b6>] ?
> >>>calc_load_account_idle+0xe/0x1d
> >>> [ 6121.326437] [<ffffffff813af047>] schedule+0x159/0x193
> >>> [ 6121.331569] [<ffffffff813af449>] schedule_timeout+0x36/0xe2
> >>> [ 6121.337223] [<ffffffff810ad9eb>] ? trace_hardirqs_on+0x9/0x20
> >>> [ 6121.343047] [<ffffffff813aee2b>] do_wait_for_common+0x97/0xe3
> >>> [ 6121.348967] [<ffffffff8103e8fa>] ? try_to_wake_up+0x200/0x200
> >>> [ 6121.354794] [<ffffffff8107a4bf>] ? __raw_spin_lock_irq+0x17/0x2f
> >>> [ 6121.360878] [<ffffffff813af2a9>] wait_for_common+0x36/0x4d
> >>> [ 6121.366441] [<ffffffff813af378>] wait_for_completion+0x1d/0x1f
> >>> [ 6121.372356] [<ffffffff8109ae6d>] synchronize_sched+0x40/0x49
> >>> [ 6121.378096] [<ffffffff810635b8>] ? find_get_pid+0x1b/0x1b
> >>> [ 6121.383574] [<ffffffff81015fe5>] mce_read+0x17f/0x25d
> >>> [ 6121.388707] [<ffffffff81111af5>] ? rw_verify_area+0xac/0xdb
> >>> [ 6121.394358] [<ffffffff811121f1>] vfs_read+0xa9/0xe1
> >>> [ 6121.399317] [<ffffffff8111248d>] sys_read+0x4c/0x70
> >>> [ 6121.404278] [<ffffffff813b6deb>] system_call_fastpath+0x16/0x1b
> >>> [ 6127.795001] INFO: rcu_sched_state detected stall on CPU
> >>>1 (t=420064 jiffies)
> >>> [ 6127.796001] INFO: rcu_sched_state detected stalls on
> >>>CPUs/tasks: { 1} (detected by 0, t=420066 jiffies)
> >>> [ 6241.410171] INFO: task mcelog:6513 blocked for more
> >>>than 120 seconds.
> >>>
> >>>Reverting commit a4dd99250dc makes the stalls go away:
> >>>
> >>> rcu: create new rcu_access_index() and use in mce
> >>>
> >>> The MCE subsystem needs to sample an RCU-protected index outside of
> >>> any protection for that index. If this was a pointer, we would use
> >>> rcu_access_pointer(), but there is no corresponding
> >>>rcu_access_index().
> >>> This commit therefore creates an rcu_access_index() and applies it
> >>> to MCE.
> >>>
> >>> Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
> >>> Tested-by: Zdenek Kabelac <zkabelac@redhat.com>
> >>
> >>Wow! This is just instructions, just wrapping the access in
> >>ACCESS_ONCE().
> >>
> >>Was the original RCU CPU stall repeatable?
> >
> >Yes. I saw it on two different machines.
> >Both were running as Ceph clients, doing sustained
> >streaming writes, if that helps any.
> >
> >I can attempt to repeat with any extra debugging
> >you'd like me to try.
>
> Evidently I cannot reproduce this after all, so my revert
> test was incorrect.
>
> If I can learn how to reproduce this I'll post with details;
> in the meantime ignore this, and sorry for the noise.
OK -- I must confess that I could not understand how this change
could cause the problem! ;-)
Thanx, Paul
prev parent reply other threads:[~2011-04-29 0:31 UTC|newest]
Thread overview: 5+ messages / expand[flat|nested] mbox.gz Atom feed top
2011-04-27 19:59 mcelog stalls on 2.6.39-rc5 Jim Schutt
2011-04-27 23:03 ` Paul E. McKenney
2011-04-28 14:06 ` Jim Schutt
2011-04-28 16:06 ` Jim Schutt
2011-04-29 0:31 ` Paul E. McKenney [this message]
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20110429003110.GC2191@linux.vnet.ibm.com \
--to=paulmck@linux.vnet.ibm.com \
--cc=jaschut@sandia.gov \
--cc=linux-kernel@vger.kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox