From: Dave Jones <davej@redhat.com>
To: "Paul E. McKenney" <paulmck@linux.vnet.ibm.com>
Cc: Linux Kernel <linux-kernel@vger.kernel.org>
Subject: Re: rcu_prempt stalls / lockup
Date: Tue, 1 Apr 2014 18:16:16 -0400 [thread overview]
Message-ID: <20140401221616.GA961@redhat.com> (raw)
In-Reply-To: <20140401183245.GA12473@linux.vnet.ibm.com>
On Tue, Apr 01, 2014 at 11:32:45AM -0700, Paul E. McKenney wrote:
> > > Given that you have CONFIG_RCU_TRACE=y, could you please enable the
> > > following trace events and dump the trace before things hang?
> > >
> > > trace_event=rcu:rcu_grace_period,rcu:rcu_grace_period_init
> > >
> > > If it is not feasible to dump the trace before things hang, let me
> > > know, and I will work out some other diagnostic regime.
> >
> > I'll give that a shot when I get back in a few hours.
ok, this is tricky, and I'm not sure how helpful the below is.
because I don't know when the hang is going to happen, in parallel to my usual workload
I did this..
while [ 1 ];
do
cat trace
echo > trace
done
and got this...
# entries-in-buffer/entries-written: 7/7 #P:4
#
# _-----=> irqs-off
# / _----=> need-resched
# | / _---=> hardirq/softirq
# || / _--=> preempt-depth
# ||| / delay
# TASK-PID CPU# |||| TIMESTAMP FUNCTION
# | | | |||| | |
<idle>-0 [000] dNs3 851.748475: rcu_grace_period: rcu_preempt 21342 cpuend
<idle>-0 [000] dNs3 851.748477: rcu_grace_period: rcu_preempt 21343 cpustart
<idle>-0 [000] dNs3 851.748477: rcu_grace_period: rcu_sched 20080 cpuend
<idle>-0 [000] dN.2 851.748480: rcu_grace_period: rcu_preempt 21343 cpuqs
rcu_preempt-9 [000] ...1 851.748485: rcu_grace_period: rcu_preempt 21343 fqsstart
rcu_preempt-9 [000] ...1 851.748487: rcu_grace_period: rcu_preempt 21343 fqsend
rcu_preempt-9 [000] ...1 851.748487: rcu_grace_period: rcu_preempt 21343 fqswait
# tracer: nop
#
# entries-in-buffer/entries-written: 13/13 #P:4
#
# _-----=> irqs-off
# / _----=> need-resched
# | / _---=> hardirq/softirq
# || / _--=> preempt-depth
# ||| / delay
# TASK-PID CPU# |||| TIMESTAMP FUNCTION
# | | | |||| | |
rcu_preempt-9 [003] d..2 851.878457: rcu_grace_period: rcu_preempt 21349 end
rcu_preempt-9 [003] d..2 851.878459: rcu_grace_period: rcu_preempt 21349 newreq
rcu_preempt-9 [003] ...1 851.878459: rcu_grace_period: rcu_preempt 21349 reqwait
rcu_preempt-9 [003] d..2 851.878460: rcu_grace_period: rcu_preempt 21350 start
rcu_preempt-9 [003] d..2 851.878461: rcu_grace_period: rcu_preempt 21350 cpustart
rcu_preempt-9 [003] d..2 851.878462: rcu_grace_period_init: rcu_preempt 21350 0 0 7 f
rcu_preempt-9 [003] ...1 851.878463: rcu_grace_period: rcu_preempt 21350 fqswait
rcu_preempt-9 [003] d..2 851.878464: rcu_grace_period: rcu_preempt 21350 cpuqs
<idle>-0 [000] dNs3 851.878482: rcu_grace_period: rcu_preempt 21349 cpuend
<idle>-0 [000] dNs3 851.878483: rcu_grace_period: rcu_preempt 21350 cpustart
<idle>-0 [000] dNs3 851.878484: rcu_grace_period: rcu_sched 20086 cpuend
<idle>-0 [000] .N.2 851.878486: rcu_grace_period: rcu_sched 20086 cpuqs
<idle>-0 [000] dN.2 851.878487: rcu_grace_period: rcu_preempt 21350 cpuqs
followed by a half dozen 'empty' traces before it totally locked up.
Dave
next prev parent reply other threads:[~2014-04-01 22:16 UTC|newest]
Thread overview: 23+ messages / expand[flat|nested] mbox.gz Atom feed top
2014-03-31 23:02 rcu_prempt stalls / lockup Dave Jones
2014-03-31 23:22 ` Paul E. McKenney
2014-03-31 23:35 ` Dave Jones
2014-04-01 0:48 ` Paul E. McKenney
2014-04-01 15:08 ` Dave Jones
2014-04-01 15:30 ` Paul E. McKenney
2014-04-01 17:22 ` Dave Jones
2014-04-01 17:55 ` Paul E. McKenney
2014-04-01 18:04 ` Dave Jones
2014-04-01 18:32 ` Paul E. McKenney
2014-04-01 22:16 ` Dave Jones [this message]
2014-04-01 23:18 ` Paul E. McKenney
2014-04-01 23:31 ` Dave Jones
2014-04-01 23:57 ` Paul E. McKenney
2014-04-02 0:07 ` Dave Jones
2014-04-02 16:20 ` Paul E. McKenney
2014-04-02 16:23 ` Dave Jones
2014-04-02 22:48 ` Dave Jones
2014-04-03 20:01 ` Dave Jones
2014-04-03 20:46 ` Paul E. McKenney
2014-04-03 21:44 ` Dave Jones
2014-04-03 22:37 ` Dave Jones
2014-04-04 17:06 ` Paul E. McKenney
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20140401221616.GA961@redhat.com \
--to=davej@redhat.com \
--cc=linux-kernel@vger.kernel.org \
--cc=paulmck@linux.vnet.ibm.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.