From: "Paul E. McKenney" <paulmck@linux.vnet.ibm.com>
To: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Cc: "Chatre, Reinette" <reinette.chatre@intel.com>,
"Jacob Pan" <jacob.jun.pan@linux.intel.com>,
"Josh Triplett" <josh@joshtriplett.org>,
"Ross Green" <rgkernel@gmail.com>,
"John Stultz" <john.stultz@linaro.org>,
"Thomas Gleixner" <tglx@linutronix.de>,
"Peter Zijlstra" <peterz@infradead.org>,
lkml <linux-kernel@vger.kernel.org>,
"Ingo Molnar" <mingo@kernel.org>,
"Lai Jiangshan" <jiangshanlai@gmail.com>,
dipankar@in.ibm.com, "Andrew Morton" <akpm@linux-foundation.org>,
rostedt <rostedt@goodmis.org>,
"David Howells" <dhowells@redhat.com>,
"Eric Dumazet" <edumazet@google.com>,
"Darren Hart" <dvhart@linux.intel.com>,
"Frédéric Weisbecker" <fweisbec@gmail.com>,
"Oleg Nesterov" <oleg@redhat.com>,
"pranith kumar" <bobby.prani@gmail.com>
Subject: Re: rcu_preempt self-detected stall on CPU from 4.5-rc3, since 3.17
Date: Sat, 26 Mar 2016 08:28:16 -0700 [thread overview]
Message-ID: <20160326152816.GW4287@linux.vnet.ibm.com> (raw)
In-Reply-To: <1370753660.36931.1458995371427.JavaMail.zimbra@efficios.com>
On Sat, Mar 26, 2016 at 12:29:31PM +0000, Mathieu Desnoyers wrote:
> ----- On Mar 25, 2016, at 5:46 PM, Paul E. McKenney paulmck@linux.vnet.ibm.com wrote:
>
> > On Fri, Mar 25, 2016 at 09:24:14PM +0000, Chatre, Reinette wrote:
> >> Hi Paul,
> >>
> >> On 2016-03-23, Paul E. McKenney wrote:
> >> > Please boot with the following parameters:
> >> >
> >> > rcu_tree.rcu_kick_kthreads ftrace
> >> > trace_event=sched_waking,sched_wakeup,sched_wake_idle_without_ipi
> >>
> >> With these parameters I expected more details to show up in the kernel logs but
> >> cannot find any. Even so, today I left the machine running again and when this
> >> happened I think I was able to capture the trace data for the event. Please
> >> find attached the trace information for the kernel message below. Since the
> >> complete trace file is very big I trimmed it to show the time around this event
> >> - hopefully this will contain the information you need. I would also like to
> >> provide some additional information. The system on which I see these events had
> >> a time that was _very_ wrong. I noticed that this issue occurs when
> >> system-timesynd was one of the tasks calling the functions of interest to your
> >> tracing and am wondering if a very out of sync time in process of being
> >> corrected could be the cause of this issue? As an experiment I ensured the
> >> system time was accurate before leaving the system idle overnight and I did not
> >> see the issue the next morning.
> >
> > Ah! Yes, a sudden jump in time or a disagreement about the time among
> > different components of the system can definitely cause these symptoms.
> > We have sometimes seen these problems occur when a pair of CPUs have
> > wildly different ideas about what time it is, for example. Please let
> > me know how it goes.
> >
> > Also, in your trace, there are no sched_waking events for the rcu_preempt
> > process that are not immediately followed by sched_wakeup, so your trace
> > isn't showing the problem that I am seeing.
>
> This is interesting.
>
> Perhaps we could try with those commits reverted ?
>
> commit e3baac47f0e82c4be632f4f97215bb93bf16b342
> Author: Peter Zijlstra <peterz@infradead.org>
> Date: Wed Jun 4 10:31:18 2014 -0700
>
> sched/idle: Optimize try-to-wake-up IPI
>
> commit fd99f91aa007ba255aac44fe6cf21c1db398243a
> Author: Peter Zijlstra <peterz@infradead.org>
> Date: Wed Apr 9 15:35:08 2014 +0200
>
> sched/idle: Avoid spurious wakeup IPIs
>
> They appeared in 3.16.
At this point, I am up for trying pretty much anything. ;-)
Will give it a go.
Thanx, Paul
> Thanks,
>
> Mathieu
>
> >
> > Still beating up on my stress test, which is not yet proving to be all
> > that stressful. :-/
> >
> > Thanx, Paul
> >
> >> [ 957.396537] INFO: rcu_preempt detected stalls on CPUs/tasks:
> >> [ 957.399933] 1-...: (0 ticks this GP) idle=4d6/0/0 softirq=6311/6311 fqs=0
> >> [ 957.403661] (detected by 0, t=60002 jiffies, g=3583, c=3582, q=47)
> >> [ 957.407227] Task dump for CPU 1:
> >> [ 957.409964] swapper/1 R running task 0 0 1 0x00200000
> >> [ 957.413770] 0000039daa9a7eb9 ffff8801785cfed0 ffffffff818af34c
> >> ffff880100000000
> >> [ 957.417696] 0000000600000003 ffff8801785d0000 ffff880072f9ea00
> >> ffffffff822dcf80
> >> [ 957.421631] ffff8801785cc000 ffff8801785cc000 ffff8801785cfee0
> >> ffffffff818af597
> >> [ 957.425562] Call Trace:
> >> [ 957.428124] [<ffffffff818af34c>] ? cpuidle_enter_state+0xfc/0x310
> >> [ 957.431713] [<ffffffff818af597>] ? cpuidle_enter+0x17/0x20
> >> [ 957.435122] [<ffffffff811515ba>] ? call_cpuidle+0x2a/0x40
> >> [ 957.438467] [<ffffffff8115198d>] ? cpu_startup_entry+0x28d/0x360
> >> [ 957.441949] [<ffffffff8108c864>] ? start_secondary+0x114/0x140
> >> [ 957.445378] rcu_preempt kthread starved for 60002 jiffies! g3583 c3582 f0x0
> >> RCU_GP_WAIT_FQS(3) ->state=0x1
> >> [ 957.449834] rcu_preempt S ffff8801785b7d68 0 7 2 0x00000000
> >> [ 957.453579] ffff8801785b7d68 ffff88017dc8cc80 ffff88016fe6bb80
> >> ffff8801785abb80
> >> [ 957.457428] ffff8801785b8000 ffff8801785b7da0 ffff88017dc8cc80
> >> ffff88017dc8cc80
> >> [ 957.461249] 0000000000000003 ffff8801785b7d80 ffffffff81ab03df
> >> 0000000100373021
> >> [ 957.465055] Call Trace:
> >> [ 957.467493] [<ffffffff81ab03df>] schedule+0x3f/0xa0
> >> [ 957.470613] [<ffffffff81ab4397>] schedule_timeout+0x127/0x270
> >> [ 957.473976] [<ffffffff81171a00>] ? detach_if_pending+0x120/0x120
> >> [ 957.477387] [<ffffffff8116d983>] rcu_gp_kthread+0x6d3/0xa40
> >> [ 957.480659] [<ffffffff811513a0>] ? wake_atomic_t_function+0x70/0x70
> >> [ 957.484123] [<ffffffff8116d2b0>] ? force_qs_rnp+0x1b0/0x1b0
> >> [ 957.487392] [<ffffffff8112f856>] kthread+0xe6/0x100
> >> [ 957.490470] [<ffffffff8112f770>] ? kthread_worker_fn+0x190/0x190
> >> [ 957.493859] [<ffffffff81ab5ccf>] ret_from_fork+0x3f/0x70
> >> [ 957.497044] [<ffffffff8112f770>] ? kthread_worker_fn+0x190/0x190
> >>
> > > Reinette
>
> --
> Mathieu Desnoyers
> EfficiOS Inc.
> http://www.efficios.com
>
next prev parent reply other threads:[~2016-03-26 15:28 UTC|newest]
Thread overview: 79+ messages / expand[flat|nested] mbox.gz Atom feed top
2016-02-09 10:11 rcu_preempt self-detected stall on CPU from 4.5-rc3, since 3.17 Ross Green
2016-02-17 5:45 ` Paul E. McKenney
2016-02-17 19:28 ` Paul E. McKenney
2016-02-17 19:45 ` Peter Zijlstra
2016-02-17 20:28 ` Paul E. McKenney
2016-02-17 23:19 ` Paul E. McKenney
2016-02-18 11:51 ` Ross Green
2016-02-18 23:14 ` Mathieu Desnoyers
2016-02-19 3:56 ` Ross Green
2016-02-19 4:13 ` John Stultz
2016-02-19 17:33 ` Paul E. McKenney
2016-02-20 4:34 ` Ross Green
2016-02-20 6:32 ` Paul E. McKenney
2016-02-21 5:04 ` Ross Green
2016-02-21 18:15 ` Ross Green
2016-02-23 20:34 ` Mathieu Desnoyers
2016-02-23 20:55 ` Paul E. McKenney
2016-02-23 21:28 ` Ross Green
2016-02-25 5:13 ` Ross Green
2016-02-26 0:56 ` Paul E. McKenney
2016-02-26 1:35 ` Paul E. McKenney
2016-03-04 5:30 ` Ross Green
2016-03-04 15:18 ` Paul E. McKenney
2016-03-18 21:00 ` Josh Triplett
2016-03-18 23:56 ` Paul E. McKenney
2016-03-21 16:22 ` Jacob Pan
2016-03-21 17:26 ` Paul E. McKenney
2016-03-22 16:35 ` Chatre, Reinette
2016-03-22 17:40 ` Paul E. McKenney
2016-03-22 21:04 ` Chatre, Reinette
2016-03-22 21:19 ` Paul E. McKenney
2016-03-23 17:15 ` Chatre, Reinette
2016-03-23 18:20 ` Paul E. McKenney
2016-03-23 18:25 ` Chatre, Reinette
2016-03-23 19:50 ` Paul E. McKenney
2016-03-25 21:24 ` Chatre, Reinette
2016-03-25 21:46 ` Paul E. McKenney
2016-03-26 12:29 ` Mathieu Desnoyers
2016-03-26 15:28 ` Paul E. McKenney [this message]
2016-03-26 18:49 ` Paul E. McKenney
2016-03-26 22:22 ` Mathieu Desnoyers
2016-03-27 1:34 ` Paul E. McKenney
2016-03-27 13:48 ` Mathieu Desnoyers
2016-03-27 15:40 ` Paul E. McKenney
2016-03-27 20:00 ` Paul E. McKenney
2016-03-27 20:45 ` Peter Zijlstra
2016-03-27 21:06 ` Paul E. McKenney
2016-03-28 6:25 ` Peter Zijlstra
2016-03-28 13:08 ` Paul E. McKenney
2016-03-29 0:25 ` Paul E. McKenney
2016-03-29 0:28 ` Paul E. McKenney
2016-03-29 13:49 ` Paul E. McKenney
2016-03-30 14:55 ` Paul E. McKenney
2016-03-31 15:42 ` Paul E. McKenney
2016-04-03 8:18 ` Paul E. McKenney
2016-05-06 6:25 ` Ross Green
2016-05-07 15:25 ` Paul E. McKenney
2016-05-10 2:36 ` Ross Green
2016-06-30 17:52 ` Paul E. McKenney
2016-03-28 1:44 ` Mathieu Desnoyers
2016-03-28 2:23 ` Mathieu Desnoyers
2016-03-28 6:13 ` Peter Zijlstra
2016-03-28 13:50 ` Paul E. McKenney
2016-03-28 14:15 ` Mathieu Desnoyers
2016-03-27 20:53 ` Peter Zijlstra
2016-03-27 21:07 ` Paul E. McKenney
2016-03-27 20:54 ` Peter Zijlstra
2016-03-27 21:09 ` Paul E. McKenney
2016-03-28 6:28 ` Peter Zijlstra
2016-03-28 13:29 ` Paul E. McKenney
2016-03-28 15:07 ` Mathieu Desnoyers
2016-03-28 15:56 ` Paul E. McKenney
2016-03-28 16:12 ` Mathieu Desnoyers
2016-03-28 16:29 ` Paul E. McKenney
2016-03-30 12:58 ` Boqun Feng
2016-03-30 13:30 ` Paul E. McKenney
2016-03-30 14:15 ` Boqun Feng
2016-02-19 4:22 ` Paul E. McKenney
2016-02-19 5:59 ` Ross Green
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20160326152816.GW4287@linux.vnet.ibm.com \
--to=paulmck@linux.vnet.ibm.com \
--cc=akpm@linux-foundation.org \
--cc=bobby.prani@gmail.com \
--cc=dhowells@redhat.com \
--cc=dipankar@in.ibm.com \
--cc=dvhart@linux.intel.com \
--cc=edumazet@google.com \
--cc=fweisbec@gmail.com \
--cc=jacob.jun.pan@linux.intel.com \
--cc=jiangshanlai@gmail.com \
--cc=john.stultz@linaro.org \
--cc=josh@joshtriplett.org \
--cc=linux-kernel@vger.kernel.org \
--cc=mathieu.desnoyers@efficios.com \
--cc=mingo@kernel.org \
--cc=oleg@redhat.com \
--cc=peterz@infradead.org \
--cc=reinette.chatre@intel.com \
--cc=rgkernel@gmail.com \
--cc=rostedt@goodmis.org \
--cc=tglx@linutronix.de \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).