From: "Paul E. McKenney" <paulmck@linux.vnet.ibm.com>
To: Paul Walmsley <paul@pwsan.com>
Cc: "Bruce, Becky" <bbruce@ti.com>,
"Paul E. McKenney" <paul.mckenney@linaro.org>,
"<linux-kernel@vger.kernel.org>" <linux-kernel@vger.kernel.org>,
"<linux-omap@vger.kernel.org>" <linux-omap@vger.kernel.org>,
"<linux-arm-kernel@lists.infradead.org>"
<linux-arm-kernel@lists.infradead.org>,
"Hilman, Kevin" <khilman@ti.com>,
"Shilimkar, Santosh" <santosh.shilimkar@ti.com>,
"Hunter, Jon" <jon-hunter@ti.com>,
"<snijsure@grid-net.com>" <snijsure@grid-net.com>
Subject: Re: rcu self-detected stall messages on OMAP3, 4 boards
Date: Sat, 22 Sep 2012 16:17:33 -0700 [thread overview]
Message-ID: <20120922231733.GI2934@linux.vnet.ibm.com> (raw)
In-Reply-To: <alpine.DEB.2.00.1209222214120.22590@utopia.booyaka.com>
On Sat, Sep 22, 2012 at 10:20:19PM +0000, Paul Walmsley wrote:
> Hi Paul
>
> On Sat, 22 Sep 2012, Paul E. McKenney wrote:
>
> > Strangely enough, I believe that I have inadvertently fixed this in
> > my -rcu tree:
> >
> > git://git.kernel.org/pub/scm/linux/kernel/git/paulmck/linux-rcu.git rcu/next
> >
> > Nevertheless, if you get a chance to try it, I would be interested to
> > hear if my guess is correct.
>
> Yes, good news: the stall warnings go away with that branch.
Very good!
> > The trick is that a kthread drives the grace period in -rcu, regardless
> > of whether or not there are callbacks.
>
> This is "rcu: Move quiescent-state forcing into kthread" ?
Yep, plus the preceding commits moving grace-period initialization and
cleanup into that same kthread. This was motivated by a bug report
last February complaining about 200-microsecond latency spikes from
RCU grace-period initialization. On systems with 4096 CPUs.
Real-time response. It is far bigger than I thought. ;-)
> Added some debugging into rcu_gp_kthread() after that commit and can
> confirm that the quiescent-state forcing loop does start a few times when
> there are zero callbacks pending (modulo any races in my measurement
> code).
Cool, thank you! Assuming it works, that indicates that there is long-term
value to the fix for this problem. On larger systems, extra grace periods
are not what you want, as their expense increases with the number of CPUs.
> > However, the backport would not be something that -stable would be happy
> > with, so I will be putting together a fix for mainline. This thing
> > has been in the kernel since about 2004, not sure why you didn't hit
> > it earlier.
>
> One other data point in that regard - noticed the warnings don't appear
> when the board is booted with:
>
> commit 4fa3b6cb1bc8c14b81b4c8ffdfd3f2500a7e9367
> Author: Paul E. McKenney <paul.mckenney@linaro.org>
> Date: Tue Jun 5 15:53:53 2012 -0700
>
> rcu: Fix qlen_lazy breakage
You lost me on this one. This is already in mainline, so if you were
using (say) 3.6-rc6, you would already have this commit applied.
Thanx, Paul
WARNING: multiple messages have this Message-ID (diff)
From: paulmck@linux.vnet.ibm.com (Paul E. McKenney)
To: linux-arm-kernel@lists.infradead.org
Subject: rcu self-detected stall messages on OMAP3, 4 boards
Date: Sat, 22 Sep 2012 16:17:33 -0700 [thread overview]
Message-ID: <20120922231733.GI2934@linux.vnet.ibm.com> (raw)
In-Reply-To: <alpine.DEB.2.00.1209222214120.22590@utopia.booyaka.com>
On Sat, Sep 22, 2012 at 10:20:19PM +0000, Paul Walmsley wrote:
> Hi Paul
>
> On Sat, 22 Sep 2012, Paul E. McKenney wrote:
>
> > Strangely enough, I believe that I have inadvertently fixed this in
> > my -rcu tree:
> >
> > git://git.kernel.org/pub/scm/linux/kernel/git/paulmck/linux-rcu.git rcu/next
> >
> > Nevertheless, if you get a chance to try it, I would be interested to
> > hear if my guess is correct.
>
> Yes, good news: the stall warnings go away with that branch.
Very good!
> > The trick is that a kthread drives the grace period in -rcu, regardless
> > of whether or not there are callbacks.
>
> This is "rcu: Move quiescent-state forcing into kthread" ?
Yep, plus the preceding commits moving grace-period initialization and
cleanup into that same kthread. This was motivated by a bug report
last February complaining about 200-microsecond latency spikes from
RCU grace-period initialization. On systems with 4096 CPUs.
Real-time response. It is far bigger than I thought. ;-)
> Added some debugging into rcu_gp_kthread() after that commit and can
> confirm that the quiescent-state forcing loop does start a few times when
> there are zero callbacks pending (modulo any races in my measurement
> code).
Cool, thank you! Assuming it works, that indicates that there is long-term
value to the fix for this problem. On larger systems, extra grace periods
are not what you want, as their expense increases with the number of CPUs.
> > However, the backport would not be something that -stable would be happy
> > with, so I will be putting together a fix for mainline. This thing
> > has been in the kernel since about 2004, not sure why you didn't hit
> > it earlier.
>
> One other data point in that regard - noticed the warnings don't appear
> when the board is booted with:
>
> commit 4fa3b6cb1bc8c14b81b4c8ffdfd3f2500a7e9367
> Author: Paul E. McKenney <paul.mckenney@linaro.org>
> Date: Tue Jun 5 15:53:53 2012 -0700
>
> rcu: Fix qlen_lazy breakage
You lost me on this one. This is already in mainline, so if you were
using (say) 3.6-rc6, you would already have this commit applied.
Thanx, Paul
next prev parent reply other threads:[~2012-09-22 23:17 UTC|newest]
Thread overview: 101+ messages / expand[flat|nested] mbox.gz Atom feed top
2012-09-12 22:51 rcu self-detected stall messages on OMAP3, 4 boards Paul Walmsley
2012-09-12 22:51 ` Paul Walmsley
2012-09-13 1:12 ` Paul E. McKenney
2012-09-13 1:12 ` Paul E. McKenney
2012-09-13 18:52 ` Paul Walmsley
2012-09-13 18:52 ` Paul Walmsley
2012-09-20 0:03 ` Paul E. McKenney
2012-09-20 0:03 ` Paul E. McKenney
2012-09-20 0:03 ` Paul E. McKenney
2012-09-20 7:56 ` Paul Walmsley
2012-09-20 7:56 ` Paul Walmsley
2012-09-20 15:03 ` Bruce, Becky
2012-09-20 15:03 ` Bruce, Becky
2012-09-20 21:49 ` Bruce, Becky
2012-09-20 21:49 ` Bruce, Becky
2012-09-20 22:01 ` Paul E. McKenney
2012-09-20 22:01 ` Paul E. McKenney
2012-09-20 22:01 ` Paul E. McKenney
2012-09-20 22:47 ` Paul Walmsley
2012-09-20 22:47 ` Paul Walmsley
2012-09-20 23:21 ` Paul E. McKenney
2012-09-20 23:21 ` Paul E. McKenney
2012-09-20 23:21 ` Paul E. McKenney
2012-09-21 18:08 ` Paul Walmsley
2012-09-21 18:08 ` Paul Walmsley
2012-09-21 18:58 ` Paul E. McKenney
2012-09-21 18:58 ` Paul E. McKenney
2012-09-21 19:11 ` Paul Walmsley
2012-09-21 19:11 ` Paul Walmsley
2012-09-21 19:57 ` Paul E. McKenney
2012-09-21 19:57 ` Paul E. McKenney
2012-09-21 20:31 ` Tony Lindgren
2012-09-21 20:31 ` Tony Lindgren
2012-09-21 22:03 ` Paul E. McKenney
2012-09-21 22:03 ` Paul E. McKenney
2012-09-22 15:45 ` Frederic Weisbecker
2012-09-22 15:45 ` Frederic Weisbecker
2012-09-22 16:00 ` Paul E. McKenney
2012-09-22 16:00 ` Paul E. McKenney
2012-09-21 22:12 ` Paul E. McKenney
2012-09-21 22:12 ` Paul E. McKenney
2012-09-22 18:42 ` Paul Walmsley
2012-09-22 18:42 ` Paul Walmsley
2012-09-22 20:10 ` Paul E. McKenney
2012-09-22 20:10 ` Paul E. McKenney
2012-09-22 21:59 ` Paul E. McKenney
2012-09-22 21:59 ` Paul E. McKenney
2012-09-22 22:25 ` Paul Walmsley
2012-09-22 22:25 ` Paul Walmsley
2012-09-22 23:11 ` Paul E. McKenney
2012-09-22 23:11 ` Paul E. McKenney
2012-09-22 23:11 ` Paul E. McKenney
2012-09-23 7:55 ` Paul Walmsley
2012-09-23 7:55 ` Paul Walmsley
2012-09-23 7:55 ` Paul Walmsley
2012-09-23 12:11 ` Paul E. McKenney
2012-09-23 12:11 ` Paul E. McKenney
2012-09-23 12:11 ` Paul E. McKenney
2012-09-23 1:42 ` Paul Walmsley
2012-09-23 1:42 ` Paul Walmsley
2012-09-23 1:56 ` Paul E. McKenney
2012-09-23 1:56 ` Paul E. McKenney
2012-09-23 1:56 ` Paul E. McKenney
2012-09-23 2:01 ` Paul Walmsley
2012-09-23 2:01 ` Paul Walmsley
2012-09-24 9:41 ` Shilimkar, Santosh
2012-09-24 9:41 ` Shilimkar, Santosh
2012-09-24 13:18 ` Paul E. McKenney
2012-09-24 13:18 ` Paul E. McKenney
2012-10-01 8:55 ` Linus Walleij
2012-10-01 8:55 ` Linus Walleij
2012-10-01 13:28 ` Paul E. McKenney
2012-10-01 13:28 ` Paul E. McKenney
2012-09-21 18:59 ` Paul Walmsley
2012-09-21 18:59 ` Paul Walmsley
2012-09-21 17:47 ` Paul Walmsley
2012-09-21 17:47 ` Paul Walmsley
2012-09-21 17:51 ` Paul Walmsley
2012-09-21 17:51 ` Paul Walmsley
2012-09-21 21:20 ` Paul E. McKenney
2012-09-21 21:20 ` Paul E. McKenney
2012-09-21 21:20 ` Paul E. McKenney
2012-09-21 22:41 ` Paul Walmsley
2012-09-21 22:41 ` Paul Walmsley
2012-09-22 0:05 ` Paul E. McKenney
2012-09-22 0:05 ` Paul E. McKenney
2012-09-22 18:16 ` Paul Walmsley
2012-09-22 18:16 ` Paul Walmsley
2012-09-22 18:16 ` Paul Walmsley
2012-09-22 19:52 ` Paul E. McKenney
2012-09-22 19:52 ` Paul E. McKenney
2012-09-22 19:52 ` Paul E. McKenney
2012-09-22 22:20 ` Paul Walmsley
2012-09-22 22:20 ` Paul Walmsley
2012-09-22 22:20 ` Paul Walmsley
2012-09-22 23:17 ` Paul E. McKenney [this message]
2012-09-22 23:17 ` Paul E. McKenney
2012-09-24 21:54 ` Paul Walmsley
2012-09-24 21:54 ` Paul Walmsley
2012-09-24 22:00 ` Paul E. McKenney
2012-09-24 22:00 ` Paul E. McKenney
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20120922231733.GI2934@linux.vnet.ibm.com \
--to=paulmck@linux.vnet.ibm.com \
--cc=bbruce@ti.com \
--cc=jon-hunter@ti.com \
--cc=khilman@ti.com \
--cc=linux-arm-kernel@lists.infradead.org \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-omap@vger.kernel.org \
--cc=paul.mckenney@linaro.org \
--cc=paul@pwsan.com \
--cc=santosh.shilimkar@ti.com \
--cc=snijsure@grid-net.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.