From: "Paul E. McKenney" <paulmck@linux.vnet.ibm.com>
To: Paul Walmsley <paul@pwsan.com>
Cc: "Hilman, Kevin" <khilman@ti.com>,
"<snijsure@grid-net.com>" <snijsure@grid-net.com>,
fweisbec@gmail.com, "Bruce, Becky" <bbruce@ti.com>,
"<linux-kernel@vger.kernel.org>" <linux-kernel@vger.kernel.org>,
"Paul E. McKenney" <paul.mckenney@linaro.org>,
"Shilimkar, Santosh" <santosh.shilimkar@ti.com>,
"Hunter, Jon" <jon-hunter@ti.com>,
"<linux-omap@vger.kernel.org>" <linux-omap@vger.kernel.org>,
"<linux-arm-kernel@lists.infradead.org>"
<linux-arm-kernel@lists.infradead.org>
Subject: Re: rcu self-detected stall messages on OMAP3, 4 boards
Date: Sat, 22 Sep 2012 16:11:22 -0700 [thread overview]
Message-ID: <20120922231121.GH2934@linux.vnet.ibm.com> (raw)
In-Reply-To: <alpine.DEB.2.00.1209222223080.22590@utopia.booyaka.com>
On Sat, Sep 22, 2012 at 10:25:59PM +0000, Paul Walmsley wrote:
> On Sat, 22 Sep 2012, Paul E. McKenney wrote:
>
> > And here is a patch. I am still having trouble reproducing the problem,
> > but figured that I should avoid serializing things.
>
> Thanks, testing this now on v3.6-rc6.
Very cool, thank you!
> One question though about the patch
> description:
>
> > All this begs the question of exactly how a callback-free grace period
> > gets started in the first place. This can happen due to the fact that
> > CPUs do not necessarily agree on which grace period is in progress.
> > If a CPU still believes that the grace period that just completed is
> > still ongoing, it will believe that it has callbacks that need to wait
> > for another grace period, never mind the fact that the grace period
> > that they were waiting for just completed. This CPU can therefore
> > erroneously decide to start a new grace period.
>
> Doesn't this imply that this bug would only affect multi-CPU systems?
Surprisingly not, at least when running TREE_RCU or TREE_PREEMPT_RCU.
In order to keep lock contention down to a dull roar on larger systems,
TREE_RCU keeps three sets of books: (1) the global state in the rcu_state
structure, (2) the combining-tree per-node state in the rcu_node
structure, and the per-CPU state in the rcu_data structure. A CPU is
not officially aware of the end of a grace period until it is reflected
in its rcu_data structure. This has the perhaps-surprising consequence
that the CPU that detected the end of the old grace period might start
a new one before becoming officially aware that the old one ended.
Why not have the CPU inform itself immediately upon noticing that the
old grace period ended? Deadlock. The rcu_node locks must be acquired
from leaf towards root, and the CPU is holding the root rcu_node lock
when it notices that the grace period has ended.
I have made this a bit less problematic in the bigrt branch, working
towards a goal of getting RCU into a state where automatic formal
validation might one day be possible. And yes, I am starting to get some
formal-validation people interested in this lofty goal, see for example:
http://sites.google.com/site/popl13grace/paper.pdf.
> The recent tests here have been on Pandaboard, which is dual-CPU, but my
> recollection is that I also observed the warnings on a single-core
> Beagleboard. Will re-test.
Anxiously awaiting the results. This has been a strange one, even by
RCU's standards.
Plus I need to add a few Reported-by lines. Next version...
Thanx, Paul
WARNING: multiple messages have this Message-ID (diff)
From: paulmck@linux.vnet.ibm.com (Paul E. McKenney)
To: linux-arm-kernel@lists.infradead.org
Subject: rcu self-detected stall messages on OMAP3, 4 boards
Date: Sat, 22 Sep 2012 16:11:22 -0700 [thread overview]
Message-ID: <20120922231121.GH2934@linux.vnet.ibm.com> (raw)
In-Reply-To: <alpine.DEB.2.00.1209222223080.22590@utopia.booyaka.com>
On Sat, Sep 22, 2012 at 10:25:59PM +0000, Paul Walmsley wrote:
> On Sat, 22 Sep 2012, Paul E. McKenney wrote:
>
> > And here is a patch. I am still having trouble reproducing the problem,
> > but figured that I should avoid serializing things.
>
> Thanks, testing this now on v3.6-rc6.
Very cool, thank you!
> One question though about the patch
> description:
>
> > All this begs the question of exactly how a callback-free grace period
> > gets started in the first place. This can happen due to the fact that
> > CPUs do not necessarily agree on which grace period is in progress.
> > If a CPU still believes that the grace period that just completed is
> > still ongoing, it will believe that it has callbacks that need to wait
> > for another grace period, never mind the fact that the grace period
> > that they were waiting for just completed. This CPU can therefore
> > erroneously decide to start a new grace period.
>
> Doesn't this imply that this bug would only affect multi-CPU systems?
Surprisingly not, at least when running TREE_RCU or TREE_PREEMPT_RCU.
In order to keep lock contention down to a dull roar on larger systems,
TREE_RCU keeps three sets of books: (1) the global state in the rcu_state
structure, (2) the combining-tree per-node state in the rcu_node
structure, and the per-CPU state in the rcu_data structure. A CPU is
not officially aware of the end of a grace period until it is reflected
in its rcu_data structure. This has the perhaps-surprising consequence
that the CPU that detected the end of the old grace period might start
a new one before becoming officially aware that the old one ended.
Why not have the CPU inform itself immediately upon noticing that the
old grace period ended? Deadlock. The rcu_node locks must be acquired
from leaf towards root, and the CPU is holding the root rcu_node lock
when it notices that the grace period has ended.
I have made this a bit less problematic in the bigrt branch, working
towards a goal of getting RCU into a state where automatic formal
validation might one day be possible. And yes, I am starting to get some
formal-validation people interested in this lofty goal, see for example:
http://sites.google.com/site/popl13grace/paper.pdf.
> The recent tests here have been on Pandaboard, which is dual-CPU, but my
> recollection is that I also observed the warnings on a single-core
> Beagleboard. Will re-test.
Anxiously awaiting the results. This has been a strange one, even by
RCU's standards.
Plus I need to add a few Reported-by lines. Next version...
Thanx, Paul
WARNING: multiple messages have this Message-ID (diff)
From: "Paul E. McKenney" <paulmck@linux.vnet.ibm.com>
To: Paul Walmsley <paul@pwsan.com>
Cc: "Bruce, Becky" <bbruce@ti.com>,
"Paul E. McKenney" <paul.mckenney@linaro.org>,
"<linux-kernel@vger.kernel.org>" <linux-kernel@vger.kernel.org>,
"<linux-omap@vger.kernel.org>" <linux-omap@vger.kernel.org>,
"<linux-arm-kernel@lists.infradead.org>"
<linux-arm-kernel@lists.infradead.org>,
"Hilman, Kevin" <khilman@ti.com>,
"Shilimkar, Santosh" <santosh.shilimkar@ti.com>,
"Hunter, Jon" <jon-hunter@ti.com>,
"<snijsure@grid-net.com>" <snijsure@grid-net.com>,
fweisbec@gmail.com
Subject: Re: rcu self-detected stall messages on OMAP3, 4 boards
Date: Sat, 22 Sep 2012 16:11:22 -0700 [thread overview]
Message-ID: <20120922231121.GH2934@linux.vnet.ibm.com> (raw)
In-Reply-To: <alpine.DEB.2.00.1209222223080.22590@utopia.booyaka.com>
On Sat, Sep 22, 2012 at 10:25:59PM +0000, Paul Walmsley wrote:
> On Sat, 22 Sep 2012, Paul E. McKenney wrote:
>
> > And here is a patch. I am still having trouble reproducing the problem,
> > but figured that I should avoid serializing things.
>
> Thanks, testing this now on v3.6-rc6.
Very cool, thank you!
> One question though about the patch
> description:
>
> > All this begs the question of exactly how a callback-free grace period
> > gets started in the first place. This can happen due to the fact that
> > CPUs do not necessarily agree on which grace period is in progress.
> > If a CPU still believes that the grace period that just completed is
> > still ongoing, it will believe that it has callbacks that need to wait
> > for another grace period, never mind the fact that the grace period
> > that they were waiting for just completed. This CPU can therefore
> > erroneously decide to start a new grace period.
>
> Doesn't this imply that this bug would only affect multi-CPU systems?
Surprisingly not, at least when running TREE_RCU or TREE_PREEMPT_RCU.
In order to keep lock contention down to a dull roar on larger systems,
TREE_RCU keeps three sets of books: (1) the global state in the rcu_state
structure, (2) the combining-tree per-node state in the rcu_node
structure, and the per-CPU state in the rcu_data structure. A CPU is
not officially aware of the end of a grace period until it is reflected
in its rcu_data structure. This has the perhaps-surprising consequence
that the CPU that detected the end of the old grace period might start
a new one before becoming officially aware that the old one ended.
Why not have the CPU inform itself immediately upon noticing that the
old grace period ended? Deadlock. The rcu_node locks must be acquired
from leaf towards root, and the CPU is holding the root rcu_node lock
when it notices that the grace period has ended.
I have made this a bit less problematic in the bigrt branch, working
towards a goal of getting RCU into a state where automatic formal
validation might one day be possible. And yes, I am starting to get some
formal-validation people interested in this lofty goal, see for example:
http://sites.google.com/site/popl13grace/paper.pdf.
> The recent tests here have been on Pandaboard, which is dual-CPU, but my
> recollection is that I also observed the warnings on a single-core
> Beagleboard. Will re-test.
Anxiously awaiting the results. This has been a strange one, even by
RCU's standards.
Plus I need to add a few Reported-by lines. Next version...
Thanx, Paul
next prev parent reply other threads:[~2012-09-22 23:11 UTC|newest]
Thread overview: 101+ messages / expand[flat|nested] mbox.gz Atom feed top
2012-09-12 22:51 rcu self-detected stall messages on OMAP3, 4 boards Paul Walmsley
2012-09-12 22:51 ` Paul Walmsley
2012-09-13 1:12 ` Paul E. McKenney
2012-09-13 1:12 ` Paul E. McKenney
2012-09-13 18:52 ` Paul Walmsley
2012-09-13 18:52 ` Paul Walmsley
2012-09-20 0:03 ` Paul E. McKenney
2012-09-20 0:03 ` Paul E. McKenney
2012-09-20 0:03 ` Paul E. McKenney
2012-09-20 7:56 ` Paul Walmsley
2012-09-20 7:56 ` Paul Walmsley
2012-09-20 15:03 ` Bruce, Becky
2012-09-20 15:03 ` Bruce, Becky
2012-09-20 21:49 ` Bruce, Becky
2012-09-20 21:49 ` Bruce, Becky
2012-09-20 22:01 ` Paul E. McKenney
2012-09-20 22:01 ` Paul E. McKenney
2012-09-20 22:01 ` Paul E. McKenney
2012-09-20 22:47 ` Paul Walmsley
2012-09-20 22:47 ` Paul Walmsley
2012-09-20 23:21 ` Paul E. McKenney
2012-09-20 23:21 ` Paul E. McKenney
2012-09-20 23:21 ` Paul E. McKenney
2012-09-21 18:08 ` Paul Walmsley
2012-09-21 18:08 ` Paul Walmsley
2012-09-21 18:58 ` Paul E. McKenney
2012-09-21 18:58 ` Paul E. McKenney
2012-09-21 19:11 ` Paul Walmsley
2012-09-21 19:11 ` Paul Walmsley
2012-09-21 19:57 ` Paul E. McKenney
2012-09-21 19:57 ` Paul E. McKenney
2012-09-21 20:31 ` Tony Lindgren
2012-09-21 20:31 ` Tony Lindgren
2012-09-21 22:03 ` Paul E. McKenney
2012-09-21 22:03 ` Paul E. McKenney
2012-09-22 15:45 ` Frederic Weisbecker
2012-09-22 15:45 ` Frederic Weisbecker
2012-09-22 16:00 ` Paul E. McKenney
2012-09-22 16:00 ` Paul E. McKenney
2012-09-21 22:12 ` Paul E. McKenney
2012-09-21 22:12 ` Paul E. McKenney
2012-09-22 18:42 ` Paul Walmsley
2012-09-22 18:42 ` Paul Walmsley
2012-09-22 20:10 ` Paul E. McKenney
2012-09-22 20:10 ` Paul E. McKenney
2012-09-22 21:59 ` Paul E. McKenney
2012-09-22 21:59 ` Paul E. McKenney
2012-09-22 22:25 ` Paul Walmsley
2012-09-22 22:25 ` Paul Walmsley
2012-09-22 23:11 ` Paul E. McKenney [this message]
2012-09-22 23:11 ` Paul E. McKenney
2012-09-22 23:11 ` Paul E. McKenney
2012-09-23 7:55 ` Paul Walmsley
2012-09-23 7:55 ` Paul Walmsley
2012-09-23 7:55 ` Paul Walmsley
2012-09-23 12:11 ` Paul E. McKenney
2012-09-23 12:11 ` Paul E. McKenney
2012-09-23 12:11 ` Paul E. McKenney
2012-09-23 1:42 ` Paul Walmsley
2012-09-23 1:42 ` Paul Walmsley
2012-09-23 1:56 ` Paul E. McKenney
2012-09-23 1:56 ` Paul E. McKenney
2012-09-23 1:56 ` Paul E. McKenney
2012-09-23 2:01 ` Paul Walmsley
2012-09-23 2:01 ` Paul Walmsley
2012-09-24 9:41 ` Shilimkar, Santosh
2012-09-24 9:41 ` Shilimkar, Santosh
2012-09-24 13:18 ` Paul E. McKenney
2012-09-24 13:18 ` Paul E. McKenney
2012-10-01 8:55 ` Linus Walleij
2012-10-01 8:55 ` Linus Walleij
2012-10-01 13:28 ` Paul E. McKenney
2012-10-01 13:28 ` Paul E. McKenney
2012-09-21 18:59 ` Paul Walmsley
2012-09-21 18:59 ` Paul Walmsley
2012-09-21 17:47 ` Paul Walmsley
2012-09-21 17:47 ` Paul Walmsley
2012-09-21 17:51 ` Paul Walmsley
2012-09-21 17:51 ` Paul Walmsley
2012-09-21 21:20 ` Paul E. McKenney
2012-09-21 21:20 ` Paul E. McKenney
2012-09-21 21:20 ` Paul E. McKenney
2012-09-21 22:41 ` Paul Walmsley
2012-09-21 22:41 ` Paul Walmsley
2012-09-22 0:05 ` Paul E. McKenney
2012-09-22 0:05 ` Paul E. McKenney
2012-09-22 18:16 ` Paul Walmsley
2012-09-22 18:16 ` Paul Walmsley
2012-09-22 18:16 ` Paul Walmsley
2012-09-22 19:52 ` Paul E. McKenney
2012-09-22 19:52 ` Paul E. McKenney
2012-09-22 19:52 ` Paul E. McKenney
2012-09-22 22:20 ` Paul Walmsley
2012-09-22 22:20 ` Paul Walmsley
2012-09-22 22:20 ` Paul Walmsley
2012-09-22 23:17 ` Paul E. McKenney
2012-09-22 23:17 ` Paul E. McKenney
2012-09-24 21:54 ` Paul Walmsley
2012-09-24 21:54 ` Paul Walmsley
2012-09-24 22:00 ` Paul E. McKenney
2012-09-24 22:00 ` Paul E. McKenney
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20120922231121.GH2934@linux.vnet.ibm.com \
--to=paulmck@linux.vnet.ibm.com \
--cc=bbruce@ti.com \
--cc=fweisbec@gmail.com \
--cc=jon-hunter@ti.com \
--cc=khilman@ti.com \
--cc=linux-arm-kernel@lists.infradead.org \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-omap@vger.kernel.org \
--cc=paul.mckenney@linaro.org \
--cc=paul@pwsan.com \
--cc=santosh.shilimkar@ti.com \
--cc=snijsure@grid-net.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.