[PATCH tip/urgent] rcu: Permit call_rcu() from CPU_DYING notifiers

[PATCH tip/urgent] rcu: Permit call_rcu() from CPU_DYING notifiers

[Paul E. McKenney]

As of commit 29494be7, RCU adopts callbacks from the dying CPU in its
CPU_DYING notifier, which means that any callbacks posted by later
CPU_DYING notifiers are ignored until the CPU comes back online.
A WARN_ON_ONCE() was added to __call_rcu() by commit e5601400 to check
for this condition.  Although this condition did not trigger (at least
as far as know) during -next testing, it did recently trigger in mainline
(https://lkml.org/lkml/2012/4/2/34).

This commit therefore causes RCU's CPU_DEAD notifier to adopt any
callbacks that were posted by CPU_DYING notifiers and removes the
WARN_ON_ONCE() from __call_rcu().  A more targeted warning for callback
posting from offline CPUs will be added as a separate commit.

Signed-off-by: Paul E. McKenney <paul.mckenney@linaro.org>
Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
Tested-by: Sergey Senozhatsky <sergey.senozhatsky@gmail.com>

 rcutree.c |   31 ++++++++++++++++++++++++++++++-
 1 file changed, 30 insertions(+), 1 deletion(-)

diff --git a/kernel/rcutree.c b/kernel/rcutree.c
index 1050d6d..4c927e6 100644
--- a/kernel/rcutree.c
+++ b/kernel/rcutree.c
@@ -1406,11 +1406,41 @@ static void rcu_cleanup_dying_cpu(struct rcu_state *rsp)
 static void rcu_cleanup_dead_cpu(int cpu, struct rcu_state *rsp)
 {
 	unsigned long flags;
+	int i;
 	unsigned long mask;
 	int need_report = 0;
 	struct rcu_data *rdp = per_cpu_ptr(rsp->rda, cpu);
 	struct rcu_node *rnp = rdp->mynode;  /* Outgoing CPU's rnp. */
 
+	/* If a CPU_DYING notifier has enqueued callbacks, adopt them. */
+	if (rdp->nxtlist != NULL) {
+		struct rcu_data *receive_rdp;
+
+		local_irq_save(flags);
+		receive_rdp = per_cpu_ptr(rsp->rda, smp_processor_id());
+
+		/* Adjust the counts. */
+		receive_rdp->qlen_lazy += rdp->qlen_lazy;
+		receive_rdp->qlen += rdp->qlen;
+		rdp->qlen_lazy = 0;
+		rdp->qlen = 0;
+
+		/*
+		 * Adopt all callbacks.  The outgoing CPU was in no shape
+		 * to advance them, so make them all go through a full
+		 * grace period.
+		 */
+		*receive_rdp->nxttail[RCU_NEXT_TAIL] = rdp->nxtlist;
+		receive_rdp->nxttail[RCU_NEXT_TAIL] =
+			rdp->nxttail[RCU_NEXT_TAIL];
+		local_irq_restore(flags);
+
+		/* Initialize the outgoing CPU's callback list. */
+		rdp->nxtlist = NULL;
+		for (i = 0; i < RCU_NEXT_SIZE; i++)
+			rdp->nxttail[i] = &rdp->nxtlist;
+	}
+
 	/* Adjust any no-longer-needed kthreads. */
 	rcu_stop_cpu_kthread(cpu);
 	rcu_node_kthread_setaffinity(rnp, -1);
@@ -1820,7 +1850,6 @@ __call_rcu(struct rcu_head *head, void (*func)(struct rcu_head *rcu),
 	 * a quiescent state betweentimes.
 	 */
 	local_irq_save(flags);
-	WARN_ON_ONCE(cpu_is_offline(smp_processor_id()));
 	rdp = this_cpu_ptr(rsp->rda);
 
 	/* Add the callback to our list. */

[tip:core/urgent] rcu: Permit call_rcu() from CPU_DYING notifiers

[tip-bot for Paul E. McKenney]

Commit-ID:  92d52de4add0e4502aeb063cd45daf314a9c2e1b
Gitweb:     http://git.kernel.org/tip/92d52de4add0e4502aeb063cd45daf314a9c2e1b
Author:     Paul E. McKenney <paulmck@linux.vnet.ibm.com>
AuthorDate: Thu, 5 Apr 2012 15:26:04 -0700
Committer:  Ingo Molnar <mingo@kernel.org>
CommitDate: Mon, 16 Apr 2012 11:17:57 +0200

rcu: Permit call_rcu() from CPU_DYING notifiers

As of:

  29494be71afe ("rcu,cleanup: simplify the code when cpu is dying")

RCU adopts callbacks from the dying CPU in its CPU_DYING notifier,
which means that any callbacks posted by later CPU_DYING notifiers
are ignored until the CPU comes back online.

A WARN_ON_ONCE() was added to __call_rcu() by:

  e56014000816 ("rcu: Simplify offline processing")

to check for this condition.  Although this condition did not trigger
(at least as far as I know) during -next testing, it did recently
trigger in mainline:

  https://lkml.org/lkml/2012/4/2/34

This commit therefore causes RCU's CPU_DEAD notifier to adopt
any callbacks that were posted by CPU_DYING notifiers and
removes the WARN_ON_ONCE() from __call_rcu().

A more targeted warning for callback posting from offline CPUs will be
added as a separate commit.

Tested-by: Sergey Senozhatsky <sergey.senozhatsky@gmail.com>
Signed-off-by: Paul E. McKenney <paul.mckenney@linaro.org>
Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
Cc: Linus Torvalds <torvalds@linux-foundation.org>
Cc: laijs@cn.fujitsu.com
Cc: dipankar@in.ibm.com
Cc: mathieu.desnoyers@efficios.com
Cc: josh@joshtriplett.org
Cc: niv@us.ibm.com
Cc: peterz@infradead.org
Cc: rostedt@goodmis.org
Cc: Valdis.Kletnieks@vt.edu
Cc: dhowells@redhat.com
Cc: eric.dumazet@gmail.com
Cc: darren@dvhart.com
Cc: fweisbec@gmail.com
Cc: patches@linaro.org
Link: http://lkml.kernel.org/r/20120405222604.GA15713@linux.vnet.ibm.com
Signed-off-by: Ingo Molnar <mingo@kernel.org>
---
 kernel/rcutree.c |   31 ++++++++++++++++++++++++++++++-
 1 files changed, 30 insertions(+), 1 deletions(-)

diff --git a/kernel/rcutree.c b/kernel/rcutree.c
index 1050d6d..4c927e6 100644
--- a/kernel/rcutree.c
+++ b/kernel/rcutree.c
@@ -1406,11 +1406,41 @@ static void rcu_cleanup_dying_cpu(struct rcu_state *rsp)
 static void rcu_cleanup_dead_cpu(int cpu, struct rcu_state *rsp)
 {
 	unsigned long flags;
+	int i;
 	unsigned long mask;
 	int need_report = 0;
 	struct rcu_data *rdp = per_cpu_ptr(rsp->rda, cpu);
 	struct rcu_node *rnp = rdp->mynode;  /* Outgoing CPU's rnp. */
 
+	/* If a CPU_DYING notifier has enqueued callbacks, adopt them. */
+	if (rdp->nxtlist != NULL) {
+		struct rcu_data *receive_rdp;
+
+		local_irq_save(flags);
+		receive_rdp = per_cpu_ptr(rsp->rda, smp_processor_id());
+
+		/* Adjust the counts. */
+		receive_rdp->qlen_lazy += rdp->qlen_lazy;
+		receive_rdp->qlen += rdp->qlen;
+		rdp->qlen_lazy = 0;
+		rdp->qlen = 0;
+
+		/*
+		 * Adopt all callbacks.  The outgoing CPU was in no shape
+		 * to advance them, so make them all go through a full
+		 * grace period.
+		 */
+		*receive_rdp->nxttail[RCU_NEXT_TAIL] = rdp->nxtlist;
+		receive_rdp->nxttail[RCU_NEXT_TAIL] =
+			rdp->nxttail[RCU_NEXT_TAIL];
+		local_irq_restore(flags);
+
+		/* Initialize the outgoing CPU's callback list. */
+		rdp->nxtlist = NULL;
+		for (i = 0; i < RCU_NEXT_SIZE; i++)
+			rdp->nxttail[i] = &rdp->nxtlist;
+	}
+
 	/* Adjust any no-longer-needed kthreads. */
 	rcu_stop_cpu_kthread(cpu);
 	rcu_node_kthread_setaffinity(rnp, -1);
@@ -1820,7 +1850,6 @@ __call_rcu(struct rcu_head *head, void (*func)(struct rcu_head *rcu),
 	 * a quiescent state betweentimes.
 	 */
 	local_irq_save(flags);
-	WARN_ON_ONCE(cpu_is_offline(smp_processor_id()));
 	rdp = this_cpu_ptr(rsp->rda);
 
 	/* Add the callback to our list. */