[PATCH tip/core/rcu 0/28] patches queued for 3.1, first installment

[PATCH tip/core/rcu 0/28] patches queued for 3.1, first installment

[Paul E. McKenney]

Hello!

This patchset contains some simplifications, bug fixes, documentation
updates, and a bunch of leftover conversions to kfree_rcu().  Almost
all of the latter need either an Acked-by or to be taken into the
appropriate maintainer tree.  The patches are as follows:

1.	Simplify RCU kthread creation.
2.	Convert from kthread_create() to kthread_create_on_node()
	(courtesy of Eric Dumazet).
3.	Improve rcu_read_lock() code generation.
4.	Restore checks for blocking within RCU read-side critical
	sections under CONFIG_PROVE_RCU.
5.	Move struct rcu_head into types.h as a first step towards
	untangling header files.
6.	Update rcutorture documentation to account for boosting,
	memory-barrier testing, and (not so) new flavors of RCU.
7.	Fix type mismatch (courtesy of Andi Kleen).
8.	Combine common grace-period-wait code.
9.	Eliminate rcutorture workarounds for previously missing
	RCU API members, and add testing of synchronize_rcu_bh_expedited().
10.	Update RCU's NMI documentation.
11-27.	Conversions to kfree_rcu() (courtesy of Lai Jiangshan,
	in need of either Acked-by or adoption).
28.	Convert call_rcu(sk_filter_release_rcu) to kfree_rcu
	(courtesy of Lai Jiangshan, already acked).

For a testing-only version of this patchset from git, please see the
following subject-to-rebase branch:

git://git.kernel.org/pub/scm/linux/kernel/git/paulmck/linux-2.6-rcu.git rcu/testing

							Thanx, Paul

------------------------------------------------------------------------

 b/Documentation/RCU/NMI-RCU.txt |    2 
 b/Documentation/RCU/torture.txt |  134 ++++++++++++++++++++++++++++----------
 b/arch/ia64/sn/kernel/irq.c     |   14 ----
 b/block/cfq-iosched.c           |    7 --
 b/block/genhd.c                 |   10 --
 b/drivers/md/linear.c           |    8 --
 b/drivers/scsi/libfc/fc_rport.c |   14 ----
 b/fs/jbd2/journal.c             |    7 --
 b/fs/nfs/delegation.c           |   14 ----
 b/include/linux/lockdep.h       |    2 
 b/include/linux/rcupdate.h      |   28 ++++++--
 b/include/linux/rcutiny.h       |   16 ++++
 b/include/linux/rcutree.h       |    2 
 b/include/linux/types.h         |   10 ++
 b/include/net/sock.h            |    4 -
 b/ipc/sem.c                     |   10 --
 b/ipc/util.c                    |   16 ----
 b/kernel/audit_tree.c           |    8 --
 b/kernel/lockdep.c              |   84 +++++++++++++-----------
 b/kernel/pid.c                  |    4 -
 b/kernel/rcupdate.c             |   21 +++++-
 b/kernel/rcutiny.c              |   28 --------
 b/kernel/rcutiny_plugin.h       |   14 ----
 b/kernel/rcutorture.c           |    3 
 b/kernel/rcutree.c              |   65 ++++++------------
 b/kernel/rcutree_plugin.h       |   11 ---
 b/kernel/rcutree_trace.c        |    2 
 b/kernel/sched.c                |    2 
 b/kernel/sysctl.c               |   11 ---
 b/mm/vmalloc.c                  |    9 --
 b/net/core/filter.c             |   12 ---
 b/net/netfilter/xt_RATEEST.c    |    8 --
 b/security/device_cgroup.c      |   10 --
 b/security/selinux/netnode.c    |   20 -----
 b/security/selinux/netport.c    |   20 -----
 include/linux/rcupdate.h        |  140 +++++++++++++++++++++-------------------
 kernel/rcutorture.c             |   55 ++++++---------
 kernel/rcutree.c                |   27 +------
 kernel/rcutree_plugin.h         |   23 ++----
 mm/vmalloc.c                    |    9 --
 40 files changed, 379 insertions(+), 505 deletions(-)

[PATCH tip/core/rcu 01/28] rcu: Simplify curing of load woes

[Paul E. McKenney]

Make the functions creating the kthreads wake them up.  Leverage the
fact that the per-node and boost kthreads can run anywhere, thus
dispensing with the need to wake them up once the incoming CPU has
gone fully online.

Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
---
 kernel/rcutree.c        |   65 +++++++++++++++-------------------------------
 kernel/rcutree_plugin.h |   11 +-------
 2 files changed, 22 insertions(+), 54 deletions(-)

diff --git a/kernel/rcutree.c b/kernel/rcutree.c
index 4cc6a94..36e79d2 100644
--- a/kernel/rcutree.c
+++ b/kernel/rcutree.c
@@ -1634,6 +1634,20 @@ static int rcu_cpu_kthread(void *arg)
  * to manipulate rcu_cpu_kthread_task.  There might be another CPU
  * attempting to access it during boot, but the locking in kthread_bind()
  * will enforce sufficient ordering.
+ *
+ * Please note that we cannot simply refuse to wake up the per-CPU
+ * kthread because kthreads are created in TASK_UNINTERRUPTIBLE state,
+ * which can result in softlockup complaints if the task ends up being
+ * idle for more than a couple of minutes.
+ *
+ * However, please note also that we cannot bind the per-CPU kthread to its
+ * CPU until that CPU is fully online.  We also cannot wait until the
+ * CPU is fully online before we create its per-CPU kthread, as this would
+ * deadlock the system when CPU notifiers tried waiting for grace
+ * periods.  So we bind the per-CPU kthread to its CPU only if the CPU
+ * is online.  If its CPU is not yet fully online, then the code in
+ * rcu_cpu_kthread() will wait until it is fully online, and then do
+ * the binding.
  */
 static int __cpuinit rcu_spawn_one_cpu_kthread(int cpu)
 {
@@ -1646,12 +1660,14 @@ static int __cpuinit rcu_spawn_one_cpu_kthread(int cpu)
 	t = kthread_create(rcu_cpu_kthread, (void *)(long)cpu, "rcuc%d", cpu);
 	if (IS_ERR(t))
 		return PTR_ERR(t);
-	kthread_bind(t, cpu);
+	if (cpu_online(cpu))
+		kthread_bind(t, cpu);
 	per_cpu(rcu_cpu_kthread_cpu, cpu) = cpu;
 	WARN_ON_ONCE(per_cpu(rcu_cpu_kthread_task, cpu) != NULL);
-	per_cpu(rcu_cpu_kthread_task, cpu) = t;
 	sp.sched_priority = RCU_KTHREAD_PRIO;
 	sched_setscheduler_nocheck(t, SCHED_FIFO, &sp);
+	per_cpu(rcu_cpu_kthread_task, cpu) = t;
+	wake_up_process(t); /* Get to TASK_INTERRUPTIBLE quickly. */
 	return 0;
 }
 
@@ -1758,12 +1774,11 @@ static int __cpuinit rcu_spawn_one_node_kthread(struct rcu_state *rsp,
 		raw_spin_unlock_irqrestore(&rnp->lock, flags);
 		sp.sched_priority = 99;
 		sched_setscheduler_nocheck(t, SCHED_FIFO, &sp);
+		wake_up_process(t); /* get to TASK_INTERRUPTIBLE quickly. */
 	}
 	return rcu_spawn_one_boost_kthread(rsp, rnp, rnp_index);
 }
 
-static void rcu_wake_one_boost_kthread(struct rcu_node *rnp);
-
 /*
  * Spawn all kthreads -- called as soon as the scheduler is running.
  */
@@ -1771,30 +1786,18 @@ static int __init rcu_spawn_kthreads(void)
 {
 	int cpu;
 	struct rcu_node *rnp;
-	struct task_struct *t;
 
 	rcu_kthreads_spawnable = 1;
 	for_each_possible_cpu(cpu) {
 		per_cpu(rcu_cpu_has_work, cpu) = 0;
-		if (cpu_online(cpu)) {
+		if (cpu_online(cpu))
 			(void)rcu_spawn_one_cpu_kthread(cpu);
-			t = per_cpu(rcu_cpu_kthread_task, cpu);
-			if (t)
-				wake_up_process(t);
-		}
 	}
 	rnp = rcu_get_root(rcu_state);
 	(void)rcu_spawn_one_node_kthread(rcu_state, rnp);
-	if (rnp->node_kthread_task)
-		wake_up_process(rnp->node_kthread_task);
 	if (NUM_RCU_NODES > 1) {
-		rcu_for_each_leaf_node(rcu_state, rnp) {
+		rcu_for_each_leaf_node(rcu_state, rnp)
 			(void)rcu_spawn_one_node_kthread(rcu_state, rnp);
-			t = rnp->node_kthread_task;
-			if (t)
-				wake_up_process(t);
-			rcu_wake_one_boost_kthread(rnp);
-		}
 	}
 	return 0;
 }
@@ -2220,31 +2223,6 @@ static void __cpuinit rcu_prepare_kthreads(int cpu)
 }
 
 /*
- * kthread_create() creates threads in TASK_UNINTERRUPTIBLE state,
- * but the RCU threads are woken on demand, and if demand is low this
- * could be a while triggering the hung task watchdog.
- *
- * In order to avoid this, poke all tasks once the CPU is fully
- * up and running.
- */
-static void __cpuinit rcu_online_kthreads(int cpu)
-{
-	struct rcu_data *rdp = per_cpu_ptr(rcu_state->rda, cpu);
-	struct rcu_node *rnp = rdp->mynode;
-	struct task_struct *t;
-
-	t = per_cpu(rcu_cpu_kthread_task, cpu);
-	if (t)
-		wake_up_process(t);
-
-	t = rnp->node_kthread_task;
-	if (t)
-		wake_up_process(t);
-
-	rcu_wake_one_boost_kthread(rnp);
-}
-
-/*
  * Handle CPU online/offline notification events.
  */
 static int __cpuinit rcu_cpu_notify(struct notifier_block *self,
@@ -2261,7 +2239,6 @@ static int __cpuinit rcu_cpu_notify(struct notifier_block *self,
 		rcu_prepare_kthreads(cpu);
 		break;
 	case CPU_ONLINE:
-		rcu_online_kthreads(cpu);
 	case CPU_DOWN_FAILED:
 		rcu_node_kthread_setaffinity(rnp, -1);
 		rcu_cpu_kthread_setrt(cpu, 1);
diff --git a/kernel/rcutree_plugin.h b/kernel/rcutree_plugin.h
index c8bff30..ea2e2fb 100644
--- a/kernel/rcutree_plugin.h
+++ b/kernel/rcutree_plugin.h
@@ -1299,15 +1299,10 @@ static int __cpuinit rcu_spawn_one_boost_kthread(struct rcu_state *rsp,
 	raw_spin_unlock_irqrestore(&rnp->lock, flags);
 	sp.sched_priority = RCU_KTHREAD_PRIO;
 	sched_setscheduler_nocheck(t, SCHED_FIFO, &sp);
+	wake_up_process(t); /* get to TASK_INTERRUPTIBLE quickly. */
 	return 0;
 }
 
-static void __cpuinit rcu_wake_one_boost_kthread(struct rcu_node *rnp)
-{
-	if (rnp->boost_kthread_task)
-		wake_up_process(rnp->boost_kthread_task);
-}
-
 #else /* #ifdef CONFIG_RCU_BOOST */
 
 static void rcu_initiate_boost(struct rcu_node *rnp, unsigned long flags)
@@ -1331,10 +1326,6 @@ static int __cpuinit rcu_spawn_one_boost_kthread(struct rcu_state *rsp,
 	return 0;
 }
 
-static void __cpuinit rcu_wake_one_boost_kthread(struct rcu_node *rnp)
-{
-}
-
 #endif /* #else #ifdef CONFIG_RCU_BOOST */
 
 #ifndef CONFIG_SMP
-- 
1.7.3.2

Re: [PATCH tip/core/rcu 01/28] rcu: Simplify curing of load woes

[Peter Zijlstra]

On Wed, 2011-06-08 at 12:29 -0700, Paul E. McKenney wrote:
> Make the functions creating the kthreads wake them up.  Leverage the
> fact that the per-node and boost kthreads can run anywhere, thus
> dispensing with the need to wake them up once the incoming CPU has
> gone fully online.

Indeed, I failed to notice the node and boost threads weren't bound.

> Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
> ---
>  kernel/rcutree.c        |   65 +++++++++++++++-------------------------------
>  kernel/rcutree_plugin.h |   11 +-------
>  2 files changed, 22 insertions(+), 54 deletions(-)
> 
> diff --git a/kernel/rcutree.c b/kernel/rcutree.c
> index 4cc6a94..36e79d2 100644
> --- a/kernel/rcutree.c
> +++ b/kernel/rcutree.c
> @@ -1634,6 +1634,20 @@ static int rcu_cpu_kthread(void *arg)
>   * to manipulate rcu_cpu_kthread_task.  There might be another CPU
>   * attempting to access it during boot, but the locking in kthread_bind()
>   * will enforce sufficient ordering.
> + *
> + * Please note that we cannot simply refuse to wake up the per-CPU
> + * kthread because kthreads are created in TASK_UNINTERRUPTIBLE state,
> + * which can result in softlockup complaints if the task ends up being
> + * idle for more than a couple of minutes.
> + *
> + * However, please note also that we cannot bind the per-CPU kthread to its
> + * CPU until that CPU is fully online.  We also cannot wait until the
> + * CPU is fully online before we create its per-CPU kthread, as this would
> + * deadlock the system when CPU notifiers tried waiting for grace
> + * periods.  So we bind the per-CPU kthread to its CPU only if the CPU
> + * is online.  If its CPU is not yet fully online, then the code in
> + * rcu_cpu_kthread() will wait until it is fully online, and then do
> + * the binding.
>   */
>  static int __cpuinit rcu_spawn_one_cpu_kthread(int cpu)
>  {
> @@ -1646,12 +1660,14 @@ static int __cpuinit rcu_spawn_one_cpu_kthread(int cpu)
>         t = kthread_create(rcu_cpu_kthread, (void *)(long)cpu, "rcuc%d", cpu);
>         if (IS_ERR(t))
>                 return PTR_ERR(t);
> -       kthread_bind(t, cpu);
> +       if (cpu_online(cpu))
> +               kthread_bind(t, cpu);
>         per_cpu(rcu_cpu_kthread_cpu, cpu) = cpu;
>         WARN_ON_ONCE(per_cpu(rcu_cpu_kthread_task, cpu) != NULL);
> -       per_cpu(rcu_cpu_kthread_task, cpu) = t;
>         sp.sched_priority = RCU_KTHREAD_PRIO;
>         sched_setscheduler_nocheck(t, SCHED_FIFO, &sp);
> +       per_cpu(rcu_cpu_kthread_task, cpu) = t;
> +       wake_up_process(t); /* Get to TASK_INTERRUPTIBLE quickly. */
>         return 0;
>  } 

I'm not quite seeing how this is working though, I cannot find any code
in rcu_cpu_kthread() that sets the thread affinity (not a hunk in this
patch that adds it).

Re: [PATCH tip/core/rcu 01/28] rcu: Simplify curing of load woes

[Paul E. McKenney]

On Fri, Jun 10, 2011 at 04:18:32PM +0200, Peter Zijlstra wrote:
> On Wed, 2011-06-08 at 12:29 -0700, Paul E. McKenney wrote:
> > Make the functions creating the kthreads wake them up.  Leverage the
> > fact that the per-node and boost kthreads can run anywhere, thus
> > dispensing with the need to wake them up once the incoming CPU has
> > gone fully online.
> 
> Indeed, I failed to notice the node and boost threads weren't bound.

Hey, you did the big fix, so I cannot complain about doing a little
cleanup!  ;-)

> > Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
> > ---
> >  kernel/rcutree.c        |   65 +++++++++++++++-------------------------------
> >  kernel/rcutree_plugin.h |   11 +-------
> >  2 files changed, 22 insertions(+), 54 deletions(-)
> > 
> > diff --git a/kernel/rcutree.c b/kernel/rcutree.c
> > index 4cc6a94..36e79d2 100644
> > --- a/kernel/rcutree.c
> > +++ b/kernel/rcutree.c
> > @@ -1634,6 +1634,20 @@ static int rcu_cpu_kthread(void *arg)
> >   * to manipulate rcu_cpu_kthread_task.  There might be another CPU
> >   * attempting to access it during boot, but the locking in kthread_bind()
> >   * will enforce sufficient ordering.
> > + *
> > + * Please note that we cannot simply refuse to wake up the per-CPU
> > + * kthread because kthreads are created in TASK_UNINTERRUPTIBLE state,
> > + * which can result in softlockup complaints if the task ends up being
> > + * idle for more than a couple of minutes.
> > + *
> > + * However, please note also that we cannot bind the per-CPU kthread to its
> > + * CPU until that CPU is fully online.  We also cannot wait until the
> > + * CPU is fully online before we create its per-CPU kthread, as this would
> > + * deadlock the system when CPU notifiers tried waiting for grace
> > + * periods.  So we bind the per-CPU kthread to its CPU only if the CPU
> > + * is online.  If its CPU is not yet fully online, then the code in
> > + * rcu_cpu_kthread() will wait until it is fully online, and then do
> > + * the binding.
> >   */
> >  static int __cpuinit rcu_spawn_one_cpu_kthread(int cpu)
> >  {
> > @@ -1646,12 +1660,14 @@ static int __cpuinit rcu_spawn_one_cpu_kthread(int cpu)
> >         t = kthread_create(rcu_cpu_kthread, (void *)(long)cpu, "rcuc%d", cpu);
> >         if (IS_ERR(t))
> >                 return PTR_ERR(t);
> > -       kthread_bind(t, cpu);
> > +       if (cpu_online(cpu))
> > +               kthread_bind(t, cpu);
> >         per_cpu(rcu_cpu_kthread_cpu, cpu) = cpu;
> >         WARN_ON_ONCE(per_cpu(rcu_cpu_kthread_task, cpu) != NULL);
> > -       per_cpu(rcu_cpu_kthread_task, cpu) = t;
> >         sp.sched_priority = RCU_KTHREAD_PRIO;
> >         sched_setscheduler_nocheck(t, SCHED_FIFO, &sp);
> > +       per_cpu(rcu_cpu_kthread_task, cpu) = t;
> > +       wake_up_process(t); /* Get to TASK_INTERRUPTIBLE quickly. */
> >         return 0;
> >  } 
> 
> I'm not quite seeing how this is working though, I cannot find any code
> in rcu_cpu_kthread() that sets the thread affinity (not a hunk in this
> patch that adds it).

This happens in rcu_cpu_kthread_should_stop(), which is called from
rcu_cpu_kthread() before it does any real work.

Here it is:

static int rcu_cpu_kthread_should_stop(int cpu)
{
	while (cpu_is_offline(cpu) ||
	       !cpumask_equal(&current->cpus_allowed, cpumask_of(cpu)) ||
	       smp_processor_id() != cpu) {
		if (kthread_should_stop())
			return 1;
		per_cpu(rcu_cpu_kthread_status, cpu) = RCU_KTHREAD_OFFCPU;
		per_cpu(rcu_cpu_kthread_cpu, cpu) = raw_smp_processor_id();
		local_bh_enable();
		schedule_timeout_uninterruptible(1);
		if (!cpumask_equal(&current->cpus_allowed, cpumask_of(cpu)))
			set_cpus_allowed_ptr(current, cpumask_of(cpu));
		local_bh_disable();
	}
	per_cpu(rcu_cpu_kthread_cpu, cpu) = cpu;
	return 0;
}

Thoughts?

							Thanx, Paul

[PATCH tip/core/rcu 02/28] rcu: Use kthread_create_on_node()

[Paul E. McKenney]

From: Eric Dumazet <eric.dumazet@gmail.com>

Commit a26ac2455ffc (move TREE_RCU from softirq to kthread) added
per-CPU kthreads.  However, kthread creation uses kthread_create(), which
can put the kthread's stack and task struct on the wrong NUMA node.
Therefore, use kthread_create_on_node() instead of kthread_create()
so that the stacks and task structs are placed on the correct NUMA node.

A similar change was carried out in commit 94dcf29a11b3 (kthread:
use kthread_create_on_node()).

Also change rcutorture's priority-boost-test kthread creation.

Signed-off-by: Eric Dumazet <eric.dumazet@gmail.com>
CC: Tejun Heo <tj@kernel.org>
CC: Rusty Russell <rusty@rustcorp.com.au>
CC: Andrew Morton <akpm@linux-foundation.org>
CC: Andi Kleen <ak@linux.intel.com>
CC: Ingo Molnar <mingo@elte.hu>
Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
---
 kernel/rcutorture.c |    3 ++-
 kernel/rcutree.c    |    5 ++++-
 2 files changed, 6 insertions(+), 2 deletions(-)

diff --git a/kernel/rcutorture.c b/kernel/rcutorture.c
index 2e138db..40d9ed2 100644
--- a/kernel/rcutorture.c
+++ b/kernel/rcutorture.c
@@ -1282,7 +1282,8 @@ static int rcutorture_booster_init(int cpu)
 	/* Don't allow time recalculation while creating a new task. */
 	mutex_lock(&boost_mutex);
 	VERBOSE_PRINTK_STRING("Creating rcu_torture_boost task");
-	boost_tasks[cpu] = kthread_create(rcu_torture_boost, NULL,
+	boost_tasks[cpu] = kthread_create_on_node(rcu_torture_boost, NULL,
+						  cpu_to_node(cpu),
 					  "rcu_torture_boost");
 	if (IS_ERR(boost_tasks[cpu])) {
 		retval = PTR_ERR(boost_tasks[cpu]);
diff --git a/kernel/rcutree.c b/kernel/rcutree.c
index 36e79d2..c67b9f4 100644
--- a/kernel/rcutree.c
+++ b/kernel/rcutree.c
@@ -1657,7 +1657,10 @@ static int __cpuinit rcu_spawn_one_cpu_kthread(int cpu)
 	if (!rcu_kthreads_spawnable ||
 	    per_cpu(rcu_cpu_kthread_task, cpu) != NULL)
 		return 0;
-	t = kthread_create(rcu_cpu_kthread, (void *)(long)cpu, "rcuc%d", cpu);
+	t = kthread_create_on_node(rcu_cpu_kthread,
+				   (void *)(long)cpu,
+				   cpu_to_node(cpu),
+				   "rcuc%d", cpu);
 	if (IS_ERR(t))
 		return PTR_ERR(t);
 	if (cpu_online(cpu))
-- 
1.7.3.2

[PATCH tip/core/rcu 03/28] rcu: Streamline code produced by __rcu_read_unlock()

[Paul E. McKenney]

Given some common flag combinations, particularly -Os, gcc will inline
rcu_read_unlock_special() despite its being in an unlikely() clause.
Use noline to prohibit this misoptimization.

In addition, move the second barrier() in __rcu_read_unlock() so that
it is not on the common-case code path.  This will allow the compiler to
generate better code for the common-case path through __rcu_read_unlock().

Finally, fix up whitespace in kernel/lockdep.c to keep checkpatch happy.

Suggested-by: Linus Torvalds <torvalds@linux-foundation.org>
Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
---
 kernel/rcutree_plugin.h |   12 ++++++------
 1 files changed, 6 insertions(+), 6 deletions(-)

diff --git a/kernel/rcutree_plugin.h b/kernel/rcutree_plugin.h
index ea2e2fb..40a6db7 100644
--- a/kernel/rcutree_plugin.h
+++ b/kernel/rcutree_plugin.h
@@ -284,7 +284,7 @@ static struct list_head *rcu_next_node_entry(struct task_struct *t,
  * notify RCU core processing or task having blocked during the RCU
  * read-side critical section.
  */
-static void rcu_read_unlock_special(struct task_struct *t)
+static noinline void rcu_read_unlock_special(struct task_struct *t)
 {
 	int empty;
 	int empty_exp;
@@ -387,11 +387,11 @@ void __rcu_read_unlock(void)
 	struct task_struct *t = current;
 
 	barrier();  /* needed if we ever invoke rcu_read_unlock in rcutree.c */
-	--t->rcu_read_lock_nesting;
-	barrier();  /* decrement before load of ->rcu_read_unlock_special */
-	if (t->rcu_read_lock_nesting == 0 &&
-	    unlikely(ACCESS_ONCE(t->rcu_read_unlock_special)))
-		rcu_read_unlock_special(t);
+	if (--t->rcu_read_lock_nesting == 0) {
+		barrier();  /* decr before ->rcu_read_unlock_special load */
+		if (unlikely(ACCESS_ONCE(t->rcu_read_unlock_special)))
+			rcu_read_unlock_special(t);
+	}
 #ifdef CONFIG_PROVE_LOCKING
 	WARN_ON_ONCE(ACCESS_ONCE(t->rcu_read_lock_nesting) < 0);
 #endif /* #ifdef CONFIG_PROVE_LOCKING */
-- 
1.7.3.2

Re: [PATCH tip/core/rcu 03/28] rcu: Streamline code produced by __rcu_read_unlock()

[Mathieu Desnoyers]

* Paul E. McKenney (paulmck@linux.vnet.ibm.com) wrote:
> Given some common flag combinations, particularly -Os, gcc will inline
> rcu_read_unlock_special() despite its being in an unlikely() clause.
> Use noline to prohibit this misoptimization.

noline -> noinline

> 
> In addition, move the second barrier() in __rcu_read_unlock() so that
> it is not on the common-case code path.  This will allow the compiler to
> generate better code for the common-case path through __rcu_read_unlock().
> 
> Finally, fix up whitespace in kernel/lockdep.c to keep checkpatch happy.

This cleanup probably moved to a separate patch, but this comment line
did not follow.

Other than that, feel free to add my

Acked-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>

> 
> Suggested-by: Linus Torvalds <torvalds@linux-foundation.org>
> Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
> ---
>  kernel/rcutree_plugin.h |   12 ++++++------
>  1 files changed, 6 insertions(+), 6 deletions(-)
> 
> diff --git a/kernel/rcutree_plugin.h b/kernel/rcutree_plugin.h
> index ea2e2fb..40a6db7 100644
> --- a/kernel/rcutree_plugin.h
> +++ b/kernel/rcutree_plugin.h
> @@ -284,7 +284,7 @@ static struct list_head *rcu_next_node_entry(struct task_struct *t,
>   * notify RCU core processing or task having blocked during the RCU
>   * read-side critical section.
>   */
> -static void rcu_read_unlock_special(struct task_struct *t)
> +static noinline void rcu_read_unlock_special(struct task_struct *t)
>  {
>  	int empty;
>  	int empty_exp;
> @@ -387,11 +387,11 @@ void __rcu_read_unlock(void)
>  	struct task_struct *t = current;
>  
>  	barrier();  /* needed if we ever invoke rcu_read_unlock in rcutree.c */
> -	--t->rcu_read_lock_nesting;
> -	barrier();  /* decrement before load of ->rcu_read_unlock_special */
> -	if (t->rcu_read_lock_nesting == 0 &&
> -	    unlikely(ACCESS_ONCE(t->rcu_read_unlock_special)))
> -		rcu_read_unlock_special(t);
> +	if (--t->rcu_read_lock_nesting == 0) {
> +		barrier();  /* decr before ->rcu_read_unlock_special load */
> +		if (unlikely(ACCESS_ONCE(t->rcu_read_unlock_special)))
> +			rcu_read_unlock_special(t);
> +	}
>  #ifdef CONFIG_PROVE_LOCKING
>  	WARN_ON_ONCE(ACCESS_ONCE(t->rcu_read_lock_nesting) < 0);
>  #endif /* #ifdef CONFIG_PROVE_LOCKING */
> -- 
> 1.7.3.2
> 

-- 
Mathieu Desnoyers
Operating System Efficiency R&D Consultant
EfficiOS Inc.
http://www.efficios.com

Re: [PATCH tip/core/rcu 03/28] rcu: Streamline code produced by __rcu_read_unlock()

[Paul E. McKenney]

On Wed, Jun 08, 2011 at 07:17:17PM -0400, Mathieu Desnoyers wrote:
> * Paul E. McKenney (paulmck@linux.vnet.ibm.com) wrote:
> > Given some common flag combinations, particularly -Os, gcc will inline
> > rcu_read_unlock_special() despite its being in an unlikely() clause.
> > Use noline to prohibit this misoptimization.
> 
> noline -> noinline

Good eyes, fixed!

> > In addition, move the second barrier() in __rcu_read_unlock() so that
> > it is not on the common-case code path.  This will allow the compiler to
> > generate better code for the common-case path through __rcu_read_unlock().
> > 
> > Finally, fix up whitespace in kernel/lockdep.c to keep checkpatch happy.
> 
> This cleanup probably moved to a separate patch, but this comment line
> did not follow.

Indeed -- I have removed it.

> Other than that, feel free to add my
> 
> Acked-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>

Thank you!

						Thanx, Paul

> > Suggested-by: Linus Torvalds <torvalds@linux-foundation.org>
> > Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
> > ---
> >  kernel/rcutree_plugin.h |   12 ++++++------
> >  1 files changed, 6 insertions(+), 6 deletions(-)
> > 
> > diff --git a/kernel/rcutree_plugin.h b/kernel/rcutree_plugin.h
> > index ea2e2fb..40a6db7 100644
> > --- a/kernel/rcutree_plugin.h
> > +++ b/kernel/rcutree_plugin.h
> > @@ -284,7 +284,7 @@ static struct list_head *rcu_next_node_entry(struct task_struct *t,
> >   * notify RCU core processing or task having blocked during the RCU
> >   * read-side critical section.
> >   */
> > -static void rcu_read_unlock_special(struct task_struct *t)
> > +static noinline void rcu_read_unlock_special(struct task_struct *t)
> >  {
> >  	int empty;
> >  	int empty_exp;
> > @@ -387,11 +387,11 @@ void __rcu_read_unlock(void)
> >  	struct task_struct *t = current;
> >  
> >  	barrier();  /* needed if we ever invoke rcu_read_unlock in rcutree.c */
> > -	--t->rcu_read_lock_nesting;
> > -	barrier();  /* decrement before load of ->rcu_read_unlock_special */
> > -	if (t->rcu_read_lock_nesting == 0 &&
> > -	    unlikely(ACCESS_ONCE(t->rcu_read_unlock_special)))
> > -		rcu_read_unlock_special(t);
> > +	if (--t->rcu_read_lock_nesting == 0) {
> > +		barrier();  /* decr before ->rcu_read_unlock_special load */
> > +		if (unlikely(ACCESS_ONCE(t->rcu_read_unlock_special)))
> > +			rcu_read_unlock_special(t);
> > +	}
> >  #ifdef CONFIG_PROVE_LOCKING
> >  	WARN_ON_ONCE(ACCESS_ONCE(t->rcu_read_lock_nesting) < 0);
> >  #endif /* #ifdef CONFIG_PROVE_LOCKING */
> > -- 
> > 1.7.3.2
> > 
> 
> -- 
> Mathieu Desnoyers
> Operating System Efficiency R&D Consultant
> EfficiOS Inc.
> http://www.efficios.com

Re: [PATCH tip/core/rcu 03/28] rcu: Streamline code produced by __rcu_read_unlock()

[Josh Triplett]

On Wed, Jun 08, 2011 at 12:29:42PM -0700, Paul E. McKenney wrote:
> Given some common flag combinations, particularly -Os, gcc will inline
> rcu_read_unlock_special() despite its being in an unlikely() clause.
> Use noline to prohibit this misoptimization.

If rcu_read_unlock_special only gets called in the one place, why does
it hurt to inline it, as long as the inlined code stays on the cold
path?  Might as well remove the overhead of the function call.

- Josh Triplett

Re: [PATCH tip/core/rcu 03/28] rcu: Streamline code produced by __rcu_read_unlock()

[Paul E. McKenney]

On Wed, Jun 08, 2011 at 08:45:07PM -0700, Josh Triplett wrote:
> On Wed, Jun 08, 2011 at 12:29:42PM -0700, Paul E. McKenney wrote:
> > Given some common flag combinations, particularly -Os, gcc will inline
> > rcu_read_unlock_special() despite its being in an unlikely() clause.
> > Use noline to prohibit this misoptimization.
> 
> If rcu_read_unlock_special only gets called in the one place, why does
> it hurt to inline it, as long as the inlined code stays on the cold
> path?  Might as well remove the overhead of the function call.

One problem with inlining rcu_read_unlock_special() is that it
increases the icache footprint on the fast path.  Because the call to
rcu_read_unlock_special() occurs only rarely (if the reader blocked
or took too long), it is better to take the overhead of the call in
the uncommon case than to take the additional cache-miss overhead
in the common case.

Or did I miss your point?

							Thanx, Paul

Re: [PATCH tip/core/rcu 03/28] rcu: Streamline code produced by __rcu_read_unlock()

[Lai Jiangshan]

On 06/09/2011 03:29 AM, Paul E. McKenney wrote:
> Given some common flag combinations, particularly -Os, gcc will inline
> rcu_read_unlock_special() despite its being in an unlikely() clause.
> Use noline to prohibit this misoptimization.
> 
> In addition, move the second barrier() in __rcu_read_unlock() so that
> it is not on the common-case code path.  This will allow the compiler to
> generate better code for the common-case path through __rcu_read_unlock().
> 
> Finally, fix up whitespace in kernel/lockdep.c to keep checkpatch happy.
> 
> Suggested-by: Linus Torvalds <torvalds@linux-foundation.org>
> Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
> ---
>  kernel/rcutree_plugin.h |   12 ++++++------
>  1 files changed, 6 insertions(+), 6 deletions(-)
> 
> diff --git a/kernel/rcutree_plugin.h b/kernel/rcutree_plugin.h
> index ea2e2fb..40a6db7 100644
> --- a/kernel/rcutree_plugin.h
> +++ b/kernel/rcutree_plugin.h
> @@ -284,7 +284,7 @@ static struct list_head *rcu_next_node_entry(struct task_struct *t,
>   * notify RCU core processing or task having blocked during the RCU
>   * read-side critical section.
>   */
> -static void rcu_read_unlock_special(struct task_struct *t)
> +static noinline void rcu_read_unlock_special(struct task_struct *t)
>  {
>  	int empty;
>  	int empty_exp;
> @@ -387,11 +387,11 @@ void __rcu_read_unlock(void)
>  	struct task_struct *t = current;
>  
>  	barrier();  /* needed if we ever invoke rcu_read_unlock in rcutree.c */
> -	--t->rcu_read_lock_nesting;
> -	barrier();  /* decrement before load of ->rcu_read_unlock_special */
> -	if (t->rcu_read_lock_nesting == 0 &&
> -	    unlikely(ACCESS_ONCE(t->rcu_read_unlock_special)))
> -		rcu_read_unlock_special(t);
> +	if (--t->rcu_read_lock_nesting == 0) {

> +		barrier();  /* decr before ->rcu_read_unlock_special load */

Since ACCESS_ONCE() is used for loading ->rcu_read_unlock_special, is the previous
barrier() still needed?

> +		if (unlikely(ACCESS_ONCE(t->rcu_read_unlock_special)))
> +			rcu_read_unlock_special(t);
> +	}
>  #ifdef CONFIG_PROVE_LOCKING
>  	WARN_ON_ONCE(ACCESS_ONCE(t->rcu_read_lock_nesting) < 0);
>  #endif /* #ifdef CONFIG_PROVE_LOCKING */

Re: [PATCH tip/core/rcu 03/28] rcu: Streamline code produced by __rcu_read_unlock()

[Paul E. McKenney]

On Fri, Jun 10, 2011 at 03:14:29PM +0800, Lai Jiangshan wrote:
> On 06/09/2011 03:29 AM, Paul E. McKenney wrote:
> > Given some common flag combinations, particularly -Os, gcc will inline
> > rcu_read_unlock_special() despite its being in an unlikely() clause.
> > Use noline to prohibit this misoptimization.
> > 
> > In addition, move the second barrier() in __rcu_read_unlock() so that
> > it is not on the common-case code path.  This will allow the compiler to
> > generate better code for the common-case path through __rcu_read_unlock().
> > 
> > Finally, fix up whitespace in kernel/lockdep.c to keep checkpatch happy.
> > 
> > Suggested-by: Linus Torvalds <torvalds@linux-foundation.org>
> > Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
> > ---
> >  kernel/rcutree_plugin.h |   12 ++++++------
> >  1 files changed, 6 insertions(+), 6 deletions(-)
> > 
> > diff --git a/kernel/rcutree_plugin.h b/kernel/rcutree_plugin.h
> > index ea2e2fb..40a6db7 100644
> > --- a/kernel/rcutree_plugin.h
> > +++ b/kernel/rcutree_plugin.h
> > @@ -284,7 +284,7 @@ static struct list_head *rcu_next_node_entry(struct task_struct *t,
> >   * notify RCU core processing or task having blocked during the RCU
> >   * read-side critical section.
> >   */
> > -static void rcu_read_unlock_special(struct task_struct *t)
> > +static noinline void rcu_read_unlock_special(struct task_struct *t)
> >  {
> >  	int empty;
> >  	int empty_exp;
> > @@ -387,11 +387,11 @@ void __rcu_read_unlock(void)
> >  	struct task_struct *t = current;
> >  
> >  	barrier();  /* needed if we ever invoke rcu_read_unlock in rcutree.c */
> > -	--t->rcu_read_lock_nesting;
> > -	barrier();  /* decrement before load of ->rcu_read_unlock_special */
> > -	if (t->rcu_read_lock_nesting == 0 &&
> > -	    unlikely(ACCESS_ONCE(t->rcu_read_unlock_special)))
> > -		rcu_read_unlock_special(t);
> > +	if (--t->rcu_read_lock_nesting == 0) {
> 
> > +		barrier();  /* decr before ->rcu_read_unlock_special load */
> 
> Since ACCESS_ONCE() is used for loading ->rcu_read_unlock_special, is the previous
> barrier() still needed?

It doesn't really matter until we can inline __rcu_read_unlock(), but
hopefully that day is coming soon.  So...

The concern is for cases where the compiler can see __rcu_read_lock() and
__rcu_read_unlock().  The compiler would then be within its rights to
cancel the increments and decrements of t->rcu_read_lock_nesting against
each other, which could turn a loop containing an RCU read-side critical
section into one big long critical section.

We could do --ACCESS_ONCE(t->rcu_read_lock_nesting), but that generates
lousy code on x86.  So, is there a way to make the compiler forget only
about t->rcu_read_lock_nesting rather than about all variables?

							Thanx, Paul

> > +		if (unlikely(ACCESS_ONCE(t->rcu_read_unlock_special)))
> > +			rcu_read_unlock_special(t);
> > +	}
> >  #ifdef CONFIG_PROVE_LOCKING
> >  	WARN_ON_ONCE(ACCESS_ONCE(t->rcu_read_lock_nesting) < 0);
> >  #endif /* #ifdef CONFIG_PROVE_LOCKING */
> 

[PATCH tip/core/rcu 04/28] rcu: Restore checks for blocking in RCU read-side critical sections

[Paul E. McKenney]

Long ago, using TREE_RCU with PREEMPT would result in "scheduling
while atomic" diagnostics if you blocked in an RCU read-side critical
section.  However, PREEMPT now implies TREE_PREEMPT_RCU, which defeats
this diagnostic.  This commit therefore adds a replacement diagnostic
based on PROVE_RCU.

Because rcu_lockdep_assert() and lockdep_rcu_dereference() are now being
used for things that have nothing to do with rcu_dereference(), rename
lockdep_rcu_dereference() to lockdep_rcu_suspicious() and add a third
argument that is a string indicating what is suspicious.  This third
argument is passed in from a new third argument to rcu_lockdep_assert().
Update all calls to rcu_lockdep_assert() to add an informative third
argument.

Finally, add a pair of rcu_lockdep_assert() calls from within
rcu_note_context_switch(), one complaining if a context switch occurs
in an RCU-bh read-side critical section and another complaining if a
context switch occurs in an RCU-sched read-side critical section.
These are present only if the PROVE_RCU kernel parameter is enabled.

Again, you must enable PROVE_RCU to see these new diagnostics.  But you
are enabling PROVE_RCU to check out new RCU uses in any case, aren't you?

Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
---
 include/linux/lockdep.h  |    2 +-
 include/linux/rcupdate.h |   28 ++++++++++++---
 kernel/lockdep.c         |   84 +++++++++++++++++++++++++--------------------
 kernel/pid.c             |    4 ++-
 kernel/sched.c           |    2 +
 5 files changed, 75 insertions(+), 45 deletions(-)

diff --git a/include/linux/lockdep.h b/include/linux/lockdep.h
index 4aef1dd..bfa66e7 100644
--- a/include/linux/lockdep.h
+++ b/include/linux/lockdep.h
@@ -545,7 +545,7 @@ do {									\
 #endif
 
 #ifdef CONFIG_PROVE_RCU
-extern void lockdep_rcu_dereference(const char *file, const int line);
+void lockdep_rcu_suspicious(const char *file, const int line, const char *s);
 #endif
 
 #endif /* __LINUX_LOCKDEP_H */
diff --git a/include/linux/rcupdate.h b/include/linux/rcupdate.h
index 99f9aa7..8be0433 100644
--- a/include/linux/rcupdate.h
+++ b/include/linux/rcupdate.h
@@ -297,19 +297,31 @@ extern int rcu_my_thread_group_empty(void);
 /**
  * rcu_lockdep_assert - emit lockdep splat if specified condition not met
  * @c: condition to check
+ * @s: informative message
  */
-#define rcu_lockdep_assert(c)						\
+#define rcu_lockdep_assert(c, s)					\
 	do {								\
 		static bool __warned;					\
 		if (debug_lockdep_rcu_enabled() && !__warned && !(c)) {	\
 			__warned = true;				\
-			lockdep_rcu_dereference(__FILE__, __LINE__);	\
+			lockdep_rcu_suspicious(__FILE__, __LINE__, s);	\
 		}							\
 	} while (0)
 
+#define rcu_sleep_check()						\
+	do {								\
+		rcu_lockdep_assert(!lock_is_held(&rcu_bh_lock_map),	\
+				   "Illegal context switch in RCU-bh"	\
+				   " read-side critical section");	\
+		rcu_lockdep_assert(!lock_is_held(&rcu_sched_lock_map),	\
+				   "Illegal context switch in RCU-sched"\
+				   " read-side critical section");	\
+	} while (0)
+
 #else /* #ifdef CONFIG_PROVE_RCU */
 
-#define rcu_lockdep_assert(c) do { } while (0)
+#define rcu_lockdep_assert(c, s) do { } while (0)
+#define rcu_sleep_check() do { } while (0)
 
 #endif /* #else #ifdef CONFIG_PROVE_RCU */
 
@@ -338,14 +350,16 @@ extern int rcu_my_thread_group_empty(void);
 #define __rcu_dereference_check(p, c, space) \
 	({ \
 		typeof(*p) *_________p1 = (typeof(*p)*__force )ACCESS_ONCE(p); \
-		rcu_lockdep_assert(c); \
+		rcu_lockdep_assert(c, "suspicious rcu_dereference_check()" \
+				      " usage"); \
 		rcu_dereference_sparse(p, space); \
 		smp_read_barrier_depends(); \
 		((typeof(*p) __force __kernel *)(_________p1)); \
 	})
 #define __rcu_dereference_protected(p, c, space) \
 	({ \
-		rcu_lockdep_assert(c); \
+		rcu_lockdep_assert(c, "suspicious rcu_dereference_protected()" \
+				      " usage"); \
 		rcu_dereference_sparse(p, space); \
 		((typeof(*p) __force __kernel *)(p)); \
 	})
@@ -359,7 +373,9 @@ extern int rcu_my_thread_group_empty(void);
 #define __rcu_dereference_index_check(p, c) \
 	({ \
 		typeof(p) _________p1 = ACCESS_ONCE(p); \
-		rcu_lockdep_assert(c); \
+		rcu_lockdep_assert(c, \
+				   "suspicious rcu_dereference_index_check()" \
+				   " usage"); \
 		smp_read_barrier_depends(); \
 		(_________p1); \
 	})
diff --git a/kernel/lockdep.c b/kernel/lockdep.c
index 53a6895..dbf453d 100644
--- a/kernel/lockdep.c
+++ b/kernel/lockdep.c
@@ -1067,10 +1067,11 @@ print_circular_bug_header(struct lock_list *entry, unsigned int depth,
 	if (debug_locks_silent)
 		return 0;
 
-	printk("\n=======================================================\n");
-	printk(  "[ INFO: possible circular locking dependency detected ]\n");
+	printk("\n");
+	printk("======================================================\n");
+	printk("[ INFO: possible circular locking dependency detected ]\n");
 	print_kernel_version();
-	printk(  "-------------------------------------------------------\n");
+	printk("-------------------------------------------------------\n");
 	printk("%s/%d is trying to acquire lock:\n",
 		curr->comm, task_pid_nr(curr));
 	print_lock(check_src);
@@ -1340,11 +1341,12 @@ print_bad_irq_dependency(struct task_struct *curr,
 	if (!debug_locks_off_graph_unlock() || debug_locks_silent)
 		return 0;
 
-	printk("\n======================================================\n");
-	printk(  "[ INFO: %s-safe -> %s-unsafe lock order detected ]\n",
+	printk("\n");
+	printk("======================================================\n");
+	printk("[ INFO: %s-safe -> %s-unsafe lock order detected ]\n",
 		irqclass, irqclass);
 	print_kernel_version();
-	printk(  "------------------------------------------------------\n");
+	printk("------------------------------------------------------\n");
 	printk("%s/%d [HC%u[%lu]:SC%u[%lu]:HE%u:SE%u] is trying to acquire:\n",
 		curr->comm, task_pid_nr(curr),
 		curr->hardirq_context, hardirq_count() >> HARDIRQ_SHIFT,
@@ -1546,10 +1548,11 @@ print_deadlock_bug(struct task_struct *curr, struct held_lock *prev,
 	if (!debug_locks_off_graph_unlock() || debug_locks_silent)
 		return 0;
 
-	printk("\n=============================================\n");
-	printk(  "[ INFO: possible recursive locking detected ]\n");
+	printk("\n");
+	printk("=============================================\n");
+	printk("[ INFO: possible recursive locking detected ]\n");
 	print_kernel_version();
-	printk(  "---------------------------------------------\n");
+	printk("---------------------------------------------\n");
 	printk("%s/%d is trying to acquire lock:\n",
 		curr->comm, task_pid_nr(curr));
 	print_lock(next);
@@ -2018,10 +2021,11 @@ print_usage_bug(struct task_struct *curr, struct held_lock *this,
 	if (!debug_locks_off_graph_unlock() || debug_locks_silent)
 		return 0;
 
-	printk("\n=================================\n");
-	printk(  "[ INFO: inconsistent lock state ]\n");
+	printk("\n");
+	printk("=================================\n");
+	printk("[ INFO: inconsistent lock state ]\n");
 	print_kernel_version();
-	printk(  "---------------------------------\n");
+	printk("---------------------------------\n");
 
 	printk("inconsistent {%s} -> {%s} usage.\n",
 		usage_str[prev_bit], usage_str[new_bit]);
@@ -2076,10 +2080,11 @@ print_irq_inversion_bug(struct task_struct *curr,
 	if (!debug_locks_off_graph_unlock() || debug_locks_silent)
 		return 0;
 
-	printk("\n=========================================================\n");
-	printk(  "[ INFO: possible irq lock inversion dependency detected ]\n");
+	printk("\n");
+	printk("=========================================================\n");
+	printk("[ INFO: possible irq lock inversion dependency detected ]\n");
 	print_kernel_version();
-	printk(  "---------------------------------------------------------\n");
+	printk("---------------------------------------------------------\n");
 	printk("%s/%d just changed the state of lock:\n",
 		curr->comm, task_pid_nr(curr));
 	print_lock(this);
@@ -2869,9 +2874,10 @@ print_unlock_inbalance_bug(struct task_struct *curr, struct lockdep_map *lock,
 	if (debug_locks_silent)
 		return 0;
 
-	printk("\n=====================================\n");
-	printk(  "[ BUG: bad unlock balance detected! ]\n");
-	printk(  "-------------------------------------\n");
+	printk("\n");
+	printk("=====================================\n");
+	printk("[ BUG: bad unlock balance detected! ]\n");
+	printk("-------------------------------------\n");
 	printk("%s/%d is trying to release lock (",
 		curr->comm, task_pid_nr(curr));
 	print_lockdep_cache(lock);
@@ -3276,9 +3282,10 @@ print_lock_contention_bug(struct task_struct *curr, struct lockdep_map *lock,
 	if (debug_locks_silent)
 		return 0;
 
-	printk("\n=================================\n");
-	printk(  "[ BUG: bad contention detected! ]\n");
-	printk(  "---------------------------------\n");
+	printk("\n");
+	printk("=================================\n");
+	printk("[ BUG: bad contention detected! ]\n");
+	printk("---------------------------------\n");
 	printk("%s/%d is trying to contend lock (",
 		curr->comm, task_pid_nr(curr));
 	print_lockdep_cache(lock);
@@ -3637,9 +3644,10 @@ print_freed_lock_bug(struct task_struct *curr, const void *mem_from,
 	if (debug_locks_silent)
 		return;
 
-	printk("\n=========================\n");
-	printk(  "[ BUG: held lock freed! ]\n");
-	printk(  "-------------------------\n");
+	printk("\n");
+	printk("=========================\n");
+	printk("[ BUG: held lock freed! ]\n");
+	printk("-------------------------\n");
 	printk("%s/%d is freeing memory %p-%p, with a lock still held there!\n",
 		curr->comm, task_pid_nr(curr), mem_from, mem_to-1);
 	print_lock(hlock);
@@ -3693,9 +3701,10 @@ static void print_held_locks_bug(struct task_struct *curr)
 	if (debug_locks_silent)
 		return;
 
-	printk("\n=====================================\n");
-	printk(  "[ BUG: lock held at task exit time! ]\n");
-	printk(  "-------------------------------------\n");
+	printk("\n");
+	printk("=====================================\n");
+	printk("[ BUG: lock held at task exit time! ]\n");
+	printk("-------------------------------------\n");
 	printk("%s/%d is exiting with locks still held!\n",
 		curr->comm, task_pid_nr(curr));
 	lockdep_print_held_locks(curr);
@@ -3789,16 +3798,17 @@ void lockdep_sys_exit(void)
 	if (unlikely(curr->lockdep_depth)) {
 		if (!debug_locks_off())
 			return;
-		printk("\n================================================\n");
-		printk(  "[ BUG: lock held when returning to user space! ]\n");
-		printk(  "------------------------------------------------\n");
+		printk("\n");
+		printk("================================================\n");
+		printk("[ BUG: lock held when returning to user space! ]\n");
+		printk("------------------------------------------------\n");
 		printk("%s/%d is leaving the kernel with locks still held!\n",
 				curr->comm, curr->pid);
 		lockdep_print_held_locks(curr);
 	}
 }
 
-void lockdep_rcu_dereference(const char *file, const int line)
+void lockdep_rcu_suspicious(const char *file, const int line, const char *s)
 {
 	struct task_struct *curr = current;
 
@@ -3807,15 +3817,15 @@ void lockdep_rcu_dereference(const char *file, const int line)
 		return;
 #endif /* #ifdef CONFIG_PROVE_RCU_REPEATEDLY */
 	/* Note: the following can be executed concurrently, so be careful. */
-	printk("\n===================================================\n");
-	printk(  "[ INFO: suspicious rcu_dereference_check() usage. ]\n");
-	printk(  "---------------------------------------------------\n");
-	printk("%s:%d invoked rcu_dereference_check() without protection!\n",
-			file, line);
+	printk("\n");
+	printk("===============================\n");
+	printk("[ INFO: suspicious RCU usage. ]\n");
+	printk("-------------------------------\n");
+	printk("%s:%d %s!\n", file, line, s);
 	printk("\nother info that might help us debug this:\n\n");
 	printk("\nrcu_scheduler_active = %d, debug_locks = %d\n", rcu_scheduler_active, debug_locks);
 	lockdep_print_held_locks(curr);
 	printk("\nstack backtrace:\n");
 	dump_stack();
 }
-EXPORT_SYMBOL_GPL(lockdep_rcu_dereference);
+EXPORT_SYMBOL_GPL(lockdep_rcu_suspicious);
diff --git a/kernel/pid.c b/kernel/pid.c
index 57a8346..a7577b3 100644
--- a/kernel/pid.c
+++ b/kernel/pid.c
@@ -419,7 +419,9 @@ EXPORT_SYMBOL(pid_task);
  */
 struct task_struct *find_task_by_pid_ns(pid_t nr, struct pid_namespace *ns)
 {
-	rcu_lockdep_assert(rcu_read_lock_held());
+	rcu_lockdep_assert(rcu_read_lock_held(),
+			   "find_task_by_pid_ns() needs rcu_read_lock()"
+			   " protection");
 	return pid_task(find_pid_ns(nr, ns), PIDTYPE_PID);
 }
 
diff --git a/kernel/sched.c b/kernel/sched.c
index 312f8b9..1352dbb 100644
--- a/kernel/sched.c
+++ b/kernel/sched.c
@@ -4021,6 +4021,7 @@ static inline void schedule_debug(struct task_struct *prev)
 	 */
 	if (unlikely(in_atomic_preempt_off() && !prev->exit_state))
 		__schedule_bug(prev);
+	rcu_sleep_check();
 
 	profile_hit(SCHED_PROFILING, __builtin_return_address(0));
 
@@ -8309,6 +8310,7 @@ void __might_sleep(const char *file, int line, int preempt_offset)
 #ifdef in_atomic
 	static unsigned long prev_jiffy;	/* ratelimiting */
 
+	rcu_sleep_check(); /* WARN_ON_ONCE() by default, no rate limit reqd. */
 	if ((preempt_count_equals(preempt_offset) && !irqs_disabled()) ||
 	    system_state != SYSTEM_RUNNING || oops_in_progress)
 		return;
-- 
1.7.3.2

Re: [PATCH tip/core/rcu 04/28] rcu: Restore checks for blocking in RCU read-side critical sections

[Frederic Weisbecker]

On Wed, Jun 08, 2011 at 12:29:43PM -0700, Paul E. McKenney wrote:
> Long ago, using TREE_RCU with PREEMPT would result in "scheduling
> while atomic" diagnostics if you blocked in an RCU read-side critical
> section.  However, PREEMPT now implies TREE_PREEMPT_RCU, which defeats
> this diagnostic.  This commit therefore adds a replacement diagnostic
> based on PROVE_RCU.
> 
> Because rcu_lockdep_assert() and lockdep_rcu_dereference() are now being
> used for things that have nothing to do with rcu_dereference(), rename
> lockdep_rcu_dereference() to lockdep_rcu_suspicious() and add a third
> argument that is a string indicating what is suspicious.  This third
> argument is passed in from a new third argument to rcu_lockdep_assert().
> Update all calls to rcu_lockdep_assert() to add an informative third
> argument.
> 
> Finally, add a pair of rcu_lockdep_assert() calls from within
> rcu_note_context_switch(), one complaining if a context switch occurs
> in an RCU-bh read-side critical section and another complaining if a
> context switch occurs in an RCU-sched read-side critical section.
> These are present only if the PROVE_RCU kernel parameter is enabled.
> 
> Again, you must enable PROVE_RCU to see these new diagnostics.  But you
> are enabling PROVE_RCU to check out new RCU uses in any case, aren't you?
> 
> Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>

So, do you think we can get rid of this patch now that we are going to have CONFIG_ATOMIC_SLEEP
working everywhere?

The last remaining piece we need is to check rcu_preempt_depth() from schedule_debug(),
which does a kind of lightweight might_sleep() check alike.

Re: [PATCH tip/core/rcu 04/28] rcu: Restore checks for blocking in RCU read-side critical sections

[Paul E. McKenney]

On Thu, Jun 09, 2011 at 01:28:35AM +0200, Frederic Weisbecker wrote:
> On Wed, Jun 08, 2011 at 12:29:43PM -0700, Paul E. McKenney wrote:
> > Long ago, using TREE_RCU with PREEMPT would result in "scheduling
> > while atomic" diagnostics if you blocked in an RCU read-side critical
> > section.  However, PREEMPT now implies TREE_PREEMPT_RCU, which defeats
> > this diagnostic.  This commit therefore adds a replacement diagnostic
> > based on PROVE_RCU.
> > 
> > Because rcu_lockdep_assert() and lockdep_rcu_dereference() are now being
> > used for things that have nothing to do with rcu_dereference(), rename
> > lockdep_rcu_dereference() to lockdep_rcu_suspicious() and add a third
> > argument that is a string indicating what is suspicious.  This third
> > argument is passed in from a new third argument to rcu_lockdep_assert().
> > Update all calls to rcu_lockdep_assert() to add an informative third
> > argument.
> > 
> > Finally, add a pair of rcu_lockdep_assert() calls from within
> > rcu_note_context_switch(), one complaining if a context switch occurs
> > in an RCU-bh read-side critical section and another complaining if a
> > context switch occurs in an RCU-sched read-side critical section.
> > These are present only if the PROVE_RCU kernel parameter is enabled.
> > 
> > Again, you must enable PROVE_RCU to see these new diagnostics.  But you
> > are enabling PROVE_RCU to check out new RCU uses in any case, aren't you?
> > 
> > Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
> 
> So, do you think we can get rid of this patch now that we are going to have CONFIG_ATOMIC_SLEEP
> working everywhere?
> 
> The last remaining piece we need is to check rcu_preempt_depth() from schedule_debug(),
> which does a kind of lightweight might_sleep() check alike.

I believe that we need them both.  Your patch provides a lightweight
check.  Mine is way heavier weight (CONFIG_PROVE_RCU is required), but
tells you in what function the offending RCU read-side critical section
was entered.

							Thanx, Paul

Re: [PATCH tip/core/rcu 04/28] rcu: Restore checks for blocking in RCU read-side critical sections

[Frederic Weisbecker]

On Wed, Jun 08, 2011 at 04:46:22PM -0700, Paul E. McKenney wrote:
> On Thu, Jun 09, 2011 at 01:28:35AM +0200, Frederic Weisbecker wrote:
> > On Wed, Jun 08, 2011 at 12:29:43PM -0700, Paul E. McKenney wrote:
> > > Long ago, using TREE_RCU with PREEMPT would result in "scheduling
> > > while atomic" diagnostics if you blocked in an RCU read-side critical
> > > section.  However, PREEMPT now implies TREE_PREEMPT_RCU, which defeats
> > > this diagnostic.  This commit therefore adds a replacement diagnostic
> > > based on PROVE_RCU.
> > > 
> > > Because rcu_lockdep_assert() and lockdep_rcu_dereference() are now being
> > > used for things that have nothing to do with rcu_dereference(), rename
> > > lockdep_rcu_dereference() to lockdep_rcu_suspicious() and add a third
> > > argument that is a string indicating what is suspicious.  This third
> > > argument is passed in from a new third argument to rcu_lockdep_assert().
> > > Update all calls to rcu_lockdep_assert() to add an informative third
> > > argument.
> > > 
> > > Finally, add a pair of rcu_lockdep_assert() calls from within
> > > rcu_note_context_switch(), one complaining if a context switch occurs
> > > in an RCU-bh read-side critical section and another complaining if a
> > > context switch occurs in an RCU-sched read-side critical section.
> > > These are present only if the PROVE_RCU kernel parameter is enabled.
> > > 
> > > Again, you must enable PROVE_RCU to see these new diagnostics.  But you
> > > are enabling PROVE_RCU to check out new RCU uses in any case, aren't you?
> > > 
> > > Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
> > 
> > So, do you think we can get rid of this patch now that we are going to have CONFIG_ATOMIC_SLEEP
> > working everywhere?
> > 
> > The last remaining piece we need is to check rcu_preempt_depth() from schedule_debug(),
> > which does a kind of lightweight might_sleep() check alike.
> 
> I believe that we need them both.  Your patch provides a lightweight
> check.  Mine is way heavier weight (CONFIG_PROVE_RCU is required), but
> tells you in what function the offending RCU read-side critical section
> was entered.

Well, that can be found easily in the stacktrace. But ok.

[PATCH tip/core/rcu 05/28] rcu: Move rcu_head definition to types.h

[Paul E. McKenney]

Take a first step towards untangling Linux kernel header files by
placing the struct rcu_head definition into include/linux/types.h.
Once this is in place, then header files that need only rcu_head
can include types.h rather than rcupdate.h.

Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
---
 include/linux/rcupdate.h |   10 +---------
 include/linux/types.h    |   10 ++++++++++
 2 files changed, 11 insertions(+), 9 deletions(-)

diff --git a/include/linux/rcupdate.h b/include/linux/rcupdate.h
index 8be0433..ad09eb8 100644
--- a/include/linux/rcupdate.h
+++ b/include/linux/rcupdate.h
@@ -64,15 +64,7 @@ static inline void rcutorture_record_progress(unsigned long vernum)
 #define ULONG_CMP_GE(a, b)	(ULONG_MAX / 2 >= (a) - (b))
 #define ULONG_CMP_LT(a, b)	(ULONG_MAX / 2 < (a) - (b))
 
-/**
- * struct rcu_head - callback structure for use with RCU
- * @next: next update requests in a list
- * @func: actual update function to call after the grace period.
- */
-struct rcu_head {
-	struct rcu_head *next;
-	void (*func)(struct rcu_head *head);
-};
+#include <linux/types.h>
 
 /* Exported common interfaces */
 extern void call_rcu_sched(struct rcu_head *head,
diff --git a/include/linux/types.h b/include/linux/types.h
index 176da8c..57a9723 100644
--- a/include/linux/types.h
+++ b/include/linux/types.h
@@ -238,6 +238,16 @@ struct ustat {
 	char			f_fpack[6];
 };
 
+/**
+ * struct rcu_head - callback structure for use with RCU
+ * @next: next update requests in a list
+ * @func: actual update function to call after the grace period.
+ */
+struct rcu_head {
+	struct rcu_head *next;
+	void (*func)(struct rcu_head *head);
+};
+
 #endif	/* __KERNEL__ */
 #endif /*  __ASSEMBLY__ */
 #endif /* _LINUX_TYPES_H */
-- 
1.7.3.2

Re: [PATCH tip/core/rcu 05/28] rcu: Move rcu_head definition to types.h

[Mathieu Desnoyers]

* Paul E. McKenney (paulmck@linux.vnet.ibm.com) wrote:
> Take a first step towards untangling Linux kernel header files by
> placing the struct rcu_head definition into include/linux/types.h.
> Once this is in place, then header files that need only rcu_head
> can include types.h rather than rcupdate.h.
> 
> Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>

Yes! I'm suddenly feeling the temperature drop by a few degrees in
#include hell ;) One detail though: I think rcupdate.h should then
include types.h.

Thanks,

Mathieu


> ---
>  include/linux/rcupdate.h |   10 +---------
>  include/linux/types.h    |   10 ++++++++++
>  2 files changed, 11 insertions(+), 9 deletions(-)
> 
> diff --git a/include/linux/rcupdate.h b/include/linux/rcupdate.h
> index 8be0433..ad09eb8 100644
> --- a/include/linux/rcupdate.h
> +++ b/include/linux/rcupdate.h
> @@ -64,15 +64,7 @@ static inline void rcutorture_record_progress(unsigned long vernum)
>  #define ULONG_CMP_GE(a, b)	(ULONG_MAX / 2 >= (a) - (b))
>  #define ULONG_CMP_LT(a, b)	(ULONG_MAX / 2 < (a) - (b))
>  
> -/**
> - * struct rcu_head - callback structure for use with RCU
> - * @next: next update requests in a list
> - * @func: actual update function to call after the grace period.
> - */
> -struct rcu_head {
> -	struct rcu_head *next;
> -	void (*func)(struct rcu_head *head);
> -};
> +#include <linux/types.h>
>  
>  /* Exported common interfaces */
>  extern void call_rcu_sched(struct rcu_head *head,
> diff --git a/include/linux/types.h b/include/linux/types.h
> index 176da8c..57a9723 100644
> --- a/include/linux/types.h
> +++ b/include/linux/types.h
> @@ -238,6 +238,16 @@ struct ustat {
>  	char			f_fpack[6];
>  };
>  
> +/**
> + * struct rcu_head - callback structure for use with RCU
> + * @next: next update requests in a list
> + * @func: actual update function to call after the grace period.
> + */
> +struct rcu_head {
> +	struct rcu_head *next;
> +	void (*func)(struct rcu_head *head);
> +};
> +
>  #endif	/* __KERNEL__ */
>  #endif /*  __ASSEMBLY__ */
>  #endif /* _LINUX_TYPES_H */
> -- 
> 1.7.3.2
> 

-- 
Mathieu Desnoyers
Operating System Efficiency R&D Consultant
EfficiOS Inc.
http://www.efficios.com

Re: [PATCH tip/core/rcu 05/28] rcu: Move rcu_head definition to types.h

[Paul E. McKenney]

On Wed, Jun 08, 2011 at 07:26:21PM -0400, Mathieu Desnoyers wrote:
> * Paul E. McKenney (paulmck@linux.vnet.ibm.com) wrote:
> > Take a first step towards untangling Linux kernel header files by
> > placing the struct rcu_head definition into include/linux/types.h.
> > Once this is in place, then header files that need only rcu_head
> > can include types.h rather than rcupdate.h.
> > 
> > Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
> 
> Yes! I'm suddenly feeling the temperature drop by a few degrees in
> #include hell ;) One detail though: I think rcupdate.h should then
> include types.h.

Glad you like it!

But please see below.

> Thanks,
> 
> Mathieu
> 
> 
> > ---
> >  include/linux/rcupdate.h |   10 +---------
> >  include/linux/types.h    |   10 ++++++++++
> >  2 files changed, 11 insertions(+), 9 deletions(-)
> > 
> > diff --git a/include/linux/rcupdate.h b/include/linux/rcupdate.h
> > index 8be0433..ad09eb8 100644
> > --- a/include/linux/rcupdate.h
> > +++ b/include/linux/rcupdate.h
> > @@ -64,15 +64,7 @@ static inline void rcutorture_record_progress(unsigned long vernum)
> >  #define ULONG_CMP_GE(a, b)	(ULONG_MAX / 2 >= (a) - (b))
> >  #define ULONG_CMP_LT(a, b)	(ULONG_MAX / 2 < (a) - (b))
> >  
> > -/**
> > - * struct rcu_head - callback structure for use with RCU
> > - * @next: next update requests in a list
> > - * @func: actual update function to call after the grace period.
> > - */
> > -struct rcu_head {
> > -	struct rcu_head *next;
> > -	void (*func)(struct rcu_head *head);
> > -};
> > +#include <linux/types.h>

And here you go!

> >  
> >  /* Exported common interfaces */
> >  extern void call_rcu_sched(struct rcu_head *head,
> > diff --git a/include/linux/types.h b/include/linux/types.h
> > index 176da8c..57a9723 100644
> > --- a/include/linux/types.h
> > +++ b/include/linux/types.h
> > @@ -238,6 +238,16 @@ struct ustat {
> >  	char			f_fpack[6];
> >  };
> >  
> > +/**
> > + * struct rcu_head - callback structure for use with RCU
> > + * @next: next update requests in a list
> > + * @func: actual update function to call after the grace period.
> > + */
> > +struct rcu_head {
> > +	struct rcu_head *next;
> > +	void (*func)(struct rcu_head *head);
> > +};
> > +
> >  #endif	/* __KERNEL__ */
> >  #endif /*  __ASSEMBLY__ */
> >  #endif /* _LINUX_TYPES_H */
> > -- 
> > 1.7.3.2
> > 
> 
> -- 
> Mathieu Desnoyers
> Operating System Efficiency R&D Consultant
> EfficiOS Inc.
> http://www.efficios.com

Re: [PATCH tip/core/rcu 05/28] rcu: Move rcu_head definition to types.h

[Mathieu Desnoyers]

* Paul E. McKenney (paulmck@linux.vnet.ibm.com) wrote:
> On Wed, Jun 08, 2011 at 07:26:21PM -0400, Mathieu Desnoyers wrote:
> > * Paul E. McKenney (paulmck@linux.vnet.ibm.com) wrote:
> > > Take a first step towards untangling Linux kernel header files by
> > > placing the struct rcu_head definition into include/linux/types.h.
> > > Once this is in place, then header files that need only rcu_head
> > > can include types.h rather than rcupdate.h.
> > > 
> > > Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
> > 
> > Yes! I'm suddenly feeling the temperature drop by a few degrees in
> > #include hell ;) One detail though: I think rcupdate.h should then
> > include types.h.
> 
> Glad you like it!
> 
> But please see below.
> 
> > Thanks,
> > 
> > Mathieu
> > 
> > 
> > > ---
> > >  include/linux/rcupdate.h |   10 +---------
> > >  include/linux/types.h    |   10 ++++++++++
> > >  2 files changed, 11 insertions(+), 9 deletions(-)
> > > 
> > > diff --git a/include/linux/rcupdate.h b/include/linux/rcupdate.h
> > > index 8be0433..ad09eb8 100644
> > > --- a/include/linux/rcupdate.h
> > > +++ b/include/linux/rcupdate.h
> > > @@ -64,15 +64,7 @@ static inline void rcutorture_record_progress(unsigned long vernum)
> > >  #define ULONG_CMP_GE(a, b)	(ULONG_MAX / 2 >= (a) - (b))
> > >  #define ULONG_CMP_LT(a, b)	(ULONG_MAX / 2 < (a) - (b))
> > >  
> > > -/**
> > > - * struct rcu_head - callback structure for use with RCU
> > > - * @next: next update requests in a list
> > > - * @func: actual update function to call after the grace period.
> > > - */
> > > -struct rcu_head {
> > > -	struct rcu_head *next;
> > > -	void (*func)(struct rcu_head *head);
> > > -};
> > > +#include <linux/types.h>
> 
> And here you go!

Hrm, I see. Here is why I missed it: I was expecting this #include to be
added at the top of rcupdate.h instead of being added in the middle like
you do here. Would it be acceptable to move this #include to the top of
the file, or is there some other obscure reason for not doing so ? (I
feel the temperature is suddenly rising) ;-)

Thanks,

Mathieu

> 
> > >  
> > >  /* Exported common interfaces */
> > >  extern void call_rcu_sched(struct rcu_head *head,
> > > diff --git a/include/linux/types.h b/include/linux/types.h
> > > index 176da8c..57a9723 100644
> > > --- a/include/linux/types.h
> > > +++ b/include/linux/types.h
> > > @@ -238,6 +238,16 @@ struct ustat {
> > >  	char			f_fpack[6];
> > >  };
> > >  
> > > +/**
> > > + * struct rcu_head - callback structure for use with RCU
> > > + * @next: next update requests in a list
> > > + * @func: actual update function to call after the grace period.
> > > + */
> > > +struct rcu_head {
> > > +	struct rcu_head *next;
> > > +	void (*func)(struct rcu_head *head);
> > > +};
> > > +
> > >  #endif	/* __KERNEL__ */
> > >  #endif /*  __ASSEMBLY__ */
> > >  #endif /* _LINUX_TYPES_H */
> > > -- 
> > > 1.7.3.2
> > > 
> > 
> > -- 
> > Mathieu Desnoyers
> > Operating System Efficiency R&D Consultant
> > EfficiOS Inc.
> > http://www.efficios.com

-- 
Mathieu Desnoyers
Operating System Efficiency R&D Consultant
EfficiOS Inc.
http://www.efficios.com

Re: [PATCH tip/core/rcu 05/28] rcu: Move rcu_head definition to types.h

[Paul E. McKenney]

On Wed, Jun 08, 2011 at 08:11:52PM -0400, Mathieu Desnoyers wrote:
> * Paul E. McKenney (paulmck@linux.vnet.ibm.com) wrote:
> > On Wed, Jun 08, 2011 at 07:26:21PM -0400, Mathieu Desnoyers wrote:
> > > * Paul E. McKenney (paulmck@linux.vnet.ibm.com) wrote:
> > > > Take a first step towards untangling Linux kernel header files by
> > > > placing the struct rcu_head definition into include/linux/types.h.
> > > > Once this is in place, then header files that need only rcu_head
> > > > can include types.h rather than rcupdate.h.
> > > > 
> > > > Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
> > > 
> > > Yes! I'm suddenly feeling the temperature drop by a few degrees in
> > > #include hell ;) One detail though: I think rcupdate.h should then
> > > include types.h.
> > 
> > Glad you like it!
> > 
> > But please see below.
> > 
> > > Thanks,
> > > 
> > > Mathieu
> > > 
> > > 
> > > > ---
> > > >  include/linux/rcupdate.h |   10 +---------
> > > >  include/linux/types.h    |   10 ++++++++++
> > > >  2 files changed, 11 insertions(+), 9 deletions(-)
> > > > 
> > > > diff --git a/include/linux/rcupdate.h b/include/linux/rcupdate.h
> > > > index 8be0433..ad09eb8 100644
> > > > --- a/include/linux/rcupdate.h
> > > > +++ b/include/linux/rcupdate.h
> > > > @@ -64,15 +64,7 @@ static inline void rcutorture_record_progress(unsigned long vernum)
> > > >  #define ULONG_CMP_GE(a, b)	(ULONG_MAX / 2 >= (a) - (b))
> > > >  #define ULONG_CMP_LT(a, b)	(ULONG_MAX / 2 < (a) - (b))
> > > >  
> > > > -/**
> > > > - * struct rcu_head - callback structure for use with RCU
> > > > - * @next: next update requests in a list
> > > > - * @func: actual update function to call after the grace period.
> > > > - */
> > > > -struct rcu_head {
> > > > -	struct rcu_head *next;
> > > > -	void (*func)(struct rcu_head *head);
> > > > -};
> > > > +#include <linux/types.h>
> > 
> > And here you go!
> 
> Hrm, I see. Here is why I missed it: I was expecting this #include to be
> added at the top of rcupdate.h instead of being added in the middle like
> you do here. Would it be acceptable to move this #include to the top of
> the file, or is there some other obscure reason for not doing so ? (I
> feel the temperature is suddenly rising) ;-)

No reason, just laziness on my part.  Seems to work OK when I move
it up -- which is no surprise, given that types.h is probably already
included by something before rcupdate.h is reached.  So consider it
moved.

							Thanx, Paul

Re: [PATCH tip/core/rcu 05/28] rcu: Move rcu_head definition to types.h

[Mathieu Desnoyers]

* Paul E. McKenney (paulmck@linux.vnet.ibm.com) wrote:
> On Wed, Jun 08, 2011 at 08:11:52PM -0400, Mathieu Desnoyers wrote:
> > * Paul E. McKenney (paulmck@linux.vnet.ibm.com) wrote:
> > > On Wed, Jun 08, 2011 at 07:26:21PM -0400, Mathieu Desnoyers wrote:
> > > > * Paul E. McKenney (paulmck@linux.vnet.ibm.com) wrote:
> > > > > Take a first step towards untangling Linux kernel header files by
> > > > > placing the struct rcu_head definition into include/linux/types.h.
> > > > > Once this is in place, then header files that need only rcu_head
> > > > > can include types.h rather than rcupdate.h.
> > > > > 
> > > > > Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
> > > > 
> > > > Yes! I'm suddenly feeling the temperature drop by a few degrees in
> > > > #include hell ;) One detail though: I think rcupdate.h should then
> > > > include types.h.
> > > 
> > > Glad you like it!
> > > 
> > > But please see below.
> > > 
> > > > Thanks,
> > > > 
> > > > Mathieu
> > > > 
> > > > 
> > > > > ---
> > > > >  include/linux/rcupdate.h |   10 +---------
> > > > >  include/linux/types.h    |   10 ++++++++++
> > > > >  2 files changed, 11 insertions(+), 9 deletions(-)
> > > > > 
> > > > > diff --git a/include/linux/rcupdate.h b/include/linux/rcupdate.h
> > > > > index 8be0433..ad09eb8 100644
> > > > > --- a/include/linux/rcupdate.h
> > > > > +++ b/include/linux/rcupdate.h
> > > > > @@ -64,15 +64,7 @@ static inline void rcutorture_record_progress(unsigned long vernum)
> > > > >  #define ULONG_CMP_GE(a, b)	(ULONG_MAX / 2 >= (a) - (b))
> > > > >  #define ULONG_CMP_LT(a, b)	(ULONG_MAX / 2 < (a) - (b))
> > > > >  
> > > > > -/**
> > > > > - * struct rcu_head - callback structure for use with RCU
> > > > > - * @next: next update requests in a list
> > > > > - * @func: actual update function to call after the grace period.
> > > > > - */
> > > > > -struct rcu_head {
> > > > > -	struct rcu_head *next;
> > > > > -	void (*func)(struct rcu_head *head);
> > > > > -};
> > > > > +#include <linux/types.h>
> > > 
> > > And here you go!
> > 
> > Hrm, I see. Here is why I missed it: I was expecting this #include to be
> > added at the top of rcupdate.h instead of being added in the middle like
> > you do here. Would it be acceptable to move this #include to the top of
> > the file, or is there some other obscure reason for not doing so ? (I
> > feel the temperature is suddenly rising) ;-)
> 
> No reason, just laziness on my part.  Seems to work OK when I move
> it up -- which is no surprise, given that types.h is probably already
> included by something before rcupdate.h is reached.  So consider it
> moved.

Considering it moved,

Acked-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>


-- 
Mathieu Desnoyers
Operating System Efficiency R&D Consultant
EfficiOS Inc.
http://www.efficios.com

[PATCH tip/core/rcu 06/28] rcu: Update rcutorture documentation

[Paul E. McKenney]

Update rcutorture documentation to account for boosting, new types of
RCU torture testing that have been added over the past few years, and
the memory-barrier testing that was added an embarrassingly long time
ago.

Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
---
 Documentation/RCU/torture.txt |  134 ++++++++++++++++++++++++++++++-----------
 1 files changed, 99 insertions(+), 35 deletions(-)

diff --git a/Documentation/RCU/torture.txt b/Documentation/RCU/torture.txt
index 5d90167..4205ed1 100644
--- a/Documentation/RCU/torture.txt
+++ b/Documentation/RCU/torture.txt
@@ -42,7 +42,7 @@ fqs_holdoff	Holdoff time (in microseconds) between consecutive calls
 fqs_stutter	Wait time (in seconds) between consecutive bursts
 		of calls to force_quiescent_state().
 
-irqreaders	Says to invoke RCU readers from irq level.  This is currently
+irqreader	Says to invoke RCU readers from irq level.  This is currently
 		done via timers.  Defaults to "1" for variants of RCU that
 		permit this.  (Or, more accurately, variants of RCU that do
 		-not- permit this know to ignore this variable.)
@@ -79,19 +79,65 @@ stutter		The length of time to run the test before pausing for this
 		Specifying "stutter=0" causes the test to run continuously
 		without pausing, which is the old default behavior.
 
+test_boost	Whether or not to test the ability of RCU to do priority
+		boosting.  Defaults to "test_boost=1", which performs
+		RCU priority-inversion testing only if the selected
+		RCU implementation supports priority boosting.  Specifying
+		"test_boost=0" never performs RCU priority-inversion
+		testing.  Specifying "test_boost=2" performs RCU
+		priority-inversion testing even if the selected RCU
+		implementation does not support RCU priority boosting,
+		which can be used to test rcutorture's ability to
+		carry out RCU priority-inversion testing.
+
+test_boost_interval
+		The number of seconds in an RCU priority-inversion test
+		cycle.	Defaults to "test_boost_interval=7".  It is
+		usually wise for this value to be relatively prime to
+		the value selected for "stutter".
+
+test_boost_duration
+		The number of seconds to do RCU priority-inversion testing
+		within any given "test_boost_interval".  Defaults to
+		"test_boost_duration=4".
+
 test_no_idle_hz	Whether or not to test the ability of RCU to operate in
 		a kernel that disables the scheduling-clock interrupt to
 		idle CPUs.  Boolean parameter, "1" to test, "0" otherwise.
 		Defaults to omitting this test.
 
-torture_type	The type of RCU to test: "rcu" for the rcu_read_lock() API,
-		"rcu_sync" for rcu_read_lock() with synchronous reclamation,
-		"rcu_bh" for the rcu_read_lock_bh() API, "rcu_bh_sync" for
-		rcu_read_lock_bh() with synchronous reclamation, "srcu" for
-		the "srcu_read_lock()" API, "sched" for the use of
-		preempt_disable() together with synchronize_sched(),
-		and "sched_expedited" for the use of preempt_disable()
-		with synchronize_sched_expedited().
+torture_type	The type of RCU to test, with string values as follows:
+
+		"rcu":  rcu_read_lock(), rcu_read_unlock() and call_rcu().
+
+		"rcu_sync":  rcu_read_lock(), rcu_read_unlock(), and
+			synchronize_rcu().
+
+		"rcu_expedited": rcu_read_lock(), rcu_read_unlock(), and
+			synchronize_rcu_expedited().
+
+		"rcu_bh": rcu_read_lock_bh(), rcu_read_unlock_bh(), and
+			call_rcu_bh().
+
+		"rcu_bh_sync": rcu_read_lock_bh(), rcu_read_unlock_bh(),
+			and synchronize_rcu_bh().
+
+		"srcu": srcu_read_lock(), srcu_read_unlock() and
+			synchronize_srcu().
+
+		"srcu_expedited": srcu_read_lock(), srcu_read_unlock() and
+			synchronize_srcu_expedited().
+
+		"sched": preempt_disable(), preempt_enable(), and
+			call_rcu_sched().
+
+		"sched_sync": preempt_disable(), preempt_enable(), and
+			synchronize_sched().
+
+		"sched_expedited": preempt_disable(), preempt_enable(), and
+			synchronize_sched_expedited().
+
+		Defaults to "rcu".
 
 verbose		Enable debug printk()s.  Default is disabled.
 
@@ -100,12 +146,12 @@ OUTPUT
 
 The statistics output is as follows:
 
-	rcu-torture: --- Start of test: nreaders=16 stat_interval=0 verbose=0
-	rcu-torture: rtc: 0000000000000000 ver: 1916 tfle: 0 rta: 1916 rtaf: 0 rtf: 1915
-	rcu-torture: Reader Pipe:  1466408 9747 0 0 0 0 0 0 0 0 0
-	rcu-torture: Reader Batch:  1464477 11678 0 0 0 0 0 0 0 0
-	rcu-torture: Free-Block Circulation:  1915 1915 1915 1915 1915 1915 1915 1915 1915 1915 0
-	rcu-torture: --- End of test
+	rcu-torture:--- Start of test: nreaders=16 nfakewriters=4 stat_interval=30 verbose=0 test_no_idle_hz=1 shuffle_interval=3 stutter=5 irqreader=1 fqs_duration=0 fqs_holdoff=0 fqs_stutter=3 test_boost=1/0 test_boost_interval=7 test_boost_duration=4
+	rcu-torture: rtc:           (null) ver: 155441 tfle: 0 rta: 155441 rtaf: 8884 rtf: 155440 rtmbe: 0 rtbke: 0 rtbre: 0 rtbf: 0 rtb: 0 nt: 3055767
+	rcu-torture: Reader Pipe:  727860534 34213 0 0 0 0 0 0 0 0 0
+	rcu-torture: Reader Batch:  727877838 17003 0 0 0 0 0 0 0 0 0
+	rcu-torture: Free-Block Circulation:  155440 155440 155440 155440 155440 155440 155440 155440 155440 155440 0
+	rcu-torture:--- End of test: SUCCESS: nreaders=16 nfakewriters=4 stat_interval=30 verbose=0 test_no_idle_hz=1 shuffle_interval=3 stutter=5 irqreader=1 fqs_duration=0 fqs_holdoff=0 fqs_stutter=3 test_boost=1/0 test_boost_interval=7 test_boost_duration=4
 
 The command "dmesg | grep torture:" will extract this information on
 most systems.  On more esoteric configurations, it may be necessary to
@@ -113,26 +159,55 @@ use other commands to access the output of the printk()s used by
 the RCU torture test.  The printk()s use KERN_ALERT, so they should
 be evident.  ;-)
 
+The first and last lines show the rcutorture module parameters, and the
+last line shows either "SUCCESS" or "FAILURE", based on rcutorture's
+automatic determination as to whether RCU operated correctly.
+
 The entries are as follows:
 
 o	"rtc": The hexadecimal address of the structure currently visible
 	to readers.
 
-o	"ver": The number of times since boot that the rcutw writer task
+o	"ver": The number of times since boot that the RCU writer task
 	has changed the structure visible to readers.
 
 o	"tfle": If non-zero, indicates that the "torture freelist"
-	containing structure to be placed into the "rtc" area is empty.
+	containing structures to be placed into the "rtc" area is empty.
 	This condition is important, since it can fool you into thinking
 	that RCU is working when it is not.  :-/
 
 o	"rta": Number of structures allocated from the torture freelist.
 
 o	"rtaf": Number of allocations from the torture freelist that have
-	failed due to the list being empty.
+	failed due to the list being empty.  It is not unusual for this
+	to be non-zero, but it is bad for it to be a large fraction of
+	the value indicated by "rta".
 
 o	"rtf": Number of frees into the torture freelist.
 
+o	"rtmbe": A non-zero value indicates that rcutorture believes that
+	rcu_assign_pointer() and rcu_dereference() are not working
+	correctly.  This value should be zero.
+
+o	"rtbke": rcutorture was unable to create the real-time kthreads
+	used to force RCU priority inversion.  This value should be zero.
+
+o	"rtbre": Although rcutorture successfully created the kthreads
+	used to force RCU priority inversion, it was unable to set them
+	to the real-time priority level of 1.  This value should be zero.
+
+o	"rtbf": The number of times that RCU priority boosting failed
+	to resolve RCU priority inversion.
+
+o	"rtb": The number of times that rcutorture attempted to force
+	an RCU priority inversion condition.  If you are testing RCU
+	priority boosting via the "test_boost" module parameter, this
+	value should be non-zero.
+
+o	"nt": The number of times rcutorture ran RCU read-side code from
+	within a timer handler.  This value should be non-zero only
+	if you specified the "irqreader" module parameter.
+
 o	"Reader Pipe": Histogram of "ages" of structures seen by readers.
 	If any entries past the first two are non-zero, RCU is broken.
 	And rcutorture prints the error flag string "!!!" to make sure
@@ -162,26 +237,15 @@ o	"Free-Block Circulation": Shows the number of torture structures
 	somehow gets incremented farther than it should.
 
 Different implementations of RCU can provide implementation-specific
-additional information.  For example, SRCU provides the following:
+additional information.  For example, SRCU provides the following
+additional line:
 
-	srcu-torture: rtc: f8cf46a8 ver: 355 tfle: 0 rta: 356 rtaf: 0 rtf: 346 rtmbe: 0
-	srcu-torture: Reader Pipe:  559738 939 0 0 0 0 0 0 0 0 0
-	srcu-torture: Reader Batch:  560434 243 0 0 0 0 0 0 0 0
-	srcu-torture: Free-Block Circulation:  355 354 353 352 351 350 349 348 347 346 0
 	srcu-torture: per-CPU(idx=1): 0(0,1) 1(0,1) 2(0,0) 3(0,1)
 
-The first four lines are similar to those for RCU.  The last line shows
-the per-CPU counter state.  The numbers in parentheses are the values
-of the "old" and "current" counters for the corresponding CPU.  The
-"idx" value maps the "old" and "current" values to the underlying array,
-and is useful for debugging.
-
-Similarly, sched_expedited RCU provides the following:
-
-	sched_expedited-torture: rtc: d0000000016c1880 ver: 1090796 tfle: 0 rta: 1090796 rtaf: 0 rtf: 1090787 rtmbe: 0 nt: 27713319
-	sched_expedited-torture: Reader Pipe:  12660320201 95875 0 0 0 0 0 0 0 0 0
-	sched_expedited-torture: Reader Batch:  12660424885 0 0 0 0 0 0 0 0 0 0
-	sched_expedited-torture: Free-Block Circulation:  1090795 1090795 1090794 1090793 1090792 1090791 1090790 1090789 1090788 1090787 0
+This line shows the per-CPU counter state.  The numbers in parentheses are
+the values of the "old" and "current" counters for the corresponding CPU.
+The "idx" value maps the "old" and "current" values to the underlying
+array, and is useful for debugging.
 
 
 USAGE
-- 
1.7.3.2

[PATCH tip/core/rcu 07/28] Fix mismatched variable in rcutree_trace.c

[Paul E. McKenney]

From: Andi Kleen <ak@linux.intel.com>

rcutree.c defines rcu_cpu_kthread_cpu as int, not unsigned int,
so the extern has to follow that.

Cc: paulmck@linux.vnet.ibm.com
Signed-off-by: Andi Kleen <ak@linux.intel.com>
Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
---
 kernel/rcutree_trace.c |    2 +-
 1 files changed, 1 insertions(+), 1 deletions(-)

diff --git a/kernel/rcutree_trace.c b/kernel/rcutree_trace.c
index 9678cc3..91c56e5 100644
--- a/kernel/rcutree_trace.c
+++ b/kernel/rcutree_trace.c
@@ -47,7 +47,7 @@
 #include "rcutree.h"
 
 DECLARE_PER_CPU(unsigned int, rcu_cpu_kthread_status);
-DECLARE_PER_CPU(unsigned int, rcu_cpu_kthread_cpu);
+DECLARE_PER_CPU(int, rcu_cpu_kthread_cpu);
 DECLARE_PER_CPU(unsigned int, rcu_cpu_kthread_loops);
 DECLARE_PER_CPU(char, rcu_cpu_has_work);
 
-- 
1.7.3.2

Re: [PATCH tip/core/rcu 07/28] Fix mismatched variable in rcutree_trace.c

[Mathieu Desnoyers]

* Paul E. McKenney (paulmck@linux.vnet.ibm.com) wrote:
> From: Andi Kleen <ak@linux.intel.com>
> 
> rcutree.c defines rcu_cpu_kthread_cpu as int, not unsigned int,
> so the extern has to follow that.

How comes this discrepancy was made possible in the first place ? This
declaration should appear in a header included by both rcutree_trace.c
and rcutree.c, which would have permitted the compiler to detect this.

Maybe kernel/rcutree.h would be a good location for these ?

Mathieu

> 
> Cc: paulmck@linux.vnet.ibm.com
> Signed-off-by: Andi Kleen <ak@linux.intel.com>
> Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
> ---
>  kernel/rcutree_trace.c |    2 +-
>  1 files changed, 1 insertions(+), 1 deletions(-)
> 
> diff --git a/kernel/rcutree_trace.c b/kernel/rcutree_trace.c
> index 9678cc3..91c56e5 100644
> --- a/kernel/rcutree_trace.c
> +++ b/kernel/rcutree_trace.c
> @@ -47,7 +47,7 @@
>  #include "rcutree.h"
>  
>  DECLARE_PER_CPU(unsigned int, rcu_cpu_kthread_status);
> -DECLARE_PER_CPU(unsigned int, rcu_cpu_kthread_cpu);
> +DECLARE_PER_CPU(int, rcu_cpu_kthread_cpu);
>  DECLARE_PER_CPU(unsigned int, rcu_cpu_kthread_loops);
>  DECLARE_PER_CPU(char, rcu_cpu_has_work);
>  
> -- 
> 1.7.3.2
> 

-- 
Mathieu Desnoyers
Operating System Efficiency R&D Consultant
EfficiOS Inc.
http://www.efficios.com

Re: [PATCH tip/core/rcu 07/28] Fix mismatched variable in rcutree_trace.c

[Paul E. McKenney]

On Wed, Jun 08, 2011 at 07:23:29PM -0400, Mathieu Desnoyers wrote:
> * Paul E. McKenney (paulmck@linux.vnet.ibm.com) wrote:
> > From: Andi Kleen <ak@linux.intel.com>
> > 
> > rcutree.c defines rcu_cpu_kthread_cpu as int, not unsigned int,
> > so the extern has to follow that.
> 
> How comes this discrepancy was made possible in the first place ? This
> declaration should appear in a header included by both rcutree_trace.c
> and rcutree.c, which would have permitted the compiler to detect this.
> 
> Maybe kernel/rcutree.h would be a good location for these ?

I vaguely recall a time where this didn't work, but will give it
another try.  It would certainly be nicer if the compiler checked
for these sorts of mismatches!

							Thanx, Paul

> Mathieu
> 
> > 
> > Cc: paulmck@linux.vnet.ibm.com
> > Signed-off-by: Andi Kleen <ak@linux.intel.com>
> > Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
> > ---
> >  kernel/rcutree_trace.c |    2 +-
> >  1 files changed, 1 insertions(+), 1 deletions(-)
> > 
> > diff --git a/kernel/rcutree_trace.c b/kernel/rcutree_trace.c
> > index 9678cc3..91c56e5 100644
> > --- a/kernel/rcutree_trace.c
> > +++ b/kernel/rcutree_trace.c
> > @@ -47,7 +47,7 @@
> >  #include "rcutree.h"
> >  
> >  DECLARE_PER_CPU(unsigned int, rcu_cpu_kthread_status);
> > -DECLARE_PER_CPU(unsigned int, rcu_cpu_kthread_cpu);
> > +DECLARE_PER_CPU(int, rcu_cpu_kthread_cpu);
> >  DECLARE_PER_CPU(unsigned int, rcu_cpu_kthread_loops);
> >  DECLARE_PER_CPU(char, rcu_cpu_has_work);
> >  
> > -- 
> > 1.7.3.2
> > 
> 
> -- 
> Mathieu Desnoyers
> Operating System Efficiency R&D Consultant
> EfficiOS Inc.
> http://www.efficios.com

[PATCH tip/core/rcu 08/28] rcu: Abstract common code for RCU grace-period-wait primitives

[Paul E. McKenney]

Pull the code that waits for an RCU grace period into a single function,
which is then called by synchronize_rcu() and friends in the case of
TREE_RCU and TREE_PREEMPT_RCU, and from rcu_barrier() and friends in
the case of TINY_RCU and TINY_PREEMPT_RCU.

Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
---
 include/linux/rcupdate.h |  130 ++++++++++++++++++++++++++--------------------
 include/linux/rcutiny.h  |   16 +++++-
 include/linux/rcutree.h  |    2 +
 kernel/rcupdate.c        |   21 +++++++-
 kernel/rcutiny.c         |   28 ----------
 kernel/rcutiny_plugin.h  |   14 -----
 kernel/rcutree.c         |   22 +-------
 kernel/rcutree_plugin.h  |   11 +----
 8 files changed, 113 insertions(+), 131 deletions(-)

diff --git a/include/linux/rcupdate.h b/include/linux/rcupdate.h
index ad09eb8..9db50df 100644
--- a/include/linux/rcupdate.h
+++ b/include/linux/rcupdate.h
@@ -67,11 +67,73 @@ static inline void rcutorture_record_progress(unsigned long vernum)
 #include <linux/types.h>
 
 /* Exported common interfaces */
+
+#ifdef CONFIG_PREEMPT_RCU
+
+/**
+ * call_rcu() - Queue an RCU callback for invocation after a grace period.
+ * @head: structure to be used for queueing the RCU updates.
+ * @func: actual callback function to be invoked after the grace period
+ *
+ * The callback function will be invoked some time after a full grace
+ * period elapses, in other words after all pre-existing RCU read-side
+ * critical sections have completed.  However, the callback function
+ * might well execute concurrently with RCU read-side critical sections
+ * that started after call_rcu() was invoked.  RCU read-side critical
+ * sections are delimited by rcu_read_lock() and rcu_read_unlock(),
+ * and may be nested.
+ */
+extern void call_rcu(struct rcu_head *head,
+			      void (*func)(struct rcu_head *head));
+
+#else /* #ifdef CONFIG_PREEMPT_RCU */
+
+/* In classic RCU, call_rcu() is just call_rcu_sched(). */
+#define	call_rcu	call_rcu_sched
+
+#endif /* #else #ifdef CONFIG_PREEMPT_RCU */
+
+/**
+ * call_rcu_bh() - Queue an RCU for invocation after a quicker grace period.
+ * @head: structure to be used for queueing the RCU updates.
+ * @func: actual callback function to be invoked after the grace period
+ *
+ * The callback function will be invoked some time after a full grace
+ * period elapses, in other words after all currently executing RCU
+ * read-side critical sections have completed. call_rcu_bh() assumes
+ * that the read-side critical sections end on completion of a softirq
+ * handler. This means that read-side critical sections in process
+ * context must not be interrupted by softirqs. This interface is to be
+ * used when most of the read-side critical sections are in softirq context.
+ * RCU read-side critical sections are delimited by :
+ *  - rcu_read_lock() and  rcu_read_unlock(), if in interrupt context.
+ *  OR
+ *  - rcu_read_lock_bh() and rcu_read_unlock_bh(), if in process context.
+ *  These may be nested.
+ */
+extern void call_rcu_bh(struct rcu_head *head,
+			void (*func)(struct rcu_head *head));
+
+/**
+ * call_rcu_sched() - Queue an RCU for invocation after sched grace period.
+ * @head: structure to be used for queueing the RCU updates.
+ * @func: actual callback function to be invoked after the grace period
+ *
+ * The callback function will be invoked some time after a full grace
+ * period elapses, in other words after all currently executing RCU
+ * read-side critical sections have completed. call_rcu_sched() assumes
+ * that the read-side critical sections end on enabling of preemption
+ * or on voluntary preemption.
+ * RCU read-side critical sections are delimited by :
+ *  - rcu_read_lock_sched() and  rcu_read_unlock_sched(),
+ *  OR
+ *  anything that disables preemption.
+ *  These may be nested.
+ */
 extern void call_rcu_sched(struct rcu_head *head,
 			   void (*func)(struct rcu_head *rcu));
+
 extern void synchronize_sched(void);
-extern void rcu_barrier_bh(void);
-extern void rcu_barrier_sched(void);
 
 static inline void __rcu_read_lock_bh(void)
 {
@@ -144,6 +206,15 @@ static inline void rcu_exit_nohz(void)
 
 #endif /* #else #ifdef CONFIG_NO_HZ */
 
+/*
+ * Infrastructure to implement the synchronize_() primitives in
+ * TREE_RCU and rcu_barrier_() primitives in TINY_RCU.
+ */
+
+typedef void call_rcu_func_t(struct rcu_head *head,
+			     void (*func)(struct rcu_head *head));
+void wait_rcu_gp(call_rcu_func_t crf);
+
 #if defined(CONFIG_TREE_RCU) || defined(CONFIG_TREE_PREEMPT_RCU)
 #include <linux/rcutree.h>
 #elif defined(CONFIG_TINY_RCU) || defined(CONFIG_TINY_PREEMPT_RCU)
@@ -724,61 +795,6 @@ static inline notrace void rcu_read_unlock_sched_notrace(void)
 #define RCU_INIT_POINTER(p, v) \
 		p = (typeof(*v) __force __rcu *)(v)
 
-/* Infrastructure to implement the synchronize_() primitives. */
-
-struct rcu_synchronize {
-	struct rcu_head head;
-	struct completion completion;
-};
-
-extern void wakeme_after_rcu(struct rcu_head  *head);
-
-#ifdef CONFIG_PREEMPT_RCU
-
-/**
- * call_rcu() - Queue an RCU callback for invocation after a grace period.
- * @head: structure to be used for queueing the RCU updates.
- * @func: actual callback function to be invoked after the grace period
- *
- * The callback function will be invoked some time after a full grace
- * period elapses, in other words after all pre-existing RCU read-side
- * critical sections have completed.  However, the callback function
- * might well execute concurrently with RCU read-side critical sections
- * that started after call_rcu() was invoked.  RCU read-side critical
- * sections are delimited by rcu_read_lock() and rcu_read_unlock(),
- * and may be nested.
- */
-extern void call_rcu(struct rcu_head *head,
-			      void (*func)(struct rcu_head *head));
-
-#else /* #ifdef CONFIG_PREEMPT_RCU */
-
-/* In classic RCU, call_rcu() is just call_rcu_sched(). */
-#define	call_rcu	call_rcu_sched
-
-#endif /* #else #ifdef CONFIG_PREEMPT_RCU */
-
-/**
- * call_rcu_bh() - Queue an RCU for invocation after a quicker grace period.
- * @head: structure to be used for queueing the RCU updates.
- * @func: actual callback function to be invoked after the grace period
- *
- * The callback function will be invoked some time after a full grace
- * period elapses, in other words after all currently executing RCU
- * read-side critical sections have completed. call_rcu_bh() assumes
- * that the read-side critical sections end on completion of a softirq
- * handler. This means that read-side critical sections in process
- * context must not be interrupted by softirqs. This interface is to be
- * used when most of the read-side critical sections are in softirq context.
- * RCU read-side critical sections are delimited by :
- *  - rcu_read_lock() and  rcu_read_unlock(), if in interrupt context.
- *  OR
- *  - rcu_read_lock_bh() and rcu_read_unlock_bh(), if in process context.
- *  These may be nested.
- */
-extern void call_rcu_bh(struct rcu_head *head,
-			void (*func)(struct rcu_head *head));
-
 /*
  * debug_rcu_head_queue()/debug_rcu_head_unqueue() are used internally
  * by call_rcu() and rcu callback execution, and are therefore not part of the
diff --git a/include/linux/rcutiny.h b/include/linux/rcutiny.h
index 52b3e02..4eab233 100644
--- a/include/linux/rcutiny.h
+++ b/include/linux/rcutiny.h
@@ -31,6 +31,16 @@ static inline void rcu_init(void)
 {
 }
 
+static inline void rcu_barrier_bh(void)
+{
+	wait_rcu_gp(call_rcu_bh);
+}
+
+static inline void rcu_barrier_sched(void)
+{
+	wait_rcu_gp(call_rcu_sched);
+}
+
 #ifdef CONFIG_TINY_RCU
 
 static inline void synchronize_rcu_expedited(void)
@@ -45,9 +55,13 @@ static inline void rcu_barrier(void)
 
 #else /* #ifdef CONFIG_TINY_RCU */
 
-void rcu_barrier(void);
 void synchronize_rcu_expedited(void);
 
+static inline void rcu_barrier(void)
+{
+	wait_rcu_gp(call_rcu);
+}
+
 #endif /* #else #ifdef CONFIG_TINY_RCU */
 
 static inline void synchronize_rcu_bh(void)
diff --git a/include/linux/rcutree.h b/include/linux/rcutree.h
index e65d066..6745846 100644
--- a/include/linux/rcutree.h
+++ b/include/linux/rcutree.h
@@ -67,6 +67,8 @@ static inline void synchronize_rcu_bh_expedited(void)
 }
 
 extern void rcu_barrier(void);
+extern void rcu_barrier_bh(void);
+extern void rcu_barrier_sched(void);
 
 extern unsigned long rcutorture_testseq;
 extern unsigned long rcutorture_vernum;
diff --git a/kernel/rcupdate.c b/kernel/rcupdate.c
index 7784bd2..a088c90 100644
--- a/kernel/rcupdate.c
+++ b/kernel/rcupdate.c
@@ -94,11 +94,16 @@ EXPORT_SYMBOL_GPL(rcu_read_lock_bh_held);
 
 #endif /* #ifdef CONFIG_DEBUG_LOCK_ALLOC */
 
+struct rcu_synchronize {
+	struct rcu_head head;
+	struct completion completion;
+};
+
 /*
  * Awaken the corresponding synchronize_rcu() instance now that a
  * grace period has elapsed.
  */
-void wakeme_after_rcu(struct rcu_head  *head)
+static void wakeme_after_rcu(struct rcu_head  *head)
 {
 	struct rcu_synchronize *rcu;
 
@@ -106,6 +111,20 @@ void wakeme_after_rcu(struct rcu_head  *head)
 	complete(&rcu->completion);
 }
 
+void wait_rcu_gp(call_rcu_func_t crf)
+{
+	struct rcu_synchronize rcu;
+
+	init_rcu_head_on_stack(&rcu.head);
+	init_completion(&rcu.completion);
+	/* Will wake me after RCU finished. */
+	crf(&rcu.head, wakeme_after_rcu);
+	/* Wait for it. */
+	wait_for_completion(&rcu.completion);
+	destroy_rcu_head_on_stack(&rcu.head);
+}
+EXPORT_SYMBOL_GPL(wait_rcu_gp);
+
 #ifdef CONFIG_PROVE_RCU
 /*
  * wrapper function to avoid #include problems.
diff --git a/kernel/rcutiny.c b/kernel/rcutiny.c
index 421abfd..775d69a 100644
--- a/kernel/rcutiny.c
+++ b/kernel/rcutiny.c
@@ -280,34 +280,6 @@ void call_rcu_bh(struct rcu_head *head, void (*func)(struct rcu_head *rcu))
 }
 EXPORT_SYMBOL_GPL(call_rcu_bh);
 
-void rcu_barrier_bh(void)
-{
-	struct rcu_synchronize rcu;
-
-	init_rcu_head_on_stack(&rcu.head);
-	init_completion(&rcu.completion);
-	/* Will wake me after RCU finished. */
-	call_rcu_bh(&rcu.head, wakeme_after_rcu);
-	/* Wait for it. */
-	wait_for_completion(&rcu.completion);
-	destroy_rcu_head_on_stack(&rcu.head);
-}
-EXPORT_SYMBOL_GPL(rcu_barrier_bh);
-
-void rcu_barrier_sched(void)
-{
-	struct rcu_synchronize rcu;
-
-	init_rcu_head_on_stack(&rcu.head);
-	init_completion(&rcu.completion);
-	/* Will wake me after RCU finished. */
-	call_rcu_sched(&rcu.head, wakeme_after_rcu);
-	/* Wait for it. */
-	wait_for_completion(&rcu.completion);
-	destroy_rcu_head_on_stack(&rcu.head);
-}
-EXPORT_SYMBOL_GPL(rcu_barrier_sched);
-
 /*
  * Spawn the kthread that invokes RCU callbacks.
  */
diff --git a/kernel/rcutiny_plugin.h b/kernel/rcutiny_plugin.h
index f259c67..6b0cedb 100644
--- a/kernel/rcutiny_plugin.h
+++ b/kernel/rcutiny_plugin.h
@@ -697,20 +697,6 @@ void call_rcu(struct rcu_head *head, void (*func)(struct rcu_head *rcu))
 }
 EXPORT_SYMBOL_GPL(call_rcu);
 
-void rcu_barrier(void)
-{
-	struct rcu_synchronize rcu;
-
-	init_rcu_head_on_stack(&rcu.head);
-	init_completion(&rcu.completion);
-	/* Will wake me after RCU finished. */
-	call_rcu(&rcu.head, wakeme_after_rcu);
-	/* Wait for it. */
-	wait_for_completion(&rcu.completion);
-	destroy_rcu_head_on_stack(&rcu.head);
-}
-EXPORT_SYMBOL_GPL(rcu_barrier);
-
 /*
  * synchronize_rcu - wait until a grace period has elapsed.
  *
diff --git a/kernel/rcutree.c b/kernel/rcutree.c
index c67b9f4..305dfae 100644
--- a/kernel/rcutree.c
+++ b/kernel/rcutree.c
@@ -1916,18 +1916,9 @@ EXPORT_SYMBOL_GPL(call_rcu_bh);
  */
 void synchronize_sched(void)
 {
-	struct rcu_synchronize rcu;
-
 	if (rcu_blocking_is_gp())
 		return;
-
-	init_rcu_head_on_stack(&rcu.head);
-	init_completion(&rcu.completion);
-	/* Will wake me after RCU finished. */
-	call_rcu_sched(&rcu.head, wakeme_after_rcu);
-	/* Wait for it. */
-	wait_for_completion(&rcu.completion);
-	destroy_rcu_head_on_stack(&rcu.head);
+	wait_rcu_gp(call_rcu_sched);
 }
 EXPORT_SYMBOL_GPL(synchronize_sched);
 
@@ -1942,18 +1933,9 @@ EXPORT_SYMBOL_GPL(synchronize_sched);
  */
 void synchronize_rcu_bh(void)
 {
-	struct rcu_synchronize rcu;
-
 	if (rcu_blocking_is_gp())
 		return;
-
-	init_rcu_head_on_stack(&rcu.head);
-	init_completion(&rcu.completion);
-	/* Will wake me after RCU finished. */
-	call_rcu_bh(&rcu.head, wakeme_after_rcu);
-	/* Wait for it. */
-	wait_for_completion(&rcu.completion);
-	destroy_rcu_head_on_stack(&rcu.head);
+	wait_rcu_gp(call_rcu_bh);
 }
 EXPORT_SYMBOL_GPL(synchronize_rcu_bh);
 
diff --git a/kernel/rcutree_plugin.h b/kernel/rcutree_plugin.h
index 40a6db7..37caa15 100644
--- a/kernel/rcutree_plugin.h
+++ b/kernel/rcutree_plugin.h
@@ -624,18 +624,9 @@ EXPORT_SYMBOL_GPL(call_rcu);
  */
 void synchronize_rcu(void)
 {
-	struct rcu_synchronize rcu;
-
 	if (!rcu_scheduler_active)
 		return;
-
-	init_rcu_head_on_stack(&rcu.head);
-	init_completion(&rcu.completion);
-	/* Will wake me after RCU finished. */
-	call_rcu(&rcu.head, wakeme_after_rcu);
-	/* Wait for it. */
-	wait_for_completion(&rcu.completion);
-	destroy_rcu_head_on_stack(&rcu.head);
+	wait_rcu_gp(call_rcu);
 }
 EXPORT_SYMBOL_GPL(synchronize_rcu);
 
-- 
1.7.3.2

[PATCH tip/core/rcu 09/28] rcu: Catch rcutorture up to new RCU API additions

[Paul E. McKenney]

Now that the RCU API contains synchronize_rcu_bh(), synchronize_sched(),
call_rcu_sched(), and rcu_bh_expedited()...

Make rcutorture test synchronize_rcu_bh(), getting rid of the old
rcu_bh_torture_synchronize() workaround.  Similarly, make rcutorture test
synchronize_sched(), getting rid of the old sched_torture_synchronize()
workaround.  Make rcutorture test call_rcu_sched() instead of wrappering
synchronize_sched().  Also add testing of rcu_bh_expedited().

Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
---
 kernel/rcutorture.c |   55 +++++++++++++++++++-------------------------------
 1 files changed, 21 insertions(+), 34 deletions(-)

diff --git a/kernel/rcutorture.c b/kernel/rcutorture.c
index 40d9ed2..f736fe3 100644
--- a/kernel/rcutorture.c
+++ b/kernel/rcutorture.c
@@ -480,30 +480,6 @@ static void rcu_bh_torture_deferred_free(struct rcu_torture *p)
 	call_rcu_bh(&p->rtort_rcu, rcu_torture_cb);
 }
 
-struct rcu_bh_torture_synchronize {
-	struct rcu_head head;
-	struct completion completion;
-};
-
-static void rcu_bh_torture_wakeme_after_cb(struct rcu_head *head)
-{
-	struct rcu_bh_torture_synchronize *rcu;
-
-	rcu = container_of(head, struct rcu_bh_torture_synchronize, head);
-	complete(&rcu->completion);
-}
-
-static void rcu_bh_torture_synchronize(void)
-{
-	struct rcu_bh_torture_synchronize rcu;
-
-	init_rcu_head_on_stack(&rcu.head);
-	init_completion(&rcu.completion);
-	call_rcu_bh(&rcu.head, rcu_bh_torture_wakeme_after_cb);
-	wait_for_completion(&rcu.completion);
-	destroy_rcu_head_on_stack(&rcu.head);
-}
-
 static struct rcu_torture_ops rcu_bh_ops = {
 	.init		= NULL,
 	.cleanup	= NULL,
@@ -512,7 +488,7 @@ static struct rcu_torture_ops rcu_bh_ops = {
 	.readunlock	= rcu_bh_torture_read_unlock,
 	.completed	= rcu_bh_torture_completed,
 	.deferred_free	= rcu_bh_torture_deferred_free,
-	.sync		= rcu_bh_torture_synchronize,
+	.sync		= synchronize_rcu_bh,
 	.cb_barrier	= rcu_barrier_bh,
 	.fqs		= rcu_bh_force_quiescent_state,
 	.stats		= NULL,
@@ -528,7 +504,7 @@ static struct rcu_torture_ops rcu_bh_sync_ops = {
 	.readunlock	= rcu_bh_torture_read_unlock,
 	.completed	= rcu_bh_torture_completed,
 	.deferred_free	= rcu_sync_torture_deferred_free,
-	.sync		= rcu_bh_torture_synchronize,
+	.sync		= synchronize_rcu_bh,
 	.cb_barrier	= NULL,
 	.fqs		= rcu_bh_force_quiescent_state,
 	.stats		= NULL,
@@ -536,6 +512,22 @@ static struct rcu_torture_ops rcu_bh_sync_ops = {
 	.name		= "rcu_bh_sync"
 };
 
+static struct rcu_torture_ops rcu_bh_expedited_ops = {
+	.init		= rcu_sync_torture_init,
+	.cleanup	= NULL,
+	.readlock	= rcu_bh_torture_read_lock,
+	.read_delay	= rcu_read_delay,  /* just reuse rcu's version. */
+	.readunlock	= rcu_bh_torture_read_unlock,
+	.completed	= rcu_bh_torture_completed,
+	.deferred_free	= rcu_sync_torture_deferred_free,
+	.sync		= synchronize_rcu_bh_expedited,
+	.cb_barrier	= NULL,
+	.fqs		= rcu_bh_force_quiescent_state,
+	.stats		= NULL,
+	.irq_capable	= 1,
+	.name		= "rcu_bh_expedited"
+};
+
 /*
  * Definitions for srcu torture testing.
  */
@@ -659,11 +651,6 @@ static void rcu_sched_torture_deferred_free(struct rcu_torture *p)
 	call_rcu_sched(&p->rtort_rcu, rcu_torture_cb);
 }
 
-static void sched_torture_synchronize(void)
-{
-	synchronize_sched();
-}
-
 static struct rcu_torture_ops sched_ops = {
 	.init		= rcu_sync_torture_init,
 	.cleanup	= NULL,
@@ -672,7 +659,7 @@ static struct rcu_torture_ops sched_ops = {
 	.readunlock	= sched_torture_read_unlock,
 	.completed	= rcu_no_completed,
 	.deferred_free	= rcu_sched_torture_deferred_free,
-	.sync		= sched_torture_synchronize,
+	.sync		= synchronize_sched,
 	.cb_barrier	= rcu_barrier_sched,
 	.fqs		= rcu_sched_force_quiescent_state,
 	.stats		= NULL,
@@ -688,7 +675,7 @@ static struct rcu_torture_ops sched_sync_ops = {
 	.readunlock	= sched_torture_read_unlock,
 	.completed	= rcu_no_completed,
 	.deferred_free	= rcu_sync_torture_deferred_free,
-	.sync		= sched_torture_synchronize,
+	.sync		= synchronize_sched,
 	.cb_barrier	= NULL,
 	.fqs		= rcu_sched_force_quiescent_state,
 	.stats		= NULL,
@@ -1427,7 +1414,7 @@ rcu_torture_init(void)
 	int firsterr = 0;
 	static struct rcu_torture_ops *torture_ops[] =
 		{ &rcu_ops, &rcu_sync_ops, &rcu_expedited_ops,
-		  &rcu_bh_ops, &rcu_bh_sync_ops,
+		  &rcu_bh_ops, &rcu_bh_sync_ops, &rcu_bh_expedited_ops,
 		  &srcu_ops, &srcu_expedited_ops,
 		  &sched_ops, &sched_sync_ops, &sched_expedited_ops, };
 
-- 
1.7.3.2

[PATCH tip/core/rcu 10/28] rcu: Fix RCU's NMI documentation

[Paul E. McKenney]

It has long been the case that the architecture must call nmi_enter()
and nmi_exit() rather than irq_enter() and irq_exit() in order to
permit RCU read-side critical sections in NMIs.  Catch the documentation
up with reality.

Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
---
 Documentation/RCU/NMI-RCU.txt |    2 +-
 1 files changed, 1 insertions(+), 1 deletions(-)

diff --git a/Documentation/RCU/NMI-RCU.txt b/Documentation/RCU/NMI-RCU.txt
index a8536cb..84e4f9c 100644
--- a/Documentation/RCU/NMI-RCU.txt
+++ b/Documentation/RCU/NMI-RCU.txt
@@ -95,7 +95,7 @@ not to return until all ongoing NMI handlers exit.  It is therefore safe
 to free up the handler's data as soon as synchronize_sched() returns.
 
 Important note: for this to work, the architecture in question must
-invoke irq_enter() and irq_exit() on NMI entry and exit, respectively.
+invoke nmi_enter() and nmi_exit() on NMI entry and exit, respectively.
 
 
 Answer to Quick Quiz
-- 
1.7.3.2

Re: [PATCH tip/core/rcu 10/28] rcu: Fix RCU's NMI documentation

[Mathieu Desnoyers]

* Paul E. McKenney (paulmck@linux.vnet.ibm.com) wrote:
> It has long been the case that the architecture must call nmi_enter()
> and nmi_exit() rather than irq_enter() and irq_exit() in order to
> permit RCU read-side critical sections in NMIs.  Catch the documentation
> up with reality.
> 
> Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>

Acked-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>

> ---
>  Documentation/RCU/NMI-RCU.txt |    2 +-
>  1 files changed, 1 insertions(+), 1 deletions(-)
> 
> diff --git a/Documentation/RCU/NMI-RCU.txt b/Documentation/RCU/NMI-RCU.txt
> index a8536cb..84e4f9c 100644
> --- a/Documentation/RCU/NMI-RCU.txt
> +++ b/Documentation/RCU/NMI-RCU.txt
> @@ -95,7 +95,7 @@ not to return until all ongoing NMI handlers exit.  It is therefore safe
>  to free up the handler's data as soon as synchronize_sched() returns.
>  
>  Important note: for this to work, the architecture in question must
> -invoke irq_enter() and irq_exit() on NMI entry and exit, respectively.
> +invoke nmi_enter() and nmi_exit() on NMI entry and exit, respectively.
>  
>  
>  Answer to Quick Quiz
> -- 
> 1.7.3.2
> 

-- 
Mathieu Desnoyers
Operating System Efficiency R&D Consultant
EfficiOS Inc.
http://www.efficios.com

[PATCH tip/core/rcu 11/28] audit_tree,rcu: Convert call_rcu(__put_tree) to kfree_rcu()

[Paul E. McKenney]

From: Lai Jiangshan <laijs@cn.fujitsu.com>

The rcu callback __put_tree() just calls a kfree(),
so we use kfree_rcu() instead of the call_rcu(__put_tree).

Signed-off-by: Lai Jiangshan <laijs@cn.fujitsu.com>
Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
Cc: Al Viro <viro@zeniv.linux.org.uk>
Cc: Eric Paris <eparis@redhat.com>
Reviewed-by: Josh Triplett <josh@joshtriplett.org>
---
 kernel/audit_tree.c |    8 +-------
 1 files changed, 1 insertions(+), 7 deletions(-)

diff --git a/kernel/audit_tree.c b/kernel/audit_tree.c
index e99dda0..5bf0790 100644
--- a/kernel/audit_tree.c
+++ b/kernel/audit_tree.c
@@ -93,16 +93,10 @@ static inline void get_tree(struct audit_tree *tree)
 	atomic_inc(&tree->count);
 }
 
-static void __put_tree(struct rcu_head *rcu)
-{
-	struct audit_tree *tree = container_of(rcu, struct audit_tree, head);
-	kfree(tree);
-}
-
 static inline void put_tree(struct audit_tree *tree)
 {
 	if (atomic_dec_and_test(&tree->count))
-		call_rcu(&tree->head, __put_tree);
+		kfree_rcu(tree, head);
 }
 
 /* to avoid bringing the entire thing in audit.h */
-- 
1.7.3.2

[PATCH tip/core/rcu 12/28] block,rcu: Convert call_rcu(cfq_cfqd_free) to kfree_rcu()

[Paul E. McKenney]

From: Lai Jiangshan <laijs@cn.fujitsu.com>

The rcu callback cfq_cfqd_free() just calls a kfree(),
so we use kfree_rcu() instead of the call_rcu(cfq_cfqd_free).

Signed-off-by: Lai Jiangshan <laijs@cn.fujitsu.com>
Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
Cc: Jens Axboe <axboe@kernel.dk>
Reviewed-by: Josh Triplett <josh@joshtriplett.org>
---
 block/cfq-iosched.c |    7 +------
 1 files changed, 1 insertions(+), 6 deletions(-)

diff --git a/block/cfq-iosched.c b/block/cfq-iosched.c
index 5b52011..6f2a460 100644
--- a/block/cfq-iosched.c
+++ b/block/cfq-iosched.c
@@ -3816,11 +3816,6 @@ static void cfq_put_async_queues(struct cfq_data *cfqd)
 		cfq_put_queue(cfqd->async_idle_cfqq);
 }
 
-static void cfq_cfqd_free(struct rcu_head *head)
-{
-	kfree(container_of(head, struct cfq_data, rcu));
-}
-
 static void cfq_exit_queue(struct elevator_queue *e)
 {
 	struct cfq_data *cfqd = e->elevator_data;
@@ -3854,7 +3849,7 @@ static void cfq_exit_queue(struct elevator_queue *e)
 	spin_unlock(&cic_index_lock);
 
 	/* Wait for cfqg->blkg->key accessors to exit their grace periods. */
-	call_rcu(&cfqd->rcu, cfq_cfqd_free);
+	kfree_rcu(cfqd, rcu);
 }
 
 static int cfq_alloc_cic_index(void)
-- 
1.7.3.2

[PATCH tip/core/rcu 13/28] nfs,rcu: Convert call_rcu(nfs_free_delegation_callback) to kfree_rcu()

[Paul E. McKenney]

From: Lai Jiangshan <laijs@cn.fujitsu.com>

The rcu callback nfs_free_delegation_callback() just calls a kfree(),
so we use kfree_rcu() instead of the call_rcu(nfs_free_delegation_callback).

Signed-off-by: Lai Jiangshan <laijs@cn.fujitsu.com>
Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
Cc: Trond Myklebust <Trond.Myklebust@netapp.com>
Reviewed-by: Josh Triplett <josh@joshtriplett.org>
---
 fs/nfs/delegation.c |   14 +-------------
 1 files changed, 1 insertions(+), 13 deletions(-)

diff --git a/fs/nfs/delegation.c b/fs/nfs/delegation.c
index bbbc6bf..dd25c2a 100644
--- a/fs/nfs/delegation.c
+++ b/fs/nfs/delegation.c
@@ -21,25 +21,13 @@
 #include "delegation.h"
 #include "internal.h"
 
-static void nfs_do_free_delegation(struct nfs_delegation *delegation)
-{
-	kfree(delegation);
-}
-
-static void nfs_free_delegation_callback(struct rcu_head *head)
-{
-	struct nfs_delegation *delegation = container_of(head, struct nfs_delegation, rcu);
-
-	nfs_do_free_delegation(delegation);
-}
-
 static void nfs_free_delegation(struct nfs_delegation *delegation)
 {
 	if (delegation->cred) {
 		put_rpccred(delegation->cred);
 		delegation->cred = NULL;
 	}
-	call_rcu(&delegation->rcu, nfs_free_delegation_callback);
+	kfree_rcu(delegation, rcu);
 }
 
 /**
-- 
1.7.3.2

Re: [PATCH tip/core/rcu 13/28] nfs,rcu: Convert call_rcu(nfs_free_delegation_callback) to kfree_rcu()

[Trond Myklebust]

On Wed, 2011-06-08 at 12:29 -0700, Paul E. McKenney wrote: 
> From: Lai Jiangshan <laijs@cn.fujitsu.com>
> 
> The rcu callback nfs_free_delegation_callback() just calls a kfree(),
> so we use kfree_rcu() instead of the call_rcu(nfs_free_delegation_callback).
> 
> Signed-off-by: Lai Jiangshan <laijs@cn.fujitsu.com>
> Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
> Cc: Trond Myklebust <Trond.Myklebust@netapp.com>
> Reviewed-by: Josh Triplett <josh@joshtriplett.org>

Acked-by: Trond Myklebust <Trond.Myklebust@netapp.com>

> ---
>  fs/nfs/delegation.c |   14 +-------------
>  1 files changed, 1 insertions(+), 13 deletions(-)
> 
> diff --git a/fs/nfs/delegation.c b/fs/nfs/delegation.c
> index bbbc6bf..dd25c2a 100644
> --- a/fs/nfs/delegation.c
> +++ b/fs/nfs/delegation.c
> @@ -21,25 +21,13 @@
>  #include "delegation.h"
>  #include "internal.h"
>  
> -static void nfs_do_free_delegation(struct nfs_delegation *delegation)
> -{
> -	kfree(delegation);
> -}
> -
> -static void nfs_free_delegation_callback(struct rcu_head *head)
> -{
> -	struct nfs_delegation *delegation = container_of(head, struct nfs_delegation, rcu);
> -
> -	nfs_do_free_delegation(delegation);
> -}
> -
>  static void nfs_free_delegation(struct nfs_delegation *delegation)
>  {
>  	if (delegation->cred) {
>  		put_rpccred(delegation->cred);
>  		delegation->cred = NULL;
>  	}
> -	call_rcu(&delegation->rcu, nfs_free_delegation_callback);
> +	kfree_rcu(delegation, rcu);
>  }
>  
>  /**

-- 
Trond Myklebust
Linux NFS client maintainer

NetApp
Trond.Myklebust@netapp.com
www.netapp.com

[PATCH tip/core/rcu 14/28] security,rcu: Convert call_rcu(whitelist_item_free) to kfree_rcu()

[Paul E. McKenney]

From: Lai Jiangshan <laijs@cn.fujitsu.com>

The rcu callback whitelist_item_free() just calls a kfree(),
so we use kfree_rcu() instead of the call_rcu(whitelist_item_free).

Signed-off-by: Lai Jiangshan <laijs@cn.fujitsu.com>
Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
Cc: James Morris <jmorris@namei.org>
Reviewed-by: Josh Triplett <josh@joshtriplett.org>
---
 security/device_cgroup.c |   10 +---------
 1 files changed, 1 insertions(+), 9 deletions(-)

diff --git a/security/device_cgroup.c b/security/device_cgroup.c
index 8d9c48f..e886ab5 100644
--- a/security/device_cgroup.c
+++ b/security/device_cgroup.c
@@ -126,14 +126,6 @@ static int dev_whitelist_add(struct dev_cgroup *dev_cgroup,
 	return 0;
 }
 
-static void whitelist_item_free(struct rcu_head *rcu)
-{
-	struct dev_whitelist_item *item;
-
-	item = container_of(rcu, struct dev_whitelist_item, rcu);
-	kfree(item);
-}
-
 /*
  * called under devcgroup_mutex
  */
@@ -156,7 +148,7 @@ remove:
 		walk->access &= ~wh->access;
 		if (!walk->access) {
 			list_del_rcu(&walk->list);
-			call_rcu(&walk->rcu, whitelist_item_free);
+			kfree_rcu(walk, rcu);
 		}
 	}
 }
-- 
1.7.3.2

Re: [PATCH tip/core/rcu 14/28] security,rcu: Convert call_rcu(whitelist_item_free) to kfree_rcu()

[James Morris]

On Wed, 8 Jun 2011, Paul E. McKenney wrote:

> From: Lai Jiangshan <laijs@cn.fujitsu.com>
> 
> The rcu callback whitelist_item_free() just calls a kfree(),
> so we use kfree_rcu() instead of the call_rcu(whitelist_item_free).
> 
> Signed-off-by: Lai Jiangshan <laijs@cn.fujitsu.com>
> Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
> Cc: James Morris <jmorris@namei.org>
> Reviewed-by: Josh Triplett <josh@joshtriplett.org>

Acked-by: James Morris <jmorris@namei.org>

-- 
James Morris
<jmorris@namei.org>

Re: [PATCH tip/core/rcu 14/28] security,rcu: Convert call_rcu(whitelist_item_free) to kfree_rcu()

[Paul E. McKenney]

On Thu, Jun 09, 2011 at 10:39:02AM +1000, James Morris wrote:
> On Wed, 8 Jun 2011, Paul E. McKenney wrote:
> 
> > From: Lai Jiangshan <laijs@cn.fujitsu.com>
> > 
> > The rcu callback whitelist_item_free() just calls a kfree(),
> > so we use kfree_rcu() instead of the call_rcu(whitelist_item_free).
> > 
> > Signed-off-by: Lai Jiangshan <laijs@cn.fujitsu.com>
> > Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
> > Cc: James Morris <jmorris@namei.org>
> > Reviewed-by: Josh Triplett <josh@joshtriplett.org>
> 
> Acked-by: James Morris <jmorris@namei.org>

Thank you, James!

							Thanx, Paul

[PATCH tip/core/rcu 15/28] scsi,rcu: Convert call_rcu(fc_rport_free_rcu) to kfree_rcu()

[Paul E. McKenney]

From: Lai Jiangshan <laijs@cn.fujitsu.com>

The rcu callback fc_rport_free_rcu() just calls a kfree(),
so we use kfree_rcu() instead of the call_rcu(fc_rport_free_rcu).

Signed-off-by: Lai Jiangshan <laijs@cn.fujitsu.com>
Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
Cc: Robert Love <robert.w.love@intel.com>
Cc: "James E.J. Bottomley" <James.Bottomley@suse.de>
Reviewed-by: Josh Triplett <josh@joshtriplett.org>
---
 drivers/scsi/libfc/fc_rport.c |   14 +-------------
 1 files changed, 1 insertions(+), 13 deletions(-)

diff --git a/drivers/scsi/libfc/fc_rport.c b/drivers/scsi/libfc/fc_rport.c
index 49e1ccc..01e13a2 100644
--- a/drivers/scsi/libfc/fc_rport.c
+++ b/drivers/scsi/libfc/fc_rport.c
@@ -153,18 +153,6 @@ static struct fc_rport_priv *fc_rport_create(struct fc_lport *lport,
 }
 
 /**
- * fc_rport_free_rcu() - Free a remote port
- * @rcu: The rcu_head structure inside the remote port
- */
-static void fc_rport_free_rcu(struct rcu_head *rcu)
-{
-	struct fc_rport_priv *rdata;
-
-	rdata = container_of(rcu, struct fc_rport_priv, rcu);
-	kfree(rdata);
-}
-
-/**
  * fc_rport_destroy() - Free a remote port after last reference is released
  * @kref: The remote port's kref
  */
@@ -173,7 +161,7 @@ static void fc_rport_destroy(struct kref *kref)
 	struct fc_rport_priv *rdata;
 
 	rdata = container_of(kref, struct fc_rport_priv, kref);
-	call_rcu(&rdata->rcu, fc_rport_free_rcu);
+	kfree_rcu(rdata, rcu);
 }
 
 /**
-- 
1.7.3.2

[PATCH tip/core/rcu 16/28] block,rcu: Convert call_rcu(disk_free_ptbl_rcu_cb) to kfree_rcu()

[Paul E. McKenney]

From: Lai Jiangshan <laijs@cn.fujitsu.com>

The rcu callback disk_free_ptbl_rcu_cb() just calls a kfree(),
so we use kfree_rcu() instead of the call_rcu(disk_free_ptbl_rcu_cb).

Signed-off-by: Lai Jiangshan <laijs@cn.fujitsu.com>
Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
Cc: Jens Axboe <axboe@kernel.dk>
Reviewed-by: Josh Triplett <josh@joshtriplett.org>
---
 block/genhd.c |   10 +---------
 1 files changed, 1 insertions(+), 9 deletions(-)

diff --git a/block/genhd.c b/block/genhd.c
index 2dd9887..f076b5b 100644
--- a/block/genhd.c
+++ b/block/genhd.c
@@ -1018,14 +1018,6 @@ static const struct attribute_group *disk_attr_groups[] = {
 	NULL
 };
 
-static void disk_free_ptbl_rcu_cb(struct rcu_head *head)
-{
-	struct disk_part_tbl *ptbl =
-		container_of(head, struct disk_part_tbl, rcu_head);
-
-	kfree(ptbl);
-}
-
 /**
  * disk_replace_part_tbl - replace disk->part_tbl in RCU-safe way
  * @disk: disk to replace part_tbl for
@@ -1046,7 +1038,7 @@ static void disk_replace_part_tbl(struct gendisk *disk,
 
 	if (old_ptbl) {
 		rcu_assign_pointer(old_ptbl->last_lookup, NULL);
-		call_rcu(&old_ptbl->rcu_head, disk_free_ptbl_rcu_cb);
+		kfree_rcu(old_ptbl, rcu_head);
 	}
 }
 
-- 
1.7.3.2

[PATCH tip/core/rcu 17/28] ia64,rcu: Convert call_rcu(sn_irq_info_free) to kfree_rcu()

[Paul E. McKenney]

From: Lai Jiangshan <laijs@cn.fujitsu.com>

The rcu callback sn_irq_info_free() just calls a kfree(),
so we use kfree_rcu() instead of the call_rcu(sn_irq_info_free).

Signed-off-by: Lai Jiangshan <laijs@cn.fujitsu.com>
Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
Cc: Jes Sorensen <jes@sgi.com>
Cc: Tony Luck <tony.luck@intel.com>
Cc: Fenghua Yu <fenghua.yu@intel.com>
Reviewed-by: Josh Triplett <josh@joshtriplett.org>
---
 arch/ia64/sn/kernel/irq.c |   14 ++------------
 1 files changed, 2 insertions(+), 12 deletions(-)

diff --git a/arch/ia64/sn/kernel/irq.c b/arch/ia64/sn/kernel/irq.c
index 81a1f4e..485c42d 100644
--- a/arch/ia64/sn/kernel/irq.c
+++ b/arch/ia64/sn/kernel/irq.c
@@ -112,8 +112,6 @@ static void sn_ack_irq(struct irq_data *data)
 	irq_move_irq(data);
 }
 
-static void sn_irq_info_free(struct rcu_head *head);
-
 struct sn_irq_info *sn_retarget_vector(struct sn_irq_info *sn_irq_info,
 				       nasid_t nasid, int slice)
 {
@@ -177,7 +175,7 @@ struct sn_irq_info *sn_retarget_vector(struct sn_irq_info *sn_irq_info,
 	spin_lock(&sn_irq_info_lock);
 	list_replace_rcu(&sn_irq_info->list, &new_irq_info->list);
 	spin_unlock(&sn_irq_info_lock);
-	call_rcu(&sn_irq_info->rcu, sn_irq_info_free);
+	kfree_rcu(sn_irq_info, rcu);
 
 
 finish_up:
@@ -338,14 +336,6 @@ static void unregister_intr_pda(struct sn_irq_info *sn_irq_info)
 	rcu_read_unlock();
 }
 
-static void sn_irq_info_free(struct rcu_head *head)
-{
-	struct sn_irq_info *sn_irq_info;
-
-	sn_irq_info = container_of(head, struct sn_irq_info, rcu);
-	kfree(sn_irq_info);
-}
-
 void sn_irq_fixup(struct pci_dev *pci_dev, struct sn_irq_info *sn_irq_info)
 {
 	nasid_t nasid = sn_irq_info->irq_nasid;
@@ -399,7 +389,7 @@ void sn_irq_unfixup(struct pci_dev *pci_dev)
 	spin_unlock(&sn_irq_info_lock);
 	if (list_empty(sn_irq_lh[sn_irq_info->irq_irq]))
 		free_irq_vector(sn_irq_info->irq_irq);
-	call_rcu(&sn_irq_info->rcu, sn_irq_info_free);
+	kfree_rcu(sn_irq_info, rcu);
 	pci_dev_put(pci_dev);
 
 }
-- 
1.7.3.2

[PATCH tip/core/rcu 18/28] jbd2,rcu: Convert call_rcu(free_devcache) to kfree_rcu()

[Paul E. McKenney]

From: Lai Jiangshan <laijs@cn.fujitsu.com>

The rcu callback free_devcache() just calls a kfree(),
so we use kfree_rcu() instead of the call_rcu(free_devcache).

Signed-off-by: Lai Jiangshan <laijs@cn.fujitsu.com>
Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
Cc: Andrew Morton <akpm@linux-foundation.org>
Cc: Jan Kara <jack@suse.cz>
Reviewed-by: Josh Triplett <josh@joshtriplett.org>
---
 fs/jbd2/journal.c |    7 +------
 1 files changed, 1 insertions(+), 6 deletions(-)

diff --git a/fs/jbd2/journal.c b/fs/jbd2/journal.c
index e0ec3db..13fb464 100644
--- a/fs/jbd2/journal.c
+++ b/fs/jbd2/journal.c
@@ -2390,11 +2390,6 @@ struct devname_cache {
 static struct devname_cache *devcache[1 << CACHE_SIZE_BITS];
 static DEFINE_SPINLOCK(devname_cache_lock);
 
-static void free_devcache(struct rcu_head *rcu)
-{
-	kfree(rcu);
-}
-
 const char *jbd2_dev_to_name(dev_t device)
 {
 	int	i = hash_32(device, CACHE_SIZE_BITS);
@@ -2423,7 +2418,7 @@ const char *jbd2_dev_to_name(dev_t device)
 			spin_unlock(&devname_cache_lock);
 			return ret;
 		}
-		call_rcu(&devcache[i]->rcu, free_devcache);
+		kfree_rcu(devcache[i], rcu);
 	}
 	devcache[i] = new_dev;
 	devcache[i]->device = device;
-- 
1.7.3.2

Re: [PATCH tip/core/rcu 18/28] jbd2,rcu: Convert call_rcu(free_devcache) to kfree_rcu()

[Jan Kara]

On Wed 08-06-11 12:29:57, Paul E. McKenney wrote:
> From: Lai Jiangshan <laijs@cn.fujitsu.com>
> 
> The rcu callback free_devcache() just calls a kfree(),
> so we use kfree_rcu() instead of the call_rcu(free_devcache).
> 
> Signed-off-by: Lai Jiangshan <laijs@cn.fujitsu.com>
> Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
> Cc: Andrew Morton <akpm@linux-foundation.org>
> Cc: Jan Kara <jack@suse.cz>
> Reviewed-by: Josh Triplett <josh@joshtriplett.org>
  Acked-by: Jan Kara <jack@suse.cz>

								Honza
> ---
>  fs/jbd2/journal.c |    7 +------
>  1 files changed, 1 insertions(+), 6 deletions(-)
> 
> diff --git a/fs/jbd2/journal.c b/fs/jbd2/journal.c
> index e0ec3db..13fb464 100644
> --- a/fs/jbd2/journal.c
> +++ b/fs/jbd2/journal.c
> @@ -2390,11 +2390,6 @@ struct devname_cache {
>  static struct devname_cache *devcache[1 << CACHE_SIZE_BITS];
>  static DEFINE_SPINLOCK(devname_cache_lock);
>  
> -static void free_devcache(struct rcu_head *rcu)
> -{
> -	kfree(rcu);
> -}
> -
>  const char *jbd2_dev_to_name(dev_t device)
>  {
>  	int	i = hash_32(device, CACHE_SIZE_BITS);
> @@ -2423,7 +2418,7 @@ const char *jbd2_dev_to_name(dev_t device)
>  			spin_unlock(&devname_cache_lock);
>  			return ret;
>  		}
> -		call_rcu(&devcache[i]->rcu, free_devcache);
> +		kfree_rcu(devcache[i], rcu);
>  	}
>  	devcache[i] = new_dev;
>  	devcache[i]->device = device;
> -- 
> 1.7.3.2
> 
-- 
Jan Kara <jack@suse.cz>
SUSE Labs, CR

Re: [PATCH tip/core/rcu 18/28] jbd2,rcu: Convert call_rcu(free_devcache) to kfree_rcu()

[Paul E. McKenney]

On Wed, Jun 08, 2011 at 09:34:31PM +0200, Jan Kara wrote:
> On Wed 08-06-11 12:29:57, Paul E. McKenney wrote:
> > From: Lai Jiangshan <laijs@cn.fujitsu.com>
> > 
> > The rcu callback free_devcache() just calls a kfree(),
> > so we use kfree_rcu() instead of the call_rcu(free_devcache).
> > 
> > Signed-off-by: Lai Jiangshan <laijs@cn.fujitsu.com>
> > Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
> > Cc: Andrew Morton <akpm@linux-foundation.org>
> > Cc: Jan Kara <jack@suse.cz>
> > Reviewed-by: Josh Triplett <josh@joshtriplett.org>
>   Acked-by: Jan Kara <jack@suse.cz>

Thank you, Jan!

						Thanx, Paul

> 								Honza
> > ---
> >  fs/jbd2/journal.c |    7 +------
> >  1 files changed, 1 insertions(+), 6 deletions(-)
> > 
> > diff --git a/fs/jbd2/journal.c b/fs/jbd2/journal.c
> > index e0ec3db..13fb464 100644
> > --- a/fs/jbd2/journal.c
> > +++ b/fs/jbd2/journal.c
> > @@ -2390,11 +2390,6 @@ struct devname_cache {
> >  static struct devname_cache *devcache[1 << CACHE_SIZE_BITS];
> >  static DEFINE_SPINLOCK(devname_cache_lock);
> >  
> > -static void free_devcache(struct rcu_head *rcu)
> > -{
> > -	kfree(rcu);
> > -}
> > -
> >  const char *jbd2_dev_to_name(dev_t device)
> >  {
> >  	int	i = hash_32(device, CACHE_SIZE_BITS);
> > @@ -2423,7 +2418,7 @@ const char *jbd2_dev_to_name(dev_t device)
> >  			spin_unlock(&devname_cache_lock);
> >  			return ret;
> >  		}
> > -		call_rcu(&devcache[i]->rcu, free_devcache);
> > +		kfree_rcu(devcache[i], rcu);
> >  	}
> >  	devcache[i] = new_dev;
> >  	devcache[i]->device = device;
> > -- 
> > 1.7.3.2
> > 
> -- 
> Jan Kara <jack@suse.cz>
> SUSE Labs, CR
> --
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/

[PATCH tip/core/rcu 19/28] md,rcu: Convert call_rcu(free_conf) to kfree_rcu()

[Paul E. McKenney]

From: Lai Jiangshan <laijs@cn.fujitsu.com>

The rcu callback free_conf() just calls a kfree(),
so we use kfree_rcu() instead of the call_rcu(free_conf).

Signed-off-by: Lai Jiangshan <laijs@cn.fujitsu.com>
Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
Cc: Neil Brown <neilb@suse.de>
Reviewed-by: Josh Triplett <josh@joshtriplett.org>
---
 drivers/md/linear.c |    8 +-------
 1 files changed, 1 insertions(+), 7 deletions(-)

diff --git a/drivers/md/linear.c b/drivers/md/linear.c
index abfb59a..6cd2c31 100644
--- a/drivers/md/linear.c
+++ b/drivers/md/linear.c
@@ -213,12 +213,6 @@ static int linear_run (mddev_t *mddev)
 	return md_integrity_register(mddev);
 }
 
-static void free_conf(struct rcu_head *head)
-{
-	linear_conf_t *conf = container_of(head, linear_conf_t, rcu);
-	kfree(conf);
-}
-
 static int linear_add(mddev_t *mddev, mdk_rdev_t *rdev)
 {
 	/* Adding a drive to a linear array allows the array to grow.
@@ -247,7 +241,7 @@ static int linear_add(mddev_t *mddev, mdk_rdev_t *rdev)
 	md_set_array_sectors(mddev, linear_size(mddev, 0, 0));
 	set_capacity(mddev->gendisk, mddev->array_sectors);
 	revalidate_disk(mddev->gendisk);
-	call_rcu(&oldconf->rcu, free_conf);
+	kfree_rcu(oldconf, rcu);
 	return 0;
 }
 
-- 
1.7.3.2

Re: [PATCH tip/core/rcu 19/28] md,rcu: Convert call_rcu(free_conf) to kfree_rcu()

[NeilBrown]

On Wed,  8 Jun 2011 12:29:58 -0700 "Paul E. McKenney"
<paulmck@linux.vnet.ibm.com> wrote:

> From: Lai Jiangshan <laijs@cn.fujitsu.com>
> 
> The rcu callback free_conf() just calls a kfree(),
> so we use kfree_rcu() instead of the call_rcu(free_conf).
> 
> Signed-off-by: Lai Jiangshan <laijs@cn.fujitsu.com>
> Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
> Cc: Neil Brown <neilb@suse.de>
> Reviewed-by: Josh Triplett <josh@joshtriplett.org>

Acked-by: NeilBrown <neilb@suse.de>

Please submit this yourself - I won't be touching that area of code at all.

Thanks,
NeilBrown


> ---
>  drivers/md/linear.c |    8 +-------
>  1 files changed, 1 insertions(+), 7 deletions(-)
> 
> diff --git a/drivers/md/linear.c b/drivers/md/linear.c
> index abfb59a..6cd2c31 100644
> --- a/drivers/md/linear.c
> +++ b/drivers/md/linear.c
> @@ -213,12 +213,6 @@ static int linear_run (mddev_t *mddev)
>  	return md_integrity_register(mddev);
>  }
>  
> -static void free_conf(struct rcu_head *head)
> -{
> -	linear_conf_t *conf = container_of(head, linear_conf_t, rcu);
> -	kfree(conf);
> -}
> -
>  static int linear_add(mddev_t *mddev, mdk_rdev_t *rdev)
>  {
>  	/* Adding a drive to a linear array allows the array to grow.
> @@ -247,7 +241,7 @@ static int linear_add(mddev_t *mddev, mdk_rdev_t *rdev)
>  	md_set_array_sectors(mddev, linear_size(mddev, 0, 0));
>  	set_capacity(mddev->gendisk, mddev->array_sectors);
>  	revalidate_disk(mddev->gendisk);
> -	call_rcu(&oldconf->rcu, free_conf);
> +	kfree_rcu(oldconf, rcu);
>  	return 0;
>  }
>  

[PATCH tip/core/rcu 20/28] security,rcu: Convert call_rcu(sel_netnode_free) to kfree_rcu()

[Paul E. McKenney]

From: Lai Jiangshan <laijs@cn.fujitsu.com>

The rcu callback sel_netnode_free() just calls a kfree(),
so we use kfree_rcu() instead of the call_rcu(sel_netnode_free).

Signed-off-by: Lai Jiangshan <laijs@cn.fujitsu.com>
Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
Cc: Stephen Smalley <sds@tycho.nsa.gov>
Cc: James Morris <jmorris@namei.org>
Cc: Eric Paris <eparis@parisplace.org>
Reviewed-by: Josh Triplett <josh@joshtriplett.org>
---
 security/selinux/netnode.c |   20 ++------------------
 1 files changed, 2 insertions(+), 18 deletions(-)

diff --git a/security/selinux/netnode.c b/security/selinux/netnode.c
index 65ebfe9..692b857 100644
--- a/security/selinux/netnode.c
+++ b/security/selinux/netnode.c
@@ -69,22 +69,6 @@ static DEFINE_SPINLOCK(sel_netnode_lock);
 static struct sel_netnode_bkt sel_netnode_hash[SEL_NETNODE_HASH_SIZE];
 
 /**
- * sel_netnode_free - Frees a node entry
- * @p: the entry's RCU field
- *
- * Description:
- * This function is designed to be used as a callback to the call_rcu()
- * function so that memory allocated to a hash table node entry can be
- * released safely.
- *
- */
-static void sel_netnode_free(struct rcu_head *p)
-{
-	struct sel_netnode *node = container_of(p, struct sel_netnode, rcu);
-	kfree(node);
-}
-
-/**
  * sel_netnode_hashfn_ipv4 - IPv4 hashing function for the node table
  * @addr: IPv4 address
  *
@@ -192,7 +176,7 @@ static void sel_netnode_insert(struct sel_netnode *node)
 			rcu_dereference(sel_netnode_hash[idx].list.prev),
 			struct sel_netnode, list);
 		list_del_rcu(&tail->list);
-		call_rcu(&tail->rcu, sel_netnode_free);
+		kfree_rcu(tail, rcu);
 	} else
 		sel_netnode_hash[idx].size++;
 }
@@ -305,7 +289,7 @@ static void sel_netnode_flush(void)
 		list_for_each_entry_safe(node, node_tmp,
 					 &sel_netnode_hash[idx].list, list) {
 				list_del_rcu(&node->list);
-				call_rcu(&node->rcu, sel_netnode_free);
+				kfree_rcu(node, rcu);
 		}
 		sel_netnode_hash[idx].size = 0;
 	}
-- 
1.7.3.2

[PATCH tip/core/rcu 21/28] security,rcu: Convert call_rcu(sel_netport_free) to kfree_rcu()

[Paul E. McKenney]

From: Lai Jiangshan <laijs@cn.fujitsu.com>

The rcu callback sel_netport_free() just calls a kfree(),
so we use kfree_rcu() instead of the call_rcu(sel_netport_free).

Signed-off-by: Lai Jiangshan <laijs@cn.fujitsu.com>
Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
Cc: Stephen Smalley <sds@tycho.nsa.gov>
Cc: James Morris <jmorris@namei.org>
Cc: Eric Paris <eparis@parisplace.org>
Reviewed-by: Josh Triplett <josh@joshtriplett.org>
---
 security/selinux/netport.c |   20 ++------------------
 1 files changed, 2 insertions(+), 18 deletions(-)

diff --git a/security/selinux/netport.c b/security/selinux/netport.c
index cfe2d72..ae76e29 100644
--- a/security/selinux/netport.c
+++ b/security/selinux/netport.c
@@ -68,22 +68,6 @@ static DEFINE_SPINLOCK(sel_netport_lock);
 static struct sel_netport_bkt sel_netport_hash[SEL_NETPORT_HASH_SIZE];
 
 /**
- * sel_netport_free - Frees a port entry
- * @p: the entry's RCU field
- *
- * Description:
- * This function is designed to be used as a callback to the call_rcu()
- * function so that memory allocated to a hash table port entry can be
- * released safely.
- *
- */
-static void sel_netport_free(struct rcu_head *p)
-{
-	struct sel_netport *port = container_of(p, struct sel_netport, rcu);
-	kfree(port);
-}
-
-/**
  * sel_netport_hashfn - Hashing function for the port table
  * @pnum: port number
  *
@@ -142,7 +126,7 @@ static void sel_netport_insert(struct sel_netport *port)
 			rcu_dereference(sel_netport_hash[idx].list.prev),
 			struct sel_netport, list);
 		list_del_rcu(&tail->list);
-		call_rcu(&tail->rcu, sel_netport_free);
+		kfree_rcu(tail, rcu);
 	} else
 		sel_netport_hash[idx].size++;
 }
@@ -241,7 +225,7 @@ static void sel_netport_flush(void)
 		list_for_each_entry_safe(port, port_tmp,
 					 &sel_netport_hash[idx].list, list) {
 			list_del_rcu(&port->list);
-			call_rcu(&port->rcu, sel_netport_free);
+			kfree_rcu(port, rcu);
 		}
 		sel_netport_hash[idx].size = 0;
 	}
-- 
1.7.3.2

[PATCH tip/core/rcu 22/28] ipc,rcu: Convert call_rcu(free_un) to kfree_rcu()

[Paul E. McKenney]

From: Lai Jiangshan <laijs@cn.fujitsu.com>

The rcu callback free_un() just calls a kfree(),
so we use kfree_rcu() instead of the call_rcu(free_un).

Signed-off-by: Lai Jiangshan <laijs@cn.fujitsu.com>
Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
Cc: Andrew Morton <akpm@linux-foundation.org>
Cc: Manfred Spraul <manfred@colorfullife.com>
Reviewed-by: Josh Triplett <josh@joshtriplett.org>
---
 ipc/sem.c |   10 ++--------
 1 files changed, 2 insertions(+), 8 deletions(-)

diff --git a/ipc/sem.c b/ipc/sem.c
index 34193ed..8b929e6 100644
--- a/ipc/sem.c
+++ b/ipc/sem.c
@@ -689,12 +689,6 @@ static int count_semzcnt (struct sem_array * sma, ushort semnum)
 	return semzcnt;
 }
 
-static void free_un(struct rcu_head *head)
-{
-	struct sem_undo *un = container_of(head, struct sem_undo, rcu);
-	kfree(un);
-}
-
 /* Free a semaphore set. freeary() is called with sem_ids.rw_mutex locked
  * as a writer and the spinlock for this semaphore set hold. sem_ids.rw_mutex
  * remains locked on exit.
@@ -714,7 +708,7 @@ static void freeary(struct ipc_namespace *ns, struct kern_ipc_perm *ipcp)
 		un->semid = -1;
 		list_del_rcu(&un->list_proc);
 		spin_unlock(&un->ulp->lock);
-		call_rcu(&un->rcu, free_un);
+		kfree_rcu(un, rcu);
 	}
 
 	/* Wake up all pending processes and let them fail with EIDRM. */
@@ -1612,7 +1606,7 @@ void exit_sem(struct task_struct *tsk)
 		sem_unlock(sma);
 		wake_up_sem_queue_do(&tasks);
 
-		call_rcu(&un->rcu, free_un);
+		kfree_rcu(un, rcu);
 	}
 	kfree(ulp);
 }
-- 
1.7.3.2

[PATCH tip/core/rcu 23/28] ipc,rcu: Convert call_rcu(ipc_immediate_free) to kfree_rcu()

[Paul E. McKenney]

From: Lai Jiangshan <laijs@cn.fujitsu.com>

The rcu callback ipc_immediate_free() just calls a kfree(),
so we use kfree_rcu() instead of the call_rcu(ipc_immediate_free).

Signed-off-by: Lai Jiangshan <laijs@cn.fujitsu.com>
Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
Cc: Andrew Morton <akpm@linux-foundation.org>
Reviewed-by: Josh Triplett <josh@joshtriplett.org>
---
 ipc/util.c |   16 +---------------
 1 files changed, 1 insertions(+), 15 deletions(-)

diff --git a/ipc/util.c b/ipc/util.c
index 5c0d289..75261a3 100644
--- a/ipc/util.c
+++ b/ipc/util.c
@@ -579,19 +579,6 @@ static void ipc_schedule_free(struct rcu_head *head)
 	schedule_work(&sched->work);
 }
 
-/**
- * ipc_immediate_free - free ipc + rcu space
- * @head: RCU callback structure that contains pointer to be freed
- *
- * Free from the RCU callback context.
- */
-static void ipc_immediate_free(struct rcu_head *head)
-{
-	struct ipc_rcu_grace *free =
-		container_of(head, struct ipc_rcu_grace, rcu);
-	kfree(free);
-}
-
 void ipc_rcu_putref(void *ptr)
 {
 	if (--container_of(ptr, struct ipc_rcu_hdr, data)->refcount > 0)
@@ -601,8 +588,7 @@ void ipc_rcu_putref(void *ptr)
 		call_rcu(&container_of(ptr, struct ipc_rcu_grace, data)->rcu,
 				ipc_schedule_free);
 	} else {
-		call_rcu(&container_of(ptr, struct ipc_rcu_grace, data)->rcu,
-				ipc_immediate_free);
+		kfree_rcu(container_of(ptr, struct ipc_rcu_grace, data), rcu);
 	}
 }
 
-- 
1.7.3.2

[PATCH tip/core/rcu 24/28] vmalloc,rcu: Convert call_rcu(rcu_free_va) to kfree_rcu()

[Paul E. McKenney]

From: Lai Jiangshan <laijs@cn.fujitsu.com>

The rcu callback rcu_free_va() just calls a kfree(),
so we use kfree_rcu() instead of the call_rcu(rcu_free_va).

Signed-off-by: Lai Jiangshan <laijs@cn.fujitsu.com>
Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
Cc: Andrew Morton <akpm@linux-foundation
Cc: Namhyung Kim <namhyung@gmail.com>
Cc: David Rientjes <rientjes@google.com>
Reviewed-by: Josh Triplett <josh@joshtriplett.org>
---
 mm/vmalloc.c |    9 +--------
 1 files changed, 1 insertions(+), 8 deletions(-)

diff --git a/mm/vmalloc.c b/mm/vmalloc.c
index 5d60302..73f56c8 100644
--- a/mm/vmalloc.c
+++ b/mm/vmalloc.c
@@ -452,13 +452,6 @@ overflow:
 	return ERR_PTR(-EBUSY);
 }
 
-static void rcu_free_va(struct rcu_head *head)
-{
-	struct vmap_area *va = container_of(head, struct vmap_area, rcu_head);
-
-	kfree(va);
-}
-
 static void __free_vmap_area(struct vmap_area *va)
 {
 	BUG_ON(RB_EMPTY_NODE(&va->rb_node));
@@ -491,7 +484,7 @@ static void __free_vmap_area(struct vmap_area *va)
 	if (va->va_end > VMALLOC_START && va->va_end <= VMALLOC_END)
 		vmap_area_pcpu_hole = max(vmap_area_pcpu_hole, va->va_end);
 
-	call_rcu(&va->rcu_head, rcu_free_va);
+	kfree_rcu(va, rcu_head);
 }
 
 /*
-- 
1.7.3.2

[PATCH tip/core/rcu 25/28] vmalloc,rcu: Convert call_rcu(rcu_free_vb) to kfree_rcu()

[Paul E. McKenney]

From: Lai Jiangshan <laijs@cn.fujitsu.com>

The rcu callback rcu_free_vb() just calls a kfree(),
so we use kfree_rcu() instead of the call_rcu(rcu_free_vb).

Signed-off-by: Lai Jiangshan <laijs@cn.fujitsu.com>
Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
Cc: Andrew Morton <akpm@linux-foundation.org>
Cc: Namhyung Kim <namhyung@gmail.com>
Cc: David Rientjes <rientjes@google.com>
Reviewed-by: Josh Triplett <josh@joshtriplett.org>
---
 mm/vmalloc.c |    9 +--------
 1 files changed, 1 insertions(+), 8 deletions(-)

diff --git a/mm/vmalloc.c b/mm/vmalloc.c
index 73f56c8..6540116 100644
--- a/mm/vmalloc.c
+++ b/mm/vmalloc.c
@@ -830,13 +830,6 @@ static struct vmap_block *new_vmap_block(gfp_t gfp_mask)
 	return vb;
 }
 
-static void rcu_free_vb(struct rcu_head *head)
-{
-	struct vmap_block *vb = container_of(head, struct vmap_block, rcu_head);
-
-	kfree(vb);
-}
-
 static void free_vmap_block(struct vmap_block *vb)
 {
 	struct vmap_block *tmp;
@@ -849,7 +842,7 @@ static void free_vmap_block(struct vmap_block *vb)
 	BUG_ON(tmp != vb);
 
 	free_vmap_area_noflush(vb->va);
-	call_rcu(&vb->rcu_head, rcu_free_vb);
+	kfree_rcu(vb, rcu_head);
 }
 
 static void purge_fragmented_blocks(int cpu)
-- 
1.7.3.2

[PATCH tip/core/rcu 26/28] sysctl,rcu: Convert call_rcu(free_head) to kfree

[Paul E. McKenney]

The RCU callback free_head just calls kfree(), so we can use kfree_rcu()
instead of call_rcu().

Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
Cc: Andrew Morton <akpm@linux-foundation.org>
Reviewed-by: Josh Triplett <josh@joshtriplett.org>
---
 kernel/sysctl.c |   11 +++--------
 1 files changed, 3 insertions(+), 8 deletions(-)

diff --git a/kernel/sysctl.c b/kernel/sysctl.c
index c0bb324..39dbad0 100644
--- a/kernel/sysctl.c
+++ b/kernel/sysctl.c
@@ -1574,16 +1574,11 @@ void sysctl_head_get(struct ctl_table_header *head)
 	spin_unlock(&sysctl_lock);
 }
 
-static void free_head(struct rcu_head *rcu)
-{
-	kfree(container_of(rcu, struct ctl_table_header, rcu));
-}
-
 void sysctl_head_put(struct ctl_table_header *head)
 {
 	spin_lock(&sysctl_lock);
 	if (!--head->count)
-		call_rcu(&head->rcu, free_head);
+		kfree_rcu(head, rcu);
 	spin_unlock(&sysctl_lock);
 }
 
@@ -1955,10 +1950,10 @@ void unregister_sysctl_table(struct ctl_table_header * header)
 	start_unregistering(header);
 	if (!--header->parent->count) {
 		WARN_ON(1);
-		call_rcu(&header->parent->rcu, free_head);
+		kfree_rcu(header->parent, rcu);
 	}
 	if (!--header->count)
-		call_rcu(&header->rcu, free_head);
+		kfree_rcu(header, rcu);
 	spin_unlock(&sysctl_lock);
 }
 
-- 
1.7.3.2

[PATCH tip/core/rcu 27/28] net,rcu: Convert call_rcu(xt_rateest_free_rcu) to kfree_rcu()

[Paul E. McKenney]

The RCU callback xt_rateest_free_rcu() just calls kfree(), so we can
use kfree_rcu() instead of call_rcu().  This also allows us to dispense
with an rcu_barrier() call, speeding up unloading of this module.

Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
Cc: Patrick McHardy <kaber@trash.net>
Reviewed-by: Josh Triplett <josh@joshtriplett.org>
---
 net/netfilter/xt_RATEEST.c |    8 +-------
 1 files changed, 1 insertions(+), 7 deletions(-)

diff --git a/net/netfilter/xt_RATEEST.c b/net/netfilter/xt_RATEEST.c
index de079abd..f264032 100644
--- a/net/netfilter/xt_RATEEST.c
+++ b/net/netfilter/xt_RATEEST.c
@@ -60,11 +60,6 @@ struct xt_rateest *xt_rateest_lookup(const char *name)
 }
 EXPORT_SYMBOL_GPL(xt_rateest_lookup);
 
-static void xt_rateest_free_rcu(struct rcu_head *head)
-{
-	kfree(container_of(head, struct xt_rateest, rcu));
-}
-
 void xt_rateest_put(struct xt_rateest *est)
 {
 	mutex_lock(&xt_rateest_mutex);
@@ -75,7 +70,7 @@ void xt_rateest_put(struct xt_rateest *est)
 		 * gen_estimator est_timer() might access est->lock or bstats,
 		 * wait a RCU grace period before freeing 'est'
 		 */
-		call_rcu(&est->rcu, xt_rateest_free_rcu);
+		kfree_rcu(est, rcu);
 	}
 	mutex_unlock(&xt_rateest_mutex);
 }
@@ -188,7 +183,6 @@ static int __init xt_rateest_tg_init(void)
 static void __exit xt_rateest_tg_fini(void)
 {
 	xt_unregister_target(&xt_rateest_tg_reg);
-	rcu_barrier(); /* Wait for completion of call_rcu()'s (xt_rateest_free_rcu) */
 }
 
 
-- 
1.7.3.2

[PATCH tip/core/rcu 28/28] net,rcu: Convert call_rcu(sk_filter_release_rcu) to kfree_rcu

[Paul E. McKenney]

The RCU callback sk_filter_release_rcu just calls kfree(), so we use
kfree_rcu() instead of call_rcu().

Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
Cc: Patrick McHardy <kaber@trash.net>
Acked-by: David S. Miller <davem@davemloft.net>
Reviewed-by: Josh Triplett <josh@joshtriplett.org>
---
 include/net/sock.h |    4 +---
 net/core/filter.c  |   12 ------------
 2 files changed, 1 insertions(+), 15 deletions(-)

diff --git a/include/net/sock.h b/include/net/sock.h
index 01810a3..60caa06 100644
--- a/include/net/sock.h
+++ b/include/net/sock.h
@@ -1179,8 +1179,6 @@ extern void sk_common_release(struct sock *sk);
 /* Initialise core socket variables */
 extern void sock_init_data(struct socket *sock, struct sock *sk);
 
-extern void sk_filter_release_rcu(struct rcu_head *rcu);
-
 /**
  *	sk_filter_release - release a socket filter
  *	@fp: filter to remove
@@ -1191,7 +1189,7 @@ extern void sk_filter_release_rcu(struct rcu_head *rcu);
 static inline void sk_filter_release(struct sk_filter *fp)
 {
 	if (atomic_dec_and_test(&fp->refcnt))
-		call_rcu(&fp->rcu, sk_filter_release_rcu);
+		kfree_rcu(fp, rcu);
 }
 
 static inline void sk_filter_uncharge(struct sock *sk, struct sk_filter *fp)
diff --git a/net/core/filter.c b/net/core/filter.c
index afb8afb..7195ce0 100644
--- a/net/core/filter.c
+++ b/net/core/filter.c
@@ -631,18 +631,6 @@ int sk_chk_filter(struct sock_filter *filter, int flen)
 EXPORT_SYMBOL(sk_chk_filter);
 
 /**
- * 	sk_filter_release_rcu - Release a socket filter by rcu_head
- *	@rcu: rcu_head that contains the sk_filter to free
- */
-void sk_filter_release_rcu(struct rcu_head *rcu)
-{
-	struct sk_filter *fp = container_of(rcu, struct sk_filter, rcu);
-
-	kfree(fp);
-}
-EXPORT_SYMBOL(sk_filter_release_rcu);
-
-/**
  *	sk_attach_filter - attach a socket filter
  *	@fprog: the filter program
  *	@sk: the socket to use
-- 
1.7.3.2