* [PATCH tip/core/rcu 02/22] rcu: Four-level hierarchy is no longer experimental
2012-06-22 15:17 ` [PATCH tip/core/rcu 01/22] rcu: Control RCU_FANOUT_LEAF from boot-time parameter Paul E. McKenney
@ 2012-06-22 15:17 ` Paul E. McKenney
2012-06-22 15:17 ` [PATCH tip/core/rcu 03/22] rcu: Size rcu_node tree from nr_cpu_ids rather than NR_CPUS Paul E. McKenney
` (20 subsequent siblings)
21 siblings, 0 replies; 27+ messages in thread
From: Paul E. McKenney @ 2012-06-22 15:17 UTC (permalink / raw)
To: linux-kernel
Cc: mingo, laijs, dipankar, akpm, mathieu.desnoyers, josh, niv, tglx,
peterz, rostedt, Valdis.Kletnieks, dhowells, eric.dumazet, darren,
fweisbec, sbw, patches, Paul E. McKenney
From: "Paul E. McKenney" <paulmck@linux.vnet.ibm.com>
Time to make the four-level-hierarchy setting less scary, so this
commit removes "Experimental" from the boot-time message. Leave the
message in order to get a heads-up on any possible need to expand to
a five-level hierarchy.
Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
---
kernel/rcutree_plugin.h | 2 +-
1 files changed, 1 insertions(+), 1 deletions(-)
diff --git a/kernel/rcutree_plugin.h b/kernel/rcutree_plugin.h
index 1f27554..61aab6f 100644
--- a/kernel/rcutree_plugin.h
+++ b/kernel/rcutree_plugin.h
@@ -68,7 +68,7 @@ static void __init rcu_bootup_announce_oddness(void)
printk(KERN_INFO "\tAdditional per-CPU info printed with stalls.\n");
#endif
#if NUM_RCU_LVL_4 != 0
- printk(KERN_INFO "\tExperimental four-level hierarchy is enabled.\n");
+ printk(KERN_INFO "\tFour-level hierarchy is enabled.\n");
#endif
if (rcu_fanout_leaf != CONFIG_RCU_FANOUT_LEAF)
printk(KERN_INFO "\tExperimental boot-time adjustment of leaf fanout to %d.\n", rcu_fanout_leaf);
--
1.7.8
^ permalink raw reply related [flat|nested] 27+ messages in thread* [PATCH tip/core/rcu 03/22] rcu: Size rcu_node tree from nr_cpu_ids rather than NR_CPUS
2012-06-22 15:17 ` [PATCH tip/core/rcu 01/22] rcu: Control RCU_FANOUT_LEAF from boot-time parameter Paul E. McKenney
2012-06-22 15:17 ` [PATCH tip/core/rcu 02/22] rcu: Four-level hierarchy is no longer experimental Paul E. McKenney
@ 2012-06-22 15:17 ` Paul E. McKenney
2012-06-22 15:17 ` [PATCH tip/core/rcu 04/22] rcu: Prevent excessive line length in RCU_STATE_INITIALIZER() Paul E. McKenney
` (19 subsequent siblings)
21 siblings, 0 replies; 27+ messages in thread
From: Paul E. McKenney @ 2012-06-22 15:17 UTC (permalink / raw)
To: linux-kernel
Cc: mingo, laijs, dipankar, akpm, mathieu.desnoyers, josh, niv, tglx,
peterz, rostedt, Valdis.Kletnieks, dhowells, eric.dumazet, darren,
fweisbec, sbw, patches, Paul E. McKenney
From: "Paul E. McKenney" <paulmck@linux.vnet.ibm.com>
The rcu_node tree array is sized based on compile-time constants,
including NR_CPUS. Although this approach has worked well in the past,
the recent trend by many distros to define NR_CPUS=4096 results in
excessive grace-period-initialization latencies.
This commit therefore substitutes the run-time computed nr_cpu_ids for
the compile-time NR_CPUS when building the tree. This can result in
much of the compile-time-allocated rcu_node array being unused. If
this is a major problem, you are in a specialized situation anyway,
so you can manually adjust the NR_CPUS, RCU_FANOUT, and RCU_FANOUT_LEAF
kernel config parameters.
Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
---
kernel/rcutree.c | 2 +-
kernel/rcutree_plugin.h | 2 ++
2 files changed, 3 insertions(+), 1 deletions(-)
diff --git a/kernel/rcutree.c b/kernel/rcutree.c
index 260c1a3..f06a655 100644
--- a/kernel/rcutree.c
+++ b/kernel/rcutree.c
@@ -2672,7 +2672,7 @@ static void __init rcu_init_geometry(void)
{
int i;
int j;
- int n = NR_CPUS;
+ int n = nr_cpu_ids;
int rcu_capacity[MAX_RCU_LVLS + 1];
/* If the compile-time values are accurate, just leave. */
diff --git a/kernel/rcutree_plugin.h b/kernel/rcutree_plugin.h
index 61aab6f..67b3b97 100644
--- a/kernel/rcutree_plugin.h
+++ b/kernel/rcutree_plugin.h
@@ -72,6 +72,8 @@ static void __init rcu_bootup_announce_oddness(void)
#endif
if (rcu_fanout_leaf != CONFIG_RCU_FANOUT_LEAF)
printk(KERN_INFO "\tExperimental boot-time adjustment of leaf fanout to %d.\n", rcu_fanout_leaf);
+ if (nr_cpu_ids != NR_CPUS)
+ printk(KERN_INFO "\tRCU restricting CPUs from NR_CPUS=%d to nr_cpu_ids=%d.\n", NR_CPUS, nr_cpu_ids);
}
#ifdef CONFIG_TREE_PREEMPT_RCU
--
1.7.8
^ permalink raw reply related [flat|nested] 27+ messages in thread* [PATCH tip/core/rcu 04/22] rcu: Prevent excessive line length in RCU_STATE_INITIALIZER()
2012-06-22 15:17 ` [PATCH tip/core/rcu 01/22] rcu: Control RCU_FANOUT_LEAF from boot-time parameter Paul E. McKenney
2012-06-22 15:17 ` [PATCH tip/core/rcu 02/22] rcu: Four-level hierarchy is no longer experimental Paul E. McKenney
2012-06-22 15:17 ` [PATCH tip/core/rcu 03/22] rcu: Size rcu_node tree from nr_cpu_ids rather than NR_CPUS Paul E. McKenney
@ 2012-06-22 15:17 ` Paul E. McKenney
2012-06-22 15:17 ` [PATCH tip/core/rcu 05/22] rcu: Place pointer to call_rcu() in rcu_data structure Paul E. McKenney
` (18 subsequent siblings)
21 siblings, 0 replies; 27+ messages in thread
From: Paul E. McKenney @ 2012-06-22 15:17 UTC (permalink / raw)
To: linux-kernel
Cc: mingo, laijs, dipankar, akpm, mathieu.desnoyers, josh, niv, tglx,
peterz, rostedt, Valdis.Kletnieks, dhowells, eric.dumazet, darren,
fweisbec, sbw, patches, Paul E. McKenney, Paul E. McKenney
From: "Paul E. McKenney" <paul.mckenney@linaro.org>
Upcoming rcu_barrier() concurrency commits will result in line lengths
greater than 80 characters in the RCU_STATE_INITIALIZER(), so this commit
shortens the name of the macro's argument to prevent this.
Signed-off-by: Paul E. McKenney <paul.mckenney@linaro.org>
Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
Reviewed-by: Josh Triplett <josh@joshtriplett.org>
---
kernel/rcutree.c | 14 +++++++-------
1 files changed, 7 insertions(+), 7 deletions(-)
diff --git a/kernel/rcutree.c b/kernel/rcutree.c
index f06a655..4580126 100644
--- a/kernel/rcutree.c
+++ b/kernel/rcutree.c
@@ -62,18 +62,18 @@
static struct lock_class_key rcu_node_class[RCU_NUM_LVLS];
-#define RCU_STATE_INITIALIZER(structname) { \
- .level = { &structname##_state.node[0] }, \
+#define RCU_STATE_INITIALIZER(sname) { \
+ .level = { &sname##_state.node[0] }, \
.fqs_state = RCU_GP_IDLE, \
.gpnum = -300, \
.completed = -300, \
- .onofflock = __RAW_SPIN_LOCK_UNLOCKED(&structname##_state.onofflock), \
- .orphan_nxttail = &structname##_state.orphan_nxtlist, \
- .orphan_donetail = &structname##_state.orphan_donelist, \
- .fqslock = __RAW_SPIN_LOCK_UNLOCKED(&structname##_state.fqslock), \
+ .onofflock = __RAW_SPIN_LOCK_UNLOCKED(&sname##_state.onofflock), \
+ .orphan_nxttail = &sname##_state.orphan_nxtlist, \
+ .orphan_donetail = &sname##_state.orphan_donelist, \
+ .fqslock = __RAW_SPIN_LOCK_UNLOCKED(&sname##_state.fqslock), \
.n_force_qs = 0, \
.n_force_qs_ngp = 0, \
- .name = #structname, \
+ .name = #sname, \
}
struct rcu_state rcu_sched_state = RCU_STATE_INITIALIZER(rcu_sched);
--
1.7.8
^ permalink raw reply related [flat|nested] 27+ messages in thread* [PATCH tip/core/rcu 05/22] rcu: Place pointer to call_rcu() in rcu_data structure
2012-06-22 15:17 ` [PATCH tip/core/rcu 01/22] rcu: Control RCU_FANOUT_LEAF from boot-time parameter Paul E. McKenney
` (2 preceding siblings ...)
2012-06-22 15:17 ` [PATCH tip/core/rcu 04/22] rcu: Prevent excessive line length in RCU_STATE_INITIALIZER() Paul E. McKenney
@ 2012-06-22 15:17 ` Paul E. McKenney
2012-06-22 15:17 ` [PATCH tip/core/rcu 06/22] rcu: Move _rcu_barrier()'s rcu_head structures to rcu_data structures Paul E. McKenney
` (17 subsequent siblings)
21 siblings, 0 replies; 27+ messages in thread
From: Paul E. McKenney @ 2012-06-22 15:17 UTC (permalink / raw)
To: linux-kernel
Cc: mingo, laijs, dipankar, akpm, mathieu.desnoyers, josh, niv, tglx,
peterz, rostedt, Valdis.Kletnieks, dhowells, eric.dumazet, darren,
fweisbec, sbw, patches, Paul E. McKenney, Paul E. McKenney
From: "Paul E. McKenney" <paul.mckenney@linaro.org>
This is a preparatory commit for increasing rcu_barrier()'s concurrency.
It adds a pointer in the rcu_data structure to the corresponding call_rcu()
function. This allows a pointer to the rcu_data structure to imply the
function pointer, which allows _rcu_barrier() state to be placed in the
rcu_state structure.
Signed-off-by: Paul E. McKenney <paul.mckenney@linaro.org>
Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
Reviewed-by: Josh Triplett <josh@joshtriplett.org>
---
kernel/rcutree.c | 27 ++++++++++++---------------
kernel/rcutree.h | 2 ++
kernel/rcutree_plugin.h | 5 +++--
3 files changed, 17 insertions(+), 17 deletions(-)
diff --git a/kernel/rcutree.c b/kernel/rcutree.c
index 4580126..24cd2f0 100644
--- a/kernel/rcutree.c
+++ b/kernel/rcutree.c
@@ -62,8 +62,9 @@
static struct lock_class_key rcu_node_class[RCU_NUM_LVLS];
-#define RCU_STATE_INITIALIZER(sname) { \
+#define RCU_STATE_INITIALIZER(sname, cr) { \
.level = { &sname##_state.node[0] }, \
+ .call = cr, \
.fqs_state = RCU_GP_IDLE, \
.gpnum = -300, \
.completed = -300, \
@@ -76,10 +77,11 @@ static struct lock_class_key rcu_node_class[RCU_NUM_LVLS];
.name = #sname, \
}
-struct rcu_state rcu_sched_state = RCU_STATE_INITIALIZER(rcu_sched);
+struct rcu_state rcu_sched_state =
+ RCU_STATE_INITIALIZER(rcu_sched, call_rcu_sched);
DEFINE_PER_CPU(struct rcu_data, rcu_sched_data);
-struct rcu_state rcu_bh_state = RCU_STATE_INITIALIZER(rcu_bh);
+struct rcu_state rcu_bh_state = RCU_STATE_INITIALIZER(rcu_bh, call_rcu_bh);
DEFINE_PER_CPU(struct rcu_data, rcu_bh_data);
static struct rcu_state *rcu_state;
@@ -2279,21 +2281,17 @@ static void rcu_barrier_func(void *type)
{
int cpu = smp_processor_id();
struct rcu_head *head = &per_cpu(rcu_barrier_head, cpu);
- void (*call_rcu_func)(struct rcu_head *head,
- void (*func)(struct rcu_head *head));
+ struct rcu_state *rsp = type;
atomic_inc(&rcu_barrier_cpu_count);
- call_rcu_func = type;
- call_rcu_func(head, rcu_barrier_callback);
+ rsp->call(head, rcu_barrier_callback);
}
/*
* Orchestrate the specified type of RCU barrier, waiting for all
* RCU callbacks of the specified type to complete.
*/
-static void _rcu_barrier(struct rcu_state *rsp,
- void (*call_rcu_func)(struct rcu_head *head,
- void (*func)(struct rcu_head *head)))
+static void _rcu_barrier(struct rcu_state *rsp)
{
int cpu;
unsigned long flags;
@@ -2345,8 +2343,7 @@ static void _rcu_barrier(struct rcu_state *rsp,
while (cpu_is_offline(cpu) && ACCESS_ONCE(rdp->qlen))
schedule_timeout_interruptible(1);
} else if (ACCESS_ONCE(rdp->qlen)) {
- smp_call_function_single(cpu, rcu_barrier_func,
- (void *)call_rcu_func, 1);
+ smp_call_function_single(cpu, rcu_barrier_func, rsp, 1);
preempt_enable();
} else {
preempt_enable();
@@ -2367,7 +2364,7 @@ static void _rcu_barrier(struct rcu_state *rsp,
raw_spin_unlock_irqrestore(&rsp->onofflock, flags);
atomic_inc(&rcu_barrier_cpu_count);
smp_mb__after_atomic_inc(); /* Ensure atomic_inc() before callback. */
- call_rcu_func(&rh, rcu_barrier_callback);
+ rsp->call(&rh, rcu_barrier_callback);
/*
* Now that we have an rcu_barrier_callback() callback on each
@@ -2390,7 +2387,7 @@ static void _rcu_barrier(struct rcu_state *rsp,
*/
void rcu_barrier_bh(void)
{
- _rcu_barrier(&rcu_bh_state, call_rcu_bh);
+ _rcu_barrier(&rcu_bh_state);
}
EXPORT_SYMBOL_GPL(rcu_barrier_bh);
@@ -2399,7 +2396,7 @@ EXPORT_SYMBOL_GPL(rcu_barrier_bh);
*/
void rcu_barrier_sched(void)
{
- _rcu_barrier(&rcu_sched_state, call_rcu_sched);
+ _rcu_barrier(&rcu_sched_state);
}
EXPORT_SYMBOL_GPL(rcu_barrier_sched);
diff --git a/kernel/rcutree.h b/kernel/rcutree.h
index df3c2c8..15837d7 100644
--- a/kernel/rcutree.h
+++ b/kernel/rcutree.h
@@ -350,6 +350,8 @@ struct rcu_state {
u32 levelcnt[MAX_RCU_LVLS + 1]; /* # nodes in each level. */
u8 levelspread[RCU_NUM_LVLS]; /* kids/node in each level. */
struct rcu_data __percpu *rda; /* pointer of percu rcu_data. */
+ void (*call)(struct rcu_head *head, /* call_rcu() flavor. */
+ void (*func)(struct rcu_head *head));
/* The following fields are guarded by the root rcu_node's lock. */
diff --git a/kernel/rcutree_plugin.h b/kernel/rcutree_plugin.h
index 67b3b97..a401508 100644
--- a/kernel/rcutree_plugin.h
+++ b/kernel/rcutree_plugin.h
@@ -78,7 +78,8 @@ static void __init rcu_bootup_announce_oddness(void)
#ifdef CONFIG_TREE_PREEMPT_RCU
-struct rcu_state rcu_preempt_state = RCU_STATE_INITIALIZER(rcu_preempt);
+struct rcu_state rcu_preempt_state =
+ RCU_STATE_INITIALIZER(rcu_preempt, call_rcu);
DEFINE_PER_CPU(struct rcu_data, rcu_preempt_data);
static struct rcu_state *rcu_state = &rcu_preempt_state;
@@ -944,7 +945,7 @@ static int rcu_preempt_cpu_has_callbacks(int cpu)
*/
void rcu_barrier(void)
{
- _rcu_barrier(&rcu_preempt_state, call_rcu);
+ _rcu_barrier(&rcu_preempt_state);
}
EXPORT_SYMBOL_GPL(rcu_barrier);
--
1.7.8
^ permalink raw reply related [flat|nested] 27+ messages in thread* [PATCH tip/core/rcu 06/22] rcu: Move _rcu_barrier()'s rcu_head structures to rcu_data structures
2012-06-22 15:17 ` [PATCH tip/core/rcu 01/22] rcu: Control RCU_FANOUT_LEAF from boot-time parameter Paul E. McKenney
` (3 preceding siblings ...)
2012-06-22 15:17 ` [PATCH tip/core/rcu 05/22] rcu: Place pointer to call_rcu() in rcu_data structure Paul E. McKenney
@ 2012-06-22 15:17 ` Paul E. McKenney
2012-06-22 15:17 ` [PATCH tip/core/rcu 07/22] rcu: Move rcu_barrier_cpu_count to rcu_state structure Paul E. McKenney
` (16 subsequent siblings)
21 siblings, 0 replies; 27+ messages in thread
From: Paul E. McKenney @ 2012-06-22 15:17 UTC (permalink / raw)
To: linux-kernel
Cc: mingo, laijs, dipankar, akpm, mathieu.desnoyers, josh, niv, tglx,
peterz, rostedt, Valdis.Kletnieks, dhowells, eric.dumazet, darren,
fweisbec, sbw, patches, Paul E. McKenney, Paul E. McKenney
From: "Paul E. McKenney" <paul.mckenney@linaro.org>
In order for multiple flavors of RCU to each concurrently run one
rcu_barrier(), each flavor needs its own per-CPU set of rcu_head
structures. This commit therefore moves _rcu_barrier()'s set of
per-CPU rcu_head structures from per-CPU variables to the existing
per-CPU and per-RCU-flavor rcu_data structures.
Signed-off-by: Paul E. McKenney <paul.mckenney@linaro.org>
Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
Reviewed-by: Josh Triplett <josh@joshtriplett.org>
---
kernel/rcutree.c | 6 ++----
kernel/rcutree.h | 3 +++
2 files changed, 5 insertions(+), 4 deletions(-)
diff --git a/kernel/rcutree.c b/kernel/rcutree.c
index 24cd2f0..cb2d35d 100644
--- a/kernel/rcutree.c
+++ b/kernel/rcutree.c
@@ -157,7 +157,6 @@ unsigned long rcutorture_vernum;
/* State information for rcu_barrier() and friends. */
-static DEFINE_PER_CPU(struct rcu_head, rcu_barrier_head) = {NULL};
static atomic_t rcu_barrier_cpu_count;
static DEFINE_MUTEX(rcu_barrier_mutex);
static struct completion rcu_barrier_completion;
@@ -2279,12 +2278,11 @@ static void rcu_barrier_callback(struct rcu_head *notused)
*/
static void rcu_barrier_func(void *type)
{
- int cpu = smp_processor_id();
- struct rcu_head *head = &per_cpu(rcu_barrier_head, cpu);
struct rcu_state *rsp = type;
+ struct rcu_data *rdp = __this_cpu_ptr(rsp->rda);
atomic_inc(&rcu_barrier_cpu_count);
- rsp->call(head, rcu_barrier_callback);
+ rsp->call(&rdp->barrier_head, rcu_barrier_callback);
}
/*
diff --git a/kernel/rcutree.h b/kernel/rcutree.h
index 15837d7..1783eae 100644
--- a/kernel/rcutree.h
+++ b/kernel/rcutree.h
@@ -300,6 +300,9 @@ struct rcu_data {
unsigned long n_rp_need_fqs;
unsigned long n_rp_need_nothing;
+ /* 6) _rcu_barrier() callback. */
+ struct rcu_head barrier_head;
+
int cpu;
struct rcu_state *rsp;
};
--
1.7.8
^ permalink raw reply related [flat|nested] 27+ messages in thread* [PATCH tip/core/rcu 07/22] rcu: Move rcu_barrier_cpu_count to rcu_state structure
2012-06-22 15:17 ` [PATCH tip/core/rcu 01/22] rcu: Control RCU_FANOUT_LEAF from boot-time parameter Paul E. McKenney
` (4 preceding siblings ...)
2012-06-22 15:17 ` [PATCH tip/core/rcu 06/22] rcu: Move _rcu_barrier()'s rcu_head structures to rcu_data structures Paul E. McKenney
@ 2012-06-22 15:17 ` Paul E. McKenney
2012-06-22 15:17 ` [PATCH tip/core/rcu 08/22] rcu: Move rcu_barrier_completion " Paul E. McKenney
` (15 subsequent siblings)
21 siblings, 0 replies; 27+ messages in thread
From: Paul E. McKenney @ 2012-06-22 15:17 UTC (permalink / raw)
To: linux-kernel
Cc: mingo, laijs, dipankar, akpm, mathieu.desnoyers, josh, niv, tglx,
peterz, rostedt, Valdis.Kletnieks, dhowells, eric.dumazet, darren,
fweisbec, sbw, patches, Paul E. McKenney, Paul E. McKenney
From: "Paul E. McKenney" <paul.mckenney@linaro.org>
In order to allow each RCU flavor to concurrently execute its rcu_barrier()
function, it is necessary to move the relevant state to the rcu_state
structure. This commit therefore moves the rcu_barrier_cpu_count global
variable to a new ->barrier_cpu_count field in the rcu_state structure.
Signed-off-by: Paul E. McKenney <paul.mckenney@linaro.org>
Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
Reviewed-by: Josh Triplett <josh@joshtriplett.org>
---
kernel/rcutree.c | 25 ++++++++++++++-----------
kernel/rcutree.h | 1 +
2 files changed, 15 insertions(+), 11 deletions(-)
diff --git a/kernel/rcutree.c b/kernel/rcutree.c
index cb2d35d..592c43f 100644
--- a/kernel/rcutree.c
+++ b/kernel/rcutree.c
@@ -157,7 +157,6 @@ unsigned long rcutorture_vernum;
/* State information for rcu_barrier() and friends. */
-static atomic_t rcu_barrier_cpu_count;
static DEFINE_MUTEX(rcu_barrier_mutex);
static struct completion rcu_barrier_completion;
@@ -2267,9 +2266,12 @@ static int rcu_cpu_has_callbacks(int cpu)
* RCU callback function for _rcu_barrier(). If we are last, wake
* up the task executing _rcu_barrier().
*/
-static void rcu_barrier_callback(struct rcu_head *notused)
+static void rcu_barrier_callback(struct rcu_head *rhp)
{
- if (atomic_dec_and_test(&rcu_barrier_cpu_count))
+ struct rcu_data *rdp = container_of(rhp, struct rcu_data, barrier_head);
+ struct rcu_state *rsp = rdp->rsp;
+
+ if (atomic_dec_and_test(&rsp->barrier_cpu_count))
complete(&rcu_barrier_completion);
}
@@ -2281,7 +2283,7 @@ static void rcu_barrier_func(void *type)
struct rcu_state *rsp = type;
struct rcu_data *rdp = __this_cpu_ptr(rsp->rda);
- atomic_inc(&rcu_barrier_cpu_count);
+ atomic_inc(&rsp->barrier_cpu_count);
rsp->call(&rdp->barrier_head, rcu_barrier_callback);
}
@@ -2294,9 +2296,9 @@ static void _rcu_barrier(struct rcu_state *rsp)
int cpu;
unsigned long flags;
struct rcu_data *rdp;
- struct rcu_head rh;
+ struct rcu_data rd;
- init_rcu_head_on_stack(&rh);
+ init_rcu_head_on_stack(&rd.barrier_head);
/* Take mutex to serialize concurrent rcu_barrier() requests. */
mutex_lock(&rcu_barrier_mutex);
@@ -2321,7 +2323,7 @@ static void _rcu_barrier(struct rcu_state *rsp)
* us -- but before CPU 1's orphaned callbacks are invoked!!!
*/
init_completion(&rcu_barrier_completion);
- atomic_set(&rcu_barrier_cpu_count, 1);
+ atomic_set(&rsp->barrier_cpu_count, 1);
raw_spin_lock_irqsave(&rsp->onofflock, flags);
rsp->rcu_barrier_in_progress = current;
raw_spin_unlock_irqrestore(&rsp->onofflock, flags);
@@ -2360,15 +2362,16 @@ static void _rcu_barrier(struct rcu_state *rsp)
rcu_adopt_orphan_cbs(rsp);
rsp->rcu_barrier_in_progress = NULL;
raw_spin_unlock_irqrestore(&rsp->onofflock, flags);
- atomic_inc(&rcu_barrier_cpu_count);
+ atomic_inc(&rsp->barrier_cpu_count);
smp_mb__after_atomic_inc(); /* Ensure atomic_inc() before callback. */
- rsp->call(&rh, rcu_barrier_callback);
+ rd.rsp = rsp;
+ rsp->call(&rd.barrier_head, rcu_barrier_callback);
/*
* Now that we have an rcu_barrier_callback() callback on each
* CPU, and thus each counted, remove the initial count.
*/
- if (atomic_dec_and_test(&rcu_barrier_cpu_count))
+ if (atomic_dec_and_test(&rsp->barrier_cpu_count))
complete(&rcu_barrier_completion);
/* Wait for all rcu_barrier_callback() callbacks to be invoked. */
@@ -2377,7 +2380,7 @@ static void _rcu_barrier(struct rcu_state *rsp)
/* Other rcu_barrier() invocations can now safely proceed. */
mutex_unlock(&rcu_barrier_mutex);
- destroy_rcu_head_on_stack(&rh);
+ destroy_rcu_head_on_stack(&rd.barrier_head);
}
/**
diff --git a/kernel/rcutree.h b/kernel/rcutree.h
index 1783eae..e7d29b7 100644
--- a/kernel/rcutree.h
+++ b/kernel/rcutree.h
@@ -386,6 +386,7 @@ struct rcu_state {
struct task_struct *rcu_barrier_in_progress;
/* Task doing rcu_barrier(), */
/* or NULL if no barrier. */
+ atomic_t barrier_cpu_count; /* # CPUs waiting on. */
raw_spinlock_t fqslock; /* Only one task forcing */
/* quiescent states. */
unsigned long jiffies_force_qs; /* Time at which to invoke */
--
1.7.8
^ permalink raw reply related [flat|nested] 27+ messages in thread* [PATCH tip/core/rcu 08/22] rcu: Move rcu_barrier_completion to rcu_state structure
2012-06-22 15:17 ` [PATCH tip/core/rcu 01/22] rcu: Control RCU_FANOUT_LEAF from boot-time parameter Paul E. McKenney
` (5 preceding siblings ...)
2012-06-22 15:17 ` [PATCH tip/core/rcu 07/22] rcu: Move rcu_barrier_cpu_count to rcu_state structure Paul E. McKenney
@ 2012-06-22 15:17 ` Paul E. McKenney
2012-06-22 15:17 ` [PATCH tip/core/rcu 09/22] rcu: Move rcu_barrier_mutex " Paul E. McKenney
` (14 subsequent siblings)
21 siblings, 0 replies; 27+ messages in thread
From: Paul E. McKenney @ 2012-06-22 15:17 UTC (permalink / raw)
To: linux-kernel
Cc: mingo, laijs, dipankar, akpm, mathieu.desnoyers, josh, niv, tglx,
peterz, rostedt, Valdis.Kletnieks, dhowells, eric.dumazet, darren,
fweisbec, sbw, patches, Paul E. McKenney, Paul E. McKenney
From: "Paul E. McKenney" <paul.mckenney@linaro.org>
In order to allow each RCU flavor to concurrently execute its
rcu_barrier() function, it is necessary to move the relevant
state to the rcu_state structure. This commit therefore moves the
rcu_barrier_completion global variable to a new ->barrier_completion
field in the rcu_state structure.
Signed-off-by: Paul E. McKenney <paul.mckenney@linaro.org>
Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
Reviewed-by: Josh Triplett <josh@joshtriplett.org>
---
kernel/rcutree.c | 9 ++++-----
kernel/rcutree.h | 1 +
2 files changed, 5 insertions(+), 5 deletions(-)
diff --git a/kernel/rcutree.c b/kernel/rcutree.c
index 592c43f..5e087f9 100644
--- a/kernel/rcutree.c
+++ b/kernel/rcutree.c
@@ -158,7 +158,6 @@ unsigned long rcutorture_vernum;
/* State information for rcu_barrier() and friends. */
static DEFINE_MUTEX(rcu_barrier_mutex);
-static struct completion rcu_barrier_completion;
/*
* Return true if an RCU grace period is in progress. The ACCESS_ONCE()s
@@ -2272,7 +2271,7 @@ static void rcu_barrier_callback(struct rcu_head *rhp)
struct rcu_state *rsp = rdp->rsp;
if (atomic_dec_and_test(&rsp->barrier_cpu_count))
- complete(&rcu_barrier_completion);
+ complete(&rsp->barrier_completion);
}
/*
@@ -2322,7 +2321,7 @@ static void _rcu_barrier(struct rcu_state *rsp)
* 6. Both rcu_barrier_callback() callbacks are invoked, awakening
* us -- but before CPU 1's orphaned callbacks are invoked!!!
*/
- init_completion(&rcu_barrier_completion);
+ init_completion(&rsp->barrier_completion);
atomic_set(&rsp->barrier_cpu_count, 1);
raw_spin_lock_irqsave(&rsp->onofflock, flags);
rsp->rcu_barrier_in_progress = current;
@@ -2372,10 +2371,10 @@ static void _rcu_barrier(struct rcu_state *rsp)
* CPU, and thus each counted, remove the initial count.
*/
if (atomic_dec_and_test(&rsp->barrier_cpu_count))
- complete(&rcu_barrier_completion);
+ complete(&rsp->barrier_completion);
/* Wait for all rcu_barrier_callback() callbacks to be invoked. */
- wait_for_completion(&rcu_barrier_completion);
+ wait_for_completion(&rsp->barrier_completion);
/* Other rcu_barrier() invocations can now safely proceed. */
mutex_unlock(&rcu_barrier_mutex);
diff --git a/kernel/rcutree.h b/kernel/rcutree.h
index e7d29b7..56fb8d4 100644
--- a/kernel/rcutree.h
+++ b/kernel/rcutree.h
@@ -387,6 +387,7 @@ struct rcu_state {
/* Task doing rcu_barrier(), */
/* or NULL if no barrier. */
atomic_t barrier_cpu_count; /* # CPUs waiting on. */
+ struct completion barrier_completion; /* Wake at barrier end. */
raw_spinlock_t fqslock; /* Only one task forcing */
/* quiescent states. */
unsigned long jiffies_force_qs; /* Time at which to invoke */
--
1.7.8
^ permalink raw reply related [flat|nested] 27+ messages in thread* [PATCH tip/core/rcu 09/22] rcu: Move rcu_barrier_mutex to rcu_state structure
2012-06-22 15:17 ` [PATCH tip/core/rcu 01/22] rcu: Control RCU_FANOUT_LEAF from boot-time parameter Paul E. McKenney
` (6 preceding siblings ...)
2012-06-22 15:17 ` [PATCH tip/core/rcu 08/22] rcu: Move rcu_barrier_completion " Paul E. McKenney
@ 2012-06-22 15:17 ` Paul E. McKenney
2012-06-22 15:17 ` [PATCH tip/core/rcu 10/22] rcu: Remove needless initialization Paul E. McKenney
` (13 subsequent siblings)
21 siblings, 0 replies; 27+ messages in thread
From: Paul E. McKenney @ 2012-06-22 15:17 UTC (permalink / raw)
To: linux-kernel
Cc: mingo, laijs, dipankar, akpm, mathieu.desnoyers, josh, niv, tglx,
peterz, rostedt, Valdis.Kletnieks, dhowells, eric.dumazet, darren,
fweisbec, sbw, patches, Paul E. McKenney, Paul E. McKenney
From: "Paul E. McKenney" <paul.mckenney@linaro.org>
In order to allow each RCU flavor to concurrently execute its
rcu_barrier() function, it is necessary to move the relevant
state to the rcu_state structure. This commit therefore moves the
rcu_barrier_mutex global variable to a new ->barrier_mutex field
in the rcu_state structure.
Signed-off-by: Paul E. McKenney <paul.mckenney@linaro.org>
Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
---
kernel/rcutree.c | 9 +++------
kernel/rcutree.h | 1 +
2 files changed, 4 insertions(+), 6 deletions(-)
diff --git a/kernel/rcutree.c b/kernel/rcutree.c
index 5e087f9..ac6e0c2 100644
--- a/kernel/rcutree.c
+++ b/kernel/rcutree.c
@@ -71,6 +71,7 @@ static struct lock_class_key rcu_node_class[RCU_NUM_LVLS];
.onofflock = __RAW_SPIN_LOCK_UNLOCKED(&sname##_state.onofflock), \
.orphan_nxttail = &sname##_state.orphan_nxtlist, \
.orphan_donetail = &sname##_state.orphan_donelist, \
+ .barrier_mutex = __MUTEX_INITIALIZER(sname##_state.barrier_mutex), \
.fqslock = __RAW_SPIN_LOCK_UNLOCKED(&sname##_state.fqslock), \
.n_force_qs = 0, \
.n_force_qs_ngp = 0, \
@@ -155,10 +156,6 @@ static void invoke_rcu_callbacks(struct rcu_state *rsp, struct rcu_data *rdp);
unsigned long rcutorture_testseq;
unsigned long rcutorture_vernum;
-/* State information for rcu_barrier() and friends. */
-
-static DEFINE_MUTEX(rcu_barrier_mutex);
-
/*
* Return true if an RCU grace period is in progress. The ACCESS_ONCE()s
* permit this function to be invoked without holding the root rcu_node
@@ -2300,7 +2297,7 @@ static void _rcu_barrier(struct rcu_state *rsp)
init_rcu_head_on_stack(&rd.barrier_head);
/* Take mutex to serialize concurrent rcu_barrier() requests. */
- mutex_lock(&rcu_barrier_mutex);
+ mutex_lock(&rsp->barrier_mutex);
smp_mb(); /* Prevent any prior operations from leaking in. */
@@ -2377,7 +2374,7 @@ static void _rcu_barrier(struct rcu_state *rsp)
wait_for_completion(&rsp->barrier_completion);
/* Other rcu_barrier() invocations can now safely proceed. */
- mutex_unlock(&rcu_barrier_mutex);
+ mutex_unlock(&rsp->barrier_mutex);
destroy_rcu_head_on_stack(&rd.barrier_head);
}
diff --git a/kernel/rcutree.h b/kernel/rcutree.h
index 56fb8d4..d9ac82f 100644
--- a/kernel/rcutree.h
+++ b/kernel/rcutree.h
@@ -386,6 +386,7 @@ struct rcu_state {
struct task_struct *rcu_barrier_in_progress;
/* Task doing rcu_barrier(), */
/* or NULL if no barrier. */
+ struct mutex barrier_mutex; /* Guards barrier fields. */
atomic_t barrier_cpu_count; /* # CPUs waiting on. */
struct completion barrier_completion; /* Wake at barrier end. */
raw_spinlock_t fqslock; /* Only one task forcing */
--
1.7.8
^ permalink raw reply related [flat|nested] 27+ messages in thread* [PATCH tip/core/rcu 10/22] rcu: Remove needless initialization
2012-06-22 15:17 ` [PATCH tip/core/rcu 01/22] rcu: Control RCU_FANOUT_LEAF from boot-time parameter Paul E. McKenney
` (7 preceding siblings ...)
2012-06-22 15:17 ` [PATCH tip/core/rcu 09/22] rcu: Move rcu_barrier_mutex " Paul E. McKenney
@ 2012-06-22 15:17 ` Paul E. McKenney
2012-06-22 15:17 ` [PATCH tip/core/rcu 11/22] rcu: Increase rcu_barrier() concurrency Paul E. McKenney
` (12 subsequent siblings)
21 siblings, 0 replies; 27+ messages in thread
From: Paul E. McKenney @ 2012-06-22 15:17 UTC (permalink / raw)
To: linux-kernel
Cc: mingo, laijs, dipankar, akpm, mathieu.desnoyers, josh, niv, tglx,
peterz, rostedt, Valdis.Kletnieks, dhowells, eric.dumazet, darren,
fweisbec, sbw, patches, Paul E. McKenney
From: "Paul E. McKenney" <paulmck@linux.vnet.ibm.com>
For global variables, C defaults all fields to zero. The initialization
of the rcu_state structure's ->n_force_qs and ->n_force_qs_ngp fields
is therefore redundant, so this commit removes these initializations.
Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
---
kernel/rcutree.c | 2 --
1 files changed, 0 insertions(+), 2 deletions(-)
diff --git a/kernel/rcutree.c b/kernel/rcutree.c
index ac6e0c2..7622073 100644
--- a/kernel/rcutree.c
+++ b/kernel/rcutree.c
@@ -73,8 +73,6 @@ static struct lock_class_key rcu_node_class[RCU_NUM_LVLS];
.orphan_donetail = &sname##_state.orphan_donelist, \
.barrier_mutex = __MUTEX_INITIALIZER(sname##_state.barrier_mutex), \
.fqslock = __RAW_SPIN_LOCK_UNLOCKED(&sname##_state.fqslock), \
- .n_force_qs = 0, \
- .n_force_qs_ngp = 0, \
.name = #sname, \
}
--
1.7.8
^ permalink raw reply related [flat|nested] 27+ messages in thread* [PATCH tip/core/rcu 11/22] rcu: Increase rcu_barrier() concurrency
2012-06-22 15:17 ` [PATCH tip/core/rcu 01/22] rcu: Control RCU_FANOUT_LEAF from boot-time parameter Paul E. McKenney
` (8 preceding siblings ...)
2012-06-22 15:17 ` [PATCH tip/core/rcu 10/22] rcu: Remove needless initialization Paul E. McKenney
@ 2012-06-22 15:17 ` Paul E. McKenney
2012-06-22 15:17 ` [PATCH tip/core/rcu 12/22] rcu: Add tracing for _rcu_barrier() Paul E. McKenney
` (11 subsequent siblings)
21 siblings, 0 replies; 27+ messages in thread
From: Paul E. McKenney @ 2012-06-22 15:17 UTC (permalink / raw)
To: linux-kernel
Cc: mingo, laijs, dipankar, akpm, mathieu.desnoyers, josh, niv, tglx,
peterz, rostedt, Valdis.Kletnieks, dhowells, eric.dumazet, darren,
fweisbec, sbw, patches, Paul E. McKenney, Paul E. McKenney
From: "Paul E. McKenney" <paul.mckenney@linaro.org>
The traditional rcu_barrier() implementation has serialized all requests,
regardless of RCU flavor, and also does not coalesce concurrent requests.
In the past, this has been good and sufficient.
However, systems are getting larger and use of rcu_barrier() has been
increasing. This commit therefore introduces a counter-based scheme
that allows _rcu_barrier() calls for the same flavor of RCU to take
advantage of each others' work.
Signed-off-by: Paul E. McKenney <paul.mckenney@linaro.org>
Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
---
kernel/rcutree.c | 36 +++++++++++++++++++++++++++++++++++-
kernel/rcutree.h | 2 ++
2 files changed, 37 insertions(+), 1 deletions(-)
diff --git a/kernel/rcutree.c b/kernel/rcutree.c
index 7622073..b0f1ab0 100644
--- a/kernel/rcutree.c
+++ b/kernel/rcutree.c
@@ -2291,13 +2291,41 @@ static void _rcu_barrier(struct rcu_state *rsp)
unsigned long flags;
struct rcu_data *rdp;
struct rcu_data rd;
+ unsigned long snap = ACCESS_ONCE(rsp->n_barrier_done);
+ unsigned long snap_done;
init_rcu_head_on_stack(&rd.barrier_head);
/* Take mutex to serialize concurrent rcu_barrier() requests. */
mutex_lock(&rsp->barrier_mutex);
- smp_mb(); /* Prevent any prior operations from leaking in. */
+ /*
+ * Ensure that all prior references, including to ->n_barrier_done,
+ * are ordered before the _rcu_barrier() machinery.
+ */
+ smp_mb(); /* See above block comment. */
+
+ /*
+ * Recheck ->n_barrier_done to see if others did our work for us.
+ * This means checking ->n_barrier_done for an even-to-odd-to-even
+ * transition. The "if" expression below therefore rounds the old
+ * value up to the next even number and adds two before comparing.
+ */
+ snap_done = ACCESS_ONCE(rsp->n_barrier_done);
+ if (ULONG_CMP_GE(snap_done, ((snap + 1) & ~0x1) + 2)) {
+ smp_mb(); /* caller's subsequent code after above check. */
+ mutex_unlock(&rsp->barrier_mutex);
+ return;
+ }
+
+ /*
+ * Increment ->n_barrier_done to avoid duplicate work. Use
+ * ACCESS_ONCE() to prevent the compiler from speculating
+ * the increment to precede the early-exit check.
+ */
+ ACCESS_ONCE(rsp->n_barrier_done)++;
+ WARN_ON_ONCE((rsp->n_barrier_done & 0x1) != 1);
+ smp_mb(); /* Order ->n_barrier_done increment with below mechanism. */
/*
* Initialize the count to one rather than to zero in order to
@@ -2368,6 +2396,12 @@ static void _rcu_barrier(struct rcu_state *rsp)
if (atomic_dec_and_test(&rsp->barrier_cpu_count))
complete(&rsp->barrier_completion);
+ /* Increment ->n_barrier_done to prevent duplicate work. */
+ smp_mb(); /* Keep increment after above mechanism. */
+ ACCESS_ONCE(rsp->n_barrier_done)++;
+ WARN_ON_ONCE((rsp->n_barrier_done & 0x1) != 0);
+ smp_mb(); /* Keep increment before caller's subsequent code. */
+
/* Wait for all rcu_barrier_callback() callbacks to be invoked. */
wait_for_completion(&rsp->barrier_completion);
diff --git a/kernel/rcutree.h b/kernel/rcutree.h
index d9ac82f..a294f7f 100644
--- a/kernel/rcutree.h
+++ b/kernel/rcutree.h
@@ -389,6 +389,8 @@ struct rcu_state {
struct mutex barrier_mutex; /* Guards barrier fields. */
atomic_t barrier_cpu_count; /* # CPUs waiting on. */
struct completion barrier_completion; /* Wake at barrier end. */
+ unsigned long n_barrier_done; /* ++ at start and end of */
+ /* _rcu_barrier(). */
raw_spinlock_t fqslock; /* Only one task forcing */
/* quiescent states. */
unsigned long jiffies_force_qs; /* Time at which to invoke */
--
1.7.8
^ permalink raw reply related [flat|nested] 27+ messages in thread* [PATCH tip/core/rcu 12/22] rcu: Add tracing for _rcu_barrier()
2012-06-22 15:17 ` [PATCH tip/core/rcu 01/22] rcu: Control RCU_FANOUT_LEAF from boot-time parameter Paul E. McKenney
` (9 preceding siblings ...)
2012-06-22 15:17 ` [PATCH tip/core/rcu 11/22] rcu: Increase rcu_barrier() concurrency Paul E. McKenney
@ 2012-06-22 15:17 ` Paul E. McKenney
2012-06-22 15:17 ` [PATCH tip/core/rcu 13/22] rcu: Add rcu_barrier() statistics to debugfs tracing Paul E. McKenney
` (10 subsequent siblings)
21 siblings, 0 replies; 27+ messages in thread
From: Paul E. McKenney @ 2012-06-22 15:17 UTC (permalink / raw)
To: linux-kernel
Cc: mingo, laijs, dipankar, akpm, mathieu.desnoyers, josh, niv, tglx,
peterz, rostedt, Valdis.Kletnieks, dhowells, eric.dumazet, darren,
fweisbec, sbw, patches, Paul E. McKenney, Paul E. McKenney
From: "Paul E. McKenney" <paul.mckenney@linaro.org>
This commit adds event tracing for _rcu_barrier() execution. This
is defined only if RCU_TRACE=y.
Signed-off-by: Paul E. McKenney <paul.mckenney@linaro.org>
Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
Reviewed-by: Josh Triplett <josh@joshtriplett.org>
---
include/trace/events/rcu.h | 45 ++++++++++++++++++++++++++++++++++++++++++++
kernel/rcutree.c | 29 +++++++++++++++++++++++++++-
2 files changed, 73 insertions(+), 1 deletions(-)
diff --git a/include/trace/events/rcu.h b/include/trace/events/rcu.h
index 1480900..cd63f79 100644
--- a/include/trace/events/rcu.h
+++ b/include/trace/events/rcu.h
@@ -540,6 +540,50 @@ TRACE_EVENT(rcu_torture_read,
__entry->rcutorturename, __entry->rhp)
);
+/*
+ * Tracepoint for _rcu_barrier() execution. The string "s" describes
+ * the _rcu_barrier phase:
+ * "Begin": rcu_barrier_callback() started.
+ * "Check": rcu_barrier_callback() checking for piggybacking.
+ * "EarlyExit": rcu_barrier_callback() piggybacked, thus early exit.
+ * "Inc1": rcu_barrier_callback() piggyback check counter incremented.
+ * "Offline": rcu_barrier_callback() found offline CPU
+ * "OnlineQ": rcu_barrier_callback() found online CPU with callbacks.
+ * "OnlineNQ": rcu_barrier_callback() found online CPU, no callbacks.
+ * "IRQ": An rcu_barrier_callback() callback posted on remote CPU.
+ * "CB": An rcu_barrier_callback() invoked a callback, not the last.
+ * "LastCB": An rcu_barrier_callback() invoked the last callback.
+ * "Inc2": rcu_barrier_callback() piggyback check counter incremented.
+ * The "cpu" argument is the CPU or -1 if meaningless, the "cnt" argument
+ * is the count of remaining callbacks, and "done" is the piggybacking count.
+ */
+TRACE_EVENT(rcu_barrier,
+
+ TP_PROTO(char *rcuname, char *s, int cpu, int cnt, unsigned long done),
+
+ TP_ARGS(rcuname, s, cpu, cnt, done),
+
+ TP_STRUCT__entry(
+ __field(char *, rcuname)
+ __field(char *, s)
+ __field(int, cpu)
+ __field(int, cnt)
+ __field(unsigned long, done)
+ ),
+
+ TP_fast_assign(
+ __entry->rcuname = rcuname;
+ __entry->s = s;
+ __entry->cpu = cpu;
+ __entry->cnt = cnt;
+ __entry->done = done;
+ ),
+
+ TP_printk("%s %s cpu %d remaining %d # %lu",
+ __entry->rcuname, __entry->s, __entry->cpu, __entry->cnt,
+ __entry->done)
+);
+
#else /* #ifdef CONFIG_RCU_TRACE */
#define trace_rcu_grace_period(rcuname, gpnum, gpevent) do { } while (0)
@@ -563,6 +607,7 @@ TRACE_EVENT(rcu_torture_read,
#define trace_rcu_batch_end(rcuname, callbacks_invoked, cb, nr, iit, risk) \
do { } while (0)
#define trace_rcu_torture_read(rcutorturename, rhp) do { } while (0)
+#define trace_rcu_barrier(name, s, cpu, cnt, done) do { } while (0)
#endif /* #else #ifdef CONFIG_RCU_TRACE */
diff --git a/kernel/rcutree.c b/kernel/rcutree.c
index b0f1ab0..5c1c6ae 100644
--- a/kernel/rcutree.c
+++ b/kernel/rcutree.c
@@ -2257,6 +2257,17 @@ static int rcu_cpu_has_callbacks(int cpu)
}
/*
+ * Helper function for _rcu_barrier() tracing. If tracing is disabled,
+ * the compiler is expected to optimize this away.
+ */
+static void _rcu_barrier_trace(struct rcu_state *rsp, char *s,
+ int cpu, unsigned long done)
+{
+ trace_rcu_barrier(rsp->name, s, cpu,
+ atomic_read(&rsp->barrier_cpu_count), done);
+}
+
+/*
* RCU callback function for _rcu_barrier(). If we are last, wake
* up the task executing _rcu_barrier().
*/
@@ -2265,8 +2276,12 @@ static void rcu_barrier_callback(struct rcu_head *rhp)
struct rcu_data *rdp = container_of(rhp, struct rcu_data, barrier_head);
struct rcu_state *rsp = rdp->rsp;
- if (atomic_dec_and_test(&rsp->barrier_cpu_count))
+ if (atomic_dec_and_test(&rsp->barrier_cpu_count)) {
+ _rcu_barrier_trace(rsp, "LastCB", -1, rsp->n_barrier_done);
complete(&rsp->barrier_completion);
+ } else {
+ _rcu_barrier_trace(rsp, "CB", -1, rsp->n_barrier_done);
+ }
}
/*
@@ -2277,6 +2292,7 @@ static void rcu_barrier_func(void *type)
struct rcu_state *rsp = type;
struct rcu_data *rdp = __this_cpu_ptr(rsp->rda);
+ _rcu_barrier_trace(rsp, "IRQ", -1, rsp->n_barrier_done);
atomic_inc(&rsp->barrier_cpu_count);
rsp->call(&rdp->barrier_head, rcu_barrier_callback);
}
@@ -2295,6 +2311,7 @@ static void _rcu_barrier(struct rcu_state *rsp)
unsigned long snap_done;
init_rcu_head_on_stack(&rd.barrier_head);
+ _rcu_barrier_trace(rsp, "Begin", -1, snap);
/* Take mutex to serialize concurrent rcu_barrier() requests. */
mutex_lock(&rsp->barrier_mutex);
@@ -2312,7 +2329,9 @@ static void _rcu_barrier(struct rcu_state *rsp)
* value up to the next even number and adds two before comparing.
*/
snap_done = ACCESS_ONCE(rsp->n_barrier_done);
+ _rcu_barrier_trace(rsp, "Check", -1, snap_done);
if (ULONG_CMP_GE(snap_done, ((snap + 1) & ~0x1) + 2)) {
+ _rcu_barrier_trace(rsp, "EarlyExit", -1, snap_done);
smp_mb(); /* caller's subsequent code after above check. */
mutex_unlock(&rsp->barrier_mutex);
return;
@@ -2325,6 +2344,7 @@ static void _rcu_barrier(struct rcu_state *rsp)
*/
ACCESS_ONCE(rsp->n_barrier_done)++;
WARN_ON_ONCE((rsp->n_barrier_done & 0x1) != 1);
+ _rcu_barrier_trace(rsp, "Inc1", -1, rsp->n_barrier_done);
smp_mb(); /* Order ->n_barrier_done increment with below mechanism. */
/*
@@ -2361,13 +2381,19 @@ static void _rcu_barrier(struct rcu_state *rsp)
preempt_disable();
rdp = per_cpu_ptr(rsp->rda, cpu);
if (cpu_is_offline(cpu)) {
+ _rcu_barrier_trace(rsp, "Offline", cpu,
+ rsp->n_barrier_done);
preempt_enable();
while (cpu_is_offline(cpu) && ACCESS_ONCE(rdp->qlen))
schedule_timeout_interruptible(1);
} else if (ACCESS_ONCE(rdp->qlen)) {
+ _rcu_barrier_trace(rsp, "OnlineQ", cpu,
+ rsp->n_barrier_done);
smp_call_function_single(cpu, rcu_barrier_func, rsp, 1);
preempt_enable();
} else {
+ _rcu_barrier_trace(rsp, "OnlineNQ", cpu,
+ rsp->n_barrier_done);
preempt_enable();
}
}
@@ -2400,6 +2426,7 @@ static void _rcu_barrier(struct rcu_state *rsp)
smp_mb(); /* Keep increment after above mechanism. */
ACCESS_ONCE(rsp->n_barrier_done)++;
WARN_ON_ONCE((rsp->n_barrier_done & 0x1) != 0);
+ _rcu_barrier_trace(rsp, "Inc2", -1, rsp->n_barrier_done);
smp_mb(); /* Keep increment before caller's subsequent code. */
/* Wait for all rcu_barrier_callback() callbacks to be invoked. */
--
1.7.8
^ permalink raw reply related [flat|nested] 27+ messages in thread* [PATCH tip/core/rcu 13/22] rcu: Add rcu_barrier() statistics to debugfs tracing
2012-06-22 15:17 ` [PATCH tip/core/rcu 01/22] rcu: Control RCU_FANOUT_LEAF from boot-time parameter Paul E. McKenney
` (10 preceding siblings ...)
2012-06-22 15:17 ` [PATCH tip/core/rcu 12/22] rcu: Add tracing for _rcu_barrier() Paul E. McKenney
@ 2012-06-22 15:17 ` Paul E. McKenney
2012-06-22 15:17 ` [PATCH tip/core/rcu 14/22] rcu: Remove unneeded __rcu_process_callbacks() argument Paul E. McKenney
` (9 subsequent siblings)
21 siblings, 0 replies; 27+ messages in thread
From: Paul E. McKenney @ 2012-06-22 15:17 UTC (permalink / raw)
To: linux-kernel
Cc: mingo, laijs, dipankar, akpm, mathieu.desnoyers, josh, niv, tglx,
peterz, rostedt, Valdis.Kletnieks, dhowells, eric.dumazet, darren,
fweisbec, sbw, patches, Paul E. McKenney, Paul E. McKenney
From: "Paul E. McKenney" <paul.mckenney@linaro.org>
This commit adds an rcubarrier file to RCU's debugfs statistical tracing
directory, providing diagnostic information on rcu_barrier().
Signed-off-by: Paul E. McKenney <paul.mckenney@linaro.org>
Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
Reviewed-by: Josh Triplett <josh@joshtriplett.org>
---
kernel/rcutree_trace.c | 39 +++++++++++++++++++++++++++++++++++++++
1 files changed, 39 insertions(+), 0 deletions(-)
diff --git a/kernel/rcutree_trace.c b/kernel/rcutree_trace.c
index a3556a2..057408b 100644
--- a/kernel/rcutree_trace.c
+++ b/kernel/rcutree_trace.c
@@ -46,6 +46,40 @@
#define RCU_TREE_NONCORE
#include "rcutree.h"
+static void print_rcubarrier(struct seq_file *m, struct rcu_state *rsp)
+{
+ seq_printf(m, "%c bcc: %d nbd: %lu\n",
+ rsp->rcu_barrier_in_progress ? 'B' : '.',
+ atomic_read(&rsp->barrier_cpu_count),
+ rsp->n_barrier_done);
+}
+
+static int show_rcubarrier(struct seq_file *m, void *unused)
+{
+#ifdef CONFIG_TREE_PREEMPT_RCU
+ seq_puts(m, "rcu_preempt: ");
+ print_rcubarrier(m, &rcu_preempt_state);
+#endif /* #ifdef CONFIG_TREE_PREEMPT_RCU */
+ seq_puts(m, "rcu_sched: ");
+ print_rcubarrier(m, &rcu_sched_state);
+ seq_puts(m, "rcu_bh: ");
+ print_rcubarrier(m, &rcu_bh_state);
+ return 0;
+}
+
+static int rcubarrier_open(struct inode *inode, struct file *file)
+{
+ return single_open(file, show_rcubarrier, NULL);
+}
+
+static const struct file_operations rcubarrier_fops = {
+ .owner = THIS_MODULE,
+ .open = rcubarrier_open,
+ .read = seq_read,
+ .llseek = seq_lseek,
+ .release = single_release,
+};
+
#ifdef CONFIG_RCU_BOOST
static char convert_kthread_status(unsigned int kthread_status)
@@ -453,6 +487,11 @@ static int __init rcutree_trace_init(void)
if (!rcudir)
goto free_out;
+ retval = debugfs_create_file("rcubarrier", 0444, rcudir,
+ NULL, &rcubarrier_fops);
+ if (!retval)
+ goto free_out;
+
retval = debugfs_create_file("rcudata", 0444, rcudir,
NULL, &rcudata_fops);
if (!retval)
--
1.7.8
^ permalink raw reply related [flat|nested] 27+ messages in thread* [PATCH tip/core/rcu 14/22] rcu: Remove unneeded __rcu_process_callbacks() argument
2012-06-22 15:17 ` [PATCH tip/core/rcu 01/22] rcu: Control RCU_FANOUT_LEAF from boot-time parameter Paul E. McKenney
` (11 preceding siblings ...)
2012-06-22 15:17 ` [PATCH tip/core/rcu 13/22] rcu: Add rcu_barrier() statistics to debugfs tracing Paul E. McKenney
@ 2012-06-22 15:17 ` Paul E. McKenney
2012-06-22 15:17 ` [PATCH tip/core/rcu 15/22] rcu: Introduce for_each_rcu_flavor() and use it Paul E. McKenney
` (8 subsequent siblings)
21 siblings, 0 replies; 27+ messages in thread
From: Paul E. McKenney @ 2012-06-22 15:17 UTC (permalink / raw)
To: linux-kernel
Cc: mingo, laijs, dipankar, akpm, mathieu.desnoyers, josh, niv, tglx,
peterz, rostedt, Valdis.Kletnieks, dhowells, eric.dumazet, darren,
fweisbec, sbw, patches, Paul E. McKenney
From: "Paul E. McKenney" <paulmck@linux.vnet.ibm.com>
With the advent of __this_cpu_ptr(), it is no longer necessary to pass
both the rcu_state and rcu_data structures into __rcu_process_callbacks().
This commit therefore computes the rcu_data pointer from the rcu_state
pointer within __rcu_process_callbacks() so that callers can pass in
only the pointer to the rcu_state structure. This paves the way for
linking the rcu_state structures together and iterating over them.
Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
Reviewed-by: Josh Triplett <josh@joshtriplett.org>
---
kernel/rcutree.c | 8 ++++----
kernel/rcutree_plugin.h | 3 +--
2 files changed, 5 insertions(+), 6 deletions(-)
diff --git a/kernel/rcutree.c b/kernel/rcutree.c
index 5c1c6ae..cf7c373 100644
--- a/kernel/rcutree.c
+++ b/kernel/rcutree.c
@@ -1785,9 +1785,10 @@ unlock_fqs_ret:
* whom the rdp belongs.
*/
static void
-__rcu_process_callbacks(struct rcu_state *rsp, struct rcu_data *rdp)
+__rcu_process_callbacks(struct rcu_state *rsp)
{
unsigned long flags;
+ struct rcu_data *rdp = __this_cpu_ptr(rsp->rda);
WARN_ON_ONCE(rdp->beenonline == 0);
@@ -1824,9 +1825,8 @@ __rcu_process_callbacks(struct rcu_state *rsp, struct rcu_data *rdp)
static void rcu_process_callbacks(struct softirq_action *unused)
{
trace_rcu_utilization("Start RCU core");
- __rcu_process_callbacks(&rcu_sched_state,
- &__get_cpu_var(rcu_sched_data));
- __rcu_process_callbacks(&rcu_bh_state, &__get_cpu_var(rcu_bh_data));
+ __rcu_process_callbacks(&rcu_sched_state);
+ __rcu_process_callbacks(&rcu_bh_state);
rcu_preempt_process_callbacks();
trace_rcu_utilization("End RCU core");
}
diff --git a/kernel/rcutree_plugin.h b/kernel/rcutree_plugin.h
index a401508..c6e986c 100644
--- a/kernel/rcutree_plugin.h
+++ b/kernel/rcutree_plugin.h
@@ -687,8 +687,7 @@ static void rcu_preempt_check_callbacks(int cpu)
*/
static void rcu_preempt_process_callbacks(void)
{
- __rcu_process_callbacks(&rcu_preempt_state,
- &__get_cpu_var(rcu_preempt_data));
+ __rcu_process_callbacks(&rcu_preempt_state);
}
#ifdef CONFIG_RCU_BOOST
--
1.7.8
^ permalink raw reply related [flat|nested] 27+ messages in thread* [PATCH tip/core/rcu 15/22] rcu: Introduce for_each_rcu_flavor() and use it
2012-06-22 15:17 ` [PATCH tip/core/rcu 01/22] rcu: Control RCU_FANOUT_LEAF from boot-time parameter Paul E. McKenney
` (12 preceding siblings ...)
2012-06-22 15:17 ` [PATCH tip/core/rcu 14/22] rcu: Remove unneeded __rcu_process_callbacks() argument Paul E. McKenney
@ 2012-06-22 15:17 ` Paul E. McKenney
2012-06-22 15:17 ` [PATCH tip/core/rcu 16/22] rcu: Use for_each_rcu_flavor() in TREE_RCU tracing Paul E. McKenney
` (7 subsequent siblings)
21 siblings, 0 replies; 27+ messages in thread
From: Paul E. McKenney @ 2012-06-22 15:17 UTC (permalink / raw)
To: linux-kernel
Cc: mingo, laijs, dipankar, akpm, mathieu.desnoyers, josh, niv, tglx,
peterz, rostedt, Valdis.Kletnieks, dhowells, eric.dumazet, darren,
fweisbec, sbw, patches, Paul E. McKenney
From: "Paul E. McKenney" <paulmck@linux.vnet.ibm.com>
The arrival of TREE_PREEMPT_RCU some years back included some ugly
code involving either #ifdef or #ifdef'ed wrapper functions to iterate
over all non-SRCU flavors of RCU. This commit therefore introduces
a for_each_rcu_flavor() iterator over the rcu_state structures for each
flavor of RCU to clean up a bit of the ugliness.
Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
---
kernel/rcutree.c | 53 +++++++++++++---------
kernel/rcutree.h | 12 ++---
kernel/rcutree_plugin.h | 116 -----------------------------------------------
3 files changed, 37 insertions(+), 144 deletions(-)
diff --git a/kernel/rcutree.c b/kernel/rcutree.c
index cf7c373..68a9d8b 100644
--- a/kernel/rcutree.c
+++ b/kernel/rcutree.c
@@ -84,6 +84,7 @@ struct rcu_state rcu_bh_state = RCU_STATE_INITIALIZER(rcu_bh, call_rcu_bh);
DEFINE_PER_CPU(struct rcu_data, rcu_bh_data);
static struct rcu_state *rcu_state;
+LIST_HEAD(rcu_struct_flavors);
/* Increase (but not decrease) the CONFIG_RCU_FANOUT_LEAF at boot time. */
static int rcu_fanout_leaf = CONFIG_RCU_FANOUT_LEAF;
@@ -859,9 +860,10 @@ static int rcu_panic(struct notifier_block *this, unsigned long ev, void *ptr)
*/
void rcu_cpu_stall_reset(void)
{
- rcu_sched_state.jiffies_stall = jiffies + ULONG_MAX / 2;
- rcu_bh_state.jiffies_stall = jiffies + ULONG_MAX / 2;
- rcu_preempt_stall_reset();
+ struct rcu_state *rsp;
+
+ for_each_rcu_flavor(rsp)
+ rsp->jiffies_stall = jiffies + ULONG_MAX / 2;
}
static struct notifier_block rcu_panic_block = {
@@ -1824,10 +1826,11 @@ __rcu_process_callbacks(struct rcu_state *rsp)
*/
static void rcu_process_callbacks(struct softirq_action *unused)
{
+ struct rcu_state *rsp;
+
trace_rcu_utilization("Start RCU core");
- __rcu_process_callbacks(&rcu_sched_state);
- __rcu_process_callbacks(&rcu_bh_state);
- rcu_preempt_process_callbacks();
+ for_each_rcu_flavor(rsp)
+ __rcu_process_callbacks(rsp);
trace_rcu_utilization("End RCU core");
}
@@ -2238,9 +2241,12 @@ static int __rcu_pending(struct rcu_state *rsp, struct rcu_data *rdp)
*/
static int rcu_pending(int cpu)
{
- return __rcu_pending(&rcu_sched_state, &per_cpu(rcu_sched_data, cpu)) ||
- __rcu_pending(&rcu_bh_state, &per_cpu(rcu_bh_data, cpu)) ||
- rcu_preempt_pending(cpu);
+ struct rcu_state *rsp;
+
+ for_each_rcu_flavor(rsp)
+ if (__rcu_pending(rsp, per_cpu_ptr(rsp->rda, cpu)))
+ return 1;
+ return 0;
}
/*
@@ -2250,10 +2256,13 @@ static int rcu_pending(int cpu)
*/
static int rcu_cpu_has_callbacks(int cpu)
{
+ struct rcu_state *rsp;
+
/* RCU callbacks either ready or pending? */
- return per_cpu(rcu_sched_data, cpu).nxtlist ||
- per_cpu(rcu_bh_data, cpu).nxtlist ||
- rcu_preempt_cpu_has_callbacks(cpu);
+ for_each_rcu_flavor(rsp)
+ if (per_cpu_ptr(rsp->rda, cpu)->nxtlist)
+ return 1;
+ return 0;
}
/*
@@ -2548,9 +2557,11 @@ rcu_init_percpu_data(int cpu, struct rcu_state *rsp, int preemptible)
static void __cpuinit rcu_prepare_cpu(int cpu)
{
- rcu_init_percpu_data(cpu, &rcu_sched_state, 0);
- rcu_init_percpu_data(cpu, &rcu_bh_state, 0);
- rcu_preempt_init_percpu_data(cpu);
+ struct rcu_state *rsp;
+
+ for_each_rcu_flavor(rsp)
+ rcu_init_percpu_data(cpu, rsp,
+ strcmp(rsp->name, "rcu_preempt") == 0);
}
/*
@@ -2562,6 +2573,7 @@ static int __cpuinit rcu_cpu_notify(struct notifier_block *self,
long cpu = (long)hcpu;
struct rcu_data *rdp = per_cpu_ptr(rcu_state->rda, cpu);
struct rcu_node *rnp = rdp->mynode;
+ struct rcu_state *rsp;
trace_rcu_utilization("Start CPU hotplug");
switch (action) {
@@ -2586,18 +2598,16 @@ static int __cpuinit rcu_cpu_notify(struct notifier_block *self,
* touch any data without introducing corruption. We send the
* dying CPU's callbacks to an arbitrarily chosen online CPU.
*/
- rcu_cleanup_dying_cpu(&rcu_bh_state);
- rcu_cleanup_dying_cpu(&rcu_sched_state);
- rcu_preempt_cleanup_dying_cpu();
+ for_each_rcu_flavor(rsp)
+ rcu_cleanup_dying_cpu(rsp);
rcu_cleanup_after_idle(cpu);
break;
case CPU_DEAD:
case CPU_DEAD_FROZEN:
case CPU_UP_CANCELED:
case CPU_UP_CANCELED_FROZEN:
- rcu_cleanup_dead_cpu(cpu, &rcu_bh_state);
- rcu_cleanup_dead_cpu(cpu, &rcu_sched_state);
- rcu_preempt_cleanup_dead_cpu(cpu);
+ for_each_rcu_flavor(rsp)
+ rcu_cleanup_dead_cpu(cpu, rsp);
break;
default:
break;
@@ -2714,6 +2724,7 @@ static void __init rcu_init_one(struct rcu_state *rsp,
per_cpu_ptr(rsp->rda, i)->mynode = rnp;
rcu_boot_init_percpu_data(i, rsp);
}
+ list_add(&rsp->flavors, &rcu_struct_flavors);
}
/*
diff --git a/kernel/rcutree.h b/kernel/rcutree.h
index a294f7f..138fb33 100644
--- a/kernel/rcutree.h
+++ b/kernel/rcutree.h
@@ -408,8 +408,13 @@ struct rcu_state {
unsigned long gp_max; /* Maximum GP duration in */
/* jiffies. */
char *name; /* Name of structure. */
+ struct list_head flavors; /* List of RCU flavors. */
};
+extern struct list_head rcu_struct_flavors;
+#define for_each_rcu_flavor(rsp) \
+ list_for_each_entry((rsp), &rcu_struct_flavors, flavors)
+
/* Return values for rcu_preempt_offline_tasks(). */
#define RCU_OFL_TASKS_NORM_GP 0x1 /* Tasks blocking normal */
@@ -451,25 +456,18 @@ static void rcu_stop_cpu_kthread(int cpu);
#endif /* #ifdef CONFIG_HOTPLUG_CPU */
static void rcu_print_detail_task_stall(struct rcu_state *rsp);
static int rcu_print_task_stall(struct rcu_node *rnp);
-static void rcu_preempt_stall_reset(void);
static void rcu_preempt_check_blocked_tasks(struct rcu_node *rnp);
#ifdef CONFIG_HOTPLUG_CPU
static int rcu_preempt_offline_tasks(struct rcu_state *rsp,
struct rcu_node *rnp,
struct rcu_data *rdp);
#endif /* #ifdef CONFIG_HOTPLUG_CPU */
-static void rcu_preempt_cleanup_dead_cpu(int cpu);
static void rcu_preempt_check_callbacks(int cpu);
-static void rcu_preempt_process_callbacks(void);
void call_rcu(struct rcu_head *head, void (*func)(struct rcu_head *rcu));
#if defined(CONFIG_HOTPLUG_CPU) || defined(CONFIG_TREE_PREEMPT_RCU)
static void rcu_report_exp_rnp(struct rcu_state *rsp, struct rcu_node *rnp,
bool wake);
#endif /* #if defined(CONFIG_HOTPLUG_CPU) || defined(CONFIG_TREE_PREEMPT_RCU) */
-static int rcu_preempt_pending(int cpu);
-static int rcu_preempt_cpu_has_callbacks(int cpu);
-static void __cpuinit rcu_preempt_init_percpu_data(int cpu);
-static void rcu_preempt_cleanup_dying_cpu(void);
static void __init __rcu_init_preempt(void);
static void rcu_initiate_boost(struct rcu_node *rnp, unsigned long flags);
static void rcu_preempt_boost_start_gp(struct rcu_node *rnp);
diff --git a/kernel/rcutree_plugin.h b/kernel/rcutree_plugin.h
index c6e986c..b7d5489 100644
--- a/kernel/rcutree_plugin.h
+++ b/kernel/rcutree_plugin.h
@@ -545,16 +545,6 @@ static int rcu_print_task_stall(struct rcu_node *rnp)
}
/*
- * Suppress preemptible RCU's CPU stall warnings by pushing the
- * time of the next stall-warning message comfortably far into the
- * future.
- */
-static void rcu_preempt_stall_reset(void)
-{
- rcu_preempt_state.jiffies_stall = jiffies + ULONG_MAX / 2;
-}
-
-/*
* Check that the list of blocked tasks for the newly completed grace
* period is in fact empty. It is a serious bug to complete a grace
* period that still has RCU readers blocked! This function must be
@@ -655,14 +645,6 @@ static int rcu_preempt_offline_tasks(struct rcu_state *rsp,
#endif /* #ifdef CONFIG_HOTPLUG_CPU */
/*
- * Do CPU-offline processing for preemptible RCU.
- */
-static void rcu_preempt_cleanup_dead_cpu(int cpu)
-{
- rcu_cleanup_dead_cpu(cpu, &rcu_preempt_state);
-}
-
-/*
* Check for a quiescent state from the current CPU. When a task blocks,
* the task is recorded in the corresponding CPU's rcu_node structure,
* which is checked elsewhere.
@@ -682,14 +664,6 @@ static void rcu_preempt_check_callbacks(int cpu)
t->rcu_read_unlock_special |= RCU_READ_UNLOCK_NEED_QS;
}
-/*
- * Process callbacks for preemptible RCU.
- */
-static void rcu_preempt_process_callbacks(void)
-{
- __rcu_process_callbacks(&rcu_preempt_state);
-}
-
#ifdef CONFIG_RCU_BOOST
static void rcu_preempt_do_callbacks(void)
@@ -921,24 +895,6 @@ mb_ret:
}
EXPORT_SYMBOL_GPL(synchronize_rcu_expedited);
-/*
- * Check to see if there is any immediate preemptible-RCU-related work
- * to be done.
- */
-static int rcu_preempt_pending(int cpu)
-{
- return __rcu_pending(&rcu_preempt_state,
- &per_cpu(rcu_preempt_data, cpu));
-}
-
-/*
- * Does preemptible RCU have callbacks on this CPU?
- */
-static int rcu_preempt_cpu_has_callbacks(int cpu)
-{
- return !!per_cpu(rcu_preempt_data, cpu).nxtlist;
-}
-
/**
* rcu_barrier - Wait until all in-flight call_rcu() callbacks complete.
*/
@@ -949,23 +905,6 @@ void rcu_barrier(void)
EXPORT_SYMBOL_GPL(rcu_barrier);
/*
- * Initialize preemptible RCU's per-CPU data.
- */
-static void __cpuinit rcu_preempt_init_percpu_data(int cpu)
-{
- rcu_init_percpu_data(cpu, &rcu_preempt_state, 1);
-}
-
-/*
- * Move preemptible RCU's callbacks from dying CPU to other online CPU
- * and record a quiescent state.
- */
-static void rcu_preempt_cleanup_dying_cpu(void)
-{
- rcu_cleanup_dying_cpu(&rcu_preempt_state);
-}
-
-/*
* Initialize preemptible RCU's state structures.
*/
static void __init __rcu_init_preempt(void)
@@ -1042,14 +981,6 @@ static int rcu_print_task_stall(struct rcu_node *rnp)
}
/*
- * Because preemptible RCU does not exist, there is no need to suppress
- * its CPU stall warnings.
- */
-static void rcu_preempt_stall_reset(void)
-{
-}
-
-/*
* Because there is no preemptible RCU, there can be no readers blocked,
* so there is no need to check for blocked tasks. So check only for
* bogus qsmask values.
@@ -1077,14 +1008,6 @@ static int rcu_preempt_offline_tasks(struct rcu_state *rsp,
#endif /* #ifdef CONFIG_HOTPLUG_CPU */
/*
- * Because preemptible RCU does not exist, it never needs CPU-offline
- * processing.
- */
-static void rcu_preempt_cleanup_dead_cpu(int cpu)
-{
-}
-
-/*
* Because preemptible RCU does not exist, it never has any callbacks
* to check.
*/
@@ -1093,14 +1016,6 @@ static void rcu_preempt_check_callbacks(int cpu)
}
/*
- * Because preemptible RCU does not exist, it never has any callbacks
- * to process.
- */
-static void rcu_preempt_process_callbacks(void)
-{
-}
-
-/*
* Queue an RCU callback for lazy invocation after a grace period.
* This will likely be later named something like "call_rcu_lazy()",
* but this change will require some way of tagging the lazy RCU
@@ -1141,22 +1056,6 @@ static void rcu_report_exp_rnp(struct rcu_state *rsp, struct rcu_node *rnp,
#endif /* #ifdef CONFIG_HOTPLUG_CPU */
/*
- * Because preemptible RCU does not exist, it never has any work to do.
- */
-static int rcu_preempt_pending(int cpu)
-{
- return 0;
-}
-
-/*
- * Because preemptible RCU does not exist, it never has callbacks
- */
-static int rcu_preempt_cpu_has_callbacks(int cpu)
-{
- return 0;
-}
-
-/*
* Because preemptible RCU does not exist, rcu_barrier() is just
* another name for rcu_barrier_sched().
*/
@@ -1167,21 +1066,6 @@ void rcu_barrier(void)
EXPORT_SYMBOL_GPL(rcu_barrier);
/*
- * Because preemptible RCU does not exist, there is no per-CPU
- * data to initialize.
- */
-static void __cpuinit rcu_preempt_init_percpu_data(int cpu)
-{
-}
-
-/*
- * Because there is no preemptible RCU, there is no cleanup to do.
- */
-static void rcu_preempt_cleanup_dying_cpu(void)
-{
-}
-
-/*
* Because preemptible RCU does not exist, it need not be initialized.
*/
static void __init __rcu_init_preempt(void)
--
1.7.8
^ permalink raw reply related [flat|nested] 27+ messages in thread* [PATCH tip/core/rcu 16/22] rcu: Use for_each_rcu_flavor() in TREE_RCU tracing
2012-06-22 15:17 ` [PATCH tip/core/rcu 01/22] rcu: Control RCU_FANOUT_LEAF from boot-time parameter Paul E. McKenney
` (13 preceding siblings ...)
2012-06-22 15:17 ` [PATCH tip/core/rcu 15/22] rcu: Introduce for_each_rcu_flavor() and use it Paul E. McKenney
@ 2012-06-22 15:17 ` Paul E. McKenney
2012-06-22 15:17 ` [PATCH tip/core/rcu 17/22] rcu: RCU_SAVE_DYNTICK code no longer ever dead Paul E. McKenney
` (6 subsequent siblings)
21 siblings, 0 replies; 27+ messages in thread
From: Paul E. McKenney @ 2012-06-22 15:17 UTC (permalink / raw)
To: linux-kernel
Cc: mingo, laijs, dipankar, akpm, mathieu.desnoyers, josh, niv, tglx,
peterz, rostedt, Valdis.Kletnieks, dhowells, eric.dumazet, darren,
fweisbec, sbw, patches, Paul E. McKenney
From: "Paul E. McKenney" <paulmck@linux.vnet.ibm.com>
This commit applies the new for_each_rcu_flavor() macro to the
kernel/rcutree_trace.c file.
Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
---
kernel/rcutree_trace.c | 116 ++++++++++++++++++------------------------------
1 files changed, 43 insertions(+), 73 deletions(-)
diff --git a/kernel/rcutree_trace.c b/kernel/rcutree_trace.c
index 057408b..a16ddbd 100644
--- a/kernel/rcutree_trace.c
+++ b/kernel/rcutree_trace.c
@@ -46,24 +46,15 @@
#define RCU_TREE_NONCORE
#include "rcutree.h"
-static void print_rcubarrier(struct seq_file *m, struct rcu_state *rsp)
-{
- seq_printf(m, "%c bcc: %d nbd: %lu\n",
- rsp->rcu_barrier_in_progress ? 'B' : '.',
- atomic_read(&rsp->barrier_cpu_count),
- rsp->n_barrier_done);
-}
-
static int show_rcubarrier(struct seq_file *m, void *unused)
{
-#ifdef CONFIG_TREE_PREEMPT_RCU
- seq_puts(m, "rcu_preempt: ");
- print_rcubarrier(m, &rcu_preempt_state);
-#endif /* #ifdef CONFIG_TREE_PREEMPT_RCU */
- seq_puts(m, "rcu_sched: ");
- print_rcubarrier(m, &rcu_sched_state);
- seq_puts(m, "rcu_bh: ");
- print_rcubarrier(m, &rcu_bh_state);
+ struct rcu_state *rsp;
+
+ for_each_rcu_flavor(rsp)
+ seq_printf(m, "%s: %c bcc: %d nbd: %lu\n",
+ rsp->name, rsp->rcu_barrier_in_progress ? 'B' : '.',
+ atomic_read(&rsp->barrier_cpu_count),
+ rsp->n_barrier_done);
return 0;
}
@@ -129,24 +120,16 @@ static void print_one_rcu_data(struct seq_file *m, struct rcu_data *rdp)
rdp->n_cbs_invoked, rdp->n_cbs_orphaned, rdp->n_cbs_adopted);
}
-#define PRINT_RCU_DATA(name, func, m) \
- do { \
- int _p_r_d_i; \
- \
- for_each_possible_cpu(_p_r_d_i) \
- func(m, &per_cpu(name, _p_r_d_i)); \
- } while (0)
-
static int show_rcudata(struct seq_file *m, void *unused)
{
-#ifdef CONFIG_TREE_PREEMPT_RCU
- seq_puts(m, "rcu_preempt:\n");
- PRINT_RCU_DATA(rcu_preempt_data, print_one_rcu_data, m);
-#endif /* #ifdef CONFIG_TREE_PREEMPT_RCU */
- seq_puts(m, "rcu_sched:\n");
- PRINT_RCU_DATA(rcu_sched_data, print_one_rcu_data, m);
- seq_puts(m, "rcu_bh:\n");
- PRINT_RCU_DATA(rcu_bh_data, print_one_rcu_data, m);
+ int cpu;
+ struct rcu_state *rsp;
+
+ for_each_rcu_flavor(rsp) {
+ seq_printf(m, "%s:\n", rsp->name);
+ for_each_possible_cpu(cpu)
+ print_one_rcu_data(m, per_cpu_ptr(rsp->rda, cpu));
+ }
return 0;
}
@@ -200,6 +183,9 @@ static void print_one_rcu_data_csv(struct seq_file *m, struct rcu_data *rdp)
static int show_rcudata_csv(struct seq_file *m, void *unused)
{
+ int cpu;
+ struct rcu_state *rsp;
+
seq_puts(m, "\"CPU\",\"Online?\",\"c\",\"g\",\"pq\",\"pgp\",\"pq\",");
seq_puts(m, "\"dt\",\"dt nesting\",\"dt NMI nesting\",\"df\",");
seq_puts(m, "\"of\",\"qll\",\"ql\",\"qs\"");
@@ -207,14 +193,11 @@ static int show_rcudata_csv(struct seq_file *m, void *unused)
seq_puts(m, "\"kt\",\"ktl\"");
#endif /* #ifdef CONFIG_RCU_BOOST */
seq_puts(m, ",\"b\",\"ci\",\"co\",\"ca\"\n");
-#ifdef CONFIG_TREE_PREEMPT_RCU
- seq_puts(m, "\"rcu_preempt:\"\n");
- PRINT_RCU_DATA(rcu_preempt_data, print_one_rcu_data_csv, m);
-#endif /* #ifdef CONFIG_TREE_PREEMPT_RCU */
- seq_puts(m, "\"rcu_sched:\"\n");
- PRINT_RCU_DATA(rcu_sched_data, print_one_rcu_data_csv, m);
- seq_puts(m, "\"rcu_bh:\"\n");
- PRINT_RCU_DATA(rcu_bh_data, print_one_rcu_data_csv, m);
+ for_each_rcu_flavor(rsp) {
+ seq_printf(m, "\"%s:\"\n", rsp->name);
+ for_each_possible_cpu(cpu)
+ print_one_rcu_data_csv(m, per_cpu_ptr(rsp->rda, cpu));
+ }
return 0;
}
@@ -304,9 +287,9 @@ static void print_one_rcu_state(struct seq_file *m, struct rcu_state *rsp)
struct rcu_node *rnp;
gpnum = rsp->gpnum;
- seq_printf(m, "c=%lu g=%lu s=%d jfq=%ld j=%x "
+ seq_printf(m, "%s: c=%lu g=%lu s=%d jfq=%ld j=%x "
"nfqs=%lu/nfqsng=%lu(%lu) fqlh=%lu oqlen=%ld/%ld\n",
- rsp->completed, gpnum, rsp->fqs_state,
+ rsp->name, rsp->completed, gpnum, rsp->fqs_state,
(long)(rsp->jiffies_force_qs - jiffies),
(int)(jiffies & 0xffff),
rsp->n_force_qs, rsp->n_force_qs_ngp,
@@ -329,14 +312,10 @@ static void print_one_rcu_state(struct seq_file *m, struct rcu_state *rsp)
static int show_rcuhier(struct seq_file *m, void *unused)
{
-#ifdef CONFIG_TREE_PREEMPT_RCU
- seq_puts(m, "rcu_preempt:\n");
- print_one_rcu_state(m, &rcu_preempt_state);
-#endif /* #ifdef CONFIG_TREE_PREEMPT_RCU */
- seq_puts(m, "rcu_sched:\n");
- print_one_rcu_state(m, &rcu_sched_state);
- seq_puts(m, "rcu_bh:\n");
- print_one_rcu_state(m, &rcu_bh_state);
+ struct rcu_state *rsp;
+
+ for_each_rcu_flavor(rsp)
+ print_one_rcu_state(m, rsp);
return 0;
}
@@ -377,11 +356,10 @@ static void show_one_rcugp(struct seq_file *m, struct rcu_state *rsp)
static int show_rcugp(struct seq_file *m, void *unused)
{
-#ifdef CONFIG_TREE_PREEMPT_RCU
- show_one_rcugp(m, &rcu_preempt_state);
-#endif /* #ifdef CONFIG_TREE_PREEMPT_RCU */
- show_one_rcugp(m, &rcu_sched_state);
- show_one_rcugp(m, &rcu_bh_state);
+ struct rcu_state *rsp;
+
+ for_each_rcu_flavor(rsp)
+ show_one_rcugp(m, rsp);
return 0;
}
@@ -416,28 +394,20 @@ static void print_one_rcu_pending(struct seq_file *m, struct rcu_data *rdp)
rdp->n_rp_need_nothing);
}
-static void print_rcu_pendings(struct seq_file *m, struct rcu_state *rsp)
+static int show_rcu_pending(struct seq_file *m, void *unused)
{
int cpu;
struct rcu_data *rdp;
-
- for_each_possible_cpu(cpu) {
- rdp = per_cpu_ptr(rsp->rda, cpu);
- if (rdp->beenonline)
- print_one_rcu_pending(m, rdp);
+ struct rcu_state *rsp;
+
+ for_each_rcu_flavor(rsp) {
+ seq_printf(m, "%s:\n", rsp->name);
+ for_each_possible_cpu(cpu) {
+ rdp = per_cpu_ptr(rsp->rda, cpu);
+ if (rdp->beenonline)
+ print_one_rcu_pending(m, rdp);
+ }
}
-}
-
-static int show_rcu_pending(struct seq_file *m, void *unused)
-{
-#ifdef CONFIG_TREE_PREEMPT_RCU
- seq_puts(m, "rcu_preempt:\n");
- print_rcu_pendings(m, &rcu_preempt_state);
-#endif /* #ifdef CONFIG_TREE_PREEMPT_RCU */
- seq_puts(m, "rcu_sched:\n");
- print_rcu_pendings(m, &rcu_sched_state);
- seq_puts(m, "rcu_bh:\n");
- print_rcu_pendings(m, &rcu_bh_state);
return 0;
}
--
1.7.8
^ permalink raw reply related [flat|nested] 27+ messages in thread* [PATCH tip/core/rcu 17/22] rcu: RCU_SAVE_DYNTICK code no longer ever dead
2012-06-22 15:17 ` [PATCH tip/core/rcu 01/22] rcu: Control RCU_FANOUT_LEAF from boot-time parameter Paul E. McKenney
` (14 preceding siblings ...)
2012-06-22 15:17 ` [PATCH tip/core/rcu 16/22] rcu: Use for_each_rcu_flavor() in TREE_RCU tracing Paul E. McKenney
@ 2012-06-22 15:17 ` Paul E. McKenney
2012-06-22 15:17 ` [PATCH tip/core/rcu 18/22] rcu: Move RCU grace-period initialization into a kthread Paul E. McKenney
` (5 subsequent siblings)
21 siblings, 0 replies; 27+ messages in thread
From: Paul E. McKenney @ 2012-06-22 15:17 UTC (permalink / raw)
To: linux-kernel
Cc: mingo, laijs, dipankar, akpm, mathieu.desnoyers, josh, niv, tglx,
peterz, rostedt, Valdis.Kletnieks, dhowells, eric.dumazet, darren,
fweisbec, sbw, patches, Paul E. McKenney
From: "Paul E. McKenney" <paulmck@linux.vnet.ibm.com>
Before RCU had unified idle, the RCU_SAVE_DYNTICK leg of the switch
statement in force_quiescent_state() was dead code for CONFIG_NO_HZ=n
kernel builds. With unified idle, the code is never dead. This commit
therefore removes the "if" statement designed to make gcc aware of when
the code was and was not dead.
Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
Reviewed-by: Josh Triplett <josh@joshtriplett.org>
---
kernel/rcutree.c | 2 --
1 files changed, 0 insertions(+), 2 deletions(-)
diff --git a/kernel/rcutree.c b/kernel/rcutree.c
index 68a9d8b..1b0e441 100644
--- a/kernel/rcutree.c
+++ b/kernel/rcutree.c
@@ -1744,8 +1744,6 @@ static void force_quiescent_state(struct rcu_state *rsp, int relaxed)
break; /* grace period idle or initializing, ignore. */
case RCU_SAVE_DYNTICK:
- if (RCU_SIGNAL_INIT != RCU_SAVE_DYNTICK)
- break; /* So gcc recognizes the dead code. */
raw_spin_unlock(&rnp->lock); /* irqs remain disabled */
--
1.7.8
^ permalink raw reply related [flat|nested] 27+ messages in thread* [PATCH tip/core/rcu 18/22] rcu: Move RCU grace-period initialization into a kthread
2012-06-22 15:17 ` [PATCH tip/core/rcu 01/22] rcu: Control RCU_FANOUT_LEAF from boot-time parameter Paul E. McKenney
` (15 preceding siblings ...)
2012-06-22 15:17 ` [PATCH tip/core/rcu 17/22] rcu: RCU_SAVE_DYNTICK code no longer ever dead Paul E. McKenney
@ 2012-06-22 15:17 ` Paul E. McKenney
2012-06-22 15:17 ` [PATCH tip/core/rcu 19/22] rcu: Allow RCU grace-period initialization to be preempted Paul E. McKenney
` (4 subsequent siblings)
21 siblings, 0 replies; 27+ messages in thread
From: Paul E. McKenney @ 2012-06-22 15:17 UTC (permalink / raw)
To: linux-kernel
Cc: mingo, laijs, dipankar, akpm, mathieu.desnoyers, josh, niv, tglx,
peterz, rostedt, Valdis.Kletnieks, dhowells, eric.dumazet, darren,
fweisbec, sbw, patches, Paul E. McKenney
From: "Paul E. McKenney" <paulmck@linux.vnet.ibm.com>
As the first step towards allowing grace-period initialization to be
preemptible, this commit moves the RCU grace-period initialization
into its own kthread. This is needed to keep large-system scheduling
latency at reasonable levels.
Reported-by: Mike Galbraith <mgalbraith@suse.de>
Reported-by: Dimitri Sivanich <sivanich@sgi.com>
Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
---
kernel/rcutree.c | 191 ++++++++++++++++++++++++++++++++++++------------------
kernel/rcutree.h | 3 +
2 files changed, 130 insertions(+), 64 deletions(-)
diff --git a/kernel/rcutree.c b/kernel/rcutree.c
index 1b0e441..faf4860 100644
--- a/kernel/rcutree.c
+++ b/kernel/rcutree.c
@@ -1030,6 +1030,103 @@ rcu_start_gp_per_cpu(struct rcu_state *rsp, struct rcu_node *rnp, struct rcu_dat
}
/*
+ * Body of kthread that handles grace periods.
+ */
+static int rcu_gp_kthread(void *arg)
+{
+ unsigned long flags;
+ struct rcu_data *rdp;
+ struct rcu_node *rnp;
+ struct rcu_state *rsp = arg;
+
+ for (;;) {
+
+ /* Handle grace-period start. */
+ rnp = rcu_get_root(rsp);
+ for (;;) {
+ wait_event_interruptible(rsp->gp_wq, rsp->gp_flags);
+ if (rsp->gp_flags)
+ break;
+ flush_signals(current);
+ }
+ raw_spin_lock_irqsave(&rnp->lock, flags);
+ rsp->gp_flags = 0;
+ rdp = this_cpu_ptr(rsp->rda);
+
+ if (rcu_gp_in_progress(rsp)) {
+ /*
+ * A grace period is already in progress, so
+ * don't start another one.
+ */
+ raw_spin_unlock_irqrestore(&rnp->lock, flags);
+ continue;
+ }
+
+ if (rsp->fqs_active) {
+ /*
+ * We need a grace period, but force_quiescent_state()
+ * is running. Tell it to start one on our behalf.
+ */
+ rsp->fqs_need_gp = 1;
+ raw_spin_unlock_irqrestore(&rnp->lock, flags);
+ continue;
+ }
+
+ /* Advance to a new grace period and initialize state. */
+ rsp->gpnum++;
+ trace_rcu_grace_period(rsp->name, rsp->gpnum, "start");
+ WARN_ON_ONCE(rsp->fqs_state == RCU_GP_INIT);
+ rsp->fqs_state = RCU_GP_INIT; /* Stop force_quiescent_state. */
+ rsp->jiffies_force_qs = jiffies + RCU_JIFFIES_TILL_FORCE_QS;
+ record_gp_stall_check_time(rsp);
+ raw_spin_unlock(&rnp->lock); /* leave irqs disabled. */
+
+ /* Exclude any concurrent CPU-hotplug operations. */
+ raw_spin_lock(&rsp->onofflock); /* irqs already disabled. */
+
+ /*
+ * Set the quiescent-state-needed bits in all the rcu_node
+ * structures for all currently online CPUs in breadth-first
+ * order, starting from the root rcu_node structure.
+ * This operation relies on the layout of the hierarchy
+ * within the rsp->node[] array. Note that other CPUs will
+ * access only the leaves of the hierarchy, which still
+ * indicate that no grace period is in progress, at least
+ * until the corresponding leaf node has been initialized.
+ * In addition, we have excluded CPU-hotplug operations.
+ *
+ * Note that the grace period cannot complete until
+ * we finish the initialization process, as there will
+ * be at least one qsmask bit set in the root node until
+ * that time, namely the one corresponding to this CPU,
+ * due to the fact that we have irqs disabled.
+ */
+ rcu_for_each_node_breadth_first(rsp, rnp) {
+ raw_spin_lock(&rnp->lock); /* irqs already disabled. */
+ rcu_preempt_check_blocked_tasks(rnp);
+ rnp->qsmask = rnp->qsmaskinit;
+ rnp->gpnum = rsp->gpnum;
+ rnp->completed = rsp->completed;
+ if (rnp == rdp->mynode)
+ rcu_start_gp_per_cpu(rsp, rnp, rdp);
+ rcu_preempt_boost_start_gp(rnp);
+ trace_rcu_grace_period_init(rsp->name, rnp->gpnum,
+ rnp->level, rnp->grplo,
+ rnp->grphi, rnp->qsmask);
+ raw_spin_unlock(&rnp->lock); /* irqs remain disabled. */
+ }
+
+ rnp = rcu_get_root(rsp);
+ raw_spin_lock(&rnp->lock); /* irqs already disabled. */
+ /* force_quiescent_state() now OK. */
+ rsp->fqs_state = RCU_SIGNAL_INIT;
+ raw_spin_unlock(&rnp->lock); /* irqs remain disabled. */
+ raw_spin_unlock_irqrestore(&rsp->onofflock, flags);
+ }
+ return 0;
+}
+
+/*
* Start a new RCU grace period if warranted, re-initializing the hierarchy
* in preparation for detecting the next grace period. The caller must hold
* the root node's ->lock, which is released before return. Hard irqs must
@@ -1046,77 +1143,20 @@ rcu_start_gp(struct rcu_state *rsp, unsigned long flags)
struct rcu_data *rdp = this_cpu_ptr(rsp->rda);
struct rcu_node *rnp = rcu_get_root(rsp);
- if (!rcu_scheduler_fully_active ||
+ if (!rsp->gp_kthread ||
!cpu_needs_another_gp(rsp, rdp)) {
/*
- * Either the scheduler hasn't yet spawned the first
- * non-idle task or this CPU does not need another
- * grace period. Either way, don't start a new grace
- * period.
- */
- raw_spin_unlock_irqrestore(&rnp->lock, flags);
- return;
- }
-
- if (rsp->fqs_active) {
- /*
- * This CPU needs a grace period, but force_quiescent_state()
- * is running. Tell it to start one on this CPU's behalf.
+ * Either we have not yet spawned the grace-period
+ * task or this CPU does not need another grace period.
+ * Either way, don't start a new grace period.
*/
- rsp->fqs_need_gp = 1;
raw_spin_unlock_irqrestore(&rnp->lock, flags);
return;
}
- /* Advance to a new grace period and initialize state. */
- rsp->gpnum++;
- trace_rcu_grace_period(rsp->name, rsp->gpnum, "start");
- WARN_ON_ONCE(rsp->fqs_state == RCU_GP_INIT);
- rsp->fqs_state = RCU_GP_INIT; /* Hold off force_quiescent_state. */
- rsp->jiffies_force_qs = jiffies + RCU_JIFFIES_TILL_FORCE_QS;
- record_gp_stall_check_time(rsp);
- raw_spin_unlock(&rnp->lock); /* leave irqs disabled. */
-
- /* Exclude any concurrent CPU-hotplug operations. */
- raw_spin_lock(&rsp->onofflock); /* irqs already disabled. */
-
- /*
- * Set the quiescent-state-needed bits in all the rcu_node
- * structures for all currently online CPUs in breadth-first
- * order, starting from the root rcu_node structure. This
- * operation relies on the layout of the hierarchy within the
- * rsp->node[] array. Note that other CPUs will access only
- * the leaves of the hierarchy, which still indicate that no
- * grace period is in progress, at least until the corresponding
- * leaf node has been initialized. In addition, we have excluded
- * CPU-hotplug operations.
- *
- * Note that the grace period cannot complete until we finish
- * the initialization process, as there will be at least one
- * qsmask bit set in the root node until that time, namely the
- * one corresponding to this CPU, due to the fact that we have
- * irqs disabled.
- */
- rcu_for_each_node_breadth_first(rsp, rnp) {
- raw_spin_lock(&rnp->lock); /* irqs already disabled. */
- rcu_preempt_check_blocked_tasks(rnp);
- rnp->qsmask = rnp->qsmaskinit;
- rnp->gpnum = rsp->gpnum;
- rnp->completed = rsp->completed;
- if (rnp == rdp->mynode)
- rcu_start_gp_per_cpu(rsp, rnp, rdp);
- rcu_preempt_boost_start_gp(rnp);
- trace_rcu_grace_period_init(rsp->name, rnp->gpnum,
- rnp->level, rnp->grplo,
- rnp->grphi, rnp->qsmask);
- raw_spin_unlock(&rnp->lock); /* irqs remain disabled. */
- }
-
- rnp = rcu_get_root(rsp);
- raw_spin_lock(&rnp->lock); /* irqs already disabled. */
- rsp->fqs_state = RCU_SIGNAL_INIT; /* force_quiescent_state now OK. */
- raw_spin_unlock(&rnp->lock); /* irqs remain disabled. */
- raw_spin_unlock_irqrestore(&rsp->onofflock, flags);
+ rsp->gp_flags = 1;
+ raw_spin_unlock_irqrestore(&rnp->lock, flags);
+ wake_up(&rsp->gp_wq);
}
/*
@@ -2615,6 +2655,28 @@ static int __cpuinit rcu_cpu_notify(struct notifier_block *self,
}
/*
+ * Spawn the kthread that handles this RCU flavor's grace periods.
+ */
+static int __init rcu_spawn_gp_kthread(void)
+{
+ unsigned long flags;
+ struct rcu_node *rnp;
+ struct rcu_state *rsp;
+ struct task_struct *t;
+
+ for_each_rcu_flavor(rsp) {
+ t = kthread_run(rcu_gp_kthread, rsp, rsp->name);
+ BUG_ON(IS_ERR(t));
+ rnp = rcu_get_root(rsp);
+ raw_spin_lock_irqsave(&rnp->lock, flags);
+ rsp->gp_kthread = t;
+ raw_spin_unlock_irqrestore(&rnp->lock, flags);
+ }
+ return 0;
+}
+early_initcall(rcu_spawn_gp_kthread);
+
+/*
* This function is invoked towards the end of the scheduler's initialization
* process. Before this is called, the idle task might contain
* RCU read-side critical sections (during which time, this idle
@@ -2715,6 +2777,7 @@ static void __init rcu_init_one(struct rcu_state *rsp,
}
rsp->rda = rda;
+ init_waitqueue_head(&rsp->gp_wq);
rnp = rsp->level[rcu_num_lvls - 1];
for_each_possible_cpu(i) {
while (i > rnp->grphi)
diff --git a/kernel/rcutree.h b/kernel/rcutree.h
index 138fb33..c7c31c2 100644
--- a/kernel/rcutree.h
+++ b/kernel/rcutree.h
@@ -370,6 +370,9 @@ struct rcu_state {
u8 boost; /* Subject to priority boost. */
unsigned long gpnum; /* Current gp number. */
unsigned long completed; /* # of last completed gp. */
+ struct task_struct *gp_kthread; /* Task for grace periods. */
+ wait_queue_head_t gp_wq; /* Where GP task waits. */
+ int gp_flags; /* Commands for GP task. */
/* End of fields guarded by root rcu_node's lock. */
--
1.7.8
^ permalink raw reply related [flat|nested] 27+ messages in thread* [PATCH tip/core/rcu 19/22] rcu: Allow RCU grace-period initialization to be preempted
2012-06-22 15:17 ` [PATCH tip/core/rcu 01/22] rcu: Control RCU_FANOUT_LEAF from boot-time parameter Paul E. McKenney
` (16 preceding siblings ...)
2012-06-22 15:17 ` [PATCH tip/core/rcu 18/22] rcu: Move RCU grace-period initialization into a kthread Paul E. McKenney
@ 2012-06-22 15:17 ` Paul E. McKenney
2012-06-22 15:17 ` [PATCH tip/core/rcu 20/22] rcu: Move RCU grace-period cleanup into kthread Paul E. McKenney
` (3 subsequent siblings)
21 siblings, 0 replies; 27+ messages in thread
From: Paul E. McKenney @ 2012-06-22 15:17 UTC (permalink / raw)
To: linux-kernel
Cc: mingo, laijs, dipankar, akpm, mathieu.desnoyers, josh, niv, tglx,
peterz, rostedt, Valdis.Kletnieks, dhowells, eric.dumazet, darren,
fweisbec, sbw, patches, Paul E. McKenney
From: "Paul E. McKenney" <paulmck@linux.vnet.ibm.com>
RCU grace-period initialization is currently carried out with interrupts
disabled, which can result in 200-microsecond latency spikes on systems
on which RCU has been configured for 4096 CPUs. This patch therefore
makes the RCU grace-period initialization be preemptible, which should
eliminate those latency spikes. Similar spikes from grace-period cleanup
and the forcing of quiescent states will be dealt with similarly by later
patches.
Reported-by: Mike Galbraith <mgalbraith@suse.de>
Reported-by: Dimitri Sivanich <sivanich@sgi.com>
Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
---
kernel/rcutree.c | 17 ++++++++++-------
1 files changed, 10 insertions(+), 7 deletions(-)
diff --git a/kernel/rcutree.c b/kernel/rcutree.c
index faf4860..e5b4532 100644
--- a/kernel/rcutree.c
+++ b/kernel/rcutree.c
@@ -1059,6 +1059,7 @@ static int rcu_gp_kthread(void *arg)
* don't start another one.
*/
raw_spin_unlock_irqrestore(&rnp->lock, flags);
+ cond_resched();
continue;
}
@@ -1069,6 +1070,7 @@ static int rcu_gp_kthread(void *arg)
*/
rsp->fqs_need_gp = 1;
raw_spin_unlock_irqrestore(&rnp->lock, flags);
+ cond_resched();
continue;
}
@@ -1079,10 +1081,10 @@ static int rcu_gp_kthread(void *arg)
rsp->fqs_state = RCU_GP_INIT; /* Stop force_quiescent_state. */
rsp->jiffies_force_qs = jiffies + RCU_JIFFIES_TILL_FORCE_QS;
record_gp_stall_check_time(rsp);
- raw_spin_unlock(&rnp->lock); /* leave irqs disabled. */
+ raw_spin_unlock_irqrestore(&rnp->lock, flags);
/* Exclude any concurrent CPU-hotplug operations. */
- raw_spin_lock(&rsp->onofflock); /* irqs already disabled. */
+ get_online_cpus();
/*
* Set the quiescent-state-needed bits in all the rcu_node
@@ -1102,7 +1104,7 @@ static int rcu_gp_kthread(void *arg)
* due to the fact that we have irqs disabled.
*/
rcu_for_each_node_breadth_first(rsp, rnp) {
- raw_spin_lock(&rnp->lock); /* irqs already disabled. */
+ raw_spin_lock_irqsave(&rnp->lock, flags);
rcu_preempt_check_blocked_tasks(rnp);
rnp->qsmask = rnp->qsmaskinit;
rnp->gpnum = rsp->gpnum;
@@ -1113,15 +1115,16 @@ static int rcu_gp_kthread(void *arg)
trace_rcu_grace_period_init(rsp->name, rnp->gpnum,
rnp->level, rnp->grplo,
rnp->grphi, rnp->qsmask);
- raw_spin_unlock(&rnp->lock); /* irqs remain disabled. */
+ raw_spin_unlock_irqrestore(&rnp->lock, flags);
+ cond_resched();
}
rnp = rcu_get_root(rsp);
- raw_spin_lock(&rnp->lock); /* irqs already disabled. */
+ raw_spin_lock_irqsave(&rnp->lock, flags);
/* force_quiescent_state() now OK. */
rsp->fqs_state = RCU_SIGNAL_INIT;
- raw_spin_unlock(&rnp->lock); /* irqs remain disabled. */
- raw_spin_unlock_irqrestore(&rsp->onofflock, flags);
+ raw_spin_unlock_irqrestore(&rnp->lock, flags);
+ put_online_cpus();
}
return 0;
}
--
1.7.8
^ permalink raw reply related [flat|nested] 27+ messages in thread* [PATCH tip/core/rcu 20/22] rcu: Move RCU grace-period cleanup into kthread
2012-06-22 15:17 ` [PATCH tip/core/rcu 01/22] rcu: Control RCU_FANOUT_LEAF from boot-time parameter Paul E. McKenney
` (17 preceding siblings ...)
2012-06-22 15:17 ` [PATCH tip/core/rcu 19/22] rcu: Allow RCU grace-period initialization to be preempted Paul E. McKenney
@ 2012-06-22 15:17 ` Paul E. McKenney
2012-06-22 15:17 ` [PATCH tip/core/rcu 21/22] rcu: Allow RCU grace-period cleanup to be preempted Paul E. McKenney
` (2 subsequent siblings)
21 siblings, 0 replies; 27+ messages in thread
From: Paul E. McKenney @ 2012-06-22 15:17 UTC (permalink / raw)
To: linux-kernel
Cc: mingo, laijs, dipankar, akpm, mathieu.desnoyers, josh, niv, tglx,
peterz, rostedt, Valdis.Kletnieks, dhowells, eric.dumazet, darren,
fweisbec, sbw, patches, Paul E. McKenney
From: "Paul E. McKenney" <paulmck@linux.vnet.ibm.com>
As a first step towards allowing grace-period cleanup to be preemptible,
this commit moves the RCU grace-period cleanup into the same kthread
that is now used to initialize grace periods. This is needed to keep
scheduling latency down to a dull roar.
Reported-by: Mike Galbraith <mgalbraith@suse.de>
Reported-by: Dimitri Sivanich <sivanich@sgi.com>
Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
---
kernel/rcutree.c | 112 ++++++++++++++++++++++++++++++------------------------
1 files changed, 62 insertions(+), 50 deletions(-)
diff --git a/kernel/rcutree.c b/kernel/rcutree.c
index e5b4532..4a3f6b0 100644
--- a/kernel/rcutree.c
+++ b/kernel/rcutree.c
@@ -1035,6 +1035,7 @@ rcu_start_gp_per_cpu(struct rcu_state *rsp, struct rcu_node *rnp, struct rcu_dat
static int rcu_gp_kthread(void *arg)
{
unsigned long flags;
+ unsigned long gp_duration;
struct rcu_data *rdp;
struct rcu_node *rnp;
struct rcu_state *rsp = arg;
@@ -1125,6 +1126,65 @@ static int rcu_gp_kthread(void *arg)
rsp->fqs_state = RCU_SIGNAL_INIT;
raw_spin_unlock_irqrestore(&rnp->lock, flags);
put_online_cpus();
+
+ /* Handle grace-period end. */
+ rnp = rcu_get_root(rsp);
+ for (;;) {
+ wait_event_interruptible(rsp->gp_wq,
+ !ACCESS_ONCE(rnp->qsmask) &&
+ !rcu_preempt_blocked_readers_cgp(rnp));
+ if (!ACCESS_ONCE(rnp->qsmask) &&
+ !rcu_preempt_blocked_readers_cgp(rnp))
+ break;
+ flush_signals(current);
+ }
+
+ raw_spin_lock_irqsave(&rnp->lock, flags);
+ gp_duration = jiffies - rsp->gp_start;
+ if (gp_duration > rsp->gp_max)
+ rsp->gp_max = gp_duration;
+
+ /*
+ * We know the grace period is complete, but to everyone else
+ * it appears to still be ongoing. But it is also the case
+ * that to everyone else it looks like there is nothing that
+ * they can do to advance the grace period. It is therefore
+ * safe for us to drop the lock in order to mark the grace
+ * period as completed in all of the rcu_node structures.
+ *
+ * But if this CPU needs another grace period, it will take
+ * care of this while initializing the next grace period.
+ * We use RCU_WAIT_TAIL instead of the usual RCU_DONE_TAIL
+ * because the callbacks have not yet been advanced: Those
+ * callbacks are waiting on the grace period that just now
+ * completed.
+ */
+ if (*rdp->nxttail[RCU_WAIT_TAIL] == NULL) {
+ raw_spin_unlock(&rnp->lock); /* irqs remain disabled. */
+
+ /*
+ * Propagate new ->completed value to rcu_node
+ * structures so that other CPUs don't have to
+ * wait until the start of the next grace period
+ * to process their callbacks.
+ */
+ rcu_for_each_node_breadth_first(rsp, rnp) {
+ /* irqs already disabled. */
+ raw_spin_lock(&rnp->lock);
+ rnp->completed = rsp->gpnum;
+ /* irqs remain disabled. */
+ raw_spin_unlock(&rnp->lock);
+ }
+ rnp = rcu_get_root(rsp);
+ raw_spin_lock(&rnp->lock); /* irqs already disabled. */
+ }
+
+ rsp->completed = rsp->gpnum; /* Declare grace period done. */
+ trace_rcu_grace_period(rsp->name, rsp->completed, "end");
+ rsp->fqs_state = RCU_GP_IDLE;
+ if (cpu_needs_another_gp(rsp, rdp))
+ rsp->gp_flags = 1;
+ raw_spin_unlock_irqrestore(&rnp->lock, flags);
}
return 0;
}
@@ -1172,57 +1232,9 @@ rcu_start_gp(struct rcu_state *rsp, unsigned long flags)
static void rcu_report_qs_rsp(struct rcu_state *rsp, unsigned long flags)
__releases(rcu_get_root(rsp)->lock)
{
- unsigned long gp_duration;
- struct rcu_node *rnp = rcu_get_root(rsp);
- struct rcu_data *rdp = this_cpu_ptr(rsp->rda);
-
WARN_ON_ONCE(!rcu_gp_in_progress(rsp));
-
- /*
- * Ensure that all grace-period and pre-grace-period activity
- * is seen before the assignment to rsp->completed.
- */
- smp_mb(); /* See above block comment. */
- gp_duration = jiffies - rsp->gp_start;
- if (gp_duration > rsp->gp_max)
- rsp->gp_max = gp_duration;
-
- /*
- * We know the grace period is complete, but to everyone else
- * it appears to still be ongoing. But it is also the case
- * that to everyone else it looks like there is nothing that
- * they can do to advance the grace period. It is therefore
- * safe for us to drop the lock in order to mark the grace
- * period as completed in all of the rcu_node structures.
- *
- * But if this CPU needs another grace period, it will take
- * care of this while initializing the next grace period.
- * We use RCU_WAIT_TAIL instead of the usual RCU_DONE_TAIL
- * because the callbacks have not yet been advanced: Those
- * callbacks are waiting on the grace period that just now
- * completed.
- */
- if (*rdp->nxttail[RCU_WAIT_TAIL] == NULL) {
- raw_spin_unlock(&rnp->lock); /* irqs remain disabled. */
-
- /*
- * Propagate new ->completed value to rcu_node structures
- * so that other CPUs don't have to wait until the start
- * of the next grace period to process their callbacks.
- */
- rcu_for_each_node_breadth_first(rsp, rnp) {
- raw_spin_lock(&rnp->lock); /* irqs already disabled. */
- rnp->completed = rsp->gpnum;
- raw_spin_unlock(&rnp->lock); /* irqs remain disabled. */
- }
- rnp = rcu_get_root(rsp);
- raw_spin_lock(&rnp->lock); /* irqs already disabled. */
- }
-
- rsp->completed = rsp->gpnum; /* Declare the grace period complete. */
- trace_rcu_grace_period(rsp->name, rsp->completed, "end");
- rsp->fqs_state = RCU_GP_IDLE;
- rcu_start_gp(rsp, flags); /* releases root node's rnp->lock. */
+ raw_spin_unlock_irqrestore(&rcu_get_root(rsp)->lock, flags);
+ wake_up(&rsp->gp_wq); /* Memory barrier implied by wake_up() path. */
}
/*
--
1.7.8
^ permalink raw reply related [flat|nested] 27+ messages in thread* [PATCH tip/core/rcu 21/22] rcu: Allow RCU grace-period cleanup to be preempted
2012-06-22 15:17 ` [PATCH tip/core/rcu 01/22] rcu: Control RCU_FANOUT_LEAF from boot-time parameter Paul E. McKenney
` (18 preceding siblings ...)
2012-06-22 15:17 ` [PATCH tip/core/rcu 20/22] rcu: Move RCU grace-period cleanup into kthread Paul E. McKenney
@ 2012-06-22 15:17 ` Paul E. McKenney
2012-06-22 15:17 ` [PATCH tip/core/rcu 22/22] rcu: Prevent offline CPUs from executing RCU core code Paul E. McKenney
2012-07-01 6:19 ` [PATCH tip/core/rcu 01/22] rcu: Control RCU_FANOUT_LEAF from boot-time parameter Rusty Russell
21 siblings, 0 replies; 27+ messages in thread
From: Paul E. McKenney @ 2012-06-22 15:17 UTC (permalink / raw)
To: linux-kernel
Cc: mingo, laijs, dipankar, akpm, mathieu.desnoyers, josh, niv, tglx,
peterz, rostedt, Valdis.Kletnieks, dhowells, eric.dumazet, darren,
fweisbec, sbw, patches, Paul E. McKenney
From: "Paul E. McKenney" <paulmck@linux.vnet.ibm.com>
RCU grace-period cleanup is currently carried out with interrupts
disabled, which can result in excessive latency spikes on large systems
(many hundreds or thousands of CPUs). This patch therefore makes the
RCU grace-period cleanup be preemptible, including voluntary preemption
points, which should eliminate those latency spikes. Similar spikes from
forcing of quiescent states will be dealt with similarly by later patches.
Reported-by: Mike Galbraith <mgalbraith@suse.de>
Reported-by: Dimitri Sivanich <sivanich@sgi.com>
Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
---
kernel/rcutree.c | 11 +++++------
1 files changed, 5 insertions(+), 6 deletions(-)
diff --git a/kernel/rcutree.c b/kernel/rcutree.c
index 4a3f6b0..d21c2d3 100644
--- a/kernel/rcutree.c
+++ b/kernel/rcutree.c
@@ -1160,7 +1160,7 @@ static int rcu_gp_kthread(void *arg)
* completed.
*/
if (*rdp->nxttail[RCU_WAIT_TAIL] == NULL) {
- raw_spin_unlock(&rnp->lock); /* irqs remain disabled. */
+ raw_spin_unlock_irqrestore(&rnp->lock, flags);
/*
* Propagate new ->completed value to rcu_node
@@ -1169,14 +1169,13 @@ static int rcu_gp_kthread(void *arg)
* to process their callbacks.
*/
rcu_for_each_node_breadth_first(rsp, rnp) {
- /* irqs already disabled. */
- raw_spin_lock(&rnp->lock);
+ raw_spin_lock_irqsave(&rnp->lock, flags);
rnp->completed = rsp->gpnum;
- /* irqs remain disabled. */
- raw_spin_unlock(&rnp->lock);
+ raw_spin_unlock_irqrestore(&rnp->lock, flags);
+ cond_resched();
}
rnp = rcu_get_root(rsp);
- raw_spin_lock(&rnp->lock); /* irqs already disabled. */
+ raw_spin_lock_irqsave(&rnp->lock, flags);
}
rsp->completed = rsp->gpnum; /* Declare grace period done. */
--
1.7.8
^ permalink raw reply related [flat|nested] 27+ messages in thread* [PATCH tip/core/rcu 22/22] rcu: Prevent offline CPUs from executing RCU core code
2012-06-22 15:17 ` [PATCH tip/core/rcu 01/22] rcu: Control RCU_FANOUT_LEAF from boot-time parameter Paul E. McKenney
` (19 preceding siblings ...)
2012-06-22 15:17 ` [PATCH tip/core/rcu 21/22] rcu: Allow RCU grace-period cleanup to be preempted Paul E. McKenney
@ 2012-06-22 15:17 ` Paul E. McKenney
2012-07-01 6:19 ` [PATCH tip/core/rcu 01/22] rcu: Control RCU_FANOUT_LEAF from boot-time parameter Rusty Russell
21 siblings, 0 replies; 27+ messages in thread
From: Paul E. McKenney @ 2012-06-22 15:17 UTC (permalink / raw)
To: linux-kernel
Cc: mingo, laijs, dipankar, akpm, mathieu.desnoyers, josh, niv, tglx,
peterz, rostedt, Valdis.Kletnieks, dhowells, eric.dumazet, darren,
fweisbec, sbw, patches, Paul E. McKenney, Paul E. McKenney
From: "Paul E. McKenney" <paul.mckenney@linaro.org>
Earlier versions of RCU invoked the RCU core from the CPU_DYING notifier
in order to note a quiescent state for the outgoing CPU. Because the
CPU is marked "offline" during the execution of the CPU_DYING notifiers,
the RCU core had to tolerate being invoked from an offline CPU. However,
commit b1420f1c (Make rcu_barrier() less disruptive) left only tracing
code in the CPU_DYING notifier, so the RCU core need no longer execute
on offline CPUs. This commit therefore enforces this restriction.
Signed-off-by: Paul E. McKenney <paul.mckenney@linaro.org>
Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
---
kernel/rcutree.c | 2 ++
1 files changed, 2 insertions(+), 0 deletions(-)
diff --git a/kernel/rcutree.c b/kernel/rcutree.c
index d21c2d3..cbf4d05 100644
--- a/kernel/rcutree.c
+++ b/kernel/rcutree.c
@@ -1880,6 +1880,8 @@ static void rcu_process_callbacks(struct softirq_action *unused)
{
struct rcu_state *rsp;
+ if (cpu_is_offline(smp_processor_id()))
+ return;
trace_rcu_utilization("Start RCU core");
for_each_rcu_flavor(rsp)
__rcu_process_callbacks(rsp);
--
1.7.8
^ permalink raw reply related [flat|nested] 27+ messages in thread* Re: [PATCH tip/core/rcu 01/22] rcu: Control RCU_FANOUT_LEAF from boot-time parameter
2012-06-22 15:17 ` [PATCH tip/core/rcu 01/22] rcu: Control RCU_FANOUT_LEAF from boot-time parameter Paul E. McKenney
` (20 preceding siblings ...)
2012-06-22 15:17 ` [PATCH tip/core/rcu 22/22] rcu: Prevent offline CPUs from executing RCU core code Paul E. McKenney
@ 2012-07-01 6:19 ` Rusty Russell
2012-07-01 7:17 ` Josh Triplett
21 siblings, 1 reply; 27+ messages in thread
From: Rusty Russell @ 2012-07-01 6:19 UTC (permalink / raw)
To: Paul E. McKenney, linux-kernel
Cc: mingo, laijs, dipankar, akpm, mathieu.desnoyers, josh, niv, tglx,
peterz, rostedt, Valdis.Kletnieks, dhowells, eric.dumazet, darren,
fweisbec, sbw, patches, Paul E. McKenney
On Fri, 22 Jun 2012 08:17:00 -0700, "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> wrote:
> From: "Paul E. McKenney" <paulmck@linux.vnet.ibm.com>
>
> Although making RCU_FANOUT_LEAF a kernel configuration parameter rather
> than a fixed constant makes it easier for people to decrease cache-miss
> overhead for large systems, it is of little help for people who must
> run a single pre-built kernel binary.
>
> This commit therefore allows the value of RCU_FANOUT_LEAF to be
> increased (but not decreased!) via a boot-time parameter named
> rcutree.rcu_fanout_leaf.
...
> +static int rcu_fanout_leaf = CONFIG_RCU_FANOUT_LEAF;
> +module_param(rcu_fanout_leaf, int, 0);
Maybe it's overkill, but 0400 or 0444 might be a nice touch.
Cheers,
Rusty.
^ permalink raw reply [flat|nested] 27+ messages in thread* Re: [PATCH tip/core/rcu 01/22] rcu: Control RCU_FANOUT_LEAF from boot-time parameter
2012-07-01 6:19 ` [PATCH tip/core/rcu 01/22] rcu: Control RCU_FANOUT_LEAF from boot-time parameter Rusty Russell
@ 2012-07-01 7:17 ` Josh Triplett
2012-07-01 12:52 ` Paul E. McKenney
0 siblings, 1 reply; 27+ messages in thread
From: Josh Triplett @ 2012-07-01 7:17 UTC (permalink / raw)
To: Rusty Russell
Cc: Paul E. McKenney, linux-kernel, mingo, laijs, dipankar, akpm,
mathieu.desnoyers, niv, tglx, peterz, rostedt, Valdis.Kletnieks,
dhowells, eric.dumazet, darren, fweisbec, sbw, patches
On Sun, Jul 01, 2012 at 03:49:42PM +0930, Rusty Russell wrote:
> On Fri, 22 Jun 2012 08:17:00 -0700, "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> wrote:
> > From: "Paul E. McKenney" <paulmck@linux.vnet.ibm.com>
> >
> > Although making RCU_FANOUT_LEAF a kernel configuration parameter rather
> > than a fixed constant makes it easier for people to decrease cache-miss
> > overhead for large systems, it is of little help for people who must
> > run a single pre-built kernel binary.
> >
> > This commit therefore allows the value of RCU_FANOUT_LEAF to be
> > increased (but not decreased!) via a boot-time parameter named
> > rcutree.rcu_fanout_leaf.
> ...
> > +static int rcu_fanout_leaf = CONFIG_RCU_FANOUT_LEAF;
> > +module_param(rcu_fanout_leaf, int, 0);
>
> Maybe it's overkill, but 0400 or 0444 might be a nice touch.
I agree. 0 almost never makes sense; root should almost always have the
ability to read module parameters. And in this case, I see no reason
not to make it 0444.
- Josh Triplett
^ permalink raw reply [flat|nested] 27+ messages in thread
* Re: [PATCH tip/core/rcu 01/22] rcu: Control RCU_FANOUT_LEAF from boot-time parameter
2012-07-01 7:17 ` Josh Triplett
@ 2012-07-01 12:52 ` Paul E. McKenney
2012-07-01 17:45 ` Josh Triplett
0 siblings, 1 reply; 27+ messages in thread
From: Paul E. McKenney @ 2012-07-01 12:52 UTC (permalink / raw)
To: Josh Triplett
Cc: Rusty Russell, linux-kernel, mingo, laijs, dipankar, akpm,
mathieu.desnoyers, niv, tglx, peterz, rostedt, Valdis.Kletnieks,
dhowells, eric.dumazet, darren, fweisbec, sbw, patches
On Sun, Jul 01, 2012 at 12:17:58AM -0700, Josh Triplett wrote:
> On Sun, Jul 01, 2012 at 03:49:42PM +0930, Rusty Russell wrote:
> > On Fri, 22 Jun 2012 08:17:00 -0700, "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> wrote:
> > > From: "Paul E. McKenney" <paulmck@linux.vnet.ibm.com>
> > >
> > > Although making RCU_FANOUT_LEAF a kernel configuration parameter rather
> > > than a fixed constant makes it easier for people to decrease cache-miss
> > > overhead for large systems, it is of little help for people who must
> > > run a single pre-built kernel binary.
> > >
> > > This commit therefore allows the value of RCU_FANOUT_LEAF to be
> > > increased (but not decreased!) via a boot-time parameter named
> > > rcutree.rcu_fanout_leaf.
> > ...
> > > +static int rcu_fanout_leaf = CONFIG_RCU_FANOUT_LEAF;
> > > +module_param(rcu_fanout_leaf, int, 0);
> >
> > Maybe it's overkill, but 0400 or 0444 might be a nice touch.
>
> I agree. 0 almost never makes sense; root should almost always have the
> ability to read module parameters. And in this case, I see no reason
> not to make it 0444.
Should I do the same for these parameters as well?
kernel/rcutree.c:module_param(blimit, int, 0);
kernel/rcutree.c:module_param(qhimark, int, 0);
kernel/rcutree.c:module_param(qlowmark, int, 0);
Cut-and-paste, it is a wonderful thing. ;-)
Thanx, Paul
^ permalink raw reply [flat|nested] 27+ messages in thread
* Re: [PATCH tip/core/rcu 01/22] rcu: Control RCU_FANOUT_LEAF from boot-time parameter
2012-07-01 12:52 ` Paul E. McKenney
@ 2012-07-01 17:45 ` Josh Triplett
0 siblings, 0 replies; 27+ messages in thread
From: Josh Triplett @ 2012-07-01 17:45 UTC (permalink / raw)
To: Paul E. McKenney
Cc: Rusty Russell, linux-kernel, mingo, laijs, dipankar, akpm,
mathieu.desnoyers, niv, tglx, peterz, rostedt, Valdis.Kletnieks,
dhowells, eric.dumazet, darren, fweisbec, sbw, patches
On Sun, Jul 01, 2012 at 05:52:22AM -0700, Paul E. McKenney wrote:
> On Sun, Jul 01, 2012 at 12:17:58AM -0700, Josh Triplett wrote:
> > On Sun, Jul 01, 2012 at 03:49:42PM +0930, Rusty Russell wrote:
> > > On Fri, 22 Jun 2012 08:17:00 -0700, "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> wrote:
> > > > From: "Paul E. McKenney" <paulmck@linux.vnet.ibm.com>
> > > >
> > > > Although making RCU_FANOUT_LEAF a kernel configuration parameter rather
> > > > than a fixed constant makes it easier for people to decrease cache-miss
> > > > overhead for large systems, it is of little help for people who must
> > > > run a single pre-built kernel binary.
> > > >
> > > > This commit therefore allows the value of RCU_FANOUT_LEAF to be
> > > > increased (but not decreased!) via a boot-time parameter named
> > > > rcutree.rcu_fanout_leaf.
> > > ...
> > > > +static int rcu_fanout_leaf = CONFIG_RCU_FANOUT_LEAF;
> > > > +module_param(rcu_fanout_leaf, int, 0);
> > >
> > > Maybe it's overkill, but 0400 or 0444 might be a nice touch.
> >
> > I agree. 0 almost never makes sense; root should almost always have the
> > ability to read module parameters. And in this case, I see no reason
> > not to make it 0444.
>
> Should I do the same for these parameters as well?
>
> kernel/rcutree.c:module_param(blimit, int, 0);
> kernel/rcutree.c:module_param(qhimark, int, 0);
> kernel/rcutree.c:module_param(qlowmark, int, 0);
Yes, all of those should become 0444.
- Josh Triplett
^ permalink raw reply [flat|nested] 27+ messages in thread