* [PATCH tip/core/rcu 02/14] rcu: Trace callback acceleration
2013-01-05 17:48 ` [PATCH tip/core/rcu 01/14] rcu: Tag callback lists with corresponding grace-period number Paul E. McKenney
@ 2013-01-05 17:48 ` Paul E. McKenney
2013-01-05 17:48 ` [PATCH tip/core/rcu 03/14] rcu: Remove restrictions on no-CBs CPUs Paul E. McKenney
` (11 subsequent siblings)
12 siblings, 0 replies; 20+ messages in thread
From: Paul E. McKenney @ 2013-01-05 17:48 UTC (permalink / raw)
To: linux-kernel
Cc: mingo, laijs, dipankar, akpm, mathieu.desnoyers, josh, niv, tglx,
peterz, rostedt, Valdis.Kletnieks, dhowells, edumazet, darren,
fweisbec, sbw, patches, Paul E. McKenney
From: "Paul E. McKenney" <paulmck@linux.vnet.ibm.com>
This commit adds event tracing for callback acceleration to allow better
tracking of callbacks through the system.
Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
---
include/trace/events/rcu.h | 6 ++++--
kernel/rcutree.c | 6 ++++++
2 files changed, 10 insertions(+), 2 deletions(-)
diff --git a/include/trace/events/rcu.h b/include/trace/events/rcu.h
index d4f559b..5678114 100644
--- a/include/trace/events/rcu.h
+++ b/include/trace/events/rcu.h
@@ -44,8 +44,10 @@ TRACE_EVENT(rcu_utilization,
* of a new grace period or the end of an old grace period ("cpustart"
* and "cpuend", respectively), a CPU passing through a quiescent
* state ("cpuqs"), a CPU coming online or going offline ("cpuonl"
- * and "cpuofl", respectively), and a CPU being kicked for being too
- * long in dyntick-idle mode ("kick").
+ * and "cpuofl", respectively), a CPU being kicked for being too
+ * long in dyntick-idle mode ("kick"), a CPU accelerating its new
+ * callbacks to RCU_NEXT_READY_TAIL ("AccReadyCB"), and a CPU
+ * accelerating its new callbacks to RCU_WAIT_TAIL ("AccWaitCB").
*/
TRACE_EVENT(rcu_grace_period,
diff --git a/kernel/rcutree.c b/kernel/rcutree.c
index ac6a75d..e9dce4f 100644
--- a/kernel/rcutree.c
+++ b/kernel/rcutree.c
@@ -1168,6 +1168,12 @@ static void rcu_accelerate_cbs(struct rcu_state *rsp, struct rcu_node *rnp,
rdp->nxttail[i] = rdp->nxttail[RCU_NEXT_TAIL];
rdp->nxtcompleted[i] = c;
}
+
+ /* Trace depending on how much we were able to accelerate. */
+ if (!*rdp->nxttail[RCU_WAIT_TAIL])
+ trace_rcu_grace_period(rsp->name, rdp->gpnum, "AccWaitCB");
+ else
+ trace_rcu_grace_period(rsp->name, rdp->gpnum, "AccReadyCB");
}
/*
--
1.7.8
^ permalink raw reply related [flat|nested] 20+ messages in thread* [PATCH tip/core/rcu 03/14] rcu: Remove restrictions on no-CBs CPUs
2013-01-05 17:48 ` [PATCH tip/core/rcu 01/14] rcu: Tag callback lists with corresponding grace-period number Paul E. McKenney
2013-01-05 17:48 ` [PATCH tip/core/rcu 02/14] rcu: Trace callback acceleration Paul E. McKenney
@ 2013-01-05 17:48 ` Paul E. McKenney
2013-01-05 17:48 ` [PATCH tip/core/rcu 04/14] rcu: Provide compile-time control for " Paul E. McKenney
` (10 subsequent siblings)
12 siblings, 0 replies; 20+ messages in thread
From: Paul E. McKenney @ 2013-01-05 17:48 UTC (permalink / raw)
To: linux-kernel
Cc: mingo, laijs, dipankar, akpm, mathieu.desnoyers, josh, niv, tglx,
peterz, rostedt, Valdis.Kletnieks, dhowells, edumazet, darren,
fweisbec, sbw, patches, Paul E. McKenney, Paul E. McKenney
From: "Paul E. McKenney" <paul.mckenney@linaro.org>
Currently, CPU 0 is constrained to not be a no-CBs CPU, and furthermore
at least one no-CBs CPU must remain online at any given time. These
restrictions are problematic in some situations, such as cases where
all CPUs must run a real-time workload that needs to be insulated from
OS jitter and latencies due to RCU callback invocation. This commit
therefore provides no-CBs CPUs a way to start and to wait for grace
periods independently of the normal RCU callback mechanisms. This
approach allows any or all of the CPUs to be designated as no-CBs CPUs,
and allows any proper subset of the CPUs (whether no-CBs CPUs or not)
to be offlined.
This commit also provides event tracing, as well as a fix for a locking
bug spotted by Xie ChanglongX <changlongx.xie@intel.com>.
Signed-off-by: Paul E. McKenney <paul.mckenney@linaro.org>
Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
---
include/trace/events/rcu.h | 55 +++++++++
init/Kconfig | 4 +-
kernel/rcutree.c | 15 ++-
kernel/rcutree.h | 18 ++--
kernel/rcutree_plugin.h | 271 +++++++++++++++++++++++++++-----------------
5 files changed, 244 insertions(+), 119 deletions(-)
diff --git a/include/trace/events/rcu.h b/include/trace/events/rcu.h
index 5678114..ef0bf31 100644
--- a/include/trace/events/rcu.h
+++ b/include/trace/events/rcu.h
@@ -72,6 +72,58 @@ TRACE_EVENT(rcu_grace_period,
);
/*
+ * Tracepoint for no-callbacks grace-period events. The caller should
+ * pull the data from the rcu_node structure, other than rcuname, which
+ * comes from the rcu_state structure, and event, which is one of the
+ * following:
+ *
+ * "Startleaf": Request a nocb grace period based on leaf-node data.
+ * "Startedleaf": Leaf-node start proved sufficient.
+ * "Startedleafroot": Leaf-node start proved sufficient after checking root.
+ * "Startedroot": Requested a nocb grace period based on root-node data.
+ * "StartWait": Start waiting for the requested grace period.
+ * "ResumeWait": Resume waiting after signal.
+ * "EndWait": Complete wait.
+ * "Cleanup": Clean up rcu_node structure after previous GP.
+ * "CleanupMore": Clean up, and another no-CB GP is needed.
+ */
+TRACE_EVENT(rcu_nocb_grace_period,
+
+ TP_PROTO(char *rcuname, unsigned long gpnum, unsigned long completed,
+ unsigned long c, u8 level, int grplo, int grphi,
+ char *gpevent),
+
+ TP_ARGS(rcuname, gpnum, completed, c, level, grplo, grphi, gpevent),
+
+ TP_STRUCT__entry(
+ __field(char *, rcuname)
+ __field(unsigned long, gpnum)
+ __field(unsigned long, completed)
+ __field(unsigned long, c)
+ __field(u8, level)
+ __field(int, grplo)
+ __field(int, grphi)
+ __field(char *, gpevent)
+ ),
+
+ TP_fast_assign(
+ __entry->rcuname = rcuname;
+ __entry->gpnum = gpnum;
+ __entry->completed = completed;
+ __entry->c = c;
+ __entry->level = level;
+ __entry->grplo = grplo;
+ __entry->grphi = grphi;
+ __entry->gpevent = gpevent;
+ ),
+
+ TP_printk("%s %lu %lu %lu %u %d %d %s",
+ __entry->rcuname, __entry->gpnum, __entry->completed,
+ __entry->c, __entry->level, __entry->grplo, __entry->grphi,
+ __entry->gpevent)
+);
+
+/*
* Tracepoint for grace-period-initialization events. These are
* distinguished by the type of RCU, the new grace-period number, the
* rcu_node structure level, the starting and ending CPU covered by the
@@ -593,6 +645,9 @@ TRACE_EVENT(rcu_barrier,
#define trace_rcu_grace_period(rcuname, gpnum, gpevent) do { } while (0)
#define trace_rcu_grace_period_init(rcuname, gpnum, level, grplo, grphi, \
qsmask) do { } while (0)
+#define trace_rcu_nocb_grace_period(rcuname, gpnum, completed, c, \
+ level, grplo, grphi, event) \
+ do { } while (0)
#define trace_rcu_preempt_task(rcuname, pid, gpnum) do { } while (0)
#define trace_rcu_unlock_preempted_task(rcuname, gpnum, pid) do { } while (0)
#define trace_rcu_quiescent_state_report(rcuname, gpnum, mask, qsmask, level, \
diff --git a/init/Kconfig b/init/Kconfig
index 7d30240..fc6a3ca 100644
--- a/init/Kconfig
+++ b/init/Kconfig
@@ -655,7 +655,7 @@ config RCU_BOOST_DELAY
Accept the default if unsure.
config RCU_NOCB_CPU
- bool "Offload RCU callback processing from boot-selected CPUs"
+ bool "Offload RCU callback processing from boot-selected CPUs (EXPERIMENTAL"
depends on TREE_RCU || TREE_PREEMPT_RCU
default n
help
@@ -673,7 +673,7 @@ config RCU_NOCB_CPU
callback, and (2) affinity or cgroups can be used to force
the kthreads to run on whatever set of CPUs is desired.
- Say Y here if you want reduced OS jitter on selected CPUs.
+ Say Y here if you want to help to debug reduced OS jitter.
Say N here if you are unsure.
endmenu # "RCU Subsystem"
diff --git a/kernel/rcutree.c b/kernel/rcutree.c
index e9dce4f..8b110fa 100644
--- a/kernel/rcutree.c
+++ b/kernel/rcutree.c
@@ -316,6 +316,8 @@ cpu_needs_another_gp(struct rcu_state *rsp, struct rcu_data *rdp)
if (rcu_gp_in_progress(rsp))
return 0; /* No, a grace period is already in progress. */
+ if (rcu_nocb_needs_gp(rsp))
+ return 1; /* Yes, a no-CBs CPU needs one. */
if (!rdp->nxttail[RCU_NEXT_TAIL])
return 0; /* No, this is a no-CBs (or offline) CPU. */
if (*rdp->nxttail[RCU_NEXT_READY_TAIL])
@@ -1400,6 +1402,7 @@ int rcu_gp_fqs(struct rcu_state *rsp, int fqs_state_in)
static void rcu_gp_cleanup(struct rcu_state *rsp)
{
unsigned long gp_duration;
+ int nocb = 0;
struct rcu_data *rdp;
struct rcu_node *rnp = rcu_get_root(rsp);
@@ -1430,11 +1433,13 @@ static void rcu_gp_cleanup(struct rcu_state *rsp)
rcu_for_each_node_breadth_first(rsp, rnp) {
raw_spin_lock_irq(&rnp->lock);
rnp->completed = rsp->gpnum;
+ nocb += rcu_nocb_gp_cleanup(rsp, rnp);
raw_spin_unlock_irq(&rnp->lock);
cond_resched();
}
rnp = rcu_get_root(rsp);
raw_spin_lock_irq(&rnp->lock);
+ rcu_nocb_gp_set(rnp, nocb);
rsp->completed = rsp->gpnum; /* Declare grace period done. */
trace_rcu_grace_period(rsp->name, rsp->completed, "end");
@@ -2951,7 +2956,6 @@ static int __cpuinit rcu_cpu_notify(struct notifier_block *self,
struct rcu_data *rdp = per_cpu_ptr(rcu_state->rda, cpu);
struct rcu_node *rnp = rdp->mynode;
struct rcu_state *rsp;
- int ret = NOTIFY_OK;
trace_rcu_utilization("Start CPU hotplug");
switch (action) {
@@ -2965,10 +2969,7 @@ static int __cpuinit rcu_cpu_notify(struct notifier_block *self,
rcu_boost_kthread_setaffinity(rnp, -1);
break;
case CPU_DOWN_PREPARE:
- if (nocb_cpu_expendable(cpu))
- rcu_boost_kthread_setaffinity(rnp, cpu);
- else
- ret = NOTIFY_BAD;
+ rcu_boost_kthread_setaffinity(rnp, cpu);
break;
case CPU_DYING:
case CPU_DYING_FROZEN:
@@ -2992,7 +2993,7 @@ static int __cpuinit rcu_cpu_notify(struct notifier_block *self,
break;
}
trace_rcu_utilization("End CPU hotplug");
- return ret;
+ return NOTIFY_OK;
}
/*
@@ -3123,6 +3124,7 @@ static void __init rcu_init_one(struct rcu_state *rsp,
}
rnp->level = i;
INIT_LIST_HEAD(&rnp->blkd_tasks);
+ rcu_init_one_nocb(rnp);
}
}
@@ -3208,7 +3210,6 @@ void __init rcu_init(void)
rcu_init_one(&rcu_sched_state, &rcu_sched_data);
rcu_init_one(&rcu_bh_state, &rcu_bh_data);
__rcu_init_preempt();
- rcu_init_nocb();
open_softirq(RCU_SOFTIRQ, rcu_process_callbacks);
/*
diff --git a/kernel/rcutree.h b/kernel/rcutree.h
index c9f362e..ef26eab 100644
--- a/kernel/rcutree.h
+++ b/kernel/rcutree.h
@@ -200,6 +200,12 @@ struct rcu_node {
/* Refused to boost: not sure why, though. */
/* This can happen due to race conditions. */
#endif /* #ifdef CONFIG_RCU_BOOST */
+#ifdef CONFIG_RCU_NOCB_CPU
+ wait_queue_head_t nocb_gp_wq[2];
+ /* Place for rcu_nocb_kthread() to wait GP. */
+ int n_nocb_gp_requests[2];
+ /* Counts of upcoming no-CB GP requests. */
+#endif /* #ifdef CONFIG_RCU_NOCB_CPU */
raw_spinlock_t fqslock ____cacheline_internodealigned_in_smp;
} ____cacheline_internodealigned_in_smp;
@@ -384,12 +390,6 @@ struct rcu_state {
struct rcu_data __percpu *rda; /* pointer of percu rcu_data. */
void (*call)(struct rcu_head *head, /* call_rcu() flavor. */
void (*func)(struct rcu_head *head));
-#ifdef CONFIG_RCU_NOCB_CPU
- void (*call_remote)(struct rcu_head *head,
- void (*func)(struct rcu_head *head));
- /* call_rcu() flavor, but for */
- /* placing on remote CPU. */
-#endif /* #ifdef CONFIG_RCU_NOCB_CPU */
/* The following fields are guarded by the root rcu_node's lock. */
@@ -538,16 +538,18 @@ static void print_cpu_stall_info(struct rcu_state *rsp, int cpu);
static void print_cpu_stall_info_end(void);
static void zero_cpu_stall_ticks(struct rcu_data *rdp);
static void increment_cpu_stall_ticks(void);
+static int rcu_nocb_needs_gp(struct rcu_state *rsp);
+static void rcu_nocb_gp_set(struct rcu_node *rnp, int nrq);
+static int rcu_nocb_gp_cleanup(struct rcu_state *rsp, struct rcu_node *rnp);
+static void rcu_init_one_nocb(struct rcu_node *rnp);
static bool is_nocb_cpu(int cpu);
static bool __call_rcu_nocb(struct rcu_data *rdp, struct rcu_head *rhp,
bool lazy);
static bool rcu_nocb_adopt_orphan_cbs(struct rcu_state *rsp,
struct rcu_data *rdp);
-static bool nocb_cpu_expendable(int cpu);
static void rcu_boot_init_nocb_percpu_data(struct rcu_data *rdp);
static void rcu_spawn_nocb_kthreads(struct rcu_state *rsp);
static void init_nocb_callback_list(struct rcu_data *rdp);
-static void __init rcu_init_nocb(void);
#endif /* #ifndef RCU_TREE_NONCORE */
diff --git a/kernel/rcutree_plugin.h b/kernel/rcutree_plugin.h
index f6e5ec2..37750bc 100644
--- a/kernel/rcutree_plugin.h
+++ b/kernel/rcutree_plugin.h
@@ -87,10 +87,6 @@ static void __init rcu_bootup_announce_oddness(void)
printk(KERN_INFO "\tRCU restricting CPUs from NR_CPUS=%d to nr_cpu_ids=%d.\n", NR_CPUS, nr_cpu_ids);
#ifdef CONFIG_RCU_NOCB_CPU
if (have_rcu_nocb_mask) {
- if (cpumask_test_cpu(0, rcu_nocb_mask)) {
- cpumask_clear_cpu(0, rcu_nocb_mask);
- pr_info("\tCPU 0: illegal no-CBs CPU (cleared).\n");
- }
cpulist_scnprintf(nocb_buf, sizeof(nocb_buf), rcu_nocb_mask);
pr_info("\tExperimental no-CBs CPUs: %s.\n", nocb_buf);
if (rcu_nocb_poll)
@@ -2159,6 +2155,57 @@ static int __init rcu_nocb_setup(char *str)
}
__setup("rcu_nocbs=", rcu_nocb_setup);
+/*
+ * Do any no-CBs CPUs need another grace period?
+ *
+ * Interrupts must be disabled. If the caller does not hold the root
+ * rnp_node structure's ->lock, the results are advisory only.
+ */
+static int rcu_nocb_needs_gp(struct rcu_state *rsp)
+{
+ struct rcu_node *rnp = rcu_get_root(rsp);
+
+ return rnp->n_nocb_gp_requests[(ACCESS_ONCE(rnp->completed) + 1) & 0x1];
+}
+
+/*
+ * Clean up this rcu_node structure's no-CBs state at the end of
+ * a grace period, and also return whether any no-CBs CPU associated
+ * with this rcu_node structure needs another grace period.
+ */
+static int rcu_nocb_gp_cleanup(struct rcu_state *rsp, struct rcu_node *rnp)
+{
+ int c = rnp->completed;
+ int needmore;
+
+ wake_up_all(&rnp->nocb_gp_wq[c & 0x1]);
+ rnp->n_nocb_gp_requests[c & 0x1] = 0;
+ needmore = rnp->n_nocb_gp_requests[(c + 1) & 0x1];
+ trace_rcu_nocb_grace_period(rsp->name, rnp->gpnum, rnp->completed,
+ c, rnp->level, rnp->grplo, rnp->grphi,
+ needmore ? "CleanupMore" : "Cleanup");
+ return needmore;
+}
+
+/*
+ * Set the root rcu_node structure's ->n_nocb_gp_requests field
+ * based on the sum of those of all rcu_node structures. This does
+ * double-count the root rcu_node structure's requests, but this
+ * is necessary to handle the possibility of a rcu_nocb_kthread()
+ * having awakened during the time that the rcu_node structures
+ * were being updated for the end of the previous grace period.
+ */
+static void rcu_nocb_gp_set(struct rcu_node *rnp, int nrq)
+{
+ rnp->n_nocb_gp_requests[(rnp->completed + 1) & 0x1] += nrq;
+}
+
+static void rcu_init_one_nocb(struct rcu_node *rnp)
+{
+ init_waitqueue_head(&rnp->nocb_gp_wq[0]);
+ init_waitqueue_head(&rnp->nocb_gp_wq[1]);
+}
+
/* Is the specified CPU a no-CPUs CPU? */
static bool is_nocb_cpu(int cpu)
{
@@ -2221,6 +2268,13 @@ static bool __call_rcu_nocb(struct rcu_data *rdp, struct rcu_head *rhp,
if (!is_nocb_cpu(rdp->cpu))
return 0;
__call_rcu_nocb_enqueue(rdp, rhp, &rhp->next, 1, lazy);
+ if (__is_kfree_rcu_offset((unsigned long)rhp->func))
+ trace_rcu_kfree_callback(rdp->rsp->name, rhp,
+ (unsigned long)rhp->func,
+ rdp->qlen_lazy, rdp->qlen);
+ else
+ trace_rcu_callback(rdp->rsp->name, rhp,
+ rdp->qlen_lazy, rdp->qlen);
return 1;
}
@@ -2259,95 +2313,108 @@ static bool __maybe_unused rcu_nocb_adopt_orphan_cbs(struct rcu_state *rsp,
}
/*
- * There must be at least one non-no-CBs CPU in operation at any given
- * time, because no-CBs CPUs are not capable of initiating grace periods
- * independently. This function therefore complains if the specified
- * CPU is the last non-no-CBs CPU, allowing the CPU-hotplug system to
- * avoid offlining the last such CPU. (Recursion is a wonderful thing,
- * but you have to have a base case!)
+ * If necessary, kick off a new grace period, and either way wait
+ * for a subsequent grace period to complete.
*/
-static bool nocb_cpu_expendable(int cpu)
+static void rcu_nocb_wait_gp(struct rcu_data *rdp)
{
- cpumask_var_t non_nocb_cpus;
- int ret;
+ unsigned long c;
+ bool d;
+ unsigned long flags;
+ unsigned long flags1;
+ struct rcu_node *rnp = rdp->mynode;
+ struct rcu_node *rnp_root = rcu_get_root(rdp->rsp);
- /*
- * If there are no no-CB CPUs or if this CPU is not a no-CB CPU,
- * then offlining this CPU is harmless. Let it happen.
- */
- if (!have_rcu_nocb_mask || is_nocb_cpu(cpu))
- return 1;
+ raw_spin_lock_irqsave(&rnp->lock, flags);
+ c = rnp->completed + 2;
- /* If no memory, play it safe and keep the CPU around. */
- if (!alloc_cpumask_var(&non_nocb_cpus, GFP_NOIO))
- return 0;
- cpumask_andnot(non_nocb_cpus, cpu_online_mask, rcu_nocb_mask);
- cpumask_clear_cpu(cpu, non_nocb_cpus);
- ret = !cpumask_empty(non_nocb_cpus);
- free_cpumask_var(non_nocb_cpus);
- return ret;
-}
+ /* Count our request for a grace period. */
+ rnp->n_nocb_gp_requests[c & 0x1]++;
+ trace_rcu_nocb_grace_period(rdp->rsp->name, rnp->gpnum, rnp->completed,
+ c, rnp->level, rnp->grplo, rnp->grphi,
+ "Startleaf");
-/*
- * Helper structure for remote registry of RCU callbacks.
- * This is needed for when a no-CBs CPU needs to start a grace period.
- * If it just invokes call_rcu(), the resulting callback will be queued,
- * which can result in deadlock.
- */
-struct rcu_head_remote {
- struct rcu_head *rhp;
- call_rcu_func_t *crf;
- void (*func)(struct rcu_head *rhp);
-};
+ if (rnp->gpnum != rnp->completed) {
-/*
- * Register a callback as specified by the rcu_head_remote struct.
- * This function is intended to be invoked via smp_call_function_single().
- */
-static void call_rcu_local(void *arg)
-{
- struct rcu_head_remote *rhrp =
- container_of(arg, struct rcu_head_remote, rhp);
+ /*
+ * This rcu_node structure believes that a grace period
+ * is in progress, so we are done. When this grace
+ * period ends, our request will be acted upon.
+ */
+ trace_rcu_nocb_grace_period(rdp->rsp->name,
+ rnp->gpnum, rnp->completed, c,
+ rnp->level, rnp->grplo, rnp->grphi,
+ "Startedleaf");
+ raw_spin_unlock_irqrestore(&rnp->lock, flags);
- rhrp->crf(rhrp->rhp, rhrp->func);
-}
+ } else {
-/*
- * Set up an rcu_head_remote structure and the invoke call_rcu_local()
- * on CPU 0 (which is guaranteed to be a non-no-CBs CPU) via
- * smp_call_function_single().
- */
-static void invoke_crf_remote(struct rcu_head *rhp,
- void (*func)(struct rcu_head *rhp),
- call_rcu_func_t crf)
-{
- struct rcu_head_remote rhr;
+ /*
+ * Might not be a grace period, check root rcu_node
+ * structure to see if we must start one.
+ */
+ if (rnp != rnp_root)
+ raw_spin_lock(&rnp_root->lock); /* irqs disabled. */
+ if (rnp_root->gpnum != rnp_root->completed) {
+ trace_rcu_nocb_grace_period(rdp->rsp->name,
+ rnp->gpnum, rnp->completed,
+ c, rnp->level,
+ rnp->grplo, rnp->grphi,
+ "Startedleafroot");
+ raw_spin_unlock(&rnp_root->lock); /* irqs disabled. */
+ } else {
- rhr.rhp = rhp;
- rhr.crf = crf;
- rhr.func = func;
- smp_call_function_single(0, call_rcu_local, &rhr, 1);
-}
+ /*
+ * No grace period, so we need to start one.
+ * The good news is that we can wait for exactly
+ * one grace period instead of part of the current
+ * grace period and all of the next grace period.
+ * Adjust counters accordingly and start the
+ * needed grace period.
+ */
+ rnp->n_nocb_gp_requests[c & 0x1]--;
+ c = rnp_root->completed + 1;
+ rnp->n_nocb_gp_requests[c & 0x1]++;
+ rnp_root->n_nocb_gp_requests[c & 0x1]++;
+ trace_rcu_nocb_grace_period(rdp->rsp->name,
+ rnp->gpnum, rnp->completed,
+ c, rnp->level,
+ rnp->grplo, rnp->grphi,
+ "Startedroot");
+ local_save_flags(flags1);
+ rcu_start_gp(rdp->rsp, flags1); /* Rlses ->lock. */
+ }
-/*
- * Helper functions to be passed to wait_rcu_gp(), each of which
- * invokes invoke_crf_remote() to register a callback appropriately.
- */
-static void __maybe_unused
-call_rcu_preempt_remote(struct rcu_head *rhp,
- void (*func)(struct rcu_head *rhp))
-{
- invoke_crf_remote(rhp, func, call_rcu);
-}
-static void call_rcu_bh_remote(struct rcu_head *rhp,
- void (*func)(struct rcu_head *rhp))
-{
- invoke_crf_remote(rhp, func, call_rcu_bh);
-}
-static void call_rcu_sched_remote(struct rcu_head *rhp,
- void (*func)(struct rcu_head *rhp))
-{
- invoke_crf_remote(rhp, func, call_rcu_sched);
+ /* Clean up locking and irq state. */
+ if (rnp != rnp_root)
+ raw_spin_unlock_irqrestore(&rnp->lock, flags);
+ else
+ local_irq_restore(flags);
+ }
+
+ /*
+ * Wait for the grace period. Do so interruptibly to avoid messing
+ * up the load average.
+ */
+ trace_rcu_nocb_grace_period(rdp->rsp->name, rnp->gpnum, rnp->completed,
+ c, rnp->level, rnp->grplo, rnp->grphi,
+ "StartWait");
+ for (;;) {
+ wait_event_interruptible(
+ rnp->nocb_gp_wq[c & 0x1],
+ (d = ULONG_CMP_GE(ACCESS_ONCE(rnp->completed), c)));
+ if (likely(d))
+ break;
+ flush_signals(current);
+ trace_rcu_nocb_grace_period(rdp->rsp->name,
+ rnp->gpnum, rnp->completed, c,
+ rnp->level, rnp->grplo, rnp->grphi,
+ "ResumeWait");
+ }
+ trace_rcu_nocb_grace_period(rdp->rsp->name, rnp->gpnum, rnp->completed,
+ c, rnp->level, rnp->grplo, rnp->grphi,
+ "EndWait");
+ smp_mb(); /* Ensure that CB invocation happens after GP end. */
}
/*
@@ -2366,10 +2433,11 @@ static int rcu_nocb_kthread(void *arg)
for (;;) {
/* If not polling, wait for next batch of callbacks. */
if (!rcu_nocb_poll)
- wait_event(rdp->nocb_wq, rdp->nocb_head);
+ wait_event_interruptible(rdp->nocb_wq, rdp->nocb_head);
list = ACCESS_ONCE(rdp->nocb_head);
if (!list) {
schedule_timeout_interruptible(1);
+ flush_signals(current);
continue;
}
@@ -2383,7 +2451,7 @@ static int rcu_nocb_kthread(void *arg)
cl = atomic_long_xchg(&rdp->nocb_q_count_lazy, 0);
ACCESS_ONCE(rdp->nocb_p_count) += c;
ACCESS_ONCE(rdp->nocb_p_count_lazy) += cl;
- wait_rcu_gp(rdp->rsp->call_remote);
+ rcu_nocb_wait_gp(rdp);
/* Each pass through the following loop invokes a callback. */
trace_rcu_batch_start(rdp->rsp->name, cl, c, -1);
@@ -2444,17 +2512,25 @@ static void init_nocb_callback_list(struct rcu_data *rdp)
rdp->nxttail[RCU_NEXT_TAIL] = NULL;
}
-/* Initialize the ->call_remote fields in the rcu_state structures. */
-static void __init rcu_init_nocb(void)
+#else /* #ifdef CONFIG_RCU_NOCB_CPU */
+
+static int rcu_nocb_needs_gp(struct rcu_state *rsp)
{
-#ifdef CONFIG_PREEMPT_RCU
- rcu_preempt_state.call_remote = call_rcu_preempt_remote;
-#endif /* #ifdef CONFIG_PREEMPT_RCU */
- rcu_bh_state.call_remote = call_rcu_bh_remote;
- rcu_sched_state.call_remote = call_rcu_sched_remote;
+ return 0;
}
-#else /* #ifdef CONFIG_RCU_NOCB_CPU */
+static int rcu_nocb_gp_cleanup(struct rcu_state *rsp, struct rcu_node *rnp)
+{
+ return 0;
+}
+
+static void rcu_nocb_gp_set(struct rcu_node *rnp, int nrq)
+{
+}
+
+static void rcu_init_one_nocb(struct rcu_node *rnp)
+{
+}
static bool is_nocb_cpu(int cpu)
{
@@ -2473,11 +2549,6 @@ static bool __maybe_unused rcu_nocb_adopt_orphan_cbs(struct rcu_state *rsp,
return 0;
}
-static bool nocb_cpu_expendable(int cpu)
-{
- return 1;
-}
-
static void __init rcu_boot_init_nocb_percpu_data(struct rcu_data *rdp)
{
}
@@ -2490,8 +2561,4 @@ static void init_nocb_callback_list(struct rcu_data *rdp)
{
}
-static void __init rcu_init_nocb(void)
-{
-}
-
#endif /* #else #ifdef CONFIG_RCU_NOCB_CPU */
--
1.7.8
^ permalink raw reply related [flat|nested] 20+ messages in thread* [PATCH tip/core/rcu 04/14] rcu: Provide compile-time control for no-CBs CPUs
2013-01-05 17:48 ` [PATCH tip/core/rcu 01/14] rcu: Tag callback lists with corresponding grace-period number Paul E. McKenney
2013-01-05 17:48 ` [PATCH tip/core/rcu 02/14] rcu: Trace callback acceleration Paul E. McKenney
2013-01-05 17:48 ` [PATCH tip/core/rcu 03/14] rcu: Remove restrictions on no-CBs CPUs Paul E. McKenney
@ 2013-01-05 17:48 ` Paul E. McKenney
2013-01-07 16:50 ` Josh Triplett
2013-01-05 17:48 ` [PATCH tip/core/rcu 05/14] rcu: Distinguish "rcuo" kthreads by RCU flavor Paul E. McKenney
` (9 subsequent siblings)
12 siblings, 1 reply; 20+ messages in thread
From: Paul E. McKenney @ 2013-01-05 17:48 UTC (permalink / raw)
To: linux-kernel
Cc: mingo, laijs, dipankar, akpm, mathieu.desnoyers, josh, niv, tglx,
peterz, rostedt, Valdis.Kletnieks, dhowells, edumazet, darren,
fweisbec, sbw, patches, Paul E. McKenney, Paul E. McKenney
From: "Paul E. McKenney" <paul.mckenney@linaro.org>
Currently, the only way to specify no-CBs CPUs is via the rcu_nocbs
kernel command-line parameter. This is inconvenient in some cases,
particularly for randconfig testing, so this commit adds a new
RCU_NOCB_CPU_DEFAULT kernel configuration parameter. Setting this
new parameter to zero (the default) retains the old behavior, setting
it to one offloads callback processing from CPU 0 (along with any
other CPUs specified by the rcu_nocbs boot-time parameter), and setting
it to two offloads callback processing from all CPUs.
Signed-off-by: Paul E. McKenney <paul.mckenney@linaro.org>
Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
---
init/Kconfig | 13 +++++++++++++
kernel/rcutree_plugin.h | 13 +++++++++++++
2 files changed, 26 insertions(+), 0 deletions(-)
diff --git a/init/Kconfig b/init/Kconfig
index fc6a3ca..35dcedb 100644
--- a/init/Kconfig
+++ b/init/Kconfig
@@ -676,6 +676,19 @@ config RCU_NOCB_CPU
Say Y here if you want to help to debug reduced OS jitter.
Say N here if you are unsure.
+config RCU_NOCB_CPU_DEFAULT
+ int "Offload RCU callback processing from compile-selected CPUs"
+ depends on RCU_NOCB_CPU
+ range 0 2
+ default 0
+ help
+ Set this option to zero to only offload RCU callback processing
+ from those CPUs specified by the boot-time rcu_nocbs kernel
+ parameter. Set it to one to offload processing from CPU 0
+ in addition to any CPUs specified at boot time. Set it to
+ two to offload processing from all CPUs, regardless of the
+ setting of the boot-time rcu_nocbs kernel parameter.
+
endmenu # "RCU Subsystem"
config IKCONFIG
diff --git a/kernel/rcutree_plugin.h b/kernel/rcutree_plugin.h
index 37750bc..eb9b473 100644
--- a/kernel/rcutree_plugin.h
+++ b/kernel/rcutree_plugin.h
@@ -86,6 +86,19 @@ static void __init rcu_bootup_announce_oddness(void)
if (nr_cpu_ids != NR_CPUS)
printk(KERN_INFO "\tRCU restricting CPUs from NR_CPUS=%d to nr_cpu_ids=%d.\n", NR_CPUS, nr_cpu_ids);
#ifdef CONFIG_RCU_NOCB_CPU
+#if CONFIG_RCU_NOCB_CPU_DEFAULT != 0
+ if (!have_rcu_nocb_mask) {
+ alloc_bootmem_cpumask_var(&rcu_nocb_mask);
+ have_rcu_nocb_mask = true;
+ }
+#if CONFIG_RCU_NOCB_CPU_DEFAULT == 1
+ pr_info("\tExperimental no-CBs CPU 0\n");
+ cpumask_set_cpu(0, rcu_nocb_mask);
+#else /* #if CONFIG_RCU_NOCB_CPU_DEFAULT == 1 */
+ pr_info("\tExperimental no-CBs for all CPUs\n");
+ cpumask_setall(rcu_nocb_mask);
+#endif /* #else #if CONFIG_RCU_NOCB_CPU_DEFAULT == 1 */
+#endif /* #if CONFIG_RCU_NOCB_CPU_DEFAULT != 0 */
if (have_rcu_nocb_mask) {
cpulist_scnprintf(nocb_buf, sizeof(nocb_buf), rcu_nocb_mask);
pr_info("\tExperimental no-CBs CPUs: %s.\n", nocb_buf);
--
1.7.8
^ permalink raw reply related [flat|nested] 20+ messages in thread* Re: [PATCH tip/core/rcu 04/14] rcu: Provide compile-time control for no-CBs CPUs
2013-01-05 17:48 ` [PATCH tip/core/rcu 04/14] rcu: Provide compile-time control for " Paul E. McKenney
@ 2013-01-07 16:50 ` Josh Triplett
2013-01-07 22:09 ` Paul E. McKenney
0 siblings, 1 reply; 20+ messages in thread
From: Josh Triplett @ 2013-01-07 16:50 UTC (permalink / raw)
To: Paul E. McKenney
Cc: linux-kernel, mingo, laijs, dipankar, akpm, mathieu.desnoyers,
niv, tglx, peterz, rostedt, Valdis.Kletnieks, dhowells, edumazet,
darren, fweisbec, sbw, patches, Paul E. McKenney
On Sat, Jan 05, 2013 at 09:48:54AM -0800, Paul E. McKenney wrote:
> From: "Paul E. McKenney" <paul.mckenney@linaro.org>
>
> Currently, the only way to specify no-CBs CPUs is via the rcu_nocbs
> kernel command-line parameter. This is inconvenient in some cases,
> particularly for randconfig testing, so this commit adds a new
> RCU_NOCB_CPU_DEFAULT kernel configuration parameter. Setting this
> new parameter to zero (the default) retains the old behavior, setting
> it to one offloads callback processing from CPU 0 (along with any
> other CPUs specified by the rcu_nocbs boot-time parameter), and setting
> it to two offloads callback processing from all CPUs.
>
> Signed-off-by: Paul E. McKenney <paul.mckenney@linaro.org>
> Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
Comments below.
> init/Kconfig | 13 +++++++++++++
> kernel/rcutree_plugin.h | 13 +++++++++++++
> 2 files changed, 26 insertions(+), 0 deletions(-)
>
> diff --git a/init/Kconfig b/init/Kconfig
> index fc6a3ca..35dcedb 100644
> --- a/init/Kconfig
> +++ b/init/Kconfig
> @@ -676,6 +676,19 @@ config RCU_NOCB_CPU
> Say Y here if you want to help to debug reduced OS jitter.
> Say N here if you are unsure.
>
> +config RCU_NOCB_CPU_DEFAULT
> + int "Offload RCU callback processing from compile-selected CPUs"
> + depends on RCU_NOCB_CPU
> + range 0 2
> + default 0
> + help
> + Set this option to zero to only offload RCU callback processing
> + from those CPUs specified by the boot-time rcu_nocbs kernel
> + parameter. Set it to one to offload processing from CPU 0
> + in addition to any CPUs specified at boot time. Set it to
> + two to offload processing from all CPUs, regardless of the
> + setting of the boot-time rcu_nocbs kernel parameter.
> +
Rather than making users translate the meanings of 0, 1, and 2, this
seems like something better done with a choice/endchoice; that would
then offer a menu of options that can each have their own description.
Take a look at the "Memory split" option in arch/x86/Kconfig for an
example.
> --- a/kernel/rcutree_plugin.h
> +++ b/kernel/rcutree_plugin.h
> @@ -86,6 +86,19 @@ static void __init rcu_bootup_announce_oddness(void)
> if (nr_cpu_ids != NR_CPUS)
> printk(KERN_INFO "\tRCU restricting CPUs from NR_CPUS=%d to nr_cpu_ids=%d.\n", NR_CPUS, nr_cpu_ids);
> #ifdef CONFIG_RCU_NOCB_CPU
> +#if CONFIG_RCU_NOCB_CPU_DEFAULT != 0
That would then make each of these conditionals self-documenting as
well.
- Josh Triplett
^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [PATCH tip/core/rcu 04/14] rcu: Provide compile-time control for no-CBs CPUs
2013-01-07 16:50 ` Josh Triplett
@ 2013-01-07 22:09 ` Paul E. McKenney
0 siblings, 0 replies; 20+ messages in thread
From: Paul E. McKenney @ 2013-01-07 22:09 UTC (permalink / raw)
To: Josh Triplett
Cc: linux-kernel, mingo, laijs, dipankar, akpm, mathieu.desnoyers,
niv, tglx, peterz, rostedt, Valdis.Kletnieks, dhowells, edumazet,
darren, fweisbec, sbw, patches, Paul E. McKenney
On Mon, Jan 07, 2013 at 08:50:23AM -0800, Josh Triplett wrote:
> On Sat, Jan 05, 2013 at 09:48:54AM -0800, Paul E. McKenney wrote:
> > From: "Paul E. McKenney" <paul.mckenney@linaro.org>
> >
> > Currently, the only way to specify no-CBs CPUs is via the rcu_nocbs
> > kernel command-line parameter. This is inconvenient in some cases,
> > particularly for randconfig testing, so this commit adds a new
> > RCU_NOCB_CPU_DEFAULT kernel configuration parameter. Setting this
> > new parameter to zero (the default) retains the old behavior, setting
> > it to one offloads callback processing from CPU 0 (along with any
> > other CPUs specified by the rcu_nocbs boot-time parameter), and setting
> > it to two offloads callback processing from all CPUs.
> >
> > Signed-off-by: Paul E. McKenney <paul.mckenney@linaro.org>
> > Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
>
> Comments below.
>
> > init/Kconfig | 13 +++++++++++++
> > kernel/rcutree_plugin.h | 13 +++++++++++++
> > 2 files changed, 26 insertions(+), 0 deletions(-)
> >
> > diff --git a/init/Kconfig b/init/Kconfig
> > index fc6a3ca..35dcedb 100644
> > --- a/init/Kconfig
> > +++ b/init/Kconfig
> > @@ -676,6 +676,19 @@ config RCU_NOCB_CPU
> > Say Y here if you want to help to debug reduced OS jitter.
> > Say N here if you are unsure.
> >
> > +config RCU_NOCB_CPU_DEFAULT
> > + int "Offload RCU callback processing from compile-selected CPUs"
> > + depends on RCU_NOCB_CPU
> > + range 0 2
> > + default 0
> > + help
> > + Set this option to zero to only offload RCU callback processing
> > + from those CPUs specified by the boot-time rcu_nocbs kernel
> > + parameter. Set it to one to offload processing from CPU 0
> > + in addition to any CPUs specified at boot time. Set it to
> > + two to offload processing from all CPUs, regardless of the
> > + setting of the boot-time rcu_nocbs kernel parameter.
> > +
>
> Rather than making users translate the meanings of 0, 1, and 2, this
> seems like something better done with a choice/endchoice; that would
> then offer a menu of options that can each have their own description.
> Take a look at the "Memory split" option in arch/x86/Kconfig for an
> example.
My first reaction was that this couldn't possibly handle the dependency
on CONFIG_RCU_NOCB_CPU, but it actually does work, suppressing all the
choices when CONFIG_RCU_NOCB_CPU=n. Cute!
> > --- a/kernel/rcutree_plugin.h
> > +++ b/kernel/rcutree_plugin.h
> > @@ -86,6 +86,19 @@ static void __init rcu_bootup_announce_oddness(void)
> > if (nr_cpu_ids != NR_CPUS)
> > printk(KERN_INFO "\tRCU restricting CPUs from NR_CPUS=%d to nr_cpu_ids=%d.\n", NR_CPUS, nr_cpu_ids);
> > #ifdef CONFIG_RCU_NOCB_CPU
> > +#if CONFIG_RCU_NOCB_CPU_DEFAULT != 0
>
> That would then make each of these conditionals self-documenting as
> well.
Indeed, much better! Thank you!
Thanx, Paul
^ permalink raw reply [flat|nested] 20+ messages in thread
* [PATCH tip/core/rcu 05/14] rcu: Distinguish "rcuo" kthreads by RCU flavor
2013-01-05 17:48 ` [PATCH tip/core/rcu 01/14] rcu: Tag callback lists with corresponding grace-period number Paul E. McKenney
` (2 preceding siblings ...)
2013-01-05 17:48 ` [PATCH tip/core/rcu 04/14] rcu: Provide compile-time control for " Paul E. McKenney
@ 2013-01-05 17:48 ` Paul E. McKenney
2013-01-06 23:34 ` Paul Gortmaker
2013-01-07 16:54 ` Josh Triplett
2013-01-05 17:48 ` [PATCH tip/core/rcu 06/14] rcu: Export RCU_FAST_NO_HZ parameters to sysfs Paul E. McKenney
` (8 subsequent siblings)
12 siblings, 2 replies; 20+ messages in thread
From: Paul E. McKenney @ 2013-01-05 17:48 UTC (permalink / raw)
To: linux-kernel
Cc: mingo, laijs, dipankar, akpm, mathieu.desnoyers, josh, niv, tglx,
peterz, rostedt, Valdis.Kletnieks, dhowells, edumazet, darren,
fweisbec, sbw, patches, Paul E. McKenney, Paul E. McKenney
From: "Paul E. McKenney" <paul.mckenney@linaro.org>
Currently, the per-no-CBs-CPU kthreads are named "rcuo" followed by
the CPU number, for example, "rcuo". This is problematic given that
there are either two or three RCU flavors, each of which gets a per-CPU
kthread with exactly the same name. This commit therefore introduces
a one-letter abbreviation for each RCU flavor, namely 'b' for RCU-bh,
'p' for RCU-preempt, and 's' for RCU-sched. This abbreviation use used
to distinguish the "rcuo" kthreads, for example, for CPU 0 we would have
"rcuo0b", "rcuo0p", and "rcuo0s".
Signed-off-by: Paul E. McKenney <paul.mckenney@linaro.org>
Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
Tested-by: Dietmar Eggemann <dietmar.eggemann@arm.com>
---
kernel/rcutree.c | 7 ++++---
kernel/rcutree.h | 1 +
kernel/rcutree_plugin.h | 5 +++--
3 files changed, 8 insertions(+), 5 deletions(-)
diff --git a/kernel/rcutree.c b/kernel/rcutree.c
index 8b110fa..4ec797e 100644
--- a/kernel/rcutree.c
+++ b/kernel/rcutree.c
@@ -64,7 +64,7 @@
static struct lock_class_key rcu_node_class[RCU_NUM_LVLS];
static struct lock_class_key rcu_fqs_class[RCU_NUM_LVLS];
-#define RCU_STATE_INITIALIZER(sname, cr) { \
+#define RCU_STATE_INITIALIZER(sname, sabbr, cr) { \
.level = { &sname##_state.node[0] }, \
.call = cr, \
.fqs_state = RCU_GP_IDLE, \
@@ -76,13 +76,14 @@ static struct lock_class_key rcu_fqs_class[RCU_NUM_LVLS];
.barrier_mutex = __MUTEX_INITIALIZER(sname##_state.barrier_mutex), \
.onoff_mutex = __MUTEX_INITIALIZER(sname##_state.onoff_mutex), \
.name = #sname, \
+ .abbr = sabbr, \
}
struct rcu_state rcu_sched_state =
- RCU_STATE_INITIALIZER(rcu_sched, call_rcu_sched);
+ RCU_STATE_INITIALIZER(rcu_sched, 's', call_rcu_sched);
DEFINE_PER_CPU(struct rcu_data, rcu_sched_data);
-struct rcu_state rcu_bh_state = RCU_STATE_INITIALIZER(rcu_bh, call_rcu_bh);
+struct rcu_state rcu_bh_state = RCU_STATE_INITIALIZER(rcu_bh, 'b', call_rcu_bh);
DEFINE_PER_CPU(struct rcu_data, rcu_bh_data);
static struct rcu_state *rcu_state;
diff --git a/kernel/rcutree.h b/kernel/rcutree.h
index ef26eab..c865117 100644
--- a/kernel/rcutree.h
+++ b/kernel/rcutree.h
@@ -452,6 +452,7 @@ struct rcu_state {
unsigned long gp_max; /* Maximum GP duration in */
/* jiffies. */
char *name; /* Name of structure. */
+ char abbr; /* Abbreviated name. */
struct list_head flavors; /* List of RCU flavors. */
};
diff --git a/kernel/rcutree_plugin.h b/kernel/rcutree_plugin.h
index eb9b473..ab1bdde 100644
--- a/kernel/rcutree_plugin.h
+++ b/kernel/rcutree_plugin.h
@@ -111,7 +111,7 @@ static void __init rcu_bootup_announce_oddness(void)
#ifdef CONFIG_TREE_PREEMPT_RCU
struct rcu_state rcu_preempt_state =
- RCU_STATE_INITIALIZER(rcu_preempt, call_rcu);
+ RCU_STATE_INITIALIZER(rcu_preempt, 'p', call_rcu);
DEFINE_PER_CPU(struct rcu_data, rcu_preempt_data);
static struct rcu_state *rcu_state = &rcu_preempt_state;
@@ -2510,7 +2510,8 @@ static void __init rcu_spawn_nocb_kthreads(struct rcu_state *rsp)
return;
for_each_cpu(cpu, rcu_nocb_mask) {
rdp = per_cpu_ptr(rsp->rda, cpu);
- t = kthread_run(rcu_nocb_kthread, rdp, "rcuo%d", cpu);
+ t = kthread_run(rcu_nocb_kthread, rdp,
+ "rcuo%d%c", cpu, rsp->abbr);
BUG_ON(IS_ERR(t));
ACCESS_ONCE(rdp->nocb_kthread) = t;
}
--
1.7.8
^ permalink raw reply related [flat|nested] 20+ messages in thread* Re: [PATCH tip/core/rcu 05/14] rcu: Distinguish "rcuo" kthreads by RCU flavor
2013-01-05 17:48 ` [PATCH tip/core/rcu 05/14] rcu: Distinguish "rcuo" kthreads by RCU flavor Paul E. McKenney
@ 2013-01-06 23:34 ` Paul Gortmaker
2013-01-07 20:53 ` Paul E. McKenney
2013-01-07 16:54 ` Josh Triplett
1 sibling, 1 reply; 20+ messages in thread
From: Paul Gortmaker @ 2013-01-06 23:34 UTC (permalink / raw)
To: Paul E. McKenney
Cc: linux-kernel, mingo, laijs, dipankar, akpm, mathieu.desnoyers,
josh, niv, tglx, peterz, rostedt, Valdis.Kletnieks, dhowells,
edumazet, darren, fweisbec, sbw, patches, Paul E. McKenney
On Sat, Jan 5, 2013 at 12:48 PM, Paul E. McKenney
<paulmck@linux.vnet.ibm.com> wrote:
> From: "Paul E. McKenney" <paul.mckenney@linaro.org>
>
> Currently, the per-no-CBs-CPU kthreads are named "rcuo" followed by
> the CPU number, for example, "rcuo". This is problematic given that
> there are either two or three RCU flavors, each of which gets a per-CPU
> kthread with exactly the same name. This commit therefore introduces
> a one-letter abbreviation for each RCU flavor, namely 'b' for RCU-bh,
> 'p' for RCU-preempt, and 's' for RCU-sched. This abbreviation use used
> to distinguish the "rcuo" kthreads, for example, for CPU 0 we would have
> "rcuo0b", "rcuo0p", and "rcuo0s".
Since these names are exposed to Joe Average when he runs ps/top
etc. -- I am inclined to favour the more full names as implemented
in this older patch:
http://goo.gl/H1Aj8
...since "rcuo0p" isn't apt to mean anything to people outside
of the to/cc list of this mail. (Catch me off guard, and I probably
might fail to be able to name the three flavours myself...)
But then again it is just a personal preference. In any case, if we stick
with the short names in your patch, we probably still should make the
similar two documentation type chunks from my patch in yours.
Paul.
--
>
> Signed-off-by: Paul E. McKenney <paul.mckenney@linaro.org>
> Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
> Tested-by: Dietmar Eggemann <dietmar.eggemann@arm.com>
> ---
> kernel/rcutree.c | 7 ++++---
> kernel/rcutree.h | 1 +
> kernel/rcutree_plugin.h | 5 +++--
> 3 files changed, 8 insertions(+), 5 deletions(-)
>
> diff --git a/kernel/rcutree.c b/kernel/rcutree.c
> index 8b110fa..4ec797e 100644
> --- a/kernel/rcutree.c
> +++ b/kernel/rcutree.c
> @@ -64,7 +64,7 @@
> static struct lock_class_key rcu_node_class[RCU_NUM_LVLS];
> static struct lock_class_key rcu_fqs_class[RCU_NUM_LVLS];
>
> -#define RCU_STATE_INITIALIZER(sname, cr) { \
> +#define RCU_STATE_INITIALIZER(sname, sabbr, cr) { \
> .level = { &sname##_state.node[0] }, \
> .call = cr, \
> .fqs_state = RCU_GP_IDLE, \
> @@ -76,13 +76,14 @@ static struct lock_class_key rcu_fqs_class[RCU_NUM_LVLS];
> .barrier_mutex = __MUTEX_INITIALIZER(sname##_state.barrier_mutex), \
> .onoff_mutex = __MUTEX_INITIALIZER(sname##_state.onoff_mutex), \
> .name = #sname, \
> + .abbr = sabbr, \
> }
>
> struct rcu_state rcu_sched_state =
> - RCU_STATE_INITIALIZER(rcu_sched, call_rcu_sched);
> + RCU_STATE_INITIALIZER(rcu_sched, 's', call_rcu_sched);
> DEFINE_PER_CPU(struct rcu_data, rcu_sched_data);
>
> -struct rcu_state rcu_bh_state = RCU_STATE_INITIALIZER(rcu_bh, call_rcu_bh);
> +struct rcu_state rcu_bh_state = RCU_STATE_INITIALIZER(rcu_bh, 'b', call_rcu_bh);
> DEFINE_PER_CPU(struct rcu_data, rcu_bh_data);
>
> static struct rcu_state *rcu_state;
> diff --git a/kernel/rcutree.h b/kernel/rcutree.h
> index ef26eab..c865117 100644
> --- a/kernel/rcutree.h
> +++ b/kernel/rcutree.h
> @@ -452,6 +452,7 @@ struct rcu_state {
> unsigned long gp_max; /* Maximum GP duration in */
> /* jiffies. */
> char *name; /* Name of structure. */
> + char abbr; /* Abbreviated name. */
> struct list_head flavors; /* List of RCU flavors. */
> };
>
> diff --git a/kernel/rcutree_plugin.h b/kernel/rcutree_plugin.h
> index eb9b473..ab1bdde 100644
> --- a/kernel/rcutree_plugin.h
> +++ b/kernel/rcutree_plugin.h
> @@ -111,7 +111,7 @@ static void __init rcu_bootup_announce_oddness(void)
> #ifdef CONFIG_TREE_PREEMPT_RCU
>
> struct rcu_state rcu_preempt_state =
> - RCU_STATE_INITIALIZER(rcu_preempt, call_rcu);
> + RCU_STATE_INITIALIZER(rcu_preempt, 'p', call_rcu);
> DEFINE_PER_CPU(struct rcu_data, rcu_preempt_data);
> static struct rcu_state *rcu_state = &rcu_preempt_state;
>
> @@ -2510,7 +2510,8 @@ static void __init rcu_spawn_nocb_kthreads(struct rcu_state *rsp)
> return;
> for_each_cpu(cpu, rcu_nocb_mask) {
> rdp = per_cpu_ptr(rsp->rda, cpu);
> - t = kthread_run(rcu_nocb_kthread, rdp, "rcuo%d", cpu);
> + t = kthread_run(rcu_nocb_kthread, rdp,
> + "rcuo%d%c", cpu, rsp->abbr);
> BUG_ON(IS_ERR(t));
> ACCESS_ONCE(rdp->nocb_kthread) = t;
> }
> --
> 1.7.8
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at http://www.tux.org/lkml/
^ permalink raw reply [flat|nested] 20+ messages in thread* Re: [PATCH tip/core/rcu 05/14] rcu: Distinguish "rcuo" kthreads by RCU flavor
2013-01-06 23:34 ` Paul Gortmaker
@ 2013-01-07 20:53 ` Paul E. McKenney
0 siblings, 0 replies; 20+ messages in thread
From: Paul E. McKenney @ 2013-01-07 20:53 UTC (permalink / raw)
To: Paul Gortmaker
Cc: linux-kernel, mingo, laijs, dipankar, akpm, mathieu.desnoyers,
josh, niv, tglx, peterz, rostedt, Valdis.Kletnieks, dhowells,
edumazet, darren, fweisbec, sbw, patches, Paul E. McKenney
On Sun, Jan 06, 2013 at 06:34:17PM -0500, Paul Gortmaker wrote:
> On Sat, Jan 5, 2013 at 12:48 PM, Paul E. McKenney
> <paulmck@linux.vnet.ibm.com> wrote:
> > From: "Paul E. McKenney" <paul.mckenney@linaro.org>
> >
> > Currently, the per-no-CBs-CPU kthreads are named "rcuo" followed by
> > the CPU number, for example, "rcuo". This is problematic given that
> > there are either two or three RCU flavors, each of which gets a per-CPU
> > kthread with exactly the same name. This commit therefore introduces
> > a one-letter abbreviation for each RCU flavor, namely 'b' for RCU-bh,
> > 'p' for RCU-preempt, and 's' for RCU-sched. This abbreviation use used
> > to distinguish the "rcuo" kthreads, for example, for CPU 0 we would have
> > "rcuo0b", "rcuo0p", and "rcuo0s".
>
> Since these names are exposed to Joe Average when he runs ps/top
> etc. -- I am inclined to favour the more full names as implemented
> in this older patch:
>
> http://goo.gl/H1Aj8
>
> ...since "rcuo0p" isn't apt to mean anything to people outside
> of the to/cc list of this mail. (Catch me off guard, and I probably
> might fail to be able to name the three flavours myself...)
>
> But then again it is just a personal preference. In any case, if we stick
> with the short names in your patch, we probably still should make the
> similar two documentation type chunks from my patch in yours.
I took yours and Josh's advice here, so the names are now rcuoX/N,
with the definitions of X and N in both the Kconfig entry and in
Documentation/kernel-parameters.txt.
Thanx, Paul
> Paul.
> --
>
> >
> > Signed-off-by: Paul E. McKenney <paul.mckenney@linaro.org>
> > Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
> > Tested-by: Dietmar Eggemann <dietmar.eggemann@arm.com>
> > ---
> > kernel/rcutree.c | 7 ++++---
> > kernel/rcutree.h | 1 +
> > kernel/rcutree_plugin.h | 5 +++--
> > 3 files changed, 8 insertions(+), 5 deletions(-)
> >
> > diff --git a/kernel/rcutree.c b/kernel/rcutree.c
> > index 8b110fa..4ec797e 100644
> > --- a/kernel/rcutree.c
> > +++ b/kernel/rcutree.c
> > @@ -64,7 +64,7 @@
> > static struct lock_class_key rcu_node_class[RCU_NUM_LVLS];
> > static struct lock_class_key rcu_fqs_class[RCU_NUM_LVLS];
> >
> > -#define RCU_STATE_INITIALIZER(sname, cr) { \
> > +#define RCU_STATE_INITIALIZER(sname, sabbr, cr) { \
> > .level = { &sname##_state.node[0] }, \
> > .call = cr, \
> > .fqs_state = RCU_GP_IDLE, \
> > @@ -76,13 +76,14 @@ static struct lock_class_key rcu_fqs_class[RCU_NUM_LVLS];
> > .barrier_mutex = __MUTEX_INITIALIZER(sname##_state.barrier_mutex), \
> > .onoff_mutex = __MUTEX_INITIALIZER(sname##_state.onoff_mutex), \
> > .name = #sname, \
> > + .abbr = sabbr, \
> > }
> >
> > struct rcu_state rcu_sched_state =
> > - RCU_STATE_INITIALIZER(rcu_sched, call_rcu_sched);
> > + RCU_STATE_INITIALIZER(rcu_sched, 's', call_rcu_sched);
> > DEFINE_PER_CPU(struct rcu_data, rcu_sched_data);
> >
> > -struct rcu_state rcu_bh_state = RCU_STATE_INITIALIZER(rcu_bh, call_rcu_bh);
> > +struct rcu_state rcu_bh_state = RCU_STATE_INITIALIZER(rcu_bh, 'b', call_rcu_bh);
> > DEFINE_PER_CPU(struct rcu_data, rcu_bh_data);
> >
> > static struct rcu_state *rcu_state;
> > diff --git a/kernel/rcutree.h b/kernel/rcutree.h
> > index ef26eab..c865117 100644
> > --- a/kernel/rcutree.h
> > +++ b/kernel/rcutree.h
> > @@ -452,6 +452,7 @@ struct rcu_state {
> > unsigned long gp_max; /* Maximum GP duration in */
> > /* jiffies. */
> > char *name; /* Name of structure. */
> > + char abbr; /* Abbreviated name. */
> > struct list_head flavors; /* List of RCU flavors. */
> > };
> >
> > diff --git a/kernel/rcutree_plugin.h b/kernel/rcutree_plugin.h
> > index eb9b473..ab1bdde 100644
> > --- a/kernel/rcutree_plugin.h
> > +++ b/kernel/rcutree_plugin.h
> > @@ -111,7 +111,7 @@ static void __init rcu_bootup_announce_oddness(void)
> > #ifdef CONFIG_TREE_PREEMPT_RCU
> >
> > struct rcu_state rcu_preempt_state =
> > - RCU_STATE_INITIALIZER(rcu_preempt, call_rcu);
> > + RCU_STATE_INITIALIZER(rcu_preempt, 'p', call_rcu);
> > DEFINE_PER_CPU(struct rcu_data, rcu_preempt_data);
> > static struct rcu_state *rcu_state = &rcu_preempt_state;
> >
> > @@ -2510,7 +2510,8 @@ static void __init rcu_spawn_nocb_kthreads(struct rcu_state *rsp)
> > return;
> > for_each_cpu(cpu, rcu_nocb_mask) {
> > rdp = per_cpu_ptr(rsp->rda, cpu);
> > - t = kthread_run(rcu_nocb_kthread, rdp, "rcuo%d", cpu);
> > + t = kthread_run(rcu_nocb_kthread, rdp,
> > + "rcuo%d%c", cpu, rsp->abbr);
> > BUG_ON(IS_ERR(t));
> > ACCESS_ONCE(rdp->nocb_kthread) = t;
> > }
> > --
> > 1.7.8
> >
> > --
> > To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> > the body of a message to majordomo@vger.kernel.org
> > More majordomo info at http://vger.kernel.org/majordomo-info.html
> > Please read the FAQ at http://www.tux.org/lkml/
>
^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [PATCH tip/core/rcu 05/14] rcu: Distinguish "rcuo" kthreads by RCU flavor
2013-01-05 17:48 ` [PATCH tip/core/rcu 05/14] rcu: Distinguish "rcuo" kthreads by RCU flavor Paul E. McKenney
2013-01-06 23:34 ` Paul Gortmaker
@ 2013-01-07 16:54 ` Josh Triplett
1 sibling, 0 replies; 20+ messages in thread
From: Josh Triplett @ 2013-01-07 16:54 UTC (permalink / raw)
To: Paul E. McKenney
Cc: linux-kernel, mingo, laijs, dipankar, akpm, mathieu.desnoyers,
niv, tglx, peterz, rostedt, Valdis.Kletnieks, dhowells, edumazet,
darren, fweisbec, sbw, patches, Paul E. McKenney
On Sat, Jan 05, 2013 at 09:48:55AM -0800, Paul E. McKenney wrote:
> From: "Paul E. McKenney" <paul.mckenney@linaro.org>
>
> Currently, the per-no-CBs-CPU kthreads are named "rcuo" followed by
> the CPU number, for example, "rcuo". This is problematic given that
I think the quoted string on this second line wanted to have a CPU
number in it.
> there are either two or three RCU flavors, each of which gets a per-CPU
> kthread with exactly the same name. This commit therefore introduces
> a one-letter abbreviation for each RCU flavor, namely 'b' for RCU-bh,
> 'p' for RCU-preempt, and 's' for RCU-sched. This abbreviation use used
> to distinguish the "rcuo" kthreads, for example, for CPU 0 we would have
> "rcuo0b", "rcuo0p", and "rcuo0s".
Don't per-CPU kthreads normally have a name ending in /$cpunum ?, rather
than just $cpunum? For instance, kworker/0, kworker/1, and so on?
Seems like these threads should follow that convention.
- Josh Triplett
^ permalink raw reply [flat|nested] 20+ messages in thread
* [PATCH tip/core/rcu 06/14] rcu: Export RCU_FAST_NO_HZ parameters to sysfs
2013-01-05 17:48 ` [PATCH tip/core/rcu 01/14] rcu: Tag callback lists with corresponding grace-period number Paul E. McKenney
` (3 preceding siblings ...)
2013-01-05 17:48 ` [PATCH tip/core/rcu 05/14] rcu: Distinguish "rcuo" kthreads by RCU flavor Paul E. McKenney
@ 2013-01-05 17:48 ` Paul E. McKenney
2013-01-05 17:48 ` [PATCH tip/core/rcu 07/14] rcu: Accelerate RCU callbacks at grace-period end Paul E. McKenney
` (7 subsequent siblings)
12 siblings, 0 replies; 20+ messages in thread
From: Paul E. McKenney @ 2013-01-05 17:48 UTC (permalink / raw)
To: linux-kernel
Cc: mingo, laijs, dipankar, akpm, mathieu.desnoyers, josh, niv, tglx,
peterz, rostedt, Valdis.Kletnieks, dhowells, edumazet, darren,
fweisbec, sbw, patches, Paul E. McKenney, Paul E. McKenney
From: "Paul E. McKenney" <paul.mckenney@linaro.org>
RCU_FAST_NO_HZ operation is controlled by four compile-time C-preprocessor
macros, but some use cases benefit greatly from runtime adjustment,
particularly when tuning devices. This commit therefore creates the
corresponding sysfs entries.
Reported-by: Robin Randhawa <robin.randhawa@arm.com>
Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
---
kernel/rcutree_plugin.h | 31 ++++++++++++++++++++-----------
1 files changed, 20 insertions(+), 11 deletions(-)
diff --git a/kernel/rcutree_plugin.h b/kernel/rcutree_plugin.h
index ab1bdde..0997e9f 100644
--- a/kernel/rcutree_plugin.h
+++ b/kernel/rcutree_plugin.h
@@ -1617,6 +1617,15 @@ static void rcu_idle_count_callbacks_posted(void)
#define RCU_IDLE_GP_DELAY 4 /* Roughly one grace period. */
#define RCU_IDLE_LAZY_GP_DELAY (6 * HZ) /* Roughly six seconds. */
+static int rcu_idle_flushes = RCU_IDLE_FLUSHES;
+module_param(rcu_idle_flushes, int, 0644);
+static int rcu_idle_opt_flushes = RCU_IDLE_OPT_FLUSHES;
+module_param(rcu_idle_opt_flushes, int, 0644);
+static int rcu_idle_gp_delay = RCU_IDLE_GP_DELAY;
+module_param(rcu_idle_gp_delay, int, 0644);
+static int rcu_idle_lazy_gp_delay = RCU_IDLE_LAZY_GP_DELAY;
+module_param(rcu_idle_lazy_gp_delay, int, 0644);
+
extern int tick_nohz_enabled;
/*
@@ -1696,10 +1705,10 @@ int rcu_needs_cpu(int cpu, unsigned long *delta_jiffies)
}
/* Set up for the possibility that RCU will post a timer. */
if (rcu_cpu_has_nonlazy_callbacks(cpu)) {
- *delta_jiffies = round_up(RCU_IDLE_GP_DELAY + jiffies,
- RCU_IDLE_GP_DELAY) - jiffies;
+ *delta_jiffies = round_up(rcu_idle_gp_delay + jiffies,
+ rcu_idle_gp_delay) - jiffies;
} else {
- *delta_jiffies = jiffies + RCU_IDLE_LAZY_GP_DELAY;
+ *delta_jiffies = jiffies + rcu_idle_lazy_gp_delay;
*delta_jiffies = round_jiffies(*delta_jiffies) - jiffies;
}
return 0;
@@ -1805,11 +1814,11 @@ static void rcu_prepare_for_idle(int cpu)
if (rcu_cpu_has_nonlazy_callbacks(cpu)) {
trace_rcu_prep_idle("User dyntick with callbacks");
rdtp->idle_gp_timer_expires =
- round_up(jiffies + RCU_IDLE_GP_DELAY,
- RCU_IDLE_GP_DELAY);
+ round_up(jiffies + rcu_idle_gp_delay,
+ rcu_idle_gp_delay);
} else if (rcu_cpu_has_callbacks(cpu)) {
rdtp->idle_gp_timer_expires =
- round_jiffies(jiffies + RCU_IDLE_LAZY_GP_DELAY);
+ round_jiffies(jiffies + rcu_idle_lazy_gp_delay);
trace_rcu_prep_idle("User dyntick with lazy callbacks");
} else {
return;
@@ -1861,8 +1870,8 @@ static void rcu_prepare_for_idle(int cpu)
/* Check and update the ->dyntick_drain sequencing. */
if (rdtp->dyntick_drain <= 0) {
/* First time through, initialize the counter. */
- rdtp->dyntick_drain = RCU_IDLE_FLUSHES;
- } else if (rdtp->dyntick_drain <= RCU_IDLE_OPT_FLUSHES &&
+ rdtp->dyntick_drain = rcu_idle_flushes;
+ } else if (rdtp->dyntick_drain <= rcu_idle_opt_flushes &&
!rcu_pending(cpu) &&
!local_softirq_pending()) {
/* Can we go dyntick-idle despite still having callbacks? */
@@ -1871,11 +1880,11 @@ static void rcu_prepare_for_idle(int cpu)
if (rcu_cpu_has_nonlazy_callbacks(cpu)) {
trace_rcu_prep_idle("Dyntick with callbacks");
rdtp->idle_gp_timer_expires =
- round_up(jiffies + RCU_IDLE_GP_DELAY,
- RCU_IDLE_GP_DELAY);
+ round_up(jiffies + rcu_idle_gp_delay,
+ rcu_idle_gp_delay);
} else {
rdtp->idle_gp_timer_expires =
- round_jiffies(jiffies + RCU_IDLE_LAZY_GP_DELAY);
+ round_jiffies(jiffies + rcu_idle_lazy_gp_delay);
trace_rcu_prep_idle("Dyntick with lazy callbacks");
}
tp = &rdtp->idle_gp_timer;
--
1.7.8
^ permalink raw reply related [flat|nested] 20+ messages in thread* [PATCH tip/core/rcu 07/14] rcu: Accelerate RCU callbacks at grace-period end
2013-01-05 17:48 ` [PATCH tip/core/rcu 01/14] rcu: Tag callback lists with corresponding grace-period number Paul E. McKenney
` (4 preceding siblings ...)
2013-01-05 17:48 ` [PATCH tip/core/rcu 06/14] rcu: Export RCU_FAST_NO_HZ parameters to sysfs Paul E. McKenney
@ 2013-01-05 17:48 ` Paul E. McKenney
2013-01-05 17:48 ` [PATCH tip/core/rcu 08/14] rcu: Make RCU_FAST_NO_HZ take advantage of numbered callbacks Paul E. McKenney
` (6 subsequent siblings)
12 siblings, 0 replies; 20+ messages in thread
From: Paul E. McKenney @ 2013-01-05 17:48 UTC (permalink / raw)
To: linux-kernel
Cc: mingo, laijs, dipankar, akpm, mathieu.desnoyers, josh, niv, tglx,
peterz, rostedt, Valdis.Kletnieks, dhowells, edumazet, darren,
fweisbec, sbw, patches, Paul E. McKenney, Paul E. McKenney
From: "Paul E. McKenney" <paul.mckenney@linaro.org>
Now that callback acceleration is idempotent, it is safe to accelerate
callbacks during grace-period cleanup on any CPUs that the kthread happens
to be running on. This commit therefore propagates the completion
of the grace period to the per-CPU data structures, and also adds an
rcu_advance_cbs() just before the cpu_needs_another_gp() check in order
to reduce false-positive grace periods.
Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
---
kernel/rcutree.c | 21 +++++++++++++--------
1 files changed, 13 insertions(+), 8 deletions(-)
diff --git a/kernel/rcutree.c b/kernel/rcutree.c
index 4ec797e..392c977 100644
--- a/kernel/rcutree.c
+++ b/kernel/rcutree.c
@@ -1434,6 +1434,9 @@ static void rcu_gp_cleanup(struct rcu_state *rsp)
rcu_for_each_node_breadth_first(rsp, rnp) {
raw_spin_lock_irq(&rnp->lock);
rnp->completed = rsp->gpnum;
+ rdp = this_cpu_ptr(rsp->rda);
+ if (rnp == rdp->mynode)
+ __rcu_process_gp_end(rsp, rnp, rdp);
nocb += rcu_nocb_gp_cleanup(rsp, rnp);
raw_spin_unlock_irq(&rnp->lock);
cond_resched();
@@ -1446,6 +1449,7 @@ static void rcu_gp_cleanup(struct rcu_state *rsp)
trace_rcu_grace_period(rsp->name, rsp->completed, "end");
rsp->fqs_state = RCU_GP_IDLE;
rdp = this_cpu_ptr(rsp->rda);
+ rcu_advance_cbs(rsp, rnp, rdp); /* Reduce false positives below. */
if (cpu_needs_another_gp(rsp, rdp))
rsp->gp_flags = 1;
raw_spin_unlock_irq(&rnp->lock);
@@ -1535,6 +1539,15 @@ rcu_start_gp(struct rcu_state *rsp, unsigned long flags)
struct rcu_data *rdp = this_cpu_ptr(rsp->rda);
struct rcu_node *rnp = rcu_get_root(rsp);
+ /*
+ * If there is no grace period in progress right now, any
+ * callbacks we have up to this point will be satisfied by the
+ * next grace period. Also, advancing the callbacks reduces the
+ * probability of false positives from cpu_needs_another_gp()
+ * resulting in pointless grace periods. So, advance callbacks!
+ */
+ rcu_advance_cbs(rsp, rnp, rdp);
+
if (!rsp->gp_kthread ||
!cpu_needs_another_gp(rsp, rdp)) {
/*
@@ -1547,14 +1560,6 @@ rcu_start_gp(struct rcu_state *rsp, unsigned long flags)
return;
}
- /*
- * Because there is no grace period in progress right now,
- * any callbacks we have up to this point will be satisfied
- * by the next grace period. So this is a good place to
- * assign a grace period number to recently posted callbacks.
- */
- rcu_accelerate_cbs(rsp, rnp, rdp);
-
rsp->gp_flags = RCU_GP_FLAG_INIT;
raw_spin_unlock(&rnp->lock); /* Interrupts remain disabled. */
--
1.7.8
^ permalink raw reply related [flat|nested] 20+ messages in thread* [PATCH tip/core/rcu 08/14] rcu: Make RCU_FAST_NO_HZ take advantage of numbered callbacks
2013-01-05 17:48 ` [PATCH tip/core/rcu 01/14] rcu: Tag callback lists with corresponding grace-period number Paul E. McKenney
` (5 preceding siblings ...)
2013-01-05 17:48 ` [PATCH tip/core/rcu 07/14] rcu: Accelerate RCU callbacks at grace-period end Paul E. McKenney
@ 2013-01-05 17:48 ` Paul E. McKenney
2013-01-05 17:48 ` [PATCH tip/core/rcu 09/14] rcu: Rearrange locking in rcu_start_gp() Paul E. McKenney
` (5 subsequent siblings)
12 siblings, 0 replies; 20+ messages in thread
From: Paul E. McKenney @ 2013-01-05 17:48 UTC (permalink / raw)
To: linux-kernel
Cc: mingo, laijs, dipankar, akpm, mathieu.desnoyers, josh, niv, tglx,
peterz, rostedt, Valdis.Kletnieks, dhowells, edumazet, darren,
fweisbec, sbw, patches, Paul E. McKenney, Paul E. McKenney
From: "Paul E. McKenney" <paul.mckenney@linaro.org>
Because RCU callbacks are now associated with the number of the grace
period that they must wait for, CPUs can now take advance callbacks
corresponding to grace periods that ended while a given CPU was in
dyntick-idle mode. This eliminates the need to try forcing the RCU
state machine while entering idle, thus reducing the CPU intensiveness
of RCU_FAST_NO_HZ, which should increase its energy efficiency.
Signed-off-by: Paul E. McKenney <paul.mckenney@linaro.org>
Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
---
include/linux/rcupdate.h | 1 +
kernel/rcutree.c | 28 +++--
kernel/rcutree.h | 12 +--
kernel/rcutree_plugin.h | 350 +++++++++++++++-------------------------------
kernel/rcutree_trace.c | 2 -
5 files changed, 131 insertions(+), 262 deletions(-)
diff --git a/include/linux/rcupdate.h b/include/linux/rcupdate.h
index 275aa3f..4b37b50 100644
--- a/include/linux/rcupdate.h
+++ b/include/linux/rcupdate.h
@@ -73,6 +73,7 @@ extern void do_trace_rcu_torture_read(char *rcutorturename,
#define UINT_CMP_LT(a, b) (UINT_MAX / 2 < (a) - (b))
#define ULONG_CMP_GE(a, b) (ULONG_MAX / 2 >= (a) - (b))
#define ULONG_CMP_LT(a, b) (ULONG_MAX / 2 < (a) - (b))
+#define ulong2long(a) (*(long *)(&(a)))
/* Exported common interfaces */
diff --git a/kernel/rcutree.c b/kernel/rcutree.c
index 392c977..4b8d91c 100644
--- a/kernel/rcutree.c
+++ b/kernel/rcutree.c
@@ -2678,19 +2678,27 @@ static int rcu_pending(int cpu)
}
/*
- * Check to see if any future RCU-related work will need to be done
- * by the current CPU, even if none need be done immediately, returning
- * 1 if so.
+ * Return true if the specified CPU has any callback. If all_lazy is
+ * non-NULL, store an indication of whether all callbacks are lazy.
+ * (If there are no callbacks, all of them are deemed to be lazy.)
*/
-static int rcu_cpu_has_callbacks(int cpu)
+static int rcu_cpu_has_callbacks(int cpu, bool *all_lazy)
{
+ bool al = true;
+ bool hc = false;
+ struct rcu_data *rdp;
struct rcu_state *rsp;
- /* RCU callbacks either ready or pending? */
- for_each_rcu_flavor(rsp)
- if (per_cpu_ptr(rsp->rda, cpu)->nxtlist)
- return 1;
- return 0;
+ for_each_rcu_flavor(rsp) {
+ rdp = per_cpu_ptr(rsp->rda, cpu);
+ if (rdp->qlen != rdp->qlen_lazy)
+ al = false;
+ if (rdp->nxtlist)
+ hc = true;
+ }
+ if (all_lazy)
+ *all_lazy = al;
+ return hc;
}
/*
@@ -2912,7 +2920,6 @@ rcu_init_percpu_data(int cpu, struct rcu_state *rsp, int preemptible)
rdp->dynticks->dynticks_nesting = DYNTICK_TASK_EXIT_IDLE;
atomic_set(&rdp->dynticks->dynticks,
(atomic_read(&rdp->dynticks->dynticks) & ~0x1) + 1);
- rcu_prepare_for_idle_init(cpu);
raw_spin_unlock(&rnp->lock); /* irqs remain disabled. */
/* Add CPU to rcu_node bitmasks. */
@@ -2986,7 +2993,6 @@ static int __cpuinit rcu_cpu_notify(struct notifier_block *self,
*/
for_each_rcu_flavor(rsp)
rcu_cleanup_dying_cpu(rsp);
- rcu_cleanup_after_idle(cpu);
break;
case CPU_DEAD:
case CPU_DEAD_FROZEN:
diff --git a/kernel/rcutree.h b/kernel/rcutree.h
index c865117..282b1d7 100644
--- a/kernel/rcutree.h
+++ b/kernel/rcutree.h
@@ -88,18 +88,13 @@ struct rcu_dynticks {
int dynticks_nmi_nesting; /* Track NMI nesting level. */
atomic_t dynticks; /* Even value for idle, else odd. */
#ifdef CONFIG_RCU_FAST_NO_HZ
- int dyntick_drain; /* Prepare-for-idle state variable. */
- unsigned long dyntick_holdoff;
- /* No retries for the jiffy of failure. */
- struct timer_list idle_gp_timer;
- /* Wake up CPU sleeping with callbacks. */
- unsigned long idle_gp_timer_expires;
- /* When to wake up CPU (for repost). */
- bool idle_first_pass; /* First pass of attempt to go idle? */
+ bool all_lazy; /* Are all CPU's CBs lazy? */
unsigned long nonlazy_posted;
/* # times non-lazy CBs posted to CPU. */
unsigned long nonlazy_posted_snap;
/* idle-period nonlazy_posted snapshot. */
+ unsigned long last_accelerate;
+ /* Last jiffy CBs were accelerated. */
int tick_nohz_enabled_snap; /* Previously seen value from sysfs. */
#endif /* #ifdef CONFIG_RCU_FAST_NO_HZ */
#ifdef CONFIG_RCU_USER_QS
@@ -530,7 +525,6 @@ static int __cpuinit rcu_spawn_one_boost_kthread(struct rcu_state *rsp,
struct rcu_node *rnp);
#endif /* #ifdef CONFIG_RCU_BOOST */
static void __cpuinit rcu_prepare_kthreads(int cpu);
-static void rcu_prepare_for_idle_init(int cpu);
static void rcu_cleanup_after_idle(int cpu);
static void rcu_prepare_for_idle(int cpu);
static void rcu_idle_count_callbacks_posted(void);
diff --git a/kernel/rcutree_plugin.h b/kernel/rcutree_plugin.h
index 0997e9f..9371bdd 100644
--- a/kernel/rcutree_plugin.h
+++ b/kernel/rcutree_plugin.h
@@ -1543,14 +1543,7 @@ static void __cpuinit rcu_prepare_kthreads(int cpu)
int rcu_needs_cpu(int cpu, unsigned long *delta_jiffies)
{
*delta_jiffies = ULONG_MAX;
- return rcu_cpu_has_callbacks(cpu);
-}
-
-/*
- * Because we do not have RCU_FAST_NO_HZ, don't bother initializing for it.
- */
-static void rcu_prepare_for_idle_init(int cpu)
-{
+ return rcu_cpu_has_callbacks(cpu, NULL);
}
/*
@@ -1587,16 +1580,6 @@ static void rcu_idle_count_callbacks_posted(void)
*
* The following three proprocessor symbols control this state machine:
*
- * RCU_IDLE_FLUSHES gives the maximum number of times that we will attempt
- * to satisfy RCU. Beyond this point, it is better to incur a periodic
- * scheduling-clock interrupt than to loop through the state machine
- * at full power.
- * RCU_IDLE_OPT_FLUSHES gives the number of RCU_IDLE_FLUSHES that are
- * optional if RCU does not need anything immediately from this
- * CPU, even if this CPU still has RCU callbacks queued. The first
- * times through the state machine are mandatory: we need to give
- * the state machine a chance to communicate a quiescent state
- * to the RCU core.
* RCU_IDLE_GP_DELAY gives the number of jiffies that a CPU is permitted
* to sleep in dyntick-idle mode with RCU callbacks pending. This
* is sized to be roughly one RCU grace period. Those energy-efficiency
@@ -1612,15 +1595,9 @@ static void rcu_idle_count_callbacks_posted(void)
* adjustment, they can be converted into kernel config parameters, though
* making the state machine smarter might be a better option.
*/
-#define RCU_IDLE_FLUSHES 5 /* Number of dyntick-idle tries. */
-#define RCU_IDLE_OPT_FLUSHES 3 /* Optional dyntick-idle tries. */
#define RCU_IDLE_GP_DELAY 4 /* Roughly one grace period. */
#define RCU_IDLE_LAZY_GP_DELAY (6 * HZ) /* Roughly six seconds. */
-static int rcu_idle_flushes = RCU_IDLE_FLUSHES;
-module_param(rcu_idle_flushes, int, 0644);
-static int rcu_idle_opt_flushes = RCU_IDLE_OPT_FLUSHES;
-module_param(rcu_idle_opt_flushes, int, 0644);
static int rcu_idle_gp_delay = RCU_IDLE_GP_DELAY;
module_param(rcu_idle_gp_delay, int, 0644);
static int rcu_idle_lazy_gp_delay = RCU_IDLE_LAZY_GP_DELAY;
@@ -1628,6 +1605,8 @@ module_param(rcu_idle_lazy_gp_delay, int, 0644);
extern int tick_nohz_enabled;
+#if 0 /*@@@*/
+
/*
* Does the specified flavor of RCU have non-lazy callbacks pending on
* the specified CPU? Both RCU flavor and CPU are specified by the
@@ -1670,137 +1649,100 @@ static bool rcu_cpu_has_nonlazy_callbacks(int cpu)
rcu_preempt_cpu_has_nonlazy_callbacks(cpu);
}
-/*
- * Allow the CPU to enter dyntick-idle mode if either: (1) There are no
- * callbacks on this CPU, (2) this CPU has not yet attempted to enter
- * dyntick-idle mode, or (3) this CPU is in the process of attempting to
- * enter dyntick-idle mode. Otherwise, if we have recently tried and failed
- * to enter dyntick-idle mode, we refuse to try to enter it. After all,
- * it is better to incur scheduling-clock interrupts than to spin
- * continuously for the same time duration!
- *
- * The delta_jiffies argument is used to store the time when RCU is
- * going to need the CPU again if it still has callbacks. The reason
- * for this is that rcu_prepare_for_idle() might need to post a timer,
- * but if so, it will do so after tick_nohz_stop_sched_tick() has set
- * the wakeup time for this CPU. This means that RCU's timer can be
- * delayed until the wakeup time, which defeats the purpose of posting
- * a timer.
- */
-int rcu_needs_cpu(int cpu, unsigned long *delta_jiffies)
-{
- struct rcu_dynticks *rdtp = &per_cpu(rcu_dynticks, cpu);
-
- /* Flag a new idle sojourn to the idle-entry state machine. */
- rdtp->idle_first_pass = 1;
- /* If no callbacks, RCU doesn't need the CPU. */
- if (!rcu_cpu_has_callbacks(cpu)) {
- *delta_jiffies = ULONG_MAX;
- return 0;
- }
- if (rdtp->dyntick_holdoff == jiffies) {
- /* RCU recently tried and failed, so don't try again. */
- *delta_jiffies = 1;
- return 1;
- }
- /* Set up for the possibility that RCU will post a timer. */
- if (rcu_cpu_has_nonlazy_callbacks(cpu)) {
- *delta_jiffies = round_up(rcu_idle_gp_delay + jiffies,
- rcu_idle_gp_delay) - jiffies;
- } else {
- *delta_jiffies = jiffies + rcu_idle_lazy_gp_delay;
- *delta_jiffies = round_jiffies(*delta_jiffies) - jiffies;
- }
- return 0;
-}
+#endif
/*
- * Handler for smp_call_function_single(). The only point of this
- * handler is to wake the CPU up, so the handler does only tracing.
+ * Try to advance callbacks for all flavors of RCU on the current CPU.
+ * Afterwards, if there are any callbacks ready for immediate invocation,
+ * return true.
*/
-void rcu_idle_demigrate(void *unused)
+static bool rcu_try_advance_all_cbs(void)
{
- trace_rcu_prep_idle("Demigrate");
-}
+ bool cbs_ready = false;
+ struct rcu_data *rdp;
+ struct rcu_node *rnp;
+ struct rcu_state *rsp;
-/*
- * Timer handler used to force CPU to start pushing its remaining RCU
- * callbacks in the case where it entered dyntick-idle mode with callbacks
- * pending. The hander doesn't really need to do anything because the
- * real work is done upon re-entry to idle, or by the next scheduling-clock
- * interrupt should idle not be re-entered.
- *
- * One special case: the timer gets migrated without awakening the CPU
- * on which the timer was scheduled on. In this case, we must wake up
- * that CPU. We do so with smp_call_function_single().
- */
-static void rcu_idle_gp_timer_func(unsigned long cpu_in)
-{
- int cpu = (int)cpu_in;
+ for_each_rcu_flavor(rsp) {
+ rdp = this_cpu_ptr(rsp->rda);
+ rnp = rdp->mynode;
- trace_rcu_prep_idle("Timer");
- if (cpu != smp_processor_id())
- smp_call_function_single(cpu, rcu_idle_demigrate, NULL, 0);
- else
- WARN_ON_ONCE(1); /* Getting here can hang the system... */
+ /*
+ * Don't bother checking unless a grace period has
+ * completed since we last checked and there are
+ * callbacks not yet ready to invoke.
+ */
+ if (rdp->completed != rnp->completed &&
+ rdp->nxttail[RCU_DONE_TAIL] != rdp->nxttail[RCU_NEXT_TAIL])
+ rcu_process_gp_end(rsp, rdp);
+
+ if (cpu_has_callbacks_ready_to_invoke(rdp))
+ cbs_ready = true;
+ }
+ return cbs_ready;
}
/*
- * Initialize the timer used to pull CPUs out of dyntick-idle mode.
+ * Allow the CPU to enter dyntick-idle mode unless it has callbacks ready
+ * to invoke. If the CPU has callbacks, try to advance them. Tell the
+ * caller to set the timeout based on whether or not there are non-lazy
+ * callbacks.
+ *
+ * The caller must have disabled interrupts.
*/
-static void rcu_prepare_for_idle_init(int cpu)
+int rcu_needs_cpu(int cpu, unsigned long *dj)
{
struct rcu_dynticks *rdtp = &per_cpu(rcu_dynticks, cpu);
- rdtp->dyntick_holdoff = jiffies - 1;
- setup_timer(&rdtp->idle_gp_timer, rcu_idle_gp_timer_func, cpu);
- rdtp->idle_gp_timer_expires = jiffies - 1;
- rdtp->idle_first_pass = 1;
-}
+ /* Snapshot to detect later posting of non-lazy callback. */
+ rdtp->nonlazy_posted_snap = rdtp->nonlazy_posted;
-/*
- * Clean up for exit from idle. Because we are exiting from idle, there
- * is no longer any point to ->idle_gp_timer, so cancel it. This will
- * do nothing if this timer is not active, so just cancel it unconditionally.
- */
-static void rcu_cleanup_after_idle(int cpu)
-{
- struct rcu_dynticks *rdtp = &per_cpu(rcu_dynticks, cpu);
+ /* If no callbacks, RCU doesn't need the CPU. */
+ if (!rcu_cpu_has_callbacks(cpu, &rdtp->all_lazy)) {
+ *dj = ULONG_MAX;
+ return 0;
+ }
- del_timer(&rdtp->idle_gp_timer);
- trace_rcu_prep_idle("Cleanup after idle");
- rdtp->tick_nohz_enabled_snap = ACCESS_ONCE(tick_nohz_enabled);
+ /* Attempt to advance callbacks. */
+ if (rcu_try_advance_all_cbs()) {
+ /* Some ready to invoke, so initiate later invocation. */
+ invoke_rcu_core();
+ return 1;
+ }
+ rdtp->last_accelerate = jiffies;
+
+ /* Request timer delay depending on laziness, and round. */
+ if (rdtp->all_lazy) {
+ *dj = round_up(rcu_idle_gp_delay + jiffies,
+ rcu_idle_gp_delay) - jiffies;
+ } else {
+ *dj = round_jiffies(rcu_idle_lazy_gp_delay + jiffies) - jiffies;
+ }
+ return 0;
}
/*
- * Check to see if any RCU-related work can be done by the current CPU,
- * and if so, schedule a softirq to get it done. This function is part
- * of the RCU implementation; it is -not- an exported member of the RCU API.
- *
- * The idea is for the current CPU to clear out all work required by the
- * RCU core for the current grace period, so that this CPU can be permitted
- * to enter dyntick-idle mode. In some cases, it will need to be awakened
- * at the end of the grace period by whatever CPU ends the grace period.
- * This allows CPUs to go dyntick-idle more quickly, and to reduce the
- * number of wakeups by a modest integer factor.
- *
- * Because it is not legal to invoke rcu_process_callbacks() with irqs
- * disabled, we do one pass of force_quiescent_state(), then do a
- * invoke_rcu_core() to cause rcu_process_callbacks() to be invoked
- * later. The ->dyntick_drain field controls the sequencing.
+ * Prepare a CPU for idle from an RCU perspective. The first major task
+ * is to sense whether nohz mode has been enabled or disabled via sysfs.
+ * The second major task is to check to see if a non-lazy callback has
+ * arrived at a CPU that previously had only lazy callbacks. The third
+ * major task is to accelerate (that is, assign grace-period numbers to)
+ * any recently arrived callbacks.
*
* The caller must have disabled interrupts.
*/
static void rcu_prepare_for_idle(int cpu)
{
- struct timer_list *tp;
+ struct rcu_data *rdp;
struct rcu_dynticks *rdtp = &per_cpu(rcu_dynticks, cpu);
+ struct rcu_node *rnp;
+ struct rcu_state *rsp;
int tne;
/* Handle nohz enablement switches conservatively. */
tne = ACCESS_ONCE(tick_nohz_enabled);
if (tne != rdtp->tick_nohz_enabled_snap) {
- if (rcu_cpu_has_callbacks(cpu))
+ if (rcu_cpu_has_callbacks(cpu, NULL))
invoke_rcu_core(); /* force nohz to see update. */
rdtp->tick_nohz_enabled_snap = tne;
return;
@@ -1808,125 +1750,56 @@ static void rcu_prepare_for_idle(int cpu)
if (!tne)
return;
- /* Adaptive-tick mode, where usermode execution is idle to RCU. */
- if (!is_idle_task(current)) {
- rdtp->dyntick_holdoff = jiffies - 1;
- if (rcu_cpu_has_nonlazy_callbacks(cpu)) {
- trace_rcu_prep_idle("User dyntick with callbacks");
- rdtp->idle_gp_timer_expires =
- round_up(jiffies + rcu_idle_gp_delay,
- rcu_idle_gp_delay);
- } else if (rcu_cpu_has_callbacks(cpu)) {
- rdtp->idle_gp_timer_expires =
- round_jiffies(jiffies + rcu_idle_lazy_gp_delay);
- trace_rcu_prep_idle("User dyntick with lazy callbacks");
- } else {
- return;
- }
- tp = &rdtp->idle_gp_timer;
- mod_timer_pinned(tp, rdtp->idle_gp_timer_expires);
+ /* If this is a no-CBs CPU, no callbacks, just return. */
+ if (is_nocb_cpu(cpu))
return;
- }
/*
- * If this is an idle re-entry, for example, due to use of
- * RCU_NONIDLE() or the new idle-loop tracing API within the idle
- * loop, then don't take any state-machine actions, unless the
- * momentary exit from idle queued additional non-lazy callbacks.
- * Instead, repost the ->idle_gp_timer if this CPU has callbacks
- * pending.
+ * If a non-lazy callback arrived at a CPU having only lazy
+ * callbacks, invoke RCU core for the side-effect of recalculating
+ * idle duration on re-entry to idle.
*/
- if (!rdtp->idle_first_pass &&
- (rdtp->nonlazy_posted == rdtp->nonlazy_posted_snap)) {
- if (rcu_cpu_has_callbacks(cpu)) {
- tp = &rdtp->idle_gp_timer;
- mod_timer_pinned(tp, rdtp->idle_gp_timer_expires);
- }
+ if (rdtp->all_lazy &&
+ rdtp->nonlazy_posted != rdtp->nonlazy_posted_snap) {
+ invoke_rcu_core();
return;
}
- rdtp->idle_first_pass = 0;
- rdtp->nonlazy_posted_snap = rdtp->nonlazy_posted - 1;
/*
- * If there are no callbacks on this CPU, enter dyntick-idle mode.
- * Also reset state to avoid prejudicing later attempts.
+ * If we have not yet accelerated this jiffy, accelerate all
+ * callbacks on this CPU.
*/
- if (!rcu_cpu_has_callbacks(cpu)) {
- rdtp->dyntick_holdoff = jiffies - 1;
- rdtp->dyntick_drain = 0;
- trace_rcu_prep_idle("No callbacks");
+ if (rdtp->last_accelerate == jiffies)
return;
+ rdtp->last_accelerate = jiffies;
+ for_each_rcu_flavor(rsp) {
+ rdp = per_cpu_ptr(rsp->rda, cpu);
+ if (!*rdp->nxttail[RCU_DONE_TAIL])
+ continue;
+ rnp = rdp->mynode;
+ raw_spin_lock(&rnp->lock); /* irqs already disabled. */
+ rcu_accelerate_cbs(rsp, rnp, rdp);
+ raw_spin_unlock(&rnp->lock); /* irqs remain disabled. */
}
+}
- /*
- * If in holdoff mode, just return. We will presumably have
- * refrained from disabling the scheduling-clock tick.
- */
- if (rdtp->dyntick_holdoff == jiffies) {
- trace_rcu_prep_idle("In holdoff");
- return;
- }
+/*
+ * Clean up for exit from idle. Attempt to advance callbacks based on
+ * any grace periods that elapsed while the CPU was idle, and if any
+ * callbacks are now ready to invoke, initiate invocation.
+ */
+static void rcu_cleanup_after_idle(int cpu)
+{
+ struct rcu_data *rdp;
+ struct rcu_state *rsp;
- /* Check and update the ->dyntick_drain sequencing. */
- if (rdtp->dyntick_drain <= 0) {
- /* First time through, initialize the counter. */
- rdtp->dyntick_drain = rcu_idle_flushes;
- } else if (rdtp->dyntick_drain <= rcu_idle_opt_flushes &&
- !rcu_pending(cpu) &&
- !local_softirq_pending()) {
- /* Can we go dyntick-idle despite still having callbacks? */
- rdtp->dyntick_drain = 0;
- rdtp->dyntick_holdoff = jiffies;
- if (rcu_cpu_has_nonlazy_callbacks(cpu)) {
- trace_rcu_prep_idle("Dyntick with callbacks");
- rdtp->idle_gp_timer_expires =
- round_up(jiffies + rcu_idle_gp_delay,
- rcu_idle_gp_delay);
- } else {
- rdtp->idle_gp_timer_expires =
- round_jiffies(jiffies + rcu_idle_lazy_gp_delay);
- trace_rcu_prep_idle("Dyntick with lazy callbacks");
- }
- tp = &rdtp->idle_gp_timer;
- mod_timer_pinned(tp, rdtp->idle_gp_timer_expires);
- rdtp->nonlazy_posted_snap = rdtp->nonlazy_posted;
- return; /* Nothing more to do immediately. */
- } else if (--(rdtp->dyntick_drain) <= 0) {
- /* We have hit the limit, so time to give up. */
- rdtp->dyntick_holdoff = jiffies;
- trace_rcu_prep_idle("Begin holdoff");
- invoke_rcu_core(); /* Force the CPU out of dyntick-idle. */
+ if (is_nocb_cpu(cpu))
return;
- }
-
- /*
- * Do one step of pushing the remaining RCU callbacks through
- * the RCU core state machine.
- */
-#ifdef CONFIG_TREE_PREEMPT_RCU
- if (per_cpu(rcu_preempt_data, cpu).nxtlist) {
- rcu_preempt_qs(cpu);
- force_quiescent_state(&rcu_preempt_state);
- }
-#endif /* #ifdef CONFIG_TREE_PREEMPT_RCU */
- if (per_cpu(rcu_sched_data, cpu).nxtlist) {
- rcu_sched_qs(cpu);
- force_quiescent_state(&rcu_sched_state);
- }
- if (per_cpu(rcu_bh_data, cpu).nxtlist) {
- rcu_bh_qs(cpu);
- force_quiescent_state(&rcu_bh_state);
- }
-
- /*
- * If RCU callbacks are still pending, RCU still needs this CPU.
- * So try forcing the callbacks through the grace period.
- */
- if (rcu_cpu_has_callbacks(cpu)) {
- trace_rcu_prep_idle("More callbacks");
- invoke_rcu_core();
- } else {
- trace_rcu_prep_idle("Callbacks drained");
+ rcu_try_advance_all_cbs();
+ for_each_rcu_flavor(rsp) {
+ rdp = per_cpu_ptr(rsp->rda, cpu);
+ if (cpu_has_callbacks_ready_to_invoke(rdp))
+ invoke_rcu_core();
}
}
@@ -2034,16 +1907,13 @@ early_initcall(rcu_register_oom_notifier);
static void print_cpu_stall_fast_no_hz(char *cp, int cpu)
{
struct rcu_dynticks *rdtp = &per_cpu(rcu_dynticks, cpu);
- struct timer_list *tltp = &rdtp->idle_gp_timer;
- char c;
+ unsigned long nlpd = rdtp->nonlazy_posted - rdtp->nonlazy_posted_snap;
- c = rdtp->dyntick_holdoff == jiffies ? 'H' : '.';
- if (timer_pending(tltp))
- sprintf(cp, "drain=%d %c timer=%lu",
- rdtp->dyntick_drain, c, tltp->expires - jiffies);
- else
- sprintf(cp, "drain=%d %c timer not pending",
- rdtp->dyntick_drain, c);
+ sprintf(cp, "last_accelerate: %04lx/%04lx, nonlazy_posted: %ld, %c%c",
+ rdtp->last_accelerate & 0xffff, jiffies & 0xffff,
+ ulong2long(nlpd),
+ rdtp->all_lazy ? 'L' : '.',
+ rdtp->tick_nohz_enabled_snap ? '.' : 'D');
}
#else /* #ifdef CONFIG_RCU_FAST_NO_HZ */
diff --git a/kernel/rcutree_trace.c b/kernel/rcutree_trace.c
index 0d095dc..49099e8 100644
--- a/kernel/rcutree_trace.c
+++ b/kernel/rcutree_trace.c
@@ -46,8 +46,6 @@
#define RCU_TREE_NONCORE
#include "rcutree.h"
-#define ulong2long(a) (*(long *)(&(a)))
-
static int r_open(struct inode *inode, struct file *file,
const struct seq_operations *op)
{
--
1.7.8
^ permalink raw reply related [flat|nested] 20+ messages in thread* [PATCH tip/core/rcu 09/14] rcu: Rearrange locking in rcu_start_gp()
2013-01-05 17:48 ` [PATCH tip/core/rcu 01/14] rcu: Tag callback lists with corresponding grace-period number Paul E. McKenney
` (6 preceding siblings ...)
2013-01-05 17:48 ` [PATCH tip/core/rcu 08/14] rcu: Make RCU_FAST_NO_HZ take advantage of numbered callbacks Paul E. McKenney
@ 2013-01-05 17:48 ` Paul E. McKenney
2013-01-05 17:49 ` [PATCH tip/core/rcu 10/14] rcu: Repurpose no-CBs event tracing to future-GP events Paul E. McKenney
` (4 subsequent siblings)
12 siblings, 0 replies; 20+ messages in thread
From: Paul E. McKenney @ 2013-01-05 17:48 UTC (permalink / raw)
To: linux-kernel
Cc: mingo, laijs, dipankar, akpm, mathieu.desnoyers, josh, niv, tglx,
peterz, rostedt, Valdis.Kletnieks, dhowells, edumazet, darren,
fweisbec, sbw, patches, Paul E. McKenney, Paul E. McKenney
From: "Paul E. McKenney" <paul.mckenney@linaro.org>
If CPUs are to give prior notice of needed grace periods, it will be
necessary to invoke rcu_start_gp() without dropping the root rcu_node
structure's ->lock. This commit takes a first step in this direction
by moving the release of this lock to the end of rcu_start_gp().
Signed-off-by: Paul E. McKenney <paul.mckenney@linaro.org>
Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
---
kernel/rcutree.c | 6 ++----
1 files changed, 2 insertions(+), 4 deletions(-)
diff --git a/kernel/rcutree.c b/kernel/rcutree.c
index 4b8d91c..7207435 100644
--- a/kernel/rcutree.c
+++ b/kernel/rcutree.c
@@ -1559,16 +1559,14 @@ rcu_start_gp(struct rcu_state *rsp, unsigned long flags)
raw_spin_unlock_irqrestore(&rnp->lock, flags);
return;
}
-
rsp->gp_flags = RCU_GP_FLAG_INIT;
- raw_spin_unlock(&rnp->lock); /* Interrupts remain disabled. */
/* Ensure that CPU is aware of completion of last grace period. */
- rcu_process_gp_end(rsp, rdp);
- local_irq_restore(flags);
+ __rcu_process_gp_end(rsp, rdp->mynode, rdp);
/* Wake up rcu_gp_kthread() to start the grace period. */
wake_up(&rsp->gp_wq);
+ raw_spin_unlock_irqrestore(&rnp->lock, flags);
}
/*
--
1.7.8
^ permalink raw reply related [flat|nested] 20+ messages in thread* [PATCH tip/core/rcu 10/14] rcu: Repurpose no-CBs event tracing to future-GP events
2013-01-05 17:48 ` [PATCH tip/core/rcu 01/14] rcu: Tag callback lists with corresponding grace-period number Paul E. McKenney
` (7 preceding siblings ...)
2013-01-05 17:48 ` [PATCH tip/core/rcu 09/14] rcu: Rearrange locking in rcu_start_gp() Paul E. McKenney
@ 2013-01-05 17:49 ` Paul E. McKenney
2013-01-05 17:49 ` [PATCH tip/core/rcu 11/14] rcu: Push lock release to rcu_start_gp()'s callers Paul E. McKenney
` (3 subsequent siblings)
12 siblings, 0 replies; 20+ messages in thread
From: Paul E. McKenney @ 2013-01-05 17:49 UTC (permalink / raw)
To: linux-kernel
Cc: mingo, laijs, dipankar, akpm, mathieu.desnoyers, josh, niv, tglx,
peterz, rostedt, Valdis.Kletnieks, dhowells, edumazet, darren,
fweisbec, sbw, patches, Paul E. McKenney, Paul E. McKenney
From: "Paul E. McKenney" <paul.mckenney@linaro.org>
Dyntick-idle CPUs need to be able to pre-announce their need for grace
periods. This can be done using something similar to the mechanism used
by no-CB CPUs to announce their need for grace periods. This commit
moves in this direction by renaming the no-CBs grace-period event tracing
to suit the new future-grace-period needs.
Signed-off-by: Paul E. McKenney <paul.mckenney@linaro.org>
Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
---
include/trace/events/rcu.h | 16 +++++-----
kernel/rcutree_plugin.h | 62 ++++++++++++++++++++++---------------------
2 files changed, 40 insertions(+), 38 deletions(-)
diff --git a/include/trace/events/rcu.h b/include/trace/events/rcu.h
index ef0bf31..0dc0177 100644
--- a/include/trace/events/rcu.h
+++ b/include/trace/events/rcu.h
@@ -72,10 +72,10 @@ TRACE_EVENT(rcu_grace_period,
);
/*
- * Tracepoint for no-callbacks grace-period events. The caller should
- * pull the data from the rcu_node structure, other than rcuname, which
- * comes from the rcu_state structure, and event, which is one of the
- * following:
+ * Tracepoint for future grace-period events, including those for no-callbacks
+ * CPUs. The caller should pull the data from the rcu_node structure,
+ * other than rcuname, which comes from the rcu_state structure, and event,
+ * which is one of the following:
*
* "Startleaf": Request a nocb grace period based on leaf-node data.
* "Startedleaf": Leaf-node start proved sufficient.
@@ -87,7 +87,7 @@ TRACE_EVENT(rcu_grace_period,
* "Cleanup": Clean up rcu_node structure after previous GP.
* "CleanupMore": Clean up, and another no-CB GP is needed.
*/
-TRACE_EVENT(rcu_nocb_grace_period,
+TRACE_EVENT(rcu_future_grace_period,
TP_PROTO(char *rcuname, unsigned long gpnum, unsigned long completed,
unsigned long c, u8 level, int grplo, int grphi,
@@ -645,9 +645,9 @@ TRACE_EVENT(rcu_barrier,
#define trace_rcu_grace_period(rcuname, gpnum, gpevent) do { } while (0)
#define trace_rcu_grace_period_init(rcuname, gpnum, level, grplo, grphi, \
qsmask) do { } while (0)
-#define trace_rcu_nocb_grace_period(rcuname, gpnum, completed, c, \
- level, grplo, grphi, event) \
- do { } while (0)
+#define trace_rcu_future_grace_period(rcuname, gpnum, completed, c, \
+ level, grplo, grphi, event) \
+ do { } while (0)
#define trace_rcu_preempt_task(rcuname, pid, gpnum) do { } while (0)
#define trace_rcu_unlock_preempted_task(rcuname, gpnum, pid) do { } while (0)
#define trace_rcu_quiescent_state_report(rcuname, gpnum, mask, qsmask, level, \
diff --git a/kernel/rcutree_plugin.h b/kernel/rcutree_plugin.h
index 9371bdd..d09acdf 100644
--- a/kernel/rcutree_plugin.h
+++ b/kernel/rcutree_plugin.h
@@ -2073,9 +2073,9 @@ static int rcu_nocb_gp_cleanup(struct rcu_state *rsp, struct rcu_node *rnp)
wake_up_all(&rnp->nocb_gp_wq[c & 0x1]);
rnp->n_nocb_gp_requests[c & 0x1] = 0;
needmore = rnp->n_nocb_gp_requests[(c + 1) & 0x1];
- trace_rcu_nocb_grace_period(rsp->name, rnp->gpnum, rnp->completed,
- c, rnp->level, rnp->grplo, rnp->grphi,
- needmore ? "CleanupMore" : "Cleanup");
+ trace_rcu_future_grace_period(rsp->name, rnp->gpnum, rnp->completed,
+ c, rnp->level, rnp->grplo, rnp->grphi,
+ needmore ? "CleanupMore" : "Cleanup");
return needmore;
}
@@ -2222,9 +2222,9 @@ static void rcu_nocb_wait_gp(struct rcu_data *rdp)
/* Count our request for a grace period. */
rnp->n_nocb_gp_requests[c & 0x1]++;
- trace_rcu_nocb_grace_period(rdp->rsp->name, rnp->gpnum, rnp->completed,
- c, rnp->level, rnp->grplo, rnp->grphi,
- "Startleaf");
+ trace_rcu_future_grace_period(rdp->rsp->name, rnp->gpnum,
+ rnp->completed, c, rnp->level,
+ rnp->grplo, rnp->grphi, "Startleaf");
if (rnp->gpnum != rnp->completed) {
@@ -2233,10 +2233,10 @@ static void rcu_nocb_wait_gp(struct rcu_data *rdp)
* is in progress, so we are done. When this grace
* period ends, our request will be acted upon.
*/
- trace_rcu_nocb_grace_period(rdp->rsp->name,
- rnp->gpnum, rnp->completed, c,
- rnp->level, rnp->grplo, rnp->grphi,
- "Startedleaf");
+ trace_rcu_future_grace_period(rdp->rsp->name, rnp->gpnum,
+ rnp->completed, c, rnp->level,
+ rnp->grplo, rnp->grphi,
+ "Startedleaf");
raw_spin_unlock_irqrestore(&rnp->lock, flags);
} else {
@@ -2248,11 +2248,12 @@ static void rcu_nocb_wait_gp(struct rcu_data *rdp)
if (rnp != rnp_root)
raw_spin_lock(&rnp_root->lock); /* irqs disabled. */
if (rnp_root->gpnum != rnp_root->completed) {
- trace_rcu_nocb_grace_period(rdp->rsp->name,
- rnp->gpnum, rnp->completed,
- c, rnp->level,
- rnp->grplo, rnp->grphi,
- "Startedleafroot");
+ trace_rcu_future_grace_period(rdp->rsp->name,
+ rnp->gpnum,
+ rnp->completed,
+ c, rnp->level,
+ rnp->grplo, rnp->grphi,
+ "Startedleafroot");
raw_spin_unlock(&rnp_root->lock); /* irqs disabled. */
} else {
@@ -2268,11 +2269,12 @@ static void rcu_nocb_wait_gp(struct rcu_data *rdp)
c = rnp_root->completed + 1;
rnp->n_nocb_gp_requests[c & 0x1]++;
rnp_root->n_nocb_gp_requests[c & 0x1]++;
- trace_rcu_nocb_grace_period(rdp->rsp->name,
- rnp->gpnum, rnp->completed,
- c, rnp->level,
- rnp->grplo, rnp->grphi,
- "Startedroot");
+ trace_rcu_future_grace_period(rdp->rsp->name,
+ rnp->gpnum,
+ rnp->completed,
+ c, rnp->level,
+ rnp->grplo, rnp->grphi,
+ "Startedroot");
local_save_flags(flags1);
rcu_start_gp(rdp->rsp, flags1); /* Rlses ->lock. */
}
@@ -2288,9 +2290,9 @@ static void rcu_nocb_wait_gp(struct rcu_data *rdp)
* Wait for the grace period. Do so interruptibly to avoid messing
* up the load average.
*/
- trace_rcu_nocb_grace_period(rdp->rsp->name, rnp->gpnum, rnp->completed,
- c, rnp->level, rnp->grplo, rnp->grphi,
- "StartWait");
+ trace_rcu_future_grace_period(rdp->rsp->name, rnp->gpnum,
+ rnp->completed, c, rnp->level,
+ rnp->grplo, rnp->grphi, "StartWait");
for (;;) {
wait_event_interruptible(
rnp->nocb_gp_wq[c & 0x1],
@@ -2298,14 +2300,14 @@ static void rcu_nocb_wait_gp(struct rcu_data *rdp)
if (likely(d))
break;
flush_signals(current);
- trace_rcu_nocb_grace_period(rdp->rsp->name,
- rnp->gpnum, rnp->completed, c,
- rnp->level, rnp->grplo, rnp->grphi,
- "ResumeWait");
+ trace_rcu_future_grace_period(rdp->rsp->name,
+ rnp->gpnum, rnp->completed, c,
+ rnp->level, rnp->grplo,
+ rnp->grphi, "ResumeWait");
}
- trace_rcu_nocb_grace_period(rdp->rsp->name, rnp->gpnum, rnp->completed,
- c, rnp->level, rnp->grplo, rnp->grphi,
- "EndWait");
+ trace_rcu_future_grace_period(rdp->rsp->name, rnp->gpnum,
+ rnp->completed, c, rnp->level,
+ rnp->grplo, rnp->grphi, "EndWait");
smp_mb(); /* Ensure that CB invocation happens after GP end. */
}
--
1.7.8
^ permalink raw reply related [flat|nested] 20+ messages in thread* [PATCH tip/core/rcu 11/14] rcu: Push lock release to rcu_start_gp()'s callers
2013-01-05 17:48 ` [PATCH tip/core/rcu 01/14] rcu: Tag callback lists with corresponding grace-period number Paul E. McKenney
` (8 preceding siblings ...)
2013-01-05 17:49 ` [PATCH tip/core/rcu 10/14] rcu: Repurpose no-CBs event tracing to future-GP events Paul E. McKenney
@ 2013-01-05 17:49 ` Paul E. McKenney
2013-01-05 17:49 ` [PATCH tip/core/rcu 12/14] rcu: Rename n_nocb_gp_requests to need_future_gp Paul E. McKenney
` (2 subsequent siblings)
12 siblings, 0 replies; 20+ messages in thread
From: Paul E. McKenney @ 2013-01-05 17:49 UTC (permalink / raw)
To: linux-kernel
Cc: mingo, laijs, dipankar, akpm, mathieu.desnoyers, josh, niv, tglx,
peterz, rostedt, Valdis.Kletnieks, dhowells, edumazet, darren,
fweisbec, sbw, patches, Paul E. McKenney, Paul E. McKenney
From: "Paul E. McKenney" <paul.mckenney@linaro.org>
If CPUs are to give prior notice of needed grace periods, it will be
necessary to invoke rcu_start_gp() without dropping the root rcu_node
structure's ->lock. This commit takes a second step in this direction
by moving the release of this lock to rcu_start_gp()'s callers.
Signed-off-by: Paul E. McKenney <paul.mckenney@linaro.org>
Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
---
kernel/rcutree.c | 24 ++++++++++--------------
kernel/rcutree_plugin.h | 5 ++---
2 files changed, 12 insertions(+), 17 deletions(-)
diff --git a/kernel/rcutree.c b/kernel/rcutree.c
index 7207435..8ca18ec 100644
--- a/kernel/rcutree.c
+++ b/kernel/rcutree.c
@@ -1525,16 +1525,14 @@ static int __noreturn rcu_gp_kthread(void *arg)
/*
* Start a new RCU grace period if warranted, re-initializing the hierarchy
* in preparation for detecting the next grace period. The caller must hold
- * the root node's ->lock, which is released before return. Hard irqs must
- * be disabled.
+ * the root node's ->lock and hard irqs must be disabled.
*
* Note that it is legal for a dying CPU (which is marked as offline) to
* invoke this function. This can happen when the dying CPU reports its
* quiescent state.
*/
static void
-rcu_start_gp(struct rcu_state *rsp, unsigned long flags)
- __releases(rcu_get_root(rsp)->lock)
+rcu_start_gp(struct rcu_state *rsp)
{
struct rcu_data *rdp = this_cpu_ptr(rsp->rda);
struct rcu_node *rnp = rcu_get_root(rsp);
@@ -1548,15 +1546,13 @@ rcu_start_gp(struct rcu_state *rsp, unsigned long flags)
*/
rcu_advance_cbs(rsp, rnp, rdp);
- if (!rsp->gp_kthread ||
- !cpu_needs_another_gp(rsp, rdp)) {
+ if (!rsp->gp_kthread || !cpu_needs_another_gp(rsp, rdp)) {
/*
* Either we have not yet spawned the grace-period
* task, this CPU does not need another grace period,
* or a grace period is already in progress.
* Either way, don't start a new grace period.
*/
- raw_spin_unlock_irqrestore(&rnp->lock, flags);
return;
}
rsp->gp_flags = RCU_GP_FLAG_INIT;
@@ -1566,15 +1562,14 @@ rcu_start_gp(struct rcu_state *rsp, unsigned long flags)
/* Wake up rcu_gp_kthread() to start the grace period. */
wake_up(&rsp->gp_wq);
- raw_spin_unlock_irqrestore(&rnp->lock, flags);
}
/*
* Report a full set of quiescent states to the specified rcu_state
* data structure. This involves cleaning up after the prior grace
* period and letting rcu_start_gp() start up the next grace period
- * if one is needed. Note that the caller must hold rnp->lock, as
- * required by rcu_start_gp(), which will release it.
+ * if one is needed. Note that the caller must hold rnp->lock, which
+ * is released before return.
*/
static void rcu_report_qs_rsp(struct rcu_state *rsp, unsigned long flags)
__releases(rcu_get_root(rsp)->lock)
@@ -2172,7 +2167,8 @@ __rcu_process_callbacks(struct rcu_state *rsp)
local_irq_save(flags);
if (cpu_needs_another_gp(rsp, rdp)) {
raw_spin_lock(&rcu_get_root(rsp)->lock); /* irqs disabled. */
- rcu_start_gp(rsp, flags); /* releases above lock */
+ rcu_start_gp(rsp);
+ raw_spin_unlock_irqrestore(&rcu_get_root(rsp)->lock, flags);
} else {
local_irq_restore(flags);
}
@@ -2252,11 +2248,11 @@ static void __call_rcu_core(struct rcu_state *rsp, struct rcu_data *rdp,
/* Start a new grace period if one not already started. */
if (!rcu_gp_in_progress(rsp)) {
- unsigned long nestflag;
struct rcu_node *rnp_root = rcu_get_root(rsp);
- raw_spin_lock_irqsave(&rnp_root->lock, nestflag);
- rcu_start_gp(rsp, nestflag); /* rlses rnp_root->lock */
+ raw_spin_lock(&rnp_root->lock);
+ rcu_start_gp(rsp);
+ raw_spin_lock(&rnp_root->lock);
} else {
/* Give the grace period a kick. */
rdp->blimit = LONG_MAX;
diff --git a/kernel/rcutree_plugin.h b/kernel/rcutree_plugin.h
index d09acdf..736dd2c 100644
--- a/kernel/rcutree_plugin.h
+++ b/kernel/rcutree_plugin.h
@@ -2213,7 +2213,6 @@ static void rcu_nocb_wait_gp(struct rcu_data *rdp)
unsigned long c;
bool d;
unsigned long flags;
- unsigned long flags1;
struct rcu_node *rnp = rdp->mynode;
struct rcu_node *rnp_root = rcu_get_root(rdp->rsp);
@@ -2275,8 +2274,8 @@ static void rcu_nocb_wait_gp(struct rcu_data *rdp)
c, rnp->level,
rnp->grplo, rnp->grphi,
"Startedroot");
- local_save_flags(flags1);
- rcu_start_gp(rdp->rsp, flags1); /* Rlses ->lock. */
+ rcu_start_gp(rdp->rsp);
+ raw_spin_unlock(&rnp->lock);
}
/* Clean up locking and irq state. */
--
1.7.8
^ permalink raw reply related [flat|nested] 20+ messages in thread* [PATCH tip/core/rcu 12/14] rcu: Rename n_nocb_gp_requests to need_future_gp
2013-01-05 17:48 ` [PATCH tip/core/rcu 01/14] rcu: Tag callback lists with corresponding grace-period number Paul E. McKenney
` (9 preceding siblings ...)
2013-01-05 17:49 ` [PATCH tip/core/rcu 11/14] rcu: Push lock release to rcu_start_gp()'s callers Paul E. McKenney
@ 2013-01-05 17:49 ` Paul E. McKenney
2013-01-05 17:49 ` [PATCH tip/core/rcu 13/14] rcu: Abstract rcu_start_future_gp() from rcu_nocb_wait_gp() Paul E. McKenney
2013-01-05 17:49 ` [PATCH tip/core/rcu 14/14] rcu: Make rcu_accelerate_cbs() note need for future grace periods Paul E. McKenney
12 siblings, 0 replies; 20+ messages in thread
From: Paul E. McKenney @ 2013-01-05 17:49 UTC (permalink / raw)
To: linux-kernel
Cc: mingo, laijs, dipankar, akpm, mathieu.desnoyers, josh, niv, tglx,
peterz, rostedt, Valdis.Kletnieks, dhowells, edumazet, darren,
fweisbec, sbw, patches, Paul E. McKenney, Paul E. McKenney
From: "Paul E. McKenney" <paul.mckenney@linaro.org>
CPUs going idle need to be able to indicate their need for future grace
periods. A mechanism for doing this already exists for no-callbacks
CPUs, so the idea is to re-use that mechanism. This commit therefore
moves the ->n_nocb_gp_requests field of the rcu_node structure out from
under the CONFIG_RCU_NOCB_CPU #ifdef and renames it to ->need_future_gp.
Signed-off-by: Paul E. McKenney <paul.mckenney@linaro.org>
Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
---
kernel/rcutree.h | 4 ++--
kernel/rcutree_plugin.h | 18 +++++++++---------
2 files changed, 11 insertions(+), 11 deletions(-)
diff --git a/kernel/rcutree.h b/kernel/rcutree.h
index 282b1d7..775d96c 100644
--- a/kernel/rcutree.h
+++ b/kernel/rcutree.h
@@ -198,9 +198,9 @@ struct rcu_node {
#ifdef CONFIG_RCU_NOCB_CPU
wait_queue_head_t nocb_gp_wq[2];
/* Place for rcu_nocb_kthread() to wait GP. */
- int n_nocb_gp_requests[2];
- /* Counts of upcoming no-CB GP requests. */
#endif /* #ifdef CONFIG_RCU_NOCB_CPU */
+ int need_future_gp[2];
+ /* Counts of upcoming no-CB GP requests. */
raw_spinlock_t fqslock ____cacheline_internodealigned_in_smp;
} ____cacheline_internodealigned_in_smp;
diff --git a/kernel/rcutree_plugin.h b/kernel/rcutree_plugin.h
index 736dd2c..e4037bd 100644
--- a/kernel/rcutree_plugin.h
+++ b/kernel/rcutree_plugin.h
@@ -2057,7 +2057,7 @@ static int rcu_nocb_needs_gp(struct rcu_state *rsp)
{
struct rcu_node *rnp = rcu_get_root(rsp);
- return rnp->n_nocb_gp_requests[(ACCESS_ONCE(rnp->completed) + 1) & 0x1];
+ return rnp->need_future_gp[(ACCESS_ONCE(rnp->completed) + 1) & 0x1];
}
/*
@@ -2071,8 +2071,8 @@ static int rcu_nocb_gp_cleanup(struct rcu_state *rsp, struct rcu_node *rnp)
int needmore;
wake_up_all(&rnp->nocb_gp_wq[c & 0x1]);
- rnp->n_nocb_gp_requests[c & 0x1] = 0;
- needmore = rnp->n_nocb_gp_requests[(c + 1) & 0x1];
+ rnp->need_future_gp[c & 0x1] = 0;
+ needmore = rnp->need_future_gp[(c + 1) & 0x1];
trace_rcu_future_grace_period(rsp->name, rnp->gpnum, rnp->completed,
c, rnp->level, rnp->grplo, rnp->grphi,
needmore ? "CleanupMore" : "Cleanup");
@@ -2080,7 +2080,7 @@ static int rcu_nocb_gp_cleanup(struct rcu_state *rsp, struct rcu_node *rnp)
}
/*
- * Set the root rcu_node structure's ->n_nocb_gp_requests field
+ * Set the root rcu_node structure's ->need_future_gp field
* based on the sum of those of all rcu_node structures. This does
* double-count the root rcu_node structure's requests, but this
* is necessary to handle the possibility of a rcu_nocb_kthread()
@@ -2089,7 +2089,7 @@ static int rcu_nocb_gp_cleanup(struct rcu_state *rsp, struct rcu_node *rnp)
*/
static void rcu_nocb_gp_set(struct rcu_node *rnp, int nrq)
{
- rnp->n_nocb_gp_requests[(rnp->completed + 1) & 0x1] += nrq;
+ rnp->need_future_gp[(rnp->completed + 1) & 0x1] += nrq;
}
static void rcu_init_one_nocb(struct rcu_node *rnp)
@@ -2220,7 +2220,7 @@ static void rcu_nocb_wait_gp(struct rcu_data *rdp)
c = rnp->completed + 2;
/* Count our request for a grace period. */
- rnp->n_nocb_gp_requests[c & 0x1]++;
+ rnp->need_future_gp[c & 0x1]++;
trace_rcu_future_grace_period(rdp->rsp->name, rnp->gpnum,
rnp->completed, c, rnp->level,
rnp->grplo, rnp->grphi, "Startleaf");
@@ -2264,10 +2264,10 @@ static void rcu_nocb_wait_gp(struct rcu_data *rdp)
* Adjust counters accordingly and start the
* needed grace period.
*/
- rnp->n_nocb_gp_requests[c & 0x1]--;
+ rnp->need_future_gp[c & 0x1]--;
c = rnp_root->completed + 1;
- rnp->n_nocb_gp_requests[c & 0x1]++;
- rnp_root->n_nocb_gp_requests[c & 0x1]++;
+ rnp->need_future_gp[c & 0x1]++;
+ rnp_root->need_future_gp[c & 0x1]++;
trace_rcu_future_grace_period(rdp->rsp->name,
rnp->gpnum,
rnp->completed,
--
1.7.8
^ permalink raw reply related [flat|nested] 20+ messages in thread* [PATCH tip/core/rcu 13/14] rcu: Abstract rcu_start_future_gp() from rcu_nocb_wait_gp()
2013-01-05 17:48 ` [PATCH tip/core/rcu 01/14] rcu: Tag callback lists with corresponding grace-period number Paul E. McKenney
` (10 preceding siblings ...)
2013-01-05 17:49 ` [PATCH tip/core/rcu 12/14] rcu: Rename n_nocb_gp_requests to need_future_gp Paul E. McKenney
@ 2013-01-05 17:49 ` Paul E. McKenney
2013-01-05 17:49 ` [PATCH tip/core/rcu 14/14] rcu: Make rcu_accelerate_cbs() note need for future grace periods Paul E. McKenney
12 siblings, 0 replies; 20+ messages in thread
From: Paul E. McKenney @ 2013-01-05 17:49 UTC (permalink / raw)
To: linux-kernel
Cc: mingo, laijs, dipankar, akpm, mathieu.desnoyers, josh, niv, tglx,
peterz, rostedt, Valdis.Kletnieks, dhowells, edumazet, darren,
fweisbec, sbw, patches, Paul E. McKenney, Paul E. McKenney
From: "Paul E. McKenney" <paul.mckenney@linaro.org>
CPUs going idle will need to record the need for a future grace
period, but won't actually need to block waiting on it. This commit
therefore splits rcu_start_future_gp(), which does the recording, from
rcu_nocb_wait_gp(), which now invokes rcu_start_future_gp() to do the
recording, after which rcu_nocb_wait_gp() does the waiting.
Signed-off-by: Paul E. McKenney <paul.mckenney@linaro.org>
Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
---
kernel/rcutree.c | 123 +++++++++++++++++++++++++++++++++++++++++++++--
kernel/rcutree.h | 2 +-
kernel/rcutree_plugin.h | 104 ++++------------------------------------
3 files changed, 130 insertions(+), 99 deletions(-)
diff --git a/kernel/rcutree.c b/kernel/rcutree.c
index 8ca18ec..bd42feb 100644
--- a/kernel/rcutree.c
+++ b/kernel/rcutree.c
@@ -230,6 +230,7 @@ static ulong jiffies_till_next_fqs = RCU_JIFFIES_TILL_FORCE_QS;
module_param(jiffies_till_first_fqs, ulong, 0644);
module_param(jiffies_till_next_fqs, ulong, 0644);
+static void rcu_start_gp(struct rcu_state *rsp);
static void force_qs_rnp(struct rcu_state *rsp, int (*f)(struct rcu_data *));
static void force_quiescent_state(struct rcu_state *rsp);
static int rcu_pending(int cpu);
@@ -1113,6 +1114,120 @@ static unsigned long rcu_cbs_completed(struct rcu_state *rsp,
}
/*
+ * Trace-event helper function for rcu_start_future_gp() and
+ * rcu_nocb_wait_gp().
+ */
+static void trace_rcu_future_gp(struct rcu_node *rnp, struct rcu_data *rdp,
+ unsigned long c, char *s)
+{
+ trace_rcu_future_grace_period(rdp->rsp->name, rnp->gpnum,
+ rnp->completed, c, rnp->level,
+ rnp->grplo, rnp->grphi, s);
+}
+
+/*
+ * Start some future grace period, as needed to handle newly arrived
+ * callbacks. The required future grace periods are recorded in each
+ * rcu_node structure's ->need_future_gp field.
+ *
+ * The caller must hold the specified rcu_node structure's ->lock.
+ */
+static unsigned long __maybe_unused
+rcu_start_future_gp(struct rcu_node *rnp, struct rcu_data *rdp)
+{
+ unsigned long c;
+ int i;
+ struct rcu_node *rnp_root = rcu_get_root(rdp->rsp);
+
+ /*
+ * Pick up grace-period number for new callbacks. If this
+ * grace period is already marked as needed, return to the caller.
+ */
+ c = rcu_cbs_completed(rdp->rsp, rnp);
+ trace_rcu_future_gp(rnp, rdp, c, "Startleaf");
+ if (rnp->need_future_gp[c & 0x1]) {
+ trace_rcu_future_gp(rnp, rdp, c, "Prestartleaf");
+ return c;
+ }
+
+ /*
+ * If either this rcu_node structure or the root rcu_node structure
+ * believe that a grace period is in progress, then we must wait
+ * for the one following, which is in "c". Because our request
+ * will be noticed at the end of the current grace period, we don't
+ * need to explicitly start one.
+ */
+ if (rnp->gpnum != rnp->completed ||
+ ACCESS_ONCE(rnp->gpnum) != ACCESS_ONCE(rnp->completed)) {
+ rnp->need_future_gp[c & 0x1]++;
+ trace_rcu_future_gp(rnp, rdp, c, "Startedleaf");
+ return c;
+ }
+
+ /*
+ * There might be no grace period in progress. If we don't already
+ * hold it, acquire the root rcu_node structure's lock in order to
+ * start one (if needed).
+ */
+ if (rnp != rnp_root)
+ raw_spin_lock(&rnp_root->lock);
+
+ /*
+ * Get a new grace-period number. If there really is no grace
+ * period in progress, it will be smaller than the one we obtained
+ * earlier. Adjust callbacks as needed. Note that even no-CBs
+ * CPUs have a ->nxtcompleted[] array, so no no-CBs checks needed.
+ */
+ c = rcu_cbs_completed(rdp->rsp, rnp_root);
+ for (i = RCU_DONE_TAIL; i < RCU_NEXT_TAIL; i++)
+ if (ULONG_CMP_LT(c, rdp->nxtcompleted[i]))
+ rdp->nxtcompleted[i] = c;
+
+ /*
+ * If the needed for the required grace period is already
+ * recorded, trace and leave.
+ */
+ if (rnp_root->need_future_gp[c & 0x1]) {
+ trace_rcu_future_gp(rnp, rdp, c, "Prestartedroot");
+ goto unlock_out;
+ }
+
+ /* Record the need for the future grace period. */
+ rnp_root->need_future_gp[c & 0x1]++;
+
+ /* If a grace period is not already in progress, start one. */
+ if (rnp_root->gpnum != rnp_root->completed) {
+ trace_rcu_future_gp(rnp, rdp, c, "Startedleafroot");
+ } else {
+ trace_rcu_future_gp(rnp, rdp, c, "Startedroot");
+ rcu_start_gp(rdp->rsp);
+ }
+unlock_out:
+ if (rnp != rnp_root)
+ raw_spin_unlock(&rnp_root->lock);
+ return c;
+}
+
+/*
+ * Clean up any old requests for the just-ended grace period. Also return
+ * whether any additional grace periods have been requested. Also invoke
+ * rcu_nocb_gp_cleanup() in order to wake up any no-callbacks kthreads
+ * waiting for this grace period to complete.
+ */
+static int rcu_future_gp_cleanup(struct rcu_state *rsp, struct rcu_node *rnp)
+{
+ int c = rnp->completed;
+ int needmore;
+ struct rcu_data *rdp = this_cpu_ptr(rsp->rda);
+
+ rcu_nocb_gp_cleanup(rsp, rnp);
+ rnp->need_future_gp[c & 0x1] = 0;
+ needmore = rnp->need_future_gp[(c + 1) & 0x1];
+ trace_rcu_future_gp(rnp, rdp, c, needmore ? "CleanupMore" : "Cleanup");
+ return needmore;
+}
+
+/*
* If there is room, assign a ->completed number to any callbacks on
* this CPU that have not already been assigned. Also accelerate any
* callbacks that were previously assigned a ->completed number that has
@@ -1350,9 +1465,9 @@ static int rcu_gp_init(struct rcu_state *rsp)
rdp = this_cpu_ptr(rsp->rda);
rcu_preempt_check_blocked_tasks(rnp);
rnp->qsmask = rnp->qsmaskinit;
- rnp->gpnum = rsp->gpnum;
+ ACCESS_ONCE(rnp->gpnum) = rsp->gpnum;
WARN_ON_ONCE(rnp->completed != rsp->completed);
- rnp->completed = rsp->completed;
+ ACCESS_ONCE(rnp->completed) = rsp->completed;
if (rnp == rdp->mynode)
rcu_start_gp_per_cpu(rsp, rnp, rdp);
rcu_preempt_boost_start_gp(rnp);
@@ -1433,11 +1548,11 @@ static void rcu_gp_cleanup(struct rcu_state *rsp)
*/
rcu_for_each_node_breadth_first(rsp, rnp) {
raw_spin_lock_irq(&rnp->lock);
- rnp->completed = rsp->gpnum;
+ ACCESS_ONCE(rnp->completed) = rsp->gpnum;
rdp = this_cpu_ptr(rsp->rda);
if (rnp == rdp->mynode)
__rcu_process_gp_end(rsp, rnp, rdp);
- nocb += rcu_nocb_gp_cleanup(rsp, rnp);
+ nocb += rcu_future_gp_cleanup(rsp, rnp);
raw_spin_unlock_irq(&rnp->lock);
cond_resched();
}
diff --git a/kernel/rcutree.h b/kernel/rcutree.h
index 775d96c..d6a043e 100644
--- a/kernel/rcutree.h
+++ b/kernel/rcutree.h
@@ -535,7 +535,7 @@ static void zero_cpu_stall_ticks(struct rcu_data *rdp);
static void increment_cpu_stall_ticks(void);
static int rcu_nocb_needs_gp(struct rcu_state *rsp);
static void rcu_nocb_gp_set(struct rcu_node *rnp, int nrq);
-static int rcu_nocb_gp_cleanup(struct rcu_state *rsp, struct rcu_node *rnp);
+static void rcu_nocb_gp_cleanup(struct rcu_state *rsp, struct rcu_node *rnp);
static void rcu_init_one_nocb(struct rcu_node *rnp);
static bool is_nocb_cpu(int cpu);
static bool __call_rcu_nocb(struct rcu_data *rdp, struct rcu_head *rhp,
diff --git a/kernel/rcutree_plugin.h b/kernel/rcutree_plugin.h
index e4037bd..71bafa8 100644
--- a/kernel/rcutree_plugin.h
+++ b/kernel/rcutree_plugin.h
@@ -2061,22 +2061,12 @@ static int rcu_nocb_needs_gp(struct rcu_state *rsp)
}
/*
- * Clean up this rcu_node structure's no-CBs state at the end of
- * a grace period, and also return whether any no-CBs CPU associated
- * with this rcu_node structure needs another grace period.
+ * Wake up any no-CBs CPUs' kthreads that were waiting on the just-ended
+ * grace period.
*/
-static int rcu_nocb_gp_cleanup(struct rcu_state *rsp, struct rcu_node *rnp)
+static void rcu_nocb_gp_cleanup(struct rcu_state *rsp, struct rcu_node *rnp)
{
- int c = rnp->completed;
- int needmore;
-
- wake_up_all(&rnp->nocb_gp_wq[c & 0x1]);
- rnp->need_future_gp[c & 0x1] = 0;
- needmore = rnp->need_future_gp[(c + 1) & 0x1];
- trace_rcu_future_grace_period(rsp->name, rnp->gpnum, rnp->completed,
- c, rnp->level, rnp->grplo, rnp->grphi,
- needmore ? "CleanupMore" : "Cleanup");
- return needmore;
+ wake_up_all(&rnp->nocb_gp_wq[rnp->completed & 0x1]);
}
/*
@@ -2214,84 +2204,16 @@ static void rcu_nocb_wait_gp(struct rcu_data *rdp)
bool d;
unsigned long flags;
struct rcu_node *rnp = rdp->mynode;
- struct rcu_node *rnp_root = rcu_get_root(rdp->rsp);
raw_spin_lock_irqsave(&rnp->lock, flags);
- c = rnp->completed + 2;
-
- /* Count our request for a grace period. */
- rnp->need_future_gp[c & 0x1]++;
- trace_rcu_future_grace_period(rdp->rsp->name, rnp->gpnum,
- rnp->completed, c, rnp->level,
- rnp->grplo, rnp->grphi, "Startleaf");
-
- if (rnp->gpnum != rnp->completed) {
-
- /*
- * This rcu_node structure believes that a grace period
- * is in progress, so we are done. When this grace
- * period ends, our request will be acted upon.
- */
- trace_rcu_future_grace_period(rdp->rsp->name, rnp->gpnum,
- rnp->completed, c, rnp->level,
- rnp->grplo, rnp->grphi,
- "Startedleaf");
- raw_spin_unlock_irqrestore(&rnp->lock, flags);
-
- } else {
-
- /*
- * Might not be a grace period, check root rcu_node
- * structure to see if we must start one.
- */
- if (rnp != rnp_root)
- raw_spin_lock(&rnp_root->lock); /* irqs disabled. */
- if (rnp_root->gpnum != rnp_root->completed) {
- trace_rcu_future_grace_period(rdp->rsp->name,
- rnp->gpnum,
- rnp->completed,
- c, rnp->level,
- rnp->grplo, rnp->grphi,
- "Startedleafroot");
- raw_spin_unlock(&rnp_root->lock); /* irqs disabled. */
- } else {
-
- /*
- * No grace period, so we need to start one.
- * The good news is that we can wait for exactly
- * one grace period instead of part of the current
- * grace period and all of the next grace period.
- * Adjust counters accordingly and start the
- * needed grace period.
- */
- rnp->need_future_gp[c & 0x1]--;
- c = rnp_root->completed + 1;
- rnp->need_future_gp[c & 0x1]++;
- rnp_root->need_future_gp[c & 0x1]++;
- trace_rcu_future_grace_period(rdp->rsp->name,
- rnp->gpnum,
- rnp->completed,
- c, rnp->level,
- rnp->grplo, rnp->grphi,
- "Startedroot");
- rcu_start_gp(rdp->rsp);
- raw_spin_unlock(&rnp->lock);
- }
-
- /* Clean up locking and irq state. */
- if (rnp != rnp_root)
- raw_spin_unlock_irqrestore(&rnp->lock, flags);
- else
- local_irq_restore(flags);
- }
+ c = rcu_start_future_gp(rnp, rdp);
+ raw_spin_unlock_irqrestore(&rnp->lock, flags);
/*
* Wait for the grace period. Do so interruptibly to avoid messing
* up the load average.
*/
- trace_rcu_future_grace_period(rdp->rsp->name, rnp->gpnum,
- rnp->completed, c, rnp->level,
- rnp->grplo, rnp->grphi, "StartWait");
+ trace_rcu_future_gp(rnp, rdp, c, "StartWait");
for (;;) {
wait_event_interruptible(
rnp->nocb_gp_wq[c & 0x1],
@@ -2299,14 +2221,9 @@ static void rcu_nocb_wait_gp(struct rcu_data *rdp)
if (likely(d))
break;
flush_signals(current);
- trace_rcu_future_grace_period(rdp->rsp->name,
- rnp->gpnum, rnp->completed, c,
- rnp->level, rnp->grplo,
- rnp->grphi, "ResumeWait");
+ trace_rcu_future_gp(rnp, rdp, c, "ResumeWait");
}
- trace_rcu_future_grace_period(rdp->rsp->name, rnp->gpnum,
- rnp->completed, c, rnp->level,
- rnp->grplo, rnp->grphi, "EndWait");
+ trace_rcu_future_gp(rnp, rdp, c, "EndWait");
smp_mb(); /* Ensure that CB invocation happens after GP end. */
}
@@ -2413,9 +2330,8 @@ static int rcu_nocb_needs_gp(struct rcu_state *rsp)
return 0;
}
-static int rcu_nocb_gp_cleanup(struct rcu_state *rsp, struct rcu_node *rnp)
+static void rcu_nocb_gp_cleanup(struct rcu_state *rsp, struct rcu_node *rnp)
{
- return 0;
}
static void rcu_nocb_gp_set(struct rcu_node *rnp, int nrq)
--
1.7.8
^ permalink raw reply related [flat|nested] 20+ messages in thread* [PATCH tip/core/rcu 14/14] rcu: Make rcu_accelerate_cbs() note need for future grace periods
2013-01-05 17:48 ` [PATCH tip/core/rcu 01/14] rcu: Tag callback lists with corresponding grace-period number Paul E. McKenney
` (11 preceding siblings ...)
2013-01-05 17:49 ` [PATCH tip/core/rcu 13/14] rcu: Abstract rcu_start_future_gp() from rcu_nocb_wait_gp() Paul E. McKenney
@ 2013-01-05 17:49 ` Paul E. McKenney
12 siblings, 0 replies; 20+ messages in thread
From: Paul E. McKenney @ 2013-01-05 17:49 UTC (permalink / raw)
To: linux-kernel
Cc: mingo, laijs, dipankar, akpm, mathieu.desnoyers, josh, niv, tglx,
peterz, rostedt, Valdis.Kletnieks, dhowells, edumazet, darren,
fweisbec, sbw, patches, Paul E. McKenney, Paul E. McKenney
From: "Paul E. McKenney" <paul.mckenney@linaro.org>
Now that rcu_start_future_gp() has been abstracted from
rcu_nocb_wait_gp(), rcu_accelerate_cbs() can invoke rcu_start_future_gp()
so as to register the need for any future grace periods needed by a
CPU about to enter dyntick-idle mode. This commit makes this change.
Note that some refactoring of rcu_start_gp() is carried out to avoid
recursion and subsequent self-deadlocks.
Signed-off-by: Paul E. McKenney <paul.mckenney@linaro.org>
Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
---
kernel/rcutree.c | 50 ++++++++++++++++++++++++++++++++------------------
1 files changed, 32 insertions(+), 18 deletions(-)
diff --git a/kernel/rcutree.c b/kernel/rcutree.c
index bd42feb..f7399b4 100644
--- a/kernel/rcutree.c
+++ b/kernel/rcutree.c
@@ -230,7 +230,8 @@ static ulong jiffies_till_next_fqs = RCU_JIFFIES_TILL_FORCE_QS;
module_param(jiffies_till_first_fqs, ulong, 0644);
module_param(jiffies_till_next_fqs, ulong, 0644);
-static void rcu_start_gp(struct rcu_state *rsp);
+static void rcu_start_gp_advanced(struct rcu_state *rsp, struct rcu_node *rnp,
+ struct rcu_data *rdp);
static void force_qs_rnp(struct rcu_state *rsp, int (*f)(struct rcu_data *));
static void force_quiescent_state(struct rcu_state *rsp);
static int rcu_pending(int cpu);
@@ -1200,7 +1201,7 @@ rcu_start_future_gp(struct rcu_node *rnp, struct rcu_data *rdp)
trace_rcu_future_gp(rnp, rdp, c, "Startedleafroot");
} else {
trace_rcu_future_gp(rnp, rdp, c, "Startedroot");
- rcu_start_gp(rdp->rsp);
+ rcu_start_gp_advanced(rdp->rsp, rnp_root, rdp);
}
unlock_out:
if (rnp != rnp_root)
@@ -1286,6 +1287,8 @@ static void rcu_accelerate_cbs(struct rcu_state *rsp, struct rcu_node *rnp,
rdp->nxttail[i] = rdp->nxttail[RCU_NEXT_TAIL];
rdp->nxtcompleted[i] = c;
}
+ /* Record any needed additional grace periods. */
+ rcu_start_future_gp(rnp, rdp);
/* Trace depending on how much we were able to accelerate. */
if (!*rdp->nxttail[RCU_WAIT_TAIL])
@@ -1647,20 +1650,9 @@ static int __noreturn rcu_gp_kthread(void *arg)
* quiescent state.
*/
static void
-rcu_start_gp(struct rcu_state *rsp)
+rcu_start_gp_advanced(struct rcu_state *rsp, struct rcu_node *rnp,
+ struct rcu_data *rdp)
{
- struct rcu_data *rdp = this_cpu_ptr(rsp->rda);
- struct rcu_node *rnp = rcu_get_root(rsp);
-
- /*
- * If there is no grace period in progress right now, any
- * callbacks we have up to this point will be satisfied by the
- * next grace period. Also, advancing the callbacks reduces the
- * probability of false positives from cpu_needs_another_gp()
- * resulting in pointless grace periods. So, advance callbacks!
- */
- rcu_advance_cbs(rsp, rnp, rdp);
-
if (!rsp->gp_kthread || !cpu_needs_another_gp(rsp, rdp)) {
/*
* Either we have not yet spawned the grace-period
@@ -1672,14 +1664,36 @@ rcu_start_gp(struct rcu_state *rsp)
}
rsp->gp_flags = RCU_GP_FLAG_INIT;
- /* Ensure that CPU is aware of completion of last grace period. */
- __rcu_process_gp_end(rsp, rdp->mynode, rdp);
-
/* Wake up rcu_gp_kthread() to start the grace period. */
wake_up(&rsp->gp_wq);
}
/*
+ * Similar to rcu_start_gp_advanced(), but also advance the calling CPU's
+ * callbacks. Note that rcu_start_gp_advanced() cannot do this because it
+ * is invoked indirectly from rcu_advance_cbs(), which would result in
+ * endless recursion -- or would do so if it wasn't for the self-deadlock
+ * that is encountered beforehand.
+ */
+static void
+rcu_start_gp(struct rcu_state *rsp)
+{
+ struct rcu_data *rdp = this_cpu_ptr(rsp->rda);
+ struct rcu_node *rnp = rcu_get_root(rsp);
+
+ /*
+ * If there is no grace period in progress right now, any
+ * callbacks we have up to this point will be satisfied by the
+ * next grace period. Also, advancing the callbacks reduces the
+ * probability of false positives from cpu_needs_another_gp()
+ * resulting in pointless grace periods. So, advance callbacks
+ * then start the grace period!
+ */
+ rcu_advance_cbs(rsp, rnp, rdp);
+ rcu_start_gp_advanced(rsp, rnp, rdp);
+}
+
+/*
* Report a full set of quiescent states to the specified rcu_state
* data structure. This involves cleaning up after the prior grace
* period and letting rcu_start_gp() start up the next grace period
--
1.7.8
^ permalink raw reply related [flat|nested] 20+ messages in thread