* [PATCH tip/core/rcu 02/10] rcu: Use DEFINE_PER_CPU_SHARED_ALIGNED for rcu_data
2014-10-28 21:53 ` [PATCH tip/core/rcu 01/10] rcu: Remove rcu_dynticks * parameters when they are always this_cpu_ptr(&rcu_dynticks) Paul E. McKenney
@ 2014-10-28 21:53 ` Paul E. McKenney
2014-10-28 21:53 ` [PATCH tip/core/rcu 03/10] rcu: Remove "cpu" argument to rcu_check_callbacks() Paul E. McKenney
` (7 subsequent siblings)
8 siblings, 0 replies; 13+ messages in thread
From: Paul E. McKenney @ 2014-10-28 21:53 UTC (permalink / raw)
To: linux-kernel
Cc: mingo, laijs, dipankar, akpm, mathieu.desnoyers, josh, tglx,
peterz, rostedt, dhowells, edumazet, dvhart, fweisbec, oleg,
bobby.prani, Paul E. McKenney
From: "Paul E. McKenney" <paulmck@linux.vnet.ibm.com>
The rcu_data per-CPU variable has a number of fields that are atomically
manipulated, potentially by any CPU. This situation can result in false
sharing with per-CPU variables that have the misfortune of being allocated
adjacent to rcu_data in memory. This commit therefore changes the
DEFINE_PER_CPU() to DEFINE_PER_CPU_SHARED_ALIGNED() in order to avoid
this false sharing.
Reported-by: Christoph Lameter <cl@linux.com>
Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
Reviewed-by: Christoph Lameter <cl@linux.com>
---
kernel/rcu/tree.c | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c
index c0673c56fb1a..ab6fcfb4fe11 100644
--- a/kernel/rcu/tree.c
+++ b/kernel/rcu/tree.c
@@ -105,7 +105,7 @@ struct rcu_state sname##_state = { \
.name = RCU_STATE_NAME(sname), \
.abbr = sabbr, \
}; \
-DEFINE_PER_CPU(struct rcu_data, sname##_data)
+DEFINE_PER_CPU_SHARED_ALIGNED(struct rcu_data, sname##_data)
RCU_STATE_INITIALIZER(rcu_sched, 's', call_rcu_sched);
RCU_STATE_INITIALIZER(rcu_bh, 'b', call_rcu_bh);
--
1.8.1.5
^ permalink raw reply related [flat|nested] 13+ messages in thread* [PATCH tip/core/rcu 03/10] rcu: Remove "cpu" argument to rcu_check_callbacks()
2014-10-28 21:53 ` [PATCH tip/core/rcu 01/10] rcu: Remove rcu_dynticks * parameters when they are always this_cpu_ptr(&rcu_dynticks) Paul E. McKenney
2014-10-28 21:53 ` [PATCH tip/core/rcu 02/10] rcu: Use DEFINE_PER_CPU_SHARED_ALIGNED for rcu_data Paul E. McKenney
@ 2014-10-28 21:53 ` Paul E. McKenney
2014-10-28 21:53 ` [PATCH tip/core/rcu 04/10] rcu: Remove "cpu" argument to rcu_pending() Paul E. McKenney
` (6 subsequent siblings)
8 siblings, 0 replies; 13+ messages in thread
From: Paul E. McKenney @ 2014-10-28 21:53 UTC (permalink / raw)
To: linux-kernel
Cc: mingo, laijs, dipankar, akpm, mathieu.desnoyers, josh, tglx,
peterz, rostedt, dhowells, edumazet, dvhart, fweisbec, oleg,
bobby.prani, Paul E. McKenney
From: "Paul E. McKenney" <paulmck@linux.vnet.ibm.com>
The "cpu" argument was kept around on the off-chance that RCU might
offload scheduler-clock interrupts. However, this offload approach
has been replaced by NO_HZ_FULL, which offloads -all- RCU processing
from qualifying CPUs. It is therefore time to remove the "cpu" argument
to rcu_check_callbacks(), which this commit does.
Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
---
include/linux/rcupdate.h | 2 +-
kernel/rcu/tiny.c | 2 +-
kernel/rcu/tree.c | 6 +++---
kernel/time/timer.c | 3 +--
4 files changed, 6 insertions(+), 7 deletions(-)
diff --git a/include/linux/rcupdate.h b/include/linux/rcupdate.h
index a4a819ffb2d1..b4ad275d90d0 100644
--- a/include/linux/rcupdate.h
+++ b/include/linux/rcupdate.h
@@ -260,7 +260,7 @@ static inline int rcu_preempt_depth(void)
void rcu_init(void);
void rcu_sched_qs(void);
void rcu_bh_qs(void);
-void rcu_check_callbacks(int cpu, int user);
+void rcu_check_callbacks(int user);
struct notifier_block;
void rcu_idle_enter(void);
void rcu_idle_exit(void);
diff --git a/kernel/rcu/tiny.c b/kernel/rcu/tiny.c
index c0623fc47125..01570c68d237 100644
--- a/kernel/rcu/tiny.c
+++ b/kernel/rcu/tiny.c
@@ -247,7 +247,7 @@ void rcu_bh_qs(void)
* be called from hardirq context. It is normally called from the
* scheduling-clock interrupt.
*/
-void rcu_check_callbacks(int cpu, int user)
+void rcu_check_callbacks(int user)
{
RCU_TRACE(check_cpu_stalls());
if (user || rcu_is_cpu_rrupt_from_idle())
diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c
index ab6fcfb4fe11..3107811bba69 100644
--- a/kernel/rcu/tree.c
+++ b/kernel/rcu/tree.c
@@ -2388,7 +2388,7 @@ static void rcu_do_batch(struct rcu_state *rsp, struct rcu_data *rdp)
* invoked from the scheduling-clock interrupt. If rcu_pending returns
* false, there is no point in invoking rcu_check_callbacks().
*/
-void rcu_check_callbacks(int cpu, int user)
+void rcu_check_callbacks(int user)
{
trace_rcu_utilization(TPS("Start scheduler-tick"));
increment_cpu_stall_ticks();
@@ -2420,8 +2420,8 @@ void rcu_check_callbacks(int cpu, int user)
rcu_bh_qs();
}
- rcu_preempt_check_callbacks(cpu);
- if (rcu_pending(cpu))
+ rcu_preempt_check_callbacks(smp_processor_id());
+ if (rcu_pending(smp_processor_id()))
invoke_rcu_core();
if (user)
rcu_note_voluntary_context_switch(current);
diff --git a/kernel/time/timer.c b/kernel/time/timer.c
index 3260ffdb368f..2d3f5c504939 100644
--- a/kernel/time/timer.c
+++ b/kernel/time/timer.c
@@ -1377,12 +1377,11 @@ unsigned long get_next_timer_interrupt(unsigned long now)
void update_process_times(int user_tick)
{
struct task_struct *p = current;
- int cpu = smp_processor_id();
/* Note: this timer irq context must be accounted for as well. */
account_process_tick(p, user_tick);
run_local_timers();
- rcu_check_callbacks(cpu, user_tick);
+ rcu_check_callbacks(user_tick);
#ifdef CONFIG_IRQ_WORK
if (in_irq())
irq_work_tick();
--
1.8.1.5
^ permalink raw reply related [flat|nested] 13+ messages in thread* [PATCH tip/core/rcu 04/10] rcu: Remove "cpu" argument to rcu_pending()
2014-10-28 21:53 ` [PATCH tip/core/rcu 01/10] rcu: Remove rcu_dynticks * parameters when they are always this_cpu_ptr(&rcu_dynticks) Paul E. McKenney
2014-10-28 21:53 ` [PATCH tip/core/rcu 02/10] rcu: Use DEFINE_PER_CPU_SHARED_ALIGNED for rcu_data Paul E. McKenney
2014-10-28 21:53 ` [PATCH tip/core/rcu 03/10] rcu: Remove "cpu" argument to rcu_check_callbacks() Paul E. McKenney
@ 2014-10-28 21:53 ` Paul E. McKenney
2014-10-28 21:53 ` [PATCH tip/core/rcu 05/10] rcu: Remove "cpu" argument to rcu_preempt_check_callbacks() Paul E. McKenney
` (5 subsequent siblings)
8 siblings, 0 replies; 13+ messages in thread
From: Paul E. McKenney @ 2014-10-28 21:53 UTC (permalink / raw)
To: linux-kernel
Cc: mingo, laijs, dipankar, akpm, mathieu.desnoyers, josh, tglx,
peterz, rostedt, dhowells, edumazet, dvhart, fweisbec, oleg,
bobby.prani, Paul E. McKenney
From: "Paul E. McKenney" <paulmck@linux.vnet.ibm.com>
Because rcu_pending()'s argument is guaranteed to always be the current
CPU, drop the argument and replace per_cpu_ptr() with this_cpu_ptr().
Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
---
kernel/rcu/tree.c | 8 ++++----
1 file changed, 4 insertions(+), 4 deletions(-)
diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c
index 3107811bba69..0670ba3a65d3 100644
--- a/kernel/rcu/tree.c
+++ b/kernel/rcu/tree.c
@@ -325,7 +325,7 @@ static void force_qs_rnp(struct rcu_state *rsp,
unsigned long *maxj),
bool *isidle, unsigned long *maxj);
static void force_quiescent_state(struct rcu_state *rsp);
-static int rcu_pending(int cpu);
+static int rcu_pending(void);
/*
* Return the number of RCU-sched batches processed thus far for debug & stats.
@@ -2421,7 +2421,7 @@ void rcu_check_callbacks(int user)
rcu_bh_qs();
}
rcu_preempt_check_callbacks(smp_processor_id());
- if (rcu_pending(smp_processor_id()))
+ if (rcu_pending())
invoke_rcu_core();
if (user)
rcu_note_voluntary_context_switch(current);
@@ -3144,12 +3144,12 @@ static int __rcu_pending(struct rcu_state *rsp, struct rcu_data *rdp)
* by the current CPU, returning 1 if so. This function is part of the
* RCU implementation; it is -not- an exported member of the RCU API.
*/
-static int rcu_pending(int cpu)
+static int rcu_pending(void)
{
struct rcu_state *rsp;
for_each_rcu_flavor(rsp)
- if (__rcu_pending(rsp, per_cpu_ptr(rsp->rda, cpu)))
+ if (__rcu_pending(rsp, this_cpu_ptr(rsp->rda)))
return 1;
return 0;
}
--
1.8.1.5
^ permalink raw reply related [flat|nested] 13+ messages in thread* [PATCH tip/core/rcu 05/10] rcu: Remove "cpu" argument to rcu_preempt_check_callbacks()
2014-10-28 21:53 ` [PATCH tip/core/rcu 01/10] rcu: Remove rcu_dynticks * parameters when they are always this_cpu_ptr(&rcu_dynticks) Paul E. McKenney
` (2 preceding siblings ...)
2014-10-28 21:53 ` [PATCH tip/core/rcu 04/10] rcu: Remove "cpu" argument to rcu_pending() Paul E. McKenney
@ 2014-10-28 21:53 ` Paul E. McKenney
2014-10-28 21:53 ` [PATCH tip/core/rcu 06/10] rcu: Remove "cpu" argument to rcu_note_context_switch() Paul E. McKenney
` (4 subsequent siblings)
8 siblings, 0 replies; 13+ messages in thread
From: Paul E. McKenney @ 2014-10-28 21:53 UTC (permalink / raw)
To: linux-kernel
Cc: mingo, laijs, dipankar, akpm, mathieu.desnoyers, josh, tglx,
peterz, rostedt, dhowells, edumazet, dvhart, fweisbec, oleg,
bobby.prani, Paul E. McKenney
From: "Paul E. McKenney" <paulmck@linux.vnet.ibm.com>
Because rcu_preempt_check_callbacks()'s argument is guaranteed to
always be the current CPU, drop the argument and replace per_cpu()
with __this_cpu_read().
Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
---
kernel/rcu/tree.c | 2 +-
kernel/rcu/tree.h | 2 +-
kernel/rcu/tree_plugin.h | 8 ++++----
3 files changed, 6 insertions(+), 6 deletions(-)
diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c
index 0670ba3a65d3..1af5e2cdcbeb 100644
--- a/kernel/rcu/tree.c
+++ b/kernel/rcu/tree.c
@@ -2420,7 +2420,7 @@ void rcu_check_callbacks(int user)
rcu_bh_qs();
}
- rcu_preempt_check_callbacks(smp_processor_id());
+ rcu_preempt_check_callbacks();
if (rcu_pending())
invoke_rcu_core();
if (user)
diff --git a/kernel/rcu/tree.h b/kernel/rcu/tree.h
index 8e90562010ec..94a26e330c1b 100644
--- a/kernel/rcu/tree.h
+++ b/kernel/rcu/tree.h
@@ -561,7 +561,7 @@ static int rcu_preempt_offline_tasks(struct rcu_state *rsp,
struct rcu_node *rnp,
struct rcu_data *rdp);
#endif /* #ifdef CONFIG_HOTPLUG_CPU */
-static void rcu_preempt_check_callbacks(int cpu);
+static void rcu_preempt_check_callbacks(void);
void call_rcu(struct rcu_head *head, void (*func)(struct rcu_head *rcu));
#if defined(CONFIG_HOTPLUG_CPU) || defined(CONFIG_TREE_PREEMPT_RCU)
static void rcu_report_exp_rnp(struct rcu_state *rsp, struct rcu_node *rnp,
diff --git a/kernel/rcu/tree_plugin.h b/kernel/rcu/tree_plugin.h
index 09547143628a..7158814b7b4a 100644
--- a/kernel/rcu/tree_plugin.h
+++ b/kernel/rcu/tree_plugin.h
@@ -621,7 +621,7 @@ static int rcu_preempt_offline_tasks(struct rcu_state *rsp,
*
* Caller must disable hard irqs.
*/
-static void rcu_preempt_check_callbacks(int cpu)
+static void rcu_preempt_check_callbacks(void)
{
struct task_struct *t = current;
@@ -630,8 +630,8 @@ static void rcu_preempt_check_callbacks(int cpu)
return;
}
if (t->rcu_read_lock_nesting > 0 &&
- per_cpu(rcu_preempt_data, cpu).qs_pending &&
- !per_cpu(rcu_preempt_data, cpu).passed_quiesce)
+ __this_cpu_read(rcu_preempt_data.qs_pending) &&
+ !__this_cpu_read(rcu_preempt_data.passed_quiesce))
t->rcu_read_unlock_special.b.need_qs = true;
}
@@ -1017,7 +1017,7 @@ static int rcu_preempt_offline_tasks(struct rcu_state *rsp,
* Because preemptible RCU does not exist, it never has any callbacks
* to check.
*/
-static void rcu_preempt_check_callbacks(int cpu)
+static void rcu_preempt_check_callbacks(void)
{
}
--
1.8.1.5
^ permalink raw reply related [flat|nested] 13+ messages in thread* [PATCH tip/core/rcu 06/10] rcu: Remove "cpu" argument to rcu_note_context_switch()
2014-10-28 21:53 ` [PATCH tip/core/rcu 01/10] rcu: Remove rcu_dynticks * parameters when they are always this_cpu_ptr(&rcu_dynticks) Paul E. McKenney
` (3 preceding siblings ...)
2014-10-28 21:53 ` [PATCH tip/core/rcu 05/10] rcu: Remove "cpu" argument to rcu_preempt_check_callbacks() Paul E. McKenney
@ 2014-10-28 21:53 ` Paul E. McKenney
2014-10-28 21:53 ` [PATCH tip/core/rcu 07/10] rcu: Remove "cpu" argument to rcu_needs_cpu() Paul E. McKenney
` (3 subsequent siblings)
8 siblings, 0 replies; 13+ messages in thread
From: Paul E. McKenney @ 2014-10-28 21:53 UTC (permalink / raw)
To: linux-kernel
Cc: mingo, laijs, dipankar, akpm, mathieu.desnoyers, josh, tglx,
peterz, rostedt, dhowells, edumazet, dvhart, fweisbec, oleg,
bobby.prani, Paul E. McKenney
From: "Paul E. McKenney" <paulmck@linux.vnet.ibm.com>
The "cpu" argument to rcu_note_context_switch() is always the current
CPU, so drop it. This in turn allows the "cpu" argument to
rcu_preempt_note_context_switch() to be removed, which allows the sole
use of "cpu" in both functions to be replaced with a this_cpu_ptr().
Again, the anticipated cross-CPU uses of these functions has been
replaced by NO_HZ_FULL.
Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
---
include/linux/rcutiny.h | 2 +-
include/linux/rcutree.h | 4 ++--
kernel/rcu/tree.c | 4 ++--
kernel/rcu/tree.h | 2 +-
kernel/rcu/tree_plugin.h | 6 +++---
kernel/sched/core.c | 2 +-
kernel/softirq.c | 2 +-
7 files changed, 11 insertions(+), 11 deletions(-)
diff --git a/include/linux/rcutiny.h b/include/linux/rcutiny.h
index 38cc5b1e252d..0e5366200154 100644
--- a/include/linux/rcutiny.h
+++ b/include/linux/rcutiny.h
@@ -78,7 +78,7 @@ static inline void kfree_call_rcu(struct rcu_head *head,
call_rcu(head, func);
}
-static inline void rcu_note_context_switch(int cpu)
+static inline void rcu_note_context_switch(void)
{
rcu_sched_qs();
}
diff --git a/include/linux/rcutree.h b/include/linux/rcutree.h
index 3e2f5d432743..7b5484db1857 100644
--- a/include/linux/rcutree.h
+++ b/include/linux/rcutree.h
@@ -30,7 +30,7 @@
#ifndef __LINUX_RCUTREE_H
#define __LINUX_RCUTREE_H
-void rcu_note_context_switch(int cpu);
+void rcu_note_context_switch(void);
#ifndef CONFIG_RCU_NOCB_CPU_ALL
int rcu_needs_cpu(int cpu, unsigned long *delta_jiffies);
#endif /* #ifndef CONFIG_RCU_NOCB_CPU_ALL */
@@ -43,7 +43,7 @@ void rcu_cpu_stall_reset(void);
*/
static inline void rcu_virt_note_context_switch(int cpu)
{
- rcu_note_context_switch(cpu);
+ rcu_note_context_switch();
}
void synchronize_rcu_bh(void);
diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c
index 1af5e2cdcbeb..b591f1459240 100644
--- a/kernel/rcu/tree.c
+++ b/kernel/rcu/tree.c
@@ -286,11 +286,11 @@ static void rcu_momentary_dyntick_idle(void)
* and requires special handling for preemptible RCU.
* The caller must have disabled preemption.
*/
-void rcu_note_context_switch(int cpu)
+void rcu_note_context_switch(void)
{
trace_rcu_utilization(TPS("Start context switch"));
rcu_sched_qs();
- rcu_preempt_note_context_switch(cpu);
+ rcu_preempt_note_context_switch();
if (unlikely(raw_cpu_read(rcu_sched_qs_mask)))
rcu_momentary_dyntick_idle();
trace_rcu_utilization(TPS("End context switch"));
diff --git a/kernel/rcu/tree.h b/kernel/rcu/tree.h
index 94a26e330c1b..238ac39053fc 100644
--- a/kernel/rcu/tree.h
+++ b/kernel/rcu/tree.h
@@ -547,7 +547,7 @@ DECLARE_PER_CPU(char, rcu_cpu_has_work);
/* Forward declarations for rcutree_plugin.h */
static void rcu_bootup_announce(void);
long rcu_batches_completed(void);
-static void rcu_preempt_note_context_switch(int cpu);
+static void rcu_preempt_note_context_switch(void);
static int rcu_preempt_blocked_readers_cgp(struct rcu_node *rnp);
#ifdef CONFIG_HOTPLUG_CPU
static void rcu_report_unblock_qs_rnp(struct rcu_node *rnp,
diff --git a/kernel/rcu/tree_plugin.h b/kernel/rcu/tree_plugin.h
index 7158814b7b4a..495d4cce47a7 100644
--- a/kernel/rcu/tree_plugin.h
+++ b/kernel/rcu/tree_plugin.h
@@ -156,7 +156,7 @@ static void rcu_preempt_qs(void)
*
* Caller must disable preemption.
*/
-static void rcu_preempt_note_context_switch(int cpu)
+static void rcu_preempt_note_context_switch(void)
{
struct task_struct *t = current;
unsigned long flags;
@@ -167,7 +167,7 @@ static void rcu_preempt_note_context_switch(int cpu)
!t->rcu_read_unlock_special.b.blocked) {
/* Possibly blocking in an RCU read-side critical section. */
- rdp = per_cpu_ptr(rcu_preempt_state.rda, cpu);
+ rdp = this_cpu_ptr(rcu_preempt_state.rda);
rnp = rdp->mynode;
raw_spin_lock_irqsave(&rnp->lock, flags);
smp_mb__after_unlock_lock();
@@ -945,7 +945,7 @@ EXPORT_SYMBOL_GPL(rcu_batches_completed);
* Because preemptible RCU does not exist, we never have to check for
* CPUs being in quiescent states.
*/
-static void rcu_preempt_note_context_switch(int cpu)
+static void rcu_preempt_note_context_switch(void)
{
}
diff --git a/kernel/sched/core.c b/kernel/sched/core.c
index 44999505e1bf..cc186945296d 100644
--- a/kernel/sched/core.c
+++ b/kernel/sched/core.c
@@ -2802,7 +2802,7 @@ need_resched:
preempt_disable();
cpu = smp_processor_id();
rq = cpu_rq(cpu);
- rcu_note_context_switch(cpu);
+ rcu_note_context_switch();
prev = rq->curr;
schedule_debug(prev);
diff --git a/kernel/softirq.c b/kernel/softirq.c
index 0699add19164..501baa9ac1be 100644
--- a/kernel/softirq.c
+++ b/kernel/softirq.c
@@ -656,7 +656,7 @@ static void run_ksoftirqd(unsigned int cpu)
* in the task stack here.
*/
__do_softirq();
- rcu_note_context_switch(cpu);
+ rcu_note_context_switch();
local_irq_enable();
cond_resched();
return;
--
1.8.1.5
^ permalink raw reply related [flat|nested] 13+ messages in thread* [PATCH tip/core/rcu 07/10] rcu: Remove "cpu" argument to rcu_needs_cpu()
2014-10-28 21:53 ` [PATCH tip/core/rcu 01/10] rcu: Remove rcu_dynticks * parameters when they are always this_cpu_ptr(&rcu_dynticks) Paul E. McKenney
` (4 preceding siblings ...)
2014-10-28 21:53 ` [PATCH tip/core/rcu 06/10] rcu: Remove "cpu" argument to rcu_note_context_switch() Paul E. McKenney
@ 2014-10-28 21:53 ` Paul E. McKenney
2014-10-28 21:53 ` [PATCH tip/core/rcu 08/10] rcu: Remove "cpu" argument to rcu_prepare_for_idle() Paul E. McKenney
` (2 subsequent siblings)
8 siblings, 0 replies; 13+ messages in thread
From: Paul E. McKenney @ 2014-10-28 21:53 UTC (permalink / raw)
To: linux-kernel
Cc: mingo, laijs, dipankar, akpm, mathieu.desnoyers, josh, tglx,
peterz, rostedt, dhowells, edumazet, dvhart, fweisbec, oleg,
bobby.prani, Paul E. McKenney
From: "Paul E. McKenney" <paulmck@linux.vnet.ibm.com>
The "cpu" argument to rcu_needs_cpu() is always the current CPU, so drop
it. This in turn allows the "cpu" argument to rcu_cpu_has_callbacks()
to be removed, which allows the uses of "cpu" in both functions to be
replaced with a this_cpu_ptr(). Again, the anticipated cross-CPU uses
of these functions has been replaced by NO_HZ_FULL.
Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
---
include/linux/rcupdate.h | 2 +-
include/linux/rcutree.h | 2 +-
kernel/rcu/tree.c | 4 ++--
kernel/rcu/tree_plugin.h | 12 ++++++------
kernel/time/tick-sched.c | 2 +-
5 files changed, 11 insertions(+), 11 deletions(-)
diff --git a/include/linux/rcupdate.h b/include/linux/rcupdate.h
index b4ad275d90d0..4eb810832b19 100644
--- a/include/linux/rcupdate.h
+++ b/include/linux/rcupdate.h
@@ -1103,7 +1103,7 @@ static inline notrace void rcu_read_unlock_sched_notrace(void)
__kfree_rcu(&((ptr)->rcu_head), offsetof(typeof(*(ptr)), rcu_head))
#if defined(CONFIG_TINY_RCU) || defined(CONFIG_RCU_NOCB_CPU_ALL)
-static inline int rcu_needs_cpu(int cpu, unsigned long *delta_jiffies)
+static inline int rcu_needs_cpu(unsigned long *delta_jiffies)
{
*delta_jiffies = ULONG_MAX;
return 0;
diff --git a/include/linux/rcutree.h b/include/linux/rcutree.h
index 7b5484db1857..52953790dcca 100644
--- a/include/linux/rcutree.h
+++ b/include/linux/rcutree.h
@@ -32,7 +32,7 @@
void rcu_note_context_switch(void);
#ifndef CONFIG_RCU_NOCB_CPU_ALL
-int rcu_needs_cpu(int cpu, unsigned long *delta_jiffies);
+int rcu_needs_cpu(unsigned long *delta_jiffies);
#endif /* #ifndef CONFIG_RCU_NOCB_CPU_ALL */
void rcu_cpu_stall_reset(void);
diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c
index b591f1459240..d678a98caf1d 100644
--- a/kernel/rcu/tree.c
+++ b/kernel/rcu/tree.c
@@ -3159,7 +3159,7 @@ static int rcu_pending(void)
* non-NULL, store an indication of whether all callbacks are lazy.
* (If there are no callbacks, all of them are deemed to be lazy.)
*/
-static int __maybe_unused rcu_cpu_has_callbacks(int cpu, bool *all_lazy)
+static int __maybe_unused rcu_cpu_has_callbacks(bool *all_lazy)
{
bool al = true;
bool hc = false;
@@ -3167,7 +3167,7 @@ static int __maybe_unused rcu_cpu_has_callbacks(int cpu, bool *all_lazy)
struct rcu_state *rsp;
for_each_rcu_flavor(rsp) {
- rdp = per_cpu_ptr(rsp->rda, cpu);
+ rdp = this_cpu_ptr(rsp->rda);
if (!rdp->nxtlist)
continue;
hc = true;
diff --git a/kernel/rcu/tree_plugin.h b/kernel/rcu/tree_plugin.h
index 495d4cce47a7..1797b76cb3ff 100644
--- a/kernel/rcu/tree_plugin.h
+++ b/kernel/rcu/tree_plugin.h
@@ -1512,10 +1512,10 @@ static void rcu_prepare_kthreads(int cpu)
* any flavor of RCU.
*/
#ifndef CONFIG_RCU_NOCB_CPU_ALL
-int rcu_needs_cpu(int cpu, unsigned long *delta_jiffies)
+int rcu_needs_cpu(unsigned long *delta_jiffies)
{
*delta_jiffies = ULONG_MAX;
- return rcu_cpu_has_callbacks(cpu, NULL);
+ return rcu_cpu_has_callbacks(NULL);
}
#endif /* #ifndef CONFIG_RCU_NOCB_CPU_ALL */
@@ -1624,15 +1624,15 @@ static bool __maybe_unused rcu_try_advance_all_cbs(void)
* The caller must have disabled interrupts.
*/
#ifndef CONFIG_RCU_NOCB_CPU_ALL
-int rcu_needs_cpu(int cpu, unsigned long *dj)
+int rcu_needs_cpu(unsigned long *dj)
{
- struct rcu_dynticks *rdtp = &per_cpu(rcu_dynticks, cpu);
+ struct rcu_dynticks *rdtp = this_cpu_ptr(&rcu_dynticks);
/* Snapshot to detect later posting of non-lazy callback. */
rdtp->nonlazy_posted_snap = rdtp->nonlazy_posted;
/* If no callbacks, RCU doesn't need the CPU. */
- if (!rcu_cpu_has_callbacks(cpu, &rdtp->all_lazy)) {
+ if (!rcu_cpu_has_callbacks(&rdtp->all_lazy)) {
*dj = ULONG_MAX;
return 0;
}
@@ -1679,7 +1679,7 @@ static void rcu_prepare_for_idle(int cpu)
/* Handle nohz enablement switches conservatively. */
tne = ACCESS_ONCE(tick_nohz_active);
if (tne != rdtp->tick_nohz_enabled_snap) {
- if (rcu_cpu_has_callbacks(cpu, NULL))
+ if (rcu_cpu_has_callbacks(NULL))
invoke_rcu_core(); /* force nohz to see update. */
rdtp->tick_nohz_enabled_snap = tne;
return;
diff --git a/kernel/time/tick-sched.c b/kernel/time/tick-sched.c
index 7b5741fc4110..1f4356037a7d 100644
--- a/kernel/time/tick-sched.c
+++ b/kernel/time/tick-sched.c
@@ -585,7 +585,7 @@ static ktime_t tick_nohz_stop_sched_tick(struct tick_sched *ts,
last_jiffies = jiffies;
} while (read_seqretry(&jiffies_lock, seq));
- if (rcu_needs_cpu(cpu, &rcu_delta_jiffies) ||
+ if (rcu_needs_cpu(&rcu_delta_jiffies) ||
arch_needs_cpu() || irq_work_needs_cpu()) {
next_jiffies = last_jiffies + 1;
delta_jiffies = 1;
--
1.8.1.5
^ permalink raw reply related [flat|nested] 13+ messages in thread* [PATCH tip/core/rcu 08/10] rcu: Remove "cpu" argument to rcu_prepare_for_idle()
2014-10-28 21:53 ` [PATCH tip/core/rcu 01/10] rcu: Remove rcu_dynticks * parameters when they are always this_cpu_ptr(&rcu_dynticks) Paul E. McKenney
` (5 preceding siblings ...)
2014-10-28 21:53 ` [PATCH tip/core/rcu 07/10] rcu: Remove "cpu" argument to rcu_needs_cpu() Paul E. McKenney
@ 2014-10-28 21:53 ` Paul E. McKenney
2014-10-28 21:53 ` [PATCH tip/core/rcu 09/10] rcu: Remove "cpu" argument to rcu_cleanup_after_idle() Paul E. McKenney
2014-10-28 21:53 ` [PATCH tip/core/rcu 10/10] cpu: Avoid puts_pending overflow Paul E. McKenney
8 siblings, 0 replies; 13+ messages in thread
From: Paul E. McKenney @ 2014-10-28 21:53 UTC (permalink / raw)
To: linux-kernel
Cc: mingo, laijs, dipankar, akpm, mathieu.desnoyers, josh, tglx,
peterz, rostedt, dhowells, edumazet, dvhart, fweisbec, oleg,
bobby.prani, Paul E. McKenney
From: "Paul E. McKenney" <paulmck@linux.vnet.ibm.com>
The "cpu" argument to rcu_prepare_for_idle() is always the current
CPU, so drop it. This in turn allows two of the uses of "cpu" in
this function to be replaced with a this_cpu_ptr() and the third by
smp_processor_id(), replacing that of the call to rcu_prepare_for_idle().
Again, the anticipated cross-CPU uses of these functions has been replaced
by NO_HZ_FULL.
Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
---
kernel/rcu/tree.c | 2 +-
kernel/rcu/tree.h | 2 +-
kernel/rcu/tree_plugin.h | 10 +++++-----
3 files changed, 7 insertions(+), 7 deletions(-)
diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c
index d678a98caf1d..8251938fb4ae 100644
--- a/kernel/rcu/tree.c
+++ b/kernel/rcu/tree.c
@@ -531,7 +531,7 @@ static void rcu_eqs_enter_common(long long oldval, bool user)
rdp = this_cpu_ptr(rsp->rda);
do_nocb_deferred_wakeup(rdp);
}
- rcu_prepare_for_idle(smp_processor_id());
+ rcu_prepare_for_idle();
/* CPUs seeing atomic_inc() must see prior RCU read-side crit sects */
smp_mb__before_atomic(); /* See above. */
atomic_inc(&rdtp->dynticks);
diff --git a/kernel/rcu/tree.h b/kernel/rcu/tree.h
index 238ac39053fc..8bc3cacbe14a 100644
--- a/kernel/rcu/tree.h
+++ b/kernel/rcu/tree.h
@@ -580,7 +580,7 @@ static int rcu_spawn_one_boost_kthread(struct rcu_state *rsp,
static void __init rcu_spawn_boost_kthreads(void);
static void rcu_prepare_kthreads(int cpu);
static void rcu_cleanup_after_idle(int cpu);
-static void rcu_prepare_for_idle(int cpu);
+static void rcu_prepare_for_idle(void);
static void rcu_idle_count_callbacks_posted(void);
static void print_cpu_stall_info_begin(void);
static void print_cpu_stall_info(struct rcu_state *rsp, int cpu);
diff --git a/kernel/rcu/tree_plugin.h b/kernel/rcu/tree_plugin.h
index 1797b76cb3ff..5e68ab5fe317 100644
--- a/kernel/rcu/tree_plugin.h
+++ b/kernel/rcu/tree_plugin.h
@@ -1531,7 +1531,7 @@ static void rcu_cleanup_after_idle(int cpu)
* Do the idle-entry grace-period work, which, because CONFIG_RCU_FAST_NO_HZ=n,
* is nothing.
*/
-static void rcu_prepare_for_idle(int cpu)
+static void rcu_prepare_for_idle(void)
{
}
@@ -1666,12 +1666,12 @@ int rcu_needs_cpu(unsigned long *dj)
*
* The caller must have disabled interrupts.
*/
-static void rcu_prepare_for_idle(int cpu)
+static void rcu_prepare_for_idle(void)
{
#ifndef CONFIG_RCU_NOCB_CPU_ALL
bool needwake;
struct rcu_data *rdp;
- struct rcu_dynticks *rdtp = &per_cpu(rcu_dynticks, cpu);
+ struct rcu_dynticks *rdtp = this_cpu_ptr(&rcu_dynticks);
struct rcu_node *rnp;
struct rcu_state *rsp;
int tne;
@@ -1688,7 +1688,7 @@ static void rcu_prepare_for_idle(int cpu)
return;
/* If this is a no-CBs CPU, no callbacks, just return. */
- if (rcu_is_nocb_cpu(cpu))
+ if (rcu_is_nocb_cpu(smp_processor_id()))
return;
/*
@@ -1712,7 +1712,7 @@ static void rcu_prepare_for_idle(int cpu)
return;
rdtp->last_accelerate = jiffies;
for_each_rcu_flavor(rsp) {
- rdp = per_cpu_ptr(rsp->rda, cpu);
+ rdp = this_cpu_ptr(rsp->rda);
if (!*rdp->nxttail[RCU_DONE_TAIL])
continue;
rnp = rdp->mynode;
--
1.8.1.5
^ permalink raw reply related [flat|nested] 13+ messages in thread* [PATCH tip/core/rcu 09/10] rcu: Remove "cpu" argument to rcu_cleanup_after_idle()
2014-10-28 21:53 ` [PATCH tip/core/rcu 01/10] rcu: Remove rcu_dynticks * parameters when they are always this_cpu_ptr(&rcu_dynticks) Paul E. McKenney
` (6 preceding siblings ...)
2014-10-28 21:53 ` [PATCH tip/core/rcu 08/10] rcu: Remove "cpu" argument to rcu_prepare_for_idle() Paul E. McKenney
@ 2014-10-28 21:53 ` Paul E. McKenney
2014-10-28 21:53 ` [PATCH tip/core/rcu 10/10] cpu: Avoid puts_pending overflow Paul E. McKenney
8 siblings, 0 replies; 13+ messages in thread
From: Paul E. McKenney @ 2014-10-28 21:53 UTC (permalink / raw)
To: linux-kernel
Cc: mingo, laijs, dipankar, akpm, mathieu.desnoyers, josh, tglx,
peterz, rostedt, dhowells, edumazet, dvhart, fweisbec, oleg,
bobby.prani, Paul E. McKenney
From: "Paul E. McKenney" <paulmck@linux.vnet.ibm.com>
The "cpu" argument to rcu_cleanup_after_idle() is always the current
CPU, so drop it. This moves the smp_processor_id() from the caller to
rcu_cleanup_after_idle(), saving argument-passing overhead. Again,
the anticipated cross-CPU uses of these functions has been replaced
by NO_HZ_FULL.
Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
---
kernel/rcu/tree.c | 2 +-
kernel/rcu/tree.h | 2 +-
kernel/rcu/tree_plugin.h | 6 +++---
3 files changed, 5 insertions(+), 5 deletions(-)
diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c
index 8251938fb4ae..fa76fd3c219c 100644
--- a/kernel/rcu/tree.c
+++ b/kernel/rcu/tree.c
@@ -661,7 +661,7 @@ static void rcu_eqs_exit_common(long long oldval, int user)
/* CPUs seeing atomic_inc() must see later RCU read-side crit sects */
smp_mb__after_atomic(); /* See above. */
WARN_ON_ONCE(!(atomic_read(&rdtp->dynticks) & 0x1));
- rcu_cleanup_after_idle(smp_processor_id());
+ rcu_cleanup_after_idle();
trace_rcu_dyntick(TPS("End"), oldval, rdtp->dynticks_nesting);
if (!user && !is_idle_task(current)) {
struct task_struct *idle __maybe_unused =
diff --git a/kernel/rcu/tree.h b/kernel/rcu/tree.h
index 8bc3cacbe14a..36c30390e4e9 100644
--- a/kernel/rcu/tree.h
+++ b/kernel/rcu/tree.h
@@ -579,7 +579,7 @@ static int rcu_spawn_one_boost_kthread(struct rcu_state *rsp,
#endif /* #ifdef CONFIG_RCU_BOOST */
static void __init rcu_spawn_boost_kthreads(void);
static void rcu_prepare_kthreads(int cpu);
-static void rcu_cleanup_after_idle(int cpu);
+static void rcu_cleanup_after_idle(void);
static void rcu_prepare_for_idle(void);
static void rcu_idle_count_callbacks_posted(void);
static void print_cpu_stall_info_begin(void);
diff --git a/kernel/rcu/tree_plugin.h b/kernel/rcu/tree_plugin.h
index 5e68ab5fe317..2443282737ba 100644
--- a/kernel/rcu/tree_plugin.h
+++ b/kernel/rcu/tree_plugin.h
@@ -1523,7 +1523,7 @@ int rcu_needs_cpu(unsigned long *delta_jiffies)
* Because we do not have RCU_FAST_NO_HZ, don't bother cleaning up
* after it.
*/
-static void rcu_cleanup_after_idle(int cpu)
+static void rcu_cleanup_after_idle(void)
{
}
@@ -1731,10 +1731,10 @@ static void rcu_prepare_for_idle(void)
* any grace periods that elapsed while the CPU was idle, and if any
* callbacks are now ready to invoke, initiate invocation.
*/
-static void rcu_cleanup_after_idle(int cpu)
+static void rcu_cleanup_after_idle(void)
{
#ifndef CONFIG_RCU_NOCB_CPU_ALL
- if (rcu_is_nocb_cpu(cpu))
+ if (rcu_is_nocb_cpu(smp_processor_id()))
return;
if (rcu_try_advance_all_cbs())
invoke_rcu_core();
--
1.8.1.5
^ permalink raw reply related [flat|nested] 13+ messages in thread* [PATCH tip/core/rcu 10/10] cpu: Avoid puts_pending overflow
2014-10-28 21:53 ` [PATCH tip/core/rcu 01/10] rcu: Remove rcu_dynticks * parameters when they are always this_cpu_ptr(&rcu_dynticks) Paul E. McKenney
` (7 preceding siblings ...)
2014-10-28 21:53 ` [PATCH tip/core/rcu 09/10] rcu: Remove "cpu" argument to rcu_cleanup_after_idle() Paul E. McKenney
@ 2014-10-28 21:53 ` Paul E. McKenney
2014-10-29 17:56 ` Pranith Kumar
8 siblings, 1 reply; 13+ messages in thread
From: Paul E. McKenney @ 2014-10-28 21:53 UTC (permalink / raw)
To: linux-kernel
Cc: mingo, laijs, dipankar, akpm, mathieu.desnoyers, josh, tglx,
peterz, rostedt, dhowells, edumazet, dvhart, fweisbec, oleg,
bobby.prani, Paul E. McKenney
From: "Paul E. McKenney" <paulmck@linux.vnet.ibm.com>
A long string of get_online_cpus() with each followed by a
put_online_cpu() that fails to acquire cpu_hotplug.lock can result in
overflow of the cpu_hotplug.puts_pending counter. Although this is
perhaps improbably, a system with absolutely no CPU-hotplug operations
will have an arbitrarily long time in which this overflow could occur.
This commit therefore adds overflow checks to get_online_cpus() and
try_get_online_cpus().
Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
---
kernel/cpu.c | 19 +++++++++++++------
1 file changed, 13 insertions(+), 6 deletions(-)
diff --git a/kernel/cpu.c b/kernel/cpu.c
index 90a3d017b90c..5d220234b3ca 100644
--- a/kernel/cpu.c
+++ b/kernel/cpu.c
@@ -86,6 +86,16 @@ static struct {
#define cpuhp_lock_acquire() lock_map_acquire(&cpu_hotplug.dep_map)
#define cpuhp_lock_release() lock_map_release(&cpu_hotplug.dep_map)
+static void apply_puts_pending(int max)
+{
+ int delta;
+
+ if (atomic_read(&cpu_hotplug.puts_pending) >= max) {
+ delta = atomic_xchg(&cpu_hotplug.puts_pending, 0);
+ cpu_hotplug.refcount -= delta;
+ }
+}
+
void get_online_cpus(void)
{
might_sleep();
@@ -93,6 +103,7 @@ void get_online_cpus(void)
return;
cpuhp_lock_acquire_read();
mutex_lock(&cpu_hotplug.lock);
+ apply_puts_pending(65536);
cpu_hotplug.refcount++;
mutex_unlock(&cpu_hotplug.lock);
}
@@ -105,6 +116,7 @@ bool try_get_online_cpus(void)
if (!mutex_trylock(&cpu_hotplug.lock))
return false;
cpuhp_lock_acquire_tryread();
+ apply_puts_pending(65536);
cpu_hotplug.refcount++;
mutex_unlock(&cpu_hotplug.lock);
return true;
@@ -161,12 +173,7 @@ void cpu_hotplug_begin(void)
cpuhp_lock_acquire();
for (;;) {
mutex_lock(&cpu_hotplug.lock);
- if (atomic_read(&cpu_hotplug.puts_pending)) {
- int delta;
-
- delta = atomic_xchg(&cpu_hotplug.puts_pending, 0);
- cpu_hotplug.refcount -= delta;
- }
+ apply_puts_pending(1);
if (likely(!cpu_hotplug.refcount))
break;
__set_current_state(TASK_UNINTERRUPTIBLE);
--
1.8.1.5
^ permalink raw reply related [flat|nested] 13+ messages in thread* Re: [PATCH tip/core/rcu 10/10] cpu: Avoid puts_pending overflow
2014-10-28 21:53 ` [PATCH tip/core/rcu 10/10] cpu: Avoid puts_pending overflow Paul E. McKenney
@ 2014-10-29 17:56 ` Pranith Kumar
0 siblings, 0 replies; 13+ messages in thread
From: Pranith Kumar @ 2014-10-29 17:56 UTC (permalink / raw)
To: Paul E. McKenney
Cc: LKML, Ingo Molnar, Lai Jiangshan, Dipankar Sarma, Andrew Morton,
Mathieu Desnoyers, Josh Triplett, Thomas Gleixner, Peter Zijlstra,
Steven Rostedt, David Howells, Eric Dumazet, dvhart,
Frédéric Weisbecker, Oleg Nesterov
On Tue, Oct 28, 2014 at 5:53 PM, Paul E. McKenney
<paulmck@linux.vnet.ibm.com> wrote:
> From: "Paul E. McKenney" <paulmck@linux.vnet.ibm.com>
>
> A long string of get_online_cpus() with each followed by a
> put_online_cpu() that fails to acquire cpu_hotplug.lock can result in
> overflow of the cpu_hotplug.puts_pending counter. Although this is
> perhaps improbably, a system with absolutely no CPU-hotplug operations
> will have an arbitrarily long time in which this overflow could occur.
> This commit therefore adds overflow checks to get_online_cpus() and
> try_get_online_cpus().
>
> Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
This patch seems to be missing in the cover letter.
Reviewed-by: Pranith Kumar <bobby.prani@gmail.com>
> ---
> kernel/cpu.c | 19 +++++++++++++------
> 1 file changed, 13 insertions(+), 6 deletions(-)
>
> diff --git a/kernel/cpu.c b/kernel/cpu.c
> index 90a3d017b90c..5d220234b3ca 100644
> --- a/kernel/cpu.c
> +++ b/kernel/cpu.c
> @@ -86,6 +86,16 @@ static struct {
> #define cpuhp_lock_acquire() lock_map_acquire(&cpu_hotplug.dep_map)
> #define cpuhp_lock_release() lock_map_release(&cpu_hotplug.dep_map)
>
> +static void apply_puts_pending(int max)
> +{
> + int delta;
> +
> + if (atomic_read(&cpu_hotplug.puts_pending) >= max) {
> + delta = atomic_xchg(&cpu_hotplug.puts_pending, 0);
> + cpu_hotplug.refcount -= delta;
> + }
> +}
> +
> void get_online_cpus(void)
> {
> might_sleep();
> @@ -93,6 +103,7 @@ void get_online_cpus(void)
> return;
> cpuhp_lock_acquire_read();
> mutex_lock(&cpu_hotplug.lock);
> + apply_puts_pending(65536);
> cpu_hotplug.refcount++;
> mutex_unlock(&cpu_hotplug.lock);
> }
> @@ -105,6 +116,7 @@ bool try_get_online_cpus(void)
> if (!mutex_trylock(&cpu_hotplug.lock))
> return false;
> cpuhp_lock_acquire_tryread();
> + apply_puts_pending(65536);
> cpu_hotplug.refcount++;
> mutex_unlock(&cpu_hotplug.lock);
> return true;
> @@ -161,12 +173,7 @@ void cpu_hotplug_begin(void)
> cpuhp_lock_acquire();
> for (;;) {
> mutex_lock(&cpu_hotplug.lock);
> - if (atomic_read(&cpu_hotplug.puts_pending)) {
> - int delta;
> -
> - delta = atomic_xchg(&cpu_hotplug.puts_pending, 0);
> - cpu_hotplug.refcount -= delta;
> - }
> + apply_puts_pending(1);
> if (likely(!cpu_hotplug.refcount))
> break;
> __set_current_state(TASK_UNINTERRUPTIBLE);
> --
> 1.8.1.5
>
--
Pranith
^ permalink raw reply [flat|nested] 13+ messages in thread