* Re: [PATCH 1/2] rcu-tasks: Make RCU-Tasks pay attention to idle tasks
2024-08-07 16:35 ` [PATCH 1/2] rcu-tasks: Make RCU-Tasks pay attention to " neeraj.upadhyay
@ 2024-08-08 17:59 ` Paul E. McKenney
2024-08-14 15:01 ` Frederic Weisbecker
1 sibling, 0 replies; 7+ messages in thread
From: Paul E. McKenney @ 2024-08-08 17:59 UTC (permalink / raw)
To: neeraj.upadhyay
Cc: rcu, linux-kernel, joel, frederic, boqun.feng, urezki, rostedt,
mathieu.desnoyers, jiangshanlai, qiang.zhang1211, neeraj.iitr10
On Wed, Aug 07, 2024 at 10:05:05PM +0530, neeraj.upadhyay@kernel.org wrote:
> From: Neeraj Upadhyay <neeraj.upadhyay@kernel.org>
>
> Currently, idle tasks are ignored by Tasks RCU and idle tasks
> are in RCU-Tasks quiescent state for Tasks RCU. Change this to
> start paying attention to idle tasks except in deep-idle functions
> where RCU is not watching. With this, for architectures where
> kernel entry/exit and deep-idle functions have been properly tagged
> noinstr, Tasks Rude RCU can be disabled.
>
> [ neeraj.upadhyay: Frederic Weisbecker and Paul E. McKenney feedback. ]
>
> Signed-off-by: Neeraj Upadhyay <neeraj.upadhyay@kernel.org>
Looks plausible, thank you!
A few questions and comments interspersed below.
Thanx, Paul
> ---
> .../RCU/Design/Requirements/Requirements.rst | 12 +-
> kernel/rcu/rcu.h | 4 +
> kernel/rcu/tasks.h | 107 ++++++++++++------
> kernel/rcu/tree.c | 12 +-
> kernel/rcu/tree_exp.h | 2 +-
> 5 files changed, 89 insertions(+), 48 deletions(-)
>
> diff --git a/Documentation/RCU/Design/Requirements/Requirements.rst b/Documentation/RCU/Design/Requirements/Requirements.rst
> index 6125e7068d2c..5016b85d53d7 100644
> --- a/Documentation/RCU/Design/Requirements/Requirements.rst
> +++ b/Documentation/RCU/Design/Requirements/Requirements.rst
> @@ -2611,8 +2611,8 @@ critical sections that are delimited by voluntary context switches, that
> is, calls to schedule(), cond_resched(), and
> synchronize_rcu_tasks(). In addition, transitions to and from
> userspace execution also delimit tasks-RCU read-side critical sections.
> -Idle tasks are ignored by Tasks RCU, and Tasks Rude RCU may be used to
> -interact with them.
> +Idle tasks which are idle from RCU's perspective are ignored by Tasks RCU,
> +and Tasks Rude RCU may be used to interact with them.
>
> Note well that involuntary context switches are *not* Tasks-RCU quiescent
> states. After all, in preemptible kernels, a task executing code in a
> @@ -2643,10 +2643,10 @@ moniker. And this operation is considered to be quite rude by real-time
> workloads that don't want their ``nohz_full`` CPUs receiving IPIs and
> by battery-powered systems that don't want their idle CPUs to be awakened.
>
> -Once kernel entry/exit and deep-idle functions have been properly tagged
> -``noinstr``, Tasks RCU can start paying attention to idle tasks (except
> -those that are idle from RCU's perspective) and then Tasks Rude RCU can
> -be removed from the kernel.
> +As Tasks RCU now pays attention to idle tasks (except those that are idle
> +from RCU's perspective), once kernel entry/exit and deep-idle functions have
> +been properly tagged ``noinstr``, Tasks Rude RCU can be removed from the
> +kernel.
>
> The tasks-rude-RCU API is also reader-marking-free and thus quite compact,
> consisting solely of synchronize_rcu_tasks_rude().
> diff --git a/kernel/rcu/rcu.h b/kernel/rcu/rcu.h
> index feb3ac1dc5d5..5fec1b52039c 100644
> --- a/kernel/rcu/rcu.h
> +++ b/kernel/rcu/rcu.h
> @@ -609,6 +609,8 @@ void srcutorture_get_gp_data(struct srcu_struct *sp, int *flags,
>
> #ifdef CONFIG_TINY_RCU
> static inline bool rcu_watching_zero_in_eqs(int cpu, int *vp) { return false; }
> +static inline bool rcu_watching_snap_in_eqs(int snap) { return false; }
> +static inline bool rcu_watching_snap_stopped_since(int cpu, int snap) { return false; }
> static inline unsigned long rcu_get_gp_seq(void) { return 0; }
> static inline unsigned long rcu_exp_batches_completed(void) { return 0; }
> static inline unsigned long
> @@ -622,6 +624,8 @@ static inline void rcu_gp_slow_register(atomic_t *rgssp) { }
> static inline void rcu_gp_slow_unregister(atomic_t *rgssp) { }
> #else /* #ifdef CONFIG_TINY_RCU */
> bool rcu_watching_zero_in_eqs(int cpu, int *vp);
> +bool rcu_watching_snap_in_eqs(int snap);
> +bool rcu_watching_snap_stopped_since(int cpu, int snap);
> unsigned long rcu_get_gp_seq(void);
> unsigned long rcu_exp_batches_completed(void);
> unsigned long srcu_batches_completed(struct srcu_struct *sp);
> diff --git a/kernel/rcu/tasks.h b/kernel/rcu/tasks.h
> index 2f8d6c8e3c4c..0c0367c8f5c8 100644
> --- a/kernel/rcu/tasks.h
> +++ b/kernel/rcu/tasks.h
> @@ -35,6 +35,7 @@ typedef void (*postgp_func_t)(struct rcu_tasks *rtp);
> * @rtp_exit_list: List of tasks in the latter portion of do_exit().
> * @cpu: CPU number corresponding to this entry.
> * @rtpp: Pointer to the rcu_tasks structure.
> + * @rcu_watching_snap: Per-GP rcu watching tracking for idle tasks.
> */
> struct rcu_tasks_percpu {
> struct rcu_segcblist cblist;
> @@ -51,6 +52,7 @@ struct rcu_tasks_percpu {
> int cpu;
> int index;
> struct rcu_tasks *rtpp;
> + int rcu_watching_snap;
> };
>
> /**
> @@ -902,14 +904,15 @@ static void rcu_tasks_wait_gp(struct rcu_tasks *rtp)
> ////////////////////////////////////////////////////////////////////////
> //
> // Simple variant of RCU whose quiescent states are voluntary context
> -// switch, cond_resched_tasks_rcu_qs(), user-space execution, and idle.
> -// As such, grace periods can take one good long time. There are no
> -// read-side primitives similar to rcu_read_lock() and rcu_read_unlock()
> -// because this implementation is intended to get the system into a safe
> -// state for some of the manipulations involved in tracing and the like.
> -// Finally, this implementation does not support high call_rcu_tasks()
> -// rates from multiple CPUs. If this is required, per-CPU callback lists
> -// will be needed.
> +// switch, cond_resched_tasks_rcu_qs(), user-space execution, and idle
> +// tasks which are in RCU-idle context. As such, grace periods can take
> +// one good long time. There are no read-side primitives similar to
> +// rcu_read_lock() and rcu_read_unlock() because this implementation is
> +// intended to get the system into a safe state for some of the
> +// manipulations involved in tracing and the like. Finally, this
> +// implementation does not support high call_rcu_tasks() rates from
> +// multiple CPUs. If this is required, per-CPU callback lists will be
> +// needed.
> //
> // The implementation uses rcu_tasks_wait_gp(), which relies on function
> // pointers in the rcu_tasks structure. The rcu_spawn_tasks_kthread()
> @@ -920,11 +923,13 @@ static void rcu_tasks_wait_gp(struct rcu_tasks *rtp)
> // Invokes synchronize_rcu() in order to wait for all in-flight
> // t->on_rq and t->nvcsw transitions to complete. This works because
> // all such transitions are carried out with interrupts disabled.
> -// rcu_tasks_pertask(), invoked on every non-idle task:
> -// For every runnable non-idle task other than the current one, use
> -// get_task_struct() to pin down that task, snapshot that task's
> -// number of voluntary context switches, and add that task to the
> -// holdout list.
> +// rcu_tasks_pertask(), invoked on every task:
> +// For idle task, snapshot the task's CPU's dynticks counter. If
> +// the CPU is in RCU-idle context, do not add it to the holdout list.
> +// For every runnable non-idle task other than the current one and
> +// idle tasks which are not in RCU-idle context, use get_task_struct()
> +// to pin down that task, snapshot that task's number of voluntary
> +// context switches, and add that task to the holdout list.
> // rcu_tasks_postscan():
> // Gather per-CPU lists of tasks in do_exit() to ensure that all
> // tasks that were in the process of exiting (and which thus might
> @@ -983,27 +988,58 @@ static bool rcu_tasks_is_holdout(struct task_struct *t)
> if (!READ_ONCE(t->on_rq))
> return false;
>
> - /*
> - * Idle tasks (or idle injection) within the idle loop are RCU-tasks
> - * quiescent states. But CPU boot code performed by the idle task
> - * isn't a quiescent state.
> - */
> - if (is_idle_task(t))
> - return false;
> -
> cpu = task_cpu(t);
>
> - /* Idle tasks on offline CPUs are RCU-tasks quiescent states. */
> - if (t == idle_task(cpu) && !rcu_cpu_online(cpu))
> +
> + if (t == idle_task(cpu)) {
> + /* Idle tasks on offline CPUs are RCU-tasks quiescent states. */
> + if (!rcu_cpu_online(cpu))
> + return false;
> +#ifdef CONFIG_SMP
> + /*
> + * As idle tasks cannot be involuntary preempted, non-running idle tasks
> + * are not in RCU-tasks critical section.
> + * synchronize_rcu() calls in rcu_tasks_pregp_step() and rcu_tasks_postgp()
> + * ensure that all ->on_cpu transitions are complete.
> + */
> + if (!t->on_cpu)
Could we do "if (IS_ENABLED(CONFIG_SMP))" here and the drop the #ifdef
and its #else clause?
> + return false;
> +#else
> + /*
> + * We are in rcu_tasks_kthread() context. Idle thread would
> + * have done a voluntary context switch.
> + */
> return false;
> +#endif
> + }
>
> return true;
> }
>
> +void call_rcu_tasks(struct rcu_head *rhp, rcu_callback_t func);
> +DEFINE_RCU_TASKS(rcu_tasks, rcu_tasks_wait_gp, call_rcu_tasks, "RCU Tasks");
> +
> /* Per-task initial processing. */
> static void rcu_tasks_pertask(struct task_struct *t, struct list_head *hop)
> {
> if (t != current && rcu_tasks_is_holdout(t)) {
> +#ifndef CONFIG_TINY_RCU
> + int cpu = task_cpu(t);
> +
> + if (t == idle_task(cpu)) {
> + struct rcu_tasks_percpu *rtpcp = per_cpu_ptr(rcu_tasks.rtpcpu, cpu);
> +
> + /*
> + * Do plain access. Ordering between remote CPU's pre idle accesses
> + * and post rcu-tasks grace period is provided by synchronize_rcu()
> + * in rcu_tasks_postgp().
> + */
> + rtpcp->rcu_watching_snap = ct_rcu_watching_cpu(cpu);
This depends on Valentin's commit:
a9fde9d1a5dd ("context_tracking, rcu: Rename ct_dynticks_cpu() into ct_rcu_watching_cpu()")
Which is not a problem, it instead just constrains what order things go
into mainline.
> + /* RCU-idle contexts are RCU-tasks quiescent state for idle tasks. */
> + if (rcu_watching_snap_in_eqs(rtpcp->rcu_watching_snap))
> + return;
> + }
> +#endif // #ifndef CONFIG_TINY_RCU
> get_task_struct(t);
> t->rcu_tasks_nvcsw = READ_ONCE(t->nvcsw);
> WRITE_ONCE(t->rcu_tasks_holdout, true);
> @@ -1011,9 +1047,6 @@ static void rcu_tasks_pertask(struct task_struct *t, struct list_head *hop)
> }
> }
>
> -void call_rcu_tasks(struct rcu_head *rhp, rcu_callback_t func);
> -DEFINE_RCU_TASKS(rcu_tasks, rcu_tasks_wait_gp, call_rcu_tasks, "RCU Tasks");
> -
> /* Processing between scanning taskslist and draining the holdout list. */
> static void rcu_tasks_postscan(struct list_head *hop)
> {
> @@ -1082,10 +1115,13 @@ static void rcu_tasks_postscan(struct list_head *hop)
> static void check_holdout_task(struct task_struct *t,
> bool needreport, bool *firstreport)
> {
> - int cpu;
> + int cpu = task_cpu(t);
> + struct rcu_tasks_percpu *rtpcp = per_cpu_ptr(rcu_tasks.rtpcpu, cpu);
>
> if (!READ_ONCE(t->rcu_tasks_holdout) ||
> t->rcu_tasks_nvcsw != READ_ONCE(t->nvcsw) ||
> + ((t == idle_task(cpu)) &&
> + rcu_watching_snap_stopped_since(cpu, rtpcp->rcu_watching_snap)) ||
> !rcu_tasks_is_holdout(t) ||
> (IS_ENABLED(CONFIG_NO_HZ_FULL) &&
> !is_idle_task(t) && READ_ONCE(t->rcu_tasks_idle_cpu) >= 0)) {
> @@ -1101,7 +1137,6 @@ static void check_holdout_task(struct task_struct *t,
> pr_err("INFO: rcu_tasks detected stalls on tasks:\n");
> *firstreport = false;
> }
> - cpu = task_cpu(t);
> pr_alert("%p: %c%c nvcsw: %lu/%lu holdout: %d idle_cpu: %d/%d\n",
> t, ".I"[is_idle_task(t)],
> "N."[cpu < 0 || !tick_nohz_full_cpu(cpu)],
> @@ -1171,11 +1206,12 @@ static void tasks_rcu_exit_srcu_stall(struct timer_list *unused)
> * period elapses, in other words after all currently executing RCU
> * read-side critical sections have completed. call_rcu_tasks() assumes
> * that the read-side critical sections end at a voluntary context
> - * switch (not a preemption!), cond_resched_tasks_rcu_qs(), entry into idle,
> - * or transition to usermode execution. As such, there are no read-side
> - * primitives analogous to rcu_read_lock() and rcu_read_unlock() because
> - * this primitive is intended to determine that all tasks have passed
> - * through a safe state, not so much for data-structure synchronization.
> + * switch (not a preemption!), cond_resched_tasks_rcu_qs(), entry into
> + * RCU-idle context or transition to usermode execution. As such, there
> + * are no read-side primitives analogous to rcu_read_lock() and
> + * rcu_read_unlock() because this primitive is intended to determine
> + * that all tasks have passed through a safe state, not so much for
> + * data-structure synchronization.
> *
> * See the description of call_rcu() for more detailed information on
> * memory ordering guarantees.
> @@ -1193,8 +1229,9 @@ EXPORT_SYMBOL_GPL(call_rcu_tasks);
> * grace period has elapsed, in other words after all currently
> * executing rcu-tasks read-side critical sections have elapsed. These
> * read-side critical sections are delimited by calls to schedule(),
> - * cond_resched_tasks_rcu_qs(), idle execution, userspace execution, calls
> - * to synchronize_rcu_tasks(), and (in theory, anyway) cond_resched().
> + * cond_resched_tasks_rcu_qs(), idle execution within RCU-idle context,
> + * userspace execution, calls to synchronize_rcu_tasks(), and (in theory,
> + * anyway) cond_resched().
> *
> * This is a very specialized primitive, intended only for a few uses in
> * tracing and other situations requiring manipulation of function
> diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c
> index 930846f06bee..1f1574200bea 100644
> --- a/kernel/rcu/tree.c
> +++ b/kernel/rcu/tree.c
> @@ -303,7 +303,7 @@ static void rcu_watching_online(void)
> * Return true if the snapshot returned from ct_rcu_watching()
> * indicates that RCU is in an extended quiescent state.
> */
I suggest breaking this commit up. One commit could just be the below
change in rcu_watching_snap_in_eqs(). Another could be the addition on
->rcu_watching_snap. And a third would be the actual change in Tasks
RCU semantics. This would make it a bit easier to review, less stuff
to track on each commit.
> -static bool rcu_watching_snap_in_eqs(int snap)
> +bool rcu_watching_snap_in_eqs(int snap)
> {
> return !(snap & CT_RCU_WATCHING);
> }
> @@ -312,16 +312,16 @@ static bool rcu_watching_snap_in_eqs(int snap)
> * rcu_watching_snap_stopped_since() - Has RCU stopped watching a given CPU
> * since the specified @snap?
> *
> - * @rdp: The rcu_data corresponding to the CPU for which to check EQS.
> + * @cpu: The CPU for which to check EQS.
> * @snap: rcu_watching snapshot taken when the CPU wasn't in an EQS.
> *
> - * Returns true if the CPU corresponding to @rdp has spent some time in an
> + * Returns true if the CPU corresponding has spent some time in an
> * extended quiescent state since @snap. Note that this doesn't check if it
> * /still/ is in an EQS, just that it went through one since @snap.
> *
> * This is meant to be used in a loop waiting for a CPU to go through an EQS.
> */
> -static bool rcu_watching_snap_stopped_since(struct rcu_data *rdp, int snap)
> +bool rcu_watching_snap_stopped_since(int cpu, int snap)
> {
> /*
> * The first failing snapshot is already ordered against the accesses
> @@ -334,7 +334,7 @@ static bool rcu_watching_snap_stopped_since(struct rcu_data *rdp, int snap)
> if (WARN_ON_ONCE(rcu_watching_snap_in_eqs(snap)))
> return true;
>
> - return snap != ct_rcu_watching_cpu_acquire(rdp->cpu);
> + return snap != ct_rcu_watching_cpu_acquire(cpu);
> }
>
> /*
> @@ -826,7 +826,7 @@ static int rcu_watching_snap_recheck(struct rcu_data *rdp)
> * read-side critical section that started before the beginning
> * of the current RCU grace period.
> */
> - if (rcu_watching_snap_stopped_since(rdp, rdp->watching_snap)) {
> + if (rcu_watching_snap_stopped_since(rdp->cpu, rdp->watching_snap)) {
> trace_rcu_fqs(rcu_state.name, rdp->gp_seq, rdp->cpu, TPS("dti"));
> rcu_gpnum_ovf(rnp, rdp);
> return 1;
> diff --git a/kernel/rcu/tree_exp.h b/kernel/rcu/tree_exp.h
> index c3266bf709d5..6202d321b2bb 100644
> --- a/kernel/rcu/tree_exp.h
> +++ b/kernel/rcu/tree_exp.h
> @@ -400,7 +400,7 @@ static void __sync_rcu_exp_select_node_cpus(struct rcu_exp_work *rewp)
> unsigned long mask = rdp->grpmask;
>
> retry_ipi:
> - if (rcu_watching_snap_stopped_since(rdp, rdp->exp_watching_snap)) {
> + if (rcu_watching_snap_stopped_since(rdp->cpu, rdp->exp_watching_snap)) {
> mask_ofl_test |= mask;
> continue;
> }
> --
> 2.40.1
>
^ permalink raw reply [flat|nested] 7+ messages in thread* Re: [PATCH 1/2] rcu-tasks: Make RCU-Tasks pay attention to idle tasks
2024-08-07 16:35 ` [PATCH 1/2] rcu-tasks: Make RCU-Tasks pay attention to " neeraj.upadhyay
2024-08-08 17:59 ` Paul E. McKenney
@ 2024-08-14 15:01 ` Frederic Weisbecker
1 sibling, 0 replies; 7+ messages in thread
From: Frederic Weisbecker @ 2024-08-14 15:01 UTC (permalink / raw)
To: neeraj.upadhyay
Cc: rcu, linux-kernel, paulmck, joel, boqun.feng, urezki, rostedt,
mathieu.desnoyers, jiangshanlai, qiang.zhang1211, neeraj.iitr10
Le Wed, Aug 07, 2024 at 10:05:05PM +0530, neeraj.upadhyay@kernel.org a écrit :
> @@ -1082,10 +1115,13 @@ static void rcu_tasks_postscan(struct list_head *hop)
> static void check_holdout_task(struct task_struct *t,
> bool needreport, bool *firstreport)
> {
> - int cpu;
> + int cpu = task_cpu(t);
> + struct rcu_tasks_percpu *rtpcp = per_cpu_ptr(rcu_tasks.rtpcpu, cpu);
>
> if (!READ_ONCE(t->rcu_tasks_holdout) ||
> t->rcu_tasks_nvcsw != READ_ONCE(t->nvcsw) ||
> + ((t == idle_task(cpu)) &&
> + rcu_watching_snap_stopped_since(cpu, rtpcp->rcu_watching_snap)) ||
> !rcu_tasks_is_holdout(t) ||
There is a potential issue, though unlikely but still RCU-TASKS can be used on
early boot. Related kthreads are even initialized before pre-smp initcalls:
// init/1 running
do_pre_smp_initcall()
random_initcall()
schedule_work(random_work)
smp_init()
// create idle task for CPU 1
fork_idle()
// create idle task, add to tasklist
copy_process()
// sched_switch init/1 ===> kworker
random_work()
synchronize_rcu_tasks()
// sched_switch kworker ===> rcu_tasks
...
rcu_tasks_wait_gp()
for_each_process_thread()
rcu_tasks_per_task()
// sees idle_task for CPU 1
// but is not idle_task(CPU 1) yet
// because init_idle() hasn't been called
list_add(&t->rcu_tasks_holdout_list, hop);
...
// sched_switch rcu_tasks ===> init/1
init_idle()
rq->idle = idle task of CPU 1
// sched_switch init/1 ===> rcu_tasks
check_all_holdout_tasks()
check_holdout_task()
// now it verifies idle_task() but there is no snapshot
// so rcu_watching_snap_stopped_since() should righfully warn
if (t == idle_task(cpu) && rcu_watching_snap_stopped_since(cpu, rtpcp->rcu_watching_snap))
...
Would something like this work? (beware, untested!)
diff --git a/Documentation/RCU/Design/Requirements/Requirements.rst b/Documentation/RCU/Design/Requirements/Requirements.rst
index 6125e7068d2c..5016b85d53d7 100644
--- a/Documentation/RCU/Design/Requirements/Requirements.rst
+++ b/Documentation/RCU/Design/Requirements/Requirements.rst
@@ -2611,8 +2611,8 @@ critical sections that are delimited by voluntary context switches, that
is, calls to schedule(), cond_resched(), and
synchronize_rcu_tasks(). In addition, transitions to and from
userspace execution also delimit tasks-RCU read-side critical sections.
-Idle tasks are ignored by Tasks RCU, and Tasks Rude RCU may be used to
-interact with them.
+Idle tasks which are idle from RCU's perspective are ignored by Tasks RCU,
+and Tasks Rude RCU may be used to interact with them.
Note well that involuntary context switches are *not* Tasks-RCU quiescent
states. After all, in preemptible kernels, a task executing code in a
@@ -2643,10 +2643,10 @@ moniker. And this operation is considered to be quite rude by real-time
workloads that don't want their ``nohz_full`` CPUs receiving IPIs and
by battery-powered systems that don't want their idle CPUs to be awakened.
-Once kernel entry/exit and deep-idle functions have been properly tagged
-``noinstr``, Tasks RCU can start paying attention to idle tasks (except
-those that are idle from RCU's perspective) and then Tasks Rude RCU can
-be removed from the kernel.
+As Tasks RCU now pays attention to idle tasks (except those that are idle
+from RCU's perspective), once kernel entry/exit and deep-idle functions have
+been properly tagged ``noinstr``, Tasks Rude RCU can be removed from the
+kernel.
The tasks-rude-RCU API is also reader-marking-free and thus quite compact,
consisting solely of synchronize_rcu_tasks_rude().
diff --git a/kernel/rcu/rcu.h b/kernel/rcu/rcu.h
index 2909662c805f..ef5f391924f6 100644
--- a/kernel/rcu/rcu.h
+++ b/kernel/rcu/rcu.h
@@ -609,6 +609,8 @@ void srcutorture_get_gp_data(struct srcu_struct *sp, int *flags,
#ifdef CONFIG_TINY_RCU
static inline bool rcu_watching_zero_in_eqs(int cpu, int *vp) { return false; }
+static inline bool rcu_watching_snap_in_eqs(int snap) { return false; }
+static inline bool rcu_watching_snap_stopped_since(int cpu, int snap) { return false; }
static inline unsigned long rcu_get_gp_seq(void) { return 0; }
static inline unsigned long rcu_exp_batches_completed(void) { return 0; }
static inline void rcu_force_quiescent_state(void) { }
@@ -620,6 +622,8 @@ static inline void rcu_gp_slow_register(atomic_t *rgssp) { }
static inline void rcu_gp_slow_unregister(atomic_t *rgssp) { }
#else /* #ifdef CONFIG_TINY_RCU */
bool rcu_watching_zero_in_eqs(int cpu, int *vp);
+bool rcu_watching_snap_in_eqs(int snap);
+bool rcu_watching_snap_stopped_since(int cpu, int snap);
unsigned long rcu_get_gp_seq(void);
unsigned long rcu_exp_batches_completed(void);
bool rcu_check_boost_fail(unsigned long gp_state, int *cpup);
diff --git a/kernel/rcu/tasks.h b/kernel/rcu/tasks.h
index 4e913e5ca737..adc86624c92b 100644
--- a/kernel/rcu/tasks.h
+++ b/kernel/rcu/tasks.h
@@ -35,6 +35,7 @@ typedef void (*postgp_func_t)(struct rcu_tasks *rtp);
* @rtp_exit_list: List of tasks in the latter portion of do_exit().
* @cpu: CPU number corresponding to this entry.
* @rtpp: Pointer to the rcu_tasks structure.
+ * @rcu_watching_snap: Per-GP rcu watching tracking for idle tasks.
*/
struct rcu_tasks_percpu {
struct rcu_segcblist cblist;
@@ -51,6 +52,8 @@ struct rcu_tasks_percpu {
int cpu;
int index;
struct rcu_tasks *rtpp;
+ int rcu_watching_snap;
+ int rcu_watching_snap_rec;
};
/**
@@ -902,14 +905,15 @@ static void rcu_tasks_wait_gp(struct rcu_tasks *rtp)
////////////////////////////////////////////////////////////////////////
//
// Simple variant of RCU whose quiescent states are voluntary context
-// switch, cond_resched_tasks_rcu_qs(), user-space execution, and idle.
-// As such, grace periods can take one good long time. There are no
-// read-side primitives similar to rcu_read_lock() and rcu_read_unlock()
-// because this implementation is intended to get the system into a safe
-// state for some of the manipulations involved in tracing and the like.
-// Finally, this implementation does not support high call_rcu_tasks()
-// rates from multiple CPUs. If this is required, per-CPU callback lists
-// will be needed.
+// switch, cond_resched_tasks_rcu_qs(), user-space execution, and idle
+// tasks which are in RCU-idle context. As such, grace periods can take
+// one good long time. There are no read-side primitives similar to
+// rcu_read_lock() and rcu_read_unlock() because this implementation is
+// intended to get the system into a safe state for some of the
+// manipulations involved in tracing and the like. Finally, this
+// implementation does not support high call_rcu_tasks() rates from
+// multiple CPUs. If this is required, per-CPU callback lists will be
+// needed.
//
// The implementation uses rcu_tasks_wait_gp(), which relies on function
// pointers in the rcu_tasks structure. The rcu_spawn_tasks_kthread()
@@ -920,11 +924,13 @@ static void rcu_tasks_wait_gp(struct rcu_tasks *rtp)
// Invokes synchronize_rcu() in order to wait for all in-flight
// t->on_rq and t->nvcsw transitions to complete. This works because
// all such transitions are carried out with interrupts disabled.
-// rcu_tasks_pertask(), invoked on every non-idle task:
-// For every runnable non-idle task other than the current one, use
-// get_task_struct() to pin down that task, snapshot that task's
-// number of voluntary context switches, and add that task to the
-// holdout list.
+// rcu_tasks_pertask(), invoked on every task:
+// For idle task, snapshot the task's CPU's dynticks counter. If
+// the CPU is in RCU-idle context, do not add it to the holdout list.
+// For every runnable non-idle task other than the current one and
+// idle tasks which are not in RCU-idle context, use get_task_struct()
+// to pin down that task, snapshot that task's number of voluntary
+// context switches, and add that task to the holdout list.
// rcu_tasks_postscan():
// Gather per-CPU lists of tasks in do_exit() to ensure that all
// tasks that were in the process of exiting (and which thus might
@@ -974,30 +980,65 @@ static void rcu_tasks_pregp_step(struct list_head *hop)
synchronize_rcu();
}
-/* Check for quiescent states since the pregp's synchronize_rcu() */
-static bool rcu_tasks_is_holdout(struct task_struct *t)
+void call_rcu_tasks(struct rcu_head *rhp, rcu_callback_t func);
+DEFINE_RCU_TASKS(rcu_tasks, rcu_tasks_wait_gp, call_rcu_tasks, "RCU Tasks");
+
+#ifndef CONFIG_TINY_RCU
+static bool rcu_idle_task_is_holdout(struct task_struct *t)
{
- int cpu;
+ int cpu = task_cpu(t);
- /* Has the task been seen voluntarily sleeping? */
- if (!READ_ONCE(t->on_rq))
- return false;
+ if (t == idle_task(cpu)) {
+ struct rcu_tasks_percpu *rtpcp = per_cpu_ptr(rcu_tasks.rtpcpu, cpu);
- /*
- * Idle tasks (or idle injection) within the idle loop are RCU-tasks
- * quiescent states. But CPU boot code performed by the idle task
- * isn't a quiescent state.
- */
- if (is_idle_task(t))
- return false;
+ /* Idle tasks on offline CPUs are RCU-tasks quiescent states. */
+ if (!rcu_cpu_online(cpu))
+ return false;
+#ifdef CONFIG_SMP
+ /*
+ * As idle tasks cannot be involuntary preempted, non-running idle tasks
+ * are not in RCU-tasks critical section.
+ * synchronize_rcu() calls in rcu_tasks_pregp_step() and rcu_tasks_postgp()
+ * ensure that all ->on_cpu transitions are complete.
+ */
+ if (!t->on_cpu)
+ return false;
+#endif
+ if (!rtpcp->rcu_watching_snap_rec) {
+ /*
+ * Do plain access. Ordering between remote CPU's pre idle accesses
+ * and post rcu-tasks grace period is provided by synchronize_rcu()
+ * in rcu_tasks_postgp().
+ */
+ rtpcp->rcu_watching_snap = ct_rcu_watching_cpu(cpu);
+ /* RCU-idle contexts are RCU-tasks quiescent state for idle tasks. */
+ if (!rcu_watching_snap_in_eqs(rtpcp->rcu_watching_snap)) {
+ rtpcp->rcu_watching_snap_rec = 1;
+ return false;
+ }
+ } else if (rcu_watching_snap_stopped_since(cpu, rtpcp->rcu_watching_snap)) {
+ return false;
+ }
+ }
- cpu = task_cpu(t);
+ return true;
+}
+
+#else /* #ifndef CONFIG_TINY_RCU */
+static inline bool rcu_idle_task_is_holdout(struct task_struct *t)
+{
+ return true;
+}
+#endif
- /* Idle tasks on offline CPUs are RCU-tasks quiescent states. */
- if (t == idle_task(cpu) && !rcu_cpu_online(cpu))
+/* Check for quiescent states since the pregp's synchronize_rcu() */
+static bool rcu_tasks_is_holdout(struct task_struct *t)
+{
+ /* Has the task been seen voluntarily sleeping? */
+ if (!READ_ONCE(t->on_rq))
return false;
- return true;
+ return rcu_idle_task_is_holdout(t);
}
/* Per-task initial processing. */
@@ -1011,9 +1052,6 @@ static void rcu_tasks_pertask(struct task_struct *t, struct list_head *hop)
}
}
-void call_rcu_tasks(struct rcu_head *rhp, rcu_callback_t func);
-DEFINE_RCU_TASKS(rcu_tasks, rcu_tasks_wait_gp, call_rcu_tasks, "RCU Tasks");
-
/* Processing between scanning taskslist and draining the holdout list. */
static void rcu_tasks_postscan(struct list_head *hop)
{
@@ -1125,6 +1162,14 @@ static void check_all_holdout_tasks(struct list_head *hop,
/* Finish off the Tasks-RCU grace period. */
static void rcu_tasks_postgp(struct rcu_tasks *rtp)
{
+ int cpu;
+
+ /* Cleanup watching snapshots for next runs */
+ for_each_possible_cpu(cpu) {
+ struct rcu_tasks_percpu *rtpcp = per_cpu_ptr(rcu_tasks.rtpcpu, cpu);
+ rtpcp->rcu_watching_snap_rec = 0;
+ }
+
/*
* Because ->on_rq and ->nvcsw are not guaranteed to have a full
* memory barriers prior to them in the schedule() path, memory
@@ -1171,11 +1216,12 @@ static void tasks_rcu_exit_srcu_stall(struct timer_list *unused)
* period elapses, in other words after all currently executing RCU
* read-side critical sections have completed. call_rcu_tasks() assumes
* that the read-side critical sections end at a voluntary context
- * switch (not a preemption!), cond_resched_tasks_rcu_qs(), entry into idle,
- * or transition to usermode execution. As such, there are no read-side
- * primitives analogous to rcu_read_lock() and rcu_read_unlock() because
- * this primitive is intended to determine that all tasks have passed
- * through a safe state, not so much for data-structure synchronization.
+ * switch (not a preemption!), cond_resched_tasks_rcu_qs(), entry into
+ * RCU-idle context or transition to usermode execution. As such, there
+ * are no read-side primitives analogous to rcu_read_lock() and
+ * rcu_read_unlock() because this primitive is intended to determine
+ * that all tasks have passed through a safe state, not so much for
+ * data-structure synchronization.
*
* See the description of call_rcu() for more detailed information on
* memory ordering guarantees.
@@ -1193,8 +1239,9 @@ EXPORT_SYMBOL_GPL(call_rcu_tasks);
* grace period has elapsed, in other words after all currently
* executing rcu-tasks read-side critical sections have elapsed. These
* read-side critical sections are delimited by calls to schedule(),
- * cond_resched_tasks_rcu_qs(), idle execution, userspace execution, calls
- * to synchronize_rcu_tasks(), and (in theory, anyway) cond_resched().
+ * cond_resched_tasks_rcu_qs(), idle execution within RCU-idle context,
+ * userspace execution, calls to synchronize_rcu_tasks(), and (in theory,
+ * anyway) cond_resched().
*
* This is a very specialized primitive, intended only for a few uses in
* tracing and other situations requiring manipulation of function
diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c
index 910d316dc27b..00cf65fbae7a 100644
--- a/kernel/rcu/tree.c
+++ b/kernel/rcu/tree.c
@@ -303,7 +303,7 @@ static void rcu_watching_online(void)
* Return true if the snapshot returned from ct_rcu_watching()
* indicates that RCU is in an extended quiescent state.
*/
-static bool rcu_watching_snap_in_eqs(int snap)
+bool rcu_watching_snap_in_eqs(int snap)
{
return !(snap & CT_RCU_WATCHING);
}
@@ -312,16 +312,16 @@ static bool rcu_watching_snap_in_eqs(int snap)
* rcu_watching_snap_stopped_since() - Has RCU stopped watching a given CPU
* since the specified @snap?
*
- * @rdp: The rcu_data corresponding to the CPU for which to check EQS.
+ * @cpu: The CPU for which to check EQS.
* @snap: rcu_watching snapshot taken when the CPU wasn't in an EQS.
*
- * Returns true if the CPU corresponding to @rdp has spent some time in an
+ * Returns true if the CPU corresponding has spent some time in an
* extended quiescent state since @snap. Note that this doesn't check if it
* /still/ is in an EQS, just that it went through one since @snap.
*
* This is meant to be used in a loop waiting for a CPU to go through an EQS.
*/
-static bool rcu_watching_snap_stopped_since(struct rcu_data *rdp, int snap)
+bool rcu_watching_snap_stopped_since(int cpu, int snap)
{
/*
* The first failing snapshot is already ordered against the accesses
@@ -334,7 +334,7 @@ static bool rcu_watching_snap_stopped_since(struct rcu_data *rdp, int snap)
if (WARN_ON_ONCE(rcu_watching_snap_in_eqs(snap)))
return true;
- return snap != ct_rcu_watching_cpu_acquire(rdp->cpu);
+ return snap != ct_rcu_watching_cpu_acquire(cpu);
}
/*
@@ -826,7 +826,7 @@ static int rcu_watching_snap_recheck(struct rcu_data *rdp)
* read-side critical section that started before the beginning
* of the current RCU grace period.
*/
- if (rcu_watching_snap_stopped_since(rdp, rdp->watching_snap)) {
+ if (rcu_watching_snap_stopped_since(rdp->cpu, rdp->watching_snap)) {
trace_rcu_fqs(rcu_state.name, rdp->gp_seq, rdp->cpu, TPS("dti"));
rcu_gpnum_ovf(rnp, rdp);
return 1;
diff --git a/kernel/rcu/tree_exp.h b/kernel/rcu/tree_exp.h
index c3266bf709d5..6202d321b2bb 100644
--- a/kernel/rcu/tree_exp.h
+++ b/kernel/rcu/tree_exp.h
@@ -400,7 +400,7 @@ static void __sync_rcu_exp_select_node_cpus(struct rcu_exp_work *rewp)
unsigned long mask = rdp->grpmask;
retry_ipi:
- if (rcu_watching_snap_stopped_since(rdp, rdp->exp_watching_snap)) {
+ if (rcu_watching_snap_stopped_since(rdp->cpu, rdp->exp_watching_snap)) {
mask_ofl_test |= mask;
continue;
}
^ permalink raw reply related [flat|nested] 7+ messages in thread