[PATCH v2] rcu-tasks: Avoid alloc_percpu() failure in rcu_tasks_rude_wait_gp()

[PATCH v2] rcu-tasks: Avoid alloc_percpu() failure in rcu_tasks_rude_wait_gp()

[Zqiang]

The RCU Tasks Rude grace period is implemented by invoking the
schedule_on_each_cpu() with an empty work function, which forces a context
switch on every online CPU and guarantees that all RCU Tasks Rude read
critical sections have completed. However, schedule_on_each_cpu() can
fail (via alloc_percpu() returning -ENOMEM) and returns immediately, in
which case those context switches never happen and also not wait for tasks
to exit the RCU Tasks Rude read critical section, too-short grace period
happen.

This commit therefore replace schedule_on_each_cpu() with a statically
defined per-CPU work_struct, which is allocated at compile time and cannot
fail in this path. rcu_tasks_rude_wait_gp() now holds CPU-hotplug readlock,
schedules rcu_tasks_be_rude() on each online CPUs, and flush each work item
before returning. This preserves the original behavior of the synchronous
API while removing both the runtime allocation.

Suggested-by: Bradley Morgan <brads@mainlining.org> 
Signed-off-by: Zqiang <qiang.zhang@linux.dev>
---
 kernel/rcu/tasks.h | 30 +++++++++++++++++++++++++-----
 1 file changed, 25 insertions(+), 5 deletions(-)

diff --git a/kernel/rcu/tasks.h b/kernel/rcu/tasks.h
index fcac7361ec51..a59aa8b6ab05 100644
--- a/kernel/rcu/tasks.h
+++ b/kernel/rcu/tasks.h
@@ -1342,11 +1342,17 @@ void exit_tasks_rcu_finish(void) { }
 ////////////////////////////////////////////////////////////////////////
 //
 // "Rude" variant of Tasks RCU, inspired by Steve Rostedt's
-// trick of passing an empty function to schedule_on_each_cpu().
+// trick of passing an empty function to per-CPUs workqueue.
 // This approach provides batching of concurrent calls to the synchronous
-// synchronize_rcu_tasks_rude() API.  This invokes schedule_on_each_cpu()
-// in order to send IPIs far and wide and induces otherwise unnecessary
-// context switches on all online CPUs, whether idle or not.
+// synchronize_rcu_tasks_rude() API.  A no-op work item is scheduled on
+// each online CPUs, in order to send IPIs far and wide and induces otherwise
+// unnecessary context switches on all online CPUs, whether idle or not.
+//
+// A statically defined per-CPU work_struct is used instead of a
+// runtime-allocated one, which can fail under memory pressure. Each
+// CPU's work item is scheduled with schedule_work_on() and waited for
+// with flush_work(), under the read side of the CPU-hotplug lock so the
+// set of online CPUs cannot change mid-iteration.
 //
 // Callback handling is provided by the rcu_tasks_kthread() function.
 //
@@ -1357,11 +1363,25 @@ static void rcu_tasks_be_rude(struct work_struct *work)
 {
 }
 
+static DEFINE_PER_CPU(struct work_struct, tasks_rude_work);
+
 // Wait for one rude RCU-tasks grace period.
 static void rcu_tasks_rude_wait_gp(struct rcu_tasks *rtp)
 {
+	int cpu;
+
+	guard(cpus_read_lock)();
 	rtp->n_ipis += cpumask_weight(cpu_online_mask);
-	schedule_on_each_cpu(rcu_tasks_be_rude);
+
+	for_each_online_cpu(cpu) {
+		struct work_struct *work = &per_cpu(tasks_rude_work, cpu);
+
+		INIT_WORK(work, rcu_tasks_be_rude);
+		schedule_work_on(cpu, work);
+	}
+
+	for_each_online_cpu(cpu)
+		flush_work(&per_cpu(tasks_rude_work, cpu));
 }
 
 static void call_rcu_tasks_rude(struct rcu_head *rhp, rcu_callback_t func);
-- 
2.17.1

[PATCH v2] doc: Update recent RCU/SRCU API additions to whatisRCU.rst

[Zqiang]

The following commit introduces some new RCU/SRCU APIs:

Commit 'd3f52f53a562 ("srcu: Create an SRCU-fast-updown API")'
Commit 'dab06d6f9379 ("srcutree: Add an atomic Tree SRCU")'
Commit 'cacadb630375 ("srcu: Add guards for notrace variants of SRCU-fast readers")'
Commit 'fa8a2d55139f ("srcu: Add lock guard for srcu_fast_updown flavor")'
Commit '954c0d741299 ("srcu: Add guards for SRCU-fast readers")'

This commit therefore update thems and some missing RCU related APIs
to whatisRCU.rst.

Signed-off-by: Zqiang <qiang.zhang@linux.dev>
---
 Documentation/RCU/whatisRCU.rst | 24 +++++++++++++++++++++---
 1 file changed, 21 insertions(+), 3 deletions(-)

diff --git a/Documentation/RCU/whatisRCU.rst b/Documentation/RCU/whatisRCU.rst
index e6767baff2d3..fe9dc3948690 100644
--- a/Documentation/RCU/whatisRCU.rst
+++ b/Documentation/RCU/whatisRCU.rst
@@ -1084,8 +1084,9 @@ RCU::
 	rcu_dereference			call_rcu
 	rcu_dereference_check		call_rcu_hurry
 	rcu_dereference_protected	kfree_rcu
-	rcu_read_lock_held		kvfree_rcu
-	rcu_read_lock_any_held		kfree_rcu_mightsleep
+	rcu_dereference_all		kvfree_rcu
+	rcu_read_lock_held		kfree_rcu_mightsleep
+	rcu_read_lock_any_held		kvfree_rcu_mightsleep
 	rcu_pointer_handoff		cond_synchronize_rcu
 	unrcu_pointer			cond_synchronize_rcu_full
 					cond_synchronize_rcu_expedited
@@ -1209,13 +1210,23 @@ SRCU::
 	srcu_read_lock_nmisafe		start_poll_synchronize_srcu
 	srcu_read_unlock_nmisafe	start_poll_synchronize_srcu_expedited
 	srcu_read_lock_notrace		poll_state_synchronize_srcu
-	srcu_read_unlock_notrace
+	srcu_read_unlock_notrace	synchronize_srcu_atomic
 	srcu_down_read
 	srcu_up_read
 	srcu_down_read_fast
 	srcu_up_read_fast
+	srcu_read_lock_fast_updown
+	srcu_read_unlock_fast_updown
+	srcu_read_lock_atomic
+	srcu_read_unlock_atomic
 	guard(srcu)()
+	guard(srcu_fast)()
+	guard(srcu_fast_notrace)()
+	guard(srcu_fast_updown)()
 	scoped_guard(srcu)
+	scoped_guard(srcu_fast)
+	scoped_guard(srcu_fast_notrace)
+	scoped_guard(srcu_fast_updown)
 	srcu_read_lock_held
 	srcu_dereference
 	srcu_dereference_check
@@ -1229,9 +1240,16 @@ SRCU: Initialization/cleanup/ordering::
 	DEFINE_STATIC_SRCU
 	DEFINE_SRCU_FAST        // for srcu_read_lock_fast() and friends
 	DEFINE_STATIC_SRCU_FAST // for srcu_read_lock_fast() and friends
+	DEFINE_SRCU_FAST_UPDOWN		// for srcu_read_lock_fast_updown() and friends
+	DEFINE_STATIC_SRCU_FAST_UPDOWN	// for srcu_read_lock_fast_updown() and friends
+	DEFINE_SRCU_ATOMIC		// for srcu_read_lock_atomic() and friends
+	DEFINE_STATIC_SRCU_ATOMIC	// for srcu_read_lock_atomic() and friends
 	init_srcu_struct
 	init_srcu_struct_fast
+	init_srcu_struct_fast_updown
+	init_srcu_struct_atomic
 	cleanup_srcu_struct
+	smp_mb__after_srcu_read_lock
 	smp_mb__after_srcu_read_unlock
 
 All: lockdep-checked RCU utility APIs::
-- 
2.17.1

Re: [PATCH v2] doc: Update recent RCU/SRCU API additions to whatisRCU.rst

[Bradley Morgan]

On 19 October 2026 13:22:09 BST, Zqiang <qiang.zhang@linux.dev> wrote:
>The following commit introduces some new RCU/SRCU APIs:
>
>Commit 'd3f52f53a562 ("srcu: Create an SRCU-fast-updown API")'
>Commit 'dab06d6f9379 ("srcutree: Add an atomic Tree SRCU")'
>Commit 'cacadb630375 ("srcu: Add guards for notrace variants of SRCU-fast
>readers")'
>Commit 'fa8a2d55139f ("srcu: Add lock guard for srcu_fast_updown flavor")'
>Commit '954c0d741299 ("srcu: Add guards for SRCU-fast readers")'
>
>This commit therefore update thems and some missing RCU related APIs
>to whatisRCU.rst.
>

Since this relates to my patch, and it looks good:

Acked-by: Bradley Morgan <brads@mainlining.org>


>Signed-off-by: Zqiang <qiang.zhang@linux.dev>
>---
> Documentation/RCU/whatisRCU.rst | 24 +++++++++++++++++++++---
> 1 file changed, 21 insertions(+), 3 deletions(-)
>
>diff --git a/Documentation/RCU/whatisRCU.rst b/Documentation/RCU/whatisRCU.rst
>index e6767baff2d3..fe9dc3948690 100644
>--- a/Documentation/RCU/whatisRCU.rst
>+++ b/Documentation/RCU/whatisRCU.rst
>@@ -1084,8 +1084,9 @@ RCU::
> 	rcu_dereference			call_rcu
> 	rcu_dereference_check		call_rcu_hurry
> 	rcu_dereference_protected	kfree_rcu
>-	rcu_read_lock_held		kvfree_rcu
>-	rcu_read_lock_any_held		kfree_rcu_mightsleep
>+	rcu_dereference_all		kvfree_rcu
>+	rcu_read_lock_held		kfree_rcu_mightsleep
>+	rcu_read_lock_any_held		kvfree_rcu_mightsleep
> 	rcu_pointer_handoff		cond_synchronize_rcu
> 	unrcu_pointer			cond_synchronize_rcu_full
> 					cond_synchronize_rcu_expedited
>@@ -1209,13 +1210,23 @@ SRCU::
> 	srcu_read_lock_nmisafe		start_poll_synchronize_srcu
> 	srcu_read_unlock_nmisafe	start_poll_synchronize_srcu_expedited
> 	srcu_read_lock_notrace		poll_state_synchronize_srcu
>-	srcu_read_unlock_notrace
>+	srcu_read_unlock_notrace	synchronize_srcu_atomic
> 	srcu_down_read
> 	srcu_up_read
> 	srcu_down_read_fast
> 	srcu_up_read_fast
>+	srcu_read_lock_fast_updown
>+	srcu_read_unlock_fast_updown
>+	srcu_read_lock_atomic
>+	srcu_read_unlock_atomic
> 	guard(srcu)()
>+	guard(srcu_fast)()
>+	guard(srcu_fast_notrace)()
>+	guard(srcu_fast_updown)()
> 	scoped_guard(srcu)
>+	scoped_guard(srcu_fast)
>+	scoped_guard(srcu_fast_notrace)
>+	scoped_guard(srcu_fast_updown)
> 	srcu_read_lock_held
> 	srcu_dereference
> 	srcu_dereference_check
>@@ -1229,9 +1240,16 @@ SRCU: Initialization/cleanup/ordering::
> 	DEFINE_STATIC_SRCU
> 	DEFINE_SRCU_FAST        // for srcu_read_lock_fast() and friends
> 	DEFINE_STATIC_SRCU_FAST // for srcu_read_lock_fast() and friends
>+	DEFINE_SRCU_FAST_UPDOWN		// for srcu_read_lock_fast_updown() and friends
>+	DEFINE_STATIC_SRCU_FAST_UPDOWN	// for srcu_read_lock_fast_updown() and friends
>+	DEFINE_SRCU_ATOMIC		// for srcu_read_lock_atomic() and friends
>+	DEFINE_STATIC_SRCU_ATOMIC	// for srcu_read_lock_atomic() and friends
> 	init_srcu_struct
> 	init_srcu_struct_fast
>+	init_srcu_struct_fast_updown
>+	init_srcu_struct_atomic
> 	cleanup_srcu_struct
>+	smp_mb__after_srcu_read_lock
> 	smp_mb__after_srcu_read_unlock
> 
> All: lockdep-checked RCU utility APIs::
>

--- Thanks!
"I'm not a very positive person" - Linus torvalds

Re: [PATCH v2] doc: Update recent RCU/SRCU API additions to whatisRCU.rst

[Paul E. McKenney]

On Mon, Oct 19, 2026 at 08:22:09PM +0800, Zqiang wrote:
> The following commit introduces some new RCU/SRCU APIs:
> 
> Commit 'd3f52f53a562 ("srcu: Create an SRCU-fast-updown API")'
> Commit 'dab06d6f9379 ("srcutree: Add an atomic Tree SRCU")'
> Commit 'cacadb630375 ("srcu: Add guards for notrace variants of SRCU-fast readers")'
> Commit 'fa8a2d55139f ("srcu: Add lock guard for srcu_fast_updown flavor")'
> Commit '954c0d741299 ("srcu: Add guards for SRCU-fast readers")'
> 
> This commit therefore update thems and some missing RCU related APIs
> to whatisRCU.rst.
> 
> Signed-off-by: Zqiang <qiang.zhang@linux.dev>

Thank you, for both patches:

Reviewed-by: Paul E. McKenney <paulmck@kernel.org>

> ---
>  Documentation/RCU/whatisRCU.rst | 24 +++++++++++++++++++++---
>  1 file changed, 21 insertions(+), 3 deletions(-)
> 
> diff --git a/Documentation/RCU/whatisRCU.rst b/Documentation/RCU/whatisRCU.rst
> index e6767baff2d3..fe9dc3948690 100644
> --- a/Documentation/RCU/whatisRCU.rst
> +++ b/Documentation/RCU/whatisRCU.rst
> @@ -1084,8 +1084,9 @@ RCU::
>  	rcu_dereference			call_rcu
>  	rcu_dereference_check		call_rcu_hurry
>  	rcu_dereference_protected	kfree_rcu
> -	rcu_read_lock_held		kvfree_rcu
> -	rcu_read_lock_any_held		kfree_rcu_mightsleep
> +	rcu_dereference_all		kvfree_rcu
> +	rcu_read_lock_held		kfree_rcu_mightsleep
> +	rcu_read_lock_any_held		kvfree_rcu_mightsleep
>  	rcu_pointer_handoff		cond_synchronize_rcu
>  	unrcu_pointer			cond_synchronize_rcu_full
>  					cond_synchronize_rcu_expedited
> @@ -1209,13 +1210,23 @@ SRCU::
>  	srcu_read_lock_nmisafe		start_poll_synchronize_srcu
>  	srcu_read_unlock_nmisafe	start_poll_synchronize_srcu_expedited
>  	srcu_read_lock_notrace		poll_state_synchronize_srcu
> -	srcu_read_unlock_notrace
> +	srcu_read_unlock_notrace	synchronize_srcu_atomic
>  	srcu_down_read
>  	srcu_up_read
>  	srcu_down_read_fast
>  	srcu_up_read_fast
> +	srcu_read_lock_fast_updown
> +	srcu_read_unlock_fast_updown
> +	srcu_read_lock_atomic
> +	srcu_read_unlock_atomic
>  	guard(srcu)()
> +	guard(srcu_fast)()
> +	guard(srcu_fast_notrace)()
> +	guard(srcu_fast_updown)()
>  	scoped_guard(srcu)
> +	scoped_guard(srcu_fast)
> +	scoped_guard(srcu_fast_notrace)
> +	scoped_guard(srcu_fast_updown)
>  	srcu_read_lock_held
>  	srcu_dereference
>  	srcu_dereference_check
> @@ -1229,9 +1240,16 @@ SRCU: Initialization/cleanup/ordering::
>  	DEFINE_STATIC_SRCU
>  	DEFINE_SRCU_FAST        // for srcu_read_lock_fast() and friends
>  	DEFINE_STATIC_SRCU_FAST // for srcu_read_lock_fast() and friends
> +	DEFINE_SRCU_FAST_UPDOWN		// for srcu_read_lock_fast_updown() and friends
> +	DEFINE_STATIC_SRCU_FAST_UPDOWN	// for srcu_read_lock_fast_updown() and friends
> +	DEFINE_SRCU_ATOMIC		// for srcu_read_lock_atomic() and friends
> +	DEFINE_STATIC_SRCU_ATOMIC	// for srcu_read_lock_atomic() and friends
>  	init_srcu_struct
>  	init_srcu_struct_fast
> +	init_srcu_struct_fast_updown
> +	init_srcu_struct_atomic
>  	cleanup_srcu_struct
> +	smp_mb__after_srcu_read_lock
>  	smp_mb__after_srcu_read_unlock
>  
>  All: lockdep-checked RCU utility APIs::
> -- 
> 2.17.1
>