[PATCH v2] rcu-tasks: Avoid alloc_percpu() failure in rcu_tasks_rude_wait_gp()

[Zqiang <qiang.zhang@linux.dev>]

The RCU Tasks Rude grace period is implemented by invoking the
schedule_on_each_cpu() with an empty work function, which forces a context
switch on every online CPU and guarantees that all RCU Tasks Rude read
critical sections have completed. However, schedule_on_each_cpu() can
fail (via alloc_percpu() returning -ENOMEM) and returns immediately, in
which case those context switches never happen and also not wait for tasks
to exit the RCU Tasks Rude read critical section, too-short grace period
happen.

This commit therefore replace schedule_on_each_cpu() with a statically
defined per-CPU work_struct, which is allocated at compile time and cannot
fail in this path. rcu_tasks_rude_wait_gp() now holds CPU-hotplug readlock,
schedules rcu_tasks_be_rude() on each online CPUs, and flush each work item
before returning. This preserves the original behavior of the synchronous
API while removing both the runtime allocation.

Suggested-by: Bradley Morgan <brads@mainlining.org> 
Signed-off-by: Zqiang <qiang.zhang@linux.dev>
---
 kernel/rcu/tasks.h | 30 +++++++++++++++++++++++++-----
 1 file changed, 25 insertions(+), 5 deletions(-)

diff --git a/kernel/rcu/tasks.h b/kernel/rcu/tasks.h
index fcac7361ec51..a59aa8b6ab05 100644
--- a/kernel/rcu/tasks.h
+++ b/kernel/rcu/tasks.h
@@ -1342,11 +1342,17 @@ void exit_tasks_rcu_finish(void) { }
 ////////////////////////////////////////////////////////////////////////
 //
 // "Rude" variant of Tasks RCU, inspired by Steve Rostedt's
-// trick of passing an empty function to schedule_on_each_cpu().
+// trick of passing an empty function to per-CPUs workqueue.
 // This approach provides batching of concurrent calls to the synchronous
-// synchronize_rcu_tasks_rude() API.  This invokes schedule_on_each_cpu()
-// in order to send IPIs far and wide and induces otherwise unnecessary
-// context switches on all online CPUs, whether idle or not.
+// synchronize_rcu_tasks_rude() API.  A no-op work item is scheduled on
+// each online CPUs, in order to send IPIs far and wide and induces otherwise
+// unnecessary context switches on all online CPUs, whether idle or not.
+//
+// A statically defined per-CPU work_struct is used instead of a
+// runtime-allocated one, which can fail under memory pressure. Each
+// CPU's work item is scheduled with schedule_work_on() and waited for
+// with flush_work(), under the read side of the CPU-hotplug lock so the
+// set of online CPUs cannot change mid-iteration.
 //
 // Callback handling is provided by the rcu_tasks_kthread() function.
 //
@@ -1357,11 +1363,25 @@ static void rcu_tasks_be_rude(struct work_struct *work)
 {
 }
 
+static DEFINE_PER_CPU(struct work_struct, tasks_rude_work);
+
 // Wait for one rude RCU-tasks grace period.
 static void rcu_tasks_rude_wait_gp(struct rcu_tasks *rtp)
 {
+	int cpu;
+
+	guard(cpus_read_lock)();
 	rtp->n_ipis += cpumask_weight(cpu_online_mask);
-	schedule_on_each_cpu(rcu_tasks_be_rude);
+
+	for_each_online_cpu(cpu) {
+		struct work_struct *work = &per_cpu(tasks_rude_work, cpu);
+
+		INIT_WORK(work, rcu_tasks_be_rude);
+		schedule_work_on(cpu, work);
+	}
+
+	for_each_online_cpu(cpu)
+		flush_work(&per_cpu(tasks_rude_work, cpu));
 }
 
 static void call_rcu_tasks_rude(struct rcu_head *rhp, rcu_callback_t func);
-- 
2.17.1