[PATCH for-v7.5 0/5] RCU core patches for v7.5

[PATCH for-v7.5 0/5] RCU core patches for v7.5

[Joel Fernandes]

This series has the RCU core patches for the v7.5 merge window.

Joel Fernandes (1):
  srcu: Select IRQ_WORK for Tiny SRCU

Song Liu (1):
  rcu-tasks: Use pr_debug() for callback contend/collapse messages

Vishal Chourasia (1):
  cpuhp: Expedite RCU when toggling system-wide SMT mode

Yury Norov (1):
  rcu: assert static storage for RCU sync and SRCU definitions

Zqiang (1):
  rcu-tasks: Avoid alloc_percpu() failure in rcu_tasks_rude_wait_gp()

 .../admin-guide/kernel-parameters.txt         |  7 ---
 include/linux/rcu_sync.h                      |  4 +-
 include/linux/rcupdate.h                      |  8 ++++
 include/linux/srcutiny.h                      |  4 +-
 include/linux/srcutree.h                      |  7 ++-
 kernel/cpu.c                                  |  4 ++
 kernel/rcu/Kconfig                            |  1 +
 kernel/rcu/rcu.h                              |  4 --
 kernel/rcu/tasks.h                            | 45 ++++++++++++-------
 9 files changed, 52 insertions(+), 32 deletions(-)

-- 
2.54.0

[PATCH for-v7.5 1/5] srcu: Select IRQ_WORK for Tiny SRCU

[Joel Fernandes]

Tiny SRCU uses irq_work_queue() to start grace periods and
irq_work_sync() to wait for pending work. However, TINY_SRCU does not
select IRQ_WORK, so configurations that do not otherwise enable it
fail to link with undefined references to these functions.

Guenter reported this failure with csky tinyconfig on the 6.12 and
6.6 stable trees. Current kernels enable IRQ_WORK indirectly through
SLUB, but Tiny SRCU should declare its own dependency.

Select IRQ_WORK from TINY_SRCU to ensure that its irq_work calls are
supported independently of other configuration options.

Fixes: a6fc88b22bc8 ("srcu: Use irq_work to start GP in tiny SRCU")
Reported-by: Guenter Roeck <linux@roeck-us.net>
Closes: https://lore.kernel.org/r/49da816c-8082-46f1-bb12-f0d6187bb06a@roeck-us.net/
Suggested-by: Guenter Roeck <linux@roeck-us.net>
Cc: stable@vger.kernel.org
Signed-off-by: Joel Fernandes <jfern@kernel.org>
---
 kernel/rcu/Kconfig | 1 +
 1 file changed, 1 insertion(+)

diff --git a/kernel/rcu/Kconfig b/kernel/rcu/Kconfig
index bc9483c0b5d8..b6d3597758fc 100644
--- a/kernel/rcu/Kconfig
+++ b/kernel/rcu/Kconfig
@@ -56,6 +56,7 @@ config RCU_EXPERT
 config TINY_SRCU
 	bool
 	default y if TINY_RCU
+	select IRQ_WORK
 	help
 	  This option selects the single-CPU non-preemptible version of SRCU.
 
-- 
2.54.0

[PATCH for-v7.5 2/5] rcu-tasks: Avoid alloc_percpu() failure in rcu_tasks_rude_wait_gp()

[Joel Fernandes]

From: Zqiang <qiang.zhang@linux.dev>

The RCU Tasks Rude grace period is implemented by invoking the
schedule_on_each_cpu() with an empty work function, which forces a context
switch on every online CPU and guarantees that all RCU Tasks Rude read
critical sections have completed. However, schedule_on_each_cpu() can
fail (via alloc_percpu() returning -ENOMEM) and returns immediately, in
which case those context switches never happen and also not wait for tasks
to exit the RCU Tasks Rude read critical section, too-short grace period
happen.

This commit therefore replace schedule_on_each_cpu() with a statically
defined per-CPU work_struct, which is allocated at compile time and cannot
fail in this path. rcu_tasks_rude_wait_gp() now holds CPU-hotplug readlock,
schedules rcu_tasks_be_rude() on each online CPUs, and flush each work item
before returning. This preserves the original behavior of the synchronous
API while removing both the runtime allocation.

Suggested-by: Bradley Morgan <brads@mainlining.org>
Signed-off-by: Zqiang <qiang.zhang@linux.dev>
Reviewed-by: Paul E. McKenney <paulmck@kernel.org>
Signed-off-by: Joel Fernandes <jfern@kernel.org>
---
 kernel/rcu/tasks.h | 30 +++++++++++++++++++++++++-----
 1 file changed, 25 insertions(+), 5 deletions(-)

diff --git a/kernel/rcu/tasks.h b/kernel/rcu/tasks.h
index fcac7361ec51..a59aa8b6ab05 100644
--- a/kernel/rcu/tasks.h
+++ b/kernel/rcu/tasks.h
@@ -1342,11 +1342,17 @@ void exit_tasks_rcu_finish(void) { }
 ////////////////////////////////////////////////////////////////////////
 //
 // "Rude" variant of Tasks RCU, inspired by Steve Rostedt's
-// trick of passing an empty function to schedule_on_each_cpu().
+// trick of passing an empty function to per-CPUs workqueue.
 // This approach provides batching of concurrent calls to the synchronous
-// synchronize_rcu_tasks_rude() API.  This invokes schedule_on_each_cpu()
-// in order to send IPIs far and wide and induces otherwise unnecessary
-// context switches on all online CPUs, whether idle or not.
+// synchronize_rcu_tasks_rude() API.  A no-op work item is scheduled on
+// each online CPUs, in order to send IPIs far and wide and induces otherwise
+// unnecessary context switches on all online CPUs, whether idle or not.
+//
+// A statically defined per-CPU work_struct is used instead of a
+// runtime-allocated one, which can fail under memory pressure. Each
+// CPU's work item is scheduled with schedule_work_on() and waited for
+// with flush_work(), under the read side of the CPU-hotplug lock so the
+// set of online CPUs cannot change mid-iteration.
 //
 // Callback handling is provided by the rcu_tasks_kthread() function.
 //
@@ -1357,11 +1363,25 @@ static void rcu_tasks_be_rude(struct work_struct *work)
 {
 }
 
+static DEFINE_PER_CPU(struct work_struct, tasks_rude_work);
+
 // Wait for one rude RCU-tasks grace period.
 static void rcu_tasks_rude_wait_gp(struct rcu_tasks *rtp)
 {
+	int cpu;
+
+	guard(cpus_read_lock)();
 	rtp->n_ipis += cpumask_weight(cpu_online_mask);
-	schedule_on_each_cpu(rcu_tasks_be_rude);
+
+	for_each_online_cpu(cpu) {
+		struct work_struct *work = &per_cpu(tasks_rude_work, cpu);
+
+		INIT_WORK(work, rcu_tasks_be_rude);
+		schedule_work_on(cpu, work);
+	}
+
+	for_each_online_cpu(cpu)
+		flush_work(&per_cpu(tasks_rude_work, cpu));
 }
 
 static void call_rcu_tasks_rude(struct rcu_head *rhp, rcu_callback_t func);
-- 
2.54.0

[PATCH for-v7.5 3/5] rcu: assert static storage for RCU sync and SRCU definitions

[Joel Fernandes]

From: Yury Norov <ynorov@nvidia.com>

DEFINE_RCU_SYNC() contains a statically initialized wait queue. SRCU
static definitions similarly rely on persistent lockdep keys and, in
Tiny SRCU, a statically initialized work item.

Add ASSERT_STATIC_STORAGE() to DEFINE_RCU_SYNC(), both Tree SRCU
__DEFINE_SRCU() variants and Tiny SRCU's common DEFINE_SRCU() declaration.
This covers the non-static FAST and FAST_UPDOWN wrappers too. Tiny SRCU's
explicitly static declarations already enforce the required duration.
The existing per-CPU restrictions on Tree SRCU declarations still apply.

Assisted-by: OpenAI Codex
Signed-off-by: Yury Norov <ynorov@nvidia.com>
Reviewed-by: Joel Fernandes <jfern@kernel.org>
Signed-off-by: Joel Fernandes <jfern@kernel.org>
---
 include/linux/rcu_sync.h | 4 +++-
 include/linux/srcutiny.h | 4 +++-
 include/linux/srcutree.h | 7 +++++--
 3 files changed, 11 insertions(+), 4 deletions(-)

diff --git a/include/linux/rcu_sync.h b/include/linux/rcu_sync.h
index 3860dbb9107a..c10b6d30561a 100644
--- a/include/linux/rcu_sync.h
+++ b/include/linux/rcu_sync.h
@@ -10,6 +10,7 @@
 #ifndef _LINUX_RCU_SYNC_H_
 #define _LINUX_RCU_SYNC_H_
 
+#include <linux/compiler.h>
 #include <linux/wait.h>
 #include <linux/rcupdate.h>
 
@@ -48,6 +49,7 @@ extern void rcu_sync_dtor(struct rcu_sync *);
 	}
 
 #define	DEFINE_RCU_SYNC(name)	\
-	struct rcu_sync name = __RCU_SYNC_INITIALIZER(name)
+	struct rcu_sync name = __RCU_SYNC_INITIALIZER(name);	\
+	ASSERT_STATIC_STORAGE(name)
 
 #endif /* _LINUX_RCU_SYNC_H_ */
diff --git a/include/linux/srcutiny.h b/include/linux/srcutiny.h
index 47a368f945e3..82901b45c9e1 100644
--- a/include/linux/srcutiny.h
+++ b/include/linux/srcutiny.h
@@ -11,6 +11,7 @@
 #ifndef _LINUX_SRCU_TINY_H
 #define _LINUX_SRCU_TINY_H
 
+#include <linux/compiler.h>
 #include <linux/irq_work_types.h>
 #include <linux/llist.h>
 #include <linux/swait.h>
@@ -56,7 +57,8 @@ void srcu_defer_drain(struct irq_work *irq_work);
  * Tree SRCU, which needs some per-CPU data.
  */
 #define DEFINE_SRCU(name) \
-	struct srcu_struct name = __SRCU_STRUCT_INIT(name, name, name, name)
+	struct srcu_struct name = __SRCU_STRUCT_INIT(name, name, name, name);	\
+	ASSERT_STATIC_STORAGE(name)
 #define DEFINE_STATIC_SRCU(name) \
 	static struct srcu_struct name = __SRCU_STRUCT_INIT(name, name, name, name)
 #define DEFINE_SRCU_FAST(name) DEFINE_SRCU(name)
diff --git a/include/linux/srcutree.h b/include/linux/srcutree.h
index 93b76e543894..332d228da270 100644
--- a/include/linux/srcutree.h
+++ b/include/linux/srcutree.h
@@ -11,6 +11,7 @@
 #ifndef _LINUX_SRCU_TREE_H
 #define _LINUX_SRCU_TREE_H
 
+#include <linux/compiler.h>
 #include <linux/rcu_node_tree.h>
 #include <linux/completion.h>
 #include <linux/irq_work_types.h>
@@ -228,13 +229,15 @@ struct srcu_struct {
 								      fast);			\
 	extern struct srcu_struct * const __srcu_struct_##name;					\
 	struct srcu_struct * const __srcu_struct_##name						\
-		__section("___srcu_struct_ptrs") = &name
+		__section("___srcu_struct_ptrs") = &name;					\
+	ASSERT_STATIC_STORAGE(name)
 #else
 # define __DEFINE_SRCU(name, fast, is_static)							\
 	static DEFINE_PER_CPU(struct srcu_data, name##_srcu_data);				\
 	static struct srcu_usage name##_srcu_usage = __SRCU_USAGE_INIT(name##_srcu_usage);	\
 	is_static struct srcu_struct name =							\
-		__SRCU_STRUCT_INIT(name, name##_srcu_usage, name##_srcu_data, fast)
+		__SRCU_STRUCT_INIT(name, name##_srcu_usage, name##_srcu_data, fast);		\
+	ASSERT_STATIC_STORAGE(name)
 #endif
 #define DEFINE_SRCU(name)		__DEFINE_SRCU(name, 0, /* !static */)
 #define DEFINE_STATIC_SRCU(name)	__DEFINE_SRCU(name, 0, static)
-- 
2.54.0

[PATCH for-v7.5 4/5] rcu-tasks: Use pr_debug() for callback contend/collapse messages

[Joel Fernandes]

From: Song Liu <song@kernel.org>

Commit 5de907d24919 ("rcu-tasks: Disable callback contend/collapse
messages by default") added the rcupdate.rcu_task_collapse_debug module
parameter to gate the three pr_info() messages that report RCU Tasks
switching between CPU-0 and per-CPU callback queuing.

The kernel already has a mechanism for debug-only messages that are off
by default: pr_debug().  Use pr_debug() for these messages, and remove
the rcu_task_collapse_debug module parameter, its documentation, and
its boot-time announcement.

The messages stay off by default.  On CONFIG_DYNAMIC_DEBUG=y kernels,
they can be enabled at boot with:

  dyndbg="file kernel/rcu/tasks.h +p"

or at runtime with:

  echo 'file kernel/rcu/tasks.h +p' > /proc/dynamic_debug/control

Assisted-by: Claude:claude-opus-5
Signed-off-by: Song Liu <song@kernel.org>
Signed-off-by: Joel Fernandes <jfern@kernel.org>
---
 Documentation/admin-guide/kernel-parameters.txt |  7 -------
 kernel/rcu/tasks.h                              | 15 +++------------
 2 files changed, 3 insertions(+), 19 deletions(-)

diff --git a/Documentation/admin-guide/kernel-parameters.txt b/Documentation/admin-guide/kernel-parameters.txt
index 0b3c140babd9..176648941aec 100644
--- a/Documentation/admin-guide/kernel-parameters.txt
+++ b/Documentation/admin-guide/kernel-parameters.txt
@@ -6369,13 +6369,6 @@ Kernel parameters
 			period to instead use normal non-expedited
 			grace-period processing.
 
-	rcupdate.rcu_task_collapse_debug= [KNL]
-			Enable debugging prints that record when RCU Tasks
-			and RCU Tasks Trace expand to per-CPU callback
-			queuing and collapse back to CPU-0 queuing.
-			This is default-disabled due to the fact that
-			some workloads can make it quite noisy.
-
 	rcupdate.rcu_task_collapse_lim= [KNL]
 			Set the maximum number of callbacks present
 			at the beginning of a grace period that allows
diff --git a/kernel/rcu/tasks.h b/kernel/rcu/tasks.h
index a59aa8b6ab05..0eec90464d64 100644
--- a/kernel/rcu/tasks.h
+++ b/kernel/rcu/tasks.h
@@ -178,8 +178,6 @@ static int rcu_task_contend_lim __read_mostly = 100;
 module_param(rcu_task_contend_lim, int, 0444);
 static int rcu_task_collapse_lim __read_mostly = 10;
 module_param(rcu_task_collapse_lim, int, 0444);
-static bool rcu_task_collapse_debug __read_mostly;
-module_param(rcu_task_collapse_debug, bool, 0644);
 static int rcu_task_lazy_lim __read_mostly = 32;
 module_param(rcu_task_lazy_lim, int, 0444);
 
@@ -392,8 +390,7 @@ static void call_rcu_tasks_generic(struct rcu_head *rhp, rcu_callback_t func,
 			WRITE_ONCE(rtp->percpu_enqueue_shift, 0);
 			WRITE_ONCE(rtp->percpu_dequeue_lim, rcu_task_cpu_ids);
 			smp_store_release(&rtp->percpu_enqueue_lim, rcu_task_cpu_ids);
-			if (data_race(rcu_task_collapse_debug))
-				pr_info("Switching %s to per-CPU callback queuing.\n", rtp->name);
+			pr_debug("Switching %s to per-CPU callback queuing.\n", rtp->name);
 		}
 		raw_spin_unlock_irqrestore(&rtp->cbs_gbl_lock, flags);
 	}
@@ -514,9 +511,7 @@ static int rcu_tasks_need_gpcb(struct rcu_tasks *rtp)
 			smp_store_release(&rtp->percpu_enqueue_lim, 1);
 			rtp->percpu_dequeue_gpseq = get_state_synchronize_rcu();
 			gpdone = false;
-			if (data_race(rcu_task_collapse_debug))
-				pr_info("Starting switch %s to CPU-0 callback queuing.\n",
-					rtp->name);
+			pr_debug("Starting switch %s to CPU-0 callback queuing.\n", rtp->name);
 		}
 		raw_spin_unlock_irqrestore(&rtp->cbs_gbl_lock, flags);
 	}
@@ -524,9 +519,7 @@ static int rcu_tasks_need_gpcb(struct rcu_tasks *rtp)
 		raw_spin_lock_irqsave(&rtp->cbs_gbl_lock, flags);
 		if (rtp->percpu_enqueue_lim < rtp->percpu_dequeue_lim) {
 			WRITE_ONCE(rtp->percpu_dequeue_lim, 1);
-			if (data_race(rcu_task_collapse_debug))
-				pr_info("Completing switch %s to CPU-0 callback queuing.\n",
-					rtp->name);
+			pr_debug("Completing switch %s to CPU-0 callback queuing.\n", rtp->name);
 		}
 		if (rtp->percpu_dequeue_lim == 1) {
 			for (cpu = rtp->percpu_dequeue_lim; cpu < rcu_task_cpu_ids; cpu++) {
@@ -711,8 +704,6 @@ static void __init rcu_tasks_bootup_oddness(void)
 		pr_info("\tTasks-RCU CPU stall info multiplier clamped to %d (rcu_task_stall_info_mult).\n", rtsimc);
 		rcu_task_stall_info_mult = rtsimc;
 	}
-	if (rcu_task_collapse_debug)
-		pr_info("\tTasks-RCU callback contend/collapse debug enabled.\n");
 #endif /* #ifdef CONFIG_TASKS_RCU */
 #ifdef CONFIG_TASKS_RCU
 	pr_info("\tTrampoline variant of Tasks RCU enabled.\n");
-- 
2.54.0

Re: [PATCH for-v7.5 4/5] rcu-tasks: Use pr_debug() for callback contend/collapse messages

[Joel Fernandes]

On Thu, Nov 12, 2026 at 7:49 PM Joel Fernandes <jfern@kernel.org> wrote:
>
> From: Song Liu <song@kernel.org>
>
> Commit 5de907d24919 ("rcu-tasks: Disable callback contend/collapse
> messages by default") added the rcupdate.rcu_task_collapse_debug module
> parameter to gate the three pr_info() messages that report RCU Tasks
> switching between CPU-0 and per-CPU callback queuing.
>
> The kernel already has a mechanism for debug-only messages that are off
> by default: pr_debug().  Use pr_debug() for these messages, and remove
> the rcu_task_collapse_debug module parameter, its documentation, and
> its boot-time announcement.
>
> The messages stay off by default.  On CONFIG_DYNAMIC_DEBUG=y kernels,
> they can be enabled at boot with:
>
>   dyndbg="file kernel/rcu/tasks.h +p"
>
> or at runtime with:
>
>   echo 'file kernel/rcu/tasks.h +p' > /proc/dynamic_debug/control
>
> Assisted-by: Claude:claude-opus-5
> Signed-off-by: Song Liu <song@kernel.org>
> Signed-off-by: Joel Fernandes <jfern@kernel.org>

Btw, even though I included this in the 'for-7.5', I am planning to
send it for 7.4-rc as we discussed, so please ignore it in this set.

thanks,

Joel Fernandes

[PATCH for-v7.5 5/5] cpuhp: Expedite RCU when toggling system-wide SMT mode

[Joel Fernandes]

From: Vishal Chourasia <vishalc@linux.ibm.com>

On large idle systems, changing system-wide SMT level using the sysfs
control interface still takes approximately 40-55 minutes. Changing SMT
levels is an user triggered operation generally done when systems are
fairly idle. This causes users to be blocked from doing other subsequent
work.

Analyzing the collected profile data during SMT level switch, showed
that CPU hotplug machinery was blocked on synchronize_rcu() calls.

Expedite RCU grace periods for the entire duration of triggered SMT
switch operation via the sysfs control interface. Individual CPU hotplug
operations via the online/offline interface are not affected.

On a PPC64 system with 400 CPUs:

  SMT8 to SMT1:
    before:  1m14s
    after:   3.2s   (~23x faster)

  SMT1 to SMT8:
    before:  2m27s
    after:   2.5s   (~58x faster)

On a large config system with 1920 CPUs, completion time improves from
~1 hour to 5-6 minutes.

Tested-by: Samir <samir@linux.ibm.com>
Reviewed-by: Shrikanth Hegde <sshegde@linux.ibm.com>
Signed-off-by: Vishal Chourasia <vishalc@linux.ibm.com>
Reviewed-by: Paul E. McKenney <paulmck@kernel.org>
Signed-off-by: Joel Fernandes <jfern@kernel.org>
---
 include/linux/rcupdate.h | 8 ++++++++
 kernel/cpu.c             | 4 ++++
 kernel/rcu/rcu.h         | 4 ----
 3 files changed, 12 insertions(+), 4 deletions(-)

diff --git a/include/linux/rcupdate.h b/include/linux/rcupdate.h
index 3f74ae6d6e1f..e498e21172da 100644
--- a/include/linux/rcupdate.h
+++ b/include/linux/rcupdate.h
@@ -1213,6 +1213,14 @@ rcu_head_after_call_rcu(struct rcu_head *rhp, rcu_callback_t f)
 extern int rcu_expedited;
 extern int rcu_normal;
 
+#ifdef CONFIG_TINY_RCU
+static inline void rcu_expedite_gp(void) { }
+static inline void rcu_unexpedite_gp(void) { }
+#else
+void rcu_expedite_gp(void);
+void rcu_unexpedite_gp(void);
+#endif
+
 DEFINE_LOCK_GUARD_0(rcu, rcu_read_lock(), rcu_read_unlock())
 DECLARE_LOCK_GUARD_0_ATTRS(rcu, __acquires_shared(RCU), __releases_shared(RCU))
 
diff --git a/kernel/cpu.c b/kernel/cpu.c
index 376d297a6292..d7dc1022bf6f 100644
--- a/kernel/cpu.c
+++ b/kernel/cpu.c
@@ -2658,6 +2658,7 @@ int cpuhp_smt_disable(enum cpuhp_smt_control ctrlval)
 	int cpu, ret = 0;
 
 	cpu_maps_update_begin();
+	rcu_expedite_gp();
 	for_each_online_cpu(cpu) {
 		if (topology_is_primary_thread(cpu))
 			continue;
@@ -2687,6 +2688,7 @@ int cpuhp_smt_disable(enum cpuhp_smt_control ctrlval)
 	}
 	if (!ret)
 		cpu_smt_control = ctrlval;
+	rcu_unexpedite_gp();
 	cpu_maps_update_done();
 	return ret;
 }
@@ -2704,6 +2706,7 @@ int cpuhp_smt_enable(void)
 	int cpu, ret = 0;
 
 	cpu_maps_update_begin();
+	rcu_expedite_gp();
 	cpu_smt_control = CPU_SMT_ENABLED;
 	for_each_present_cpu(cpu) {
 		/* Skip online CPUs and CPUs on offline nodes */
@@ -2717,6 +2720,7 @@ int cpuhp_smt_enable(void)
 		/* See comment in cpuhp_smt_disable() */
 		cpuhp_online_cpu_device(cpu);
 	}
+	rcu_unexpedite_gp();
 	cpu_maps_update_done();
 	return ret;
 }
diff --git a/kernel/rcu/rcu.h b/kernel/rcu/rcu.h
index d60444bf3a02..7cae7e64e96d 100644
--- a/kernel/rcu/rcu.h
+++ b/kernel/rcu/rcu.h
@@ -530,8 +530,6 @@ do {									\
 static inline bool rcu_gp_is_normal(void) { return true; }
 static inline bool rcu_gp_is_expedited(void) { return false; }
 static inline bool rcu_async_should_hurry(void) { return false; }
-static inline void rcu_expedite_gp(void) { }
-static inline void rcu_unexpedite_gp(void) { }
 static inline void rcu_async_hurry(void) { }
 static inline void rcu_async_relax(void) { }
 static inline bool rcu_cpu_online(int cpu) { return true; }
@@ -539,8 +537,6 @@ static inline bool rcu_cpu_online(int cpu) { return true; }
 bool rcu_gp_is_normal(void);     /* Internal RCU use. */
 bool rcu_gp_is_expedited(void);  /* Internal RCU use. */
 bool rcu_async_should_hurry(void);  /* Internal RCU use. */
-void rcu_expedite_gp(void);
-void rcu_unexpedite_gp(void);
 void rcu_async_hurry(void);
 void rcu_async_relax(void);
 void rcupdate_announce_bootup_oddness(void);
-- 
2.54.0