* [PATCH RFC 01/16] rcu: Use task_state_to_char() in stall-warning prints
2026-07-31 1:01 ` [PATCH v2 0/16] Miscellaneous RCU updates for v7.3 Paul E. McKenney
@ 2026-07-31 1:01 ` Paul E. McKenney
2026-07-31 1:01 ` [PATCH RFC 02/16] rcu: Mark __rcu_access_pointer() as context_unsafe() Paul E. McKenney
` (14 subsequent siblings)
15 siblings, 0 replies; 28+ messages in thread
From: Paul E. McKenney @ 2026-07-31 1:01 UTC (permalink / raw)
To: rcu
Cc: linux-kernel, kernel-team, rostedt, Kunwu Chan, Zqiang, Wang Lian,
Paul E . McKenney
From: Kunwu Chan <kunwu.chan@gmail.com>
RCU stall warnings currently print task states as raw hexadecimal
values, requiring developers to manually decode them.
Use task_state_to_char() so that stall warnings show the same symbolic
task-state representation used elsewhere in the kernel.
For example:
->state=0x402 becomes ->state=I
->state=0x0 becomes ->state=R
->state=0x2 becomes ->state=D
This improves readability while preserving the underlying diagnostic
information.
Suggested-by: Zqiang <qiang.zhang@linux.dev>
Co-developed-by: Wang Lian <lianux.mm@gmail.com>
Signed-off-by: Wang Lian <lianux.mm@gmail.com>
Signed-off-by: Kunwu Chan <kunwu.chan@gmail.com>
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
---
kernel/rcu/tree_stall.h | 12 ++++++------
1 file changed, 6 insertions(+), 6 deletions(-)
diff --git a/kernel/rcu/tree_stall.h b/kernel/rcu/tree_stall.h
index cf7ae51cba4033..45b9856ccd2b23 100644
--- a/kernel/rcu/tree_stall.h
+++ b/kernel/rcu/tree_stall.h
@@ -573,13 +573,13 @@ static void rcu_check_gp_kthread_starvation(void)
if (rcu_is_gp_kthread_starving(&j)) {
cpu = gpk ? task_cpu(gpk) : -1;
- pr_err("%s kthread starved for %ld jiffies! g%ld f%#x %s(%d) ->state=%#x ->cpu=%d\n",
+ pr_err("%s kthread starved for %ld jiffies! g%ld f%#x %s(%d) ->state=%c ->cpu=%d\n",
rcu_state.name, j,
(long)rcu_seq_current(&rcu_state.gp_seq),
data_race(READ_ONCE(rcu_state.gp_flags)),
gp_state_getname(rcu_state.gp_state),
data_race(READ_ONCE(rcu_state.gp_state)),
- gpk ? data_race(READ_ONCE(gpk->__state)) : ~0, cpu);
+ gpk ? task_state_to_char(gpk) : '?', cpu);
if (gpk) {
struct rcu_data *rdp = per_cpu_ptr(&rcu_data, cpu);
@@ -616,12 +616,12 @@ static void rcu_check_gp_kthread_expired_fqs_timer(void)
time_after(jiffies, jiffies_fqs + RCU_STALL_MIGHT_MIN) &&
gpk && !READ_ONCE(gpk->on_rq)) {
cpu = task_cpu(gpk);
- pr_err("%s kthread timer wakeup didn't happen for %ld jiffies! g%ld f%#x %s(%d) ->state=%#x\n",
+ pr_err("%s kthread timer wakeup didn't happen for %ld jiffies! g%ld f%#x %s(%d) ->state=%c\n",
rcu_state.name, (jiffies - jiffies_fqs),
(long)rcu_seq_current(&rcu_state.gp_seq),
data_race(READ_ONCE(rcu_state.gp_flags)), // Diagnostic read
gp_state_getname(RCU_GP_WAIT_FQS), RCU_GP_WAIT_FQS,
- data_race(READ_ONCE(gpk->__state)));
+ task_state_to_char(gpk));
pr_err("\tPossible timer handling issue on cpu=%d timer-softirq=%u\n",
cpu, kstat_softirqs_cpu(TIMER_SOFTIRQ, cpu));
}
@@ -948,10 +948,10 @@ void show_rcu_gp_kthreads(void)
jr = j - data_race(READ_ONCE(rcu_state.gp_req_activity));
js = j - data_race(READ_ONCE(rcu_state.gp_start));
jw = j - data_race(READ_ONCE(rcu_state.gp_wake_time));
- pr_info("%s: wait state: %s(%d) ->state: %#x ->rt_priority %u delta ->gp_start %lu ->gp_activity %lu ->gp_req_activity %lu ->gp_wake_time %lu ->gp_wake_seq %ld ->gp_seq %ld ->gp_seq_needed %ld ->gp_max %lu ->gp_flags %#x\n",
+ pr_info("%s: wait state: %s(%d) ->state: %c ->rt_priority %u delta ->gp_start %lu ->gp_activity %lu ->gp_req_activity %lu ->gp_wake_time %lu ->gp_wake_seq %ld ->gp_seq %ld ->gp_seq_needed %ld ->gp_max %lu ->gp_flags %#x\n",
rcu_state.name, gp_state_getname(rcu_state.gp_state),
data_race(READ_ONCE(rcu_state.gp_state)),
- t ? data_race(READ_ONCE(t->__state)) : 0x1ffff, t ? t->rt_priority : 0xffU,
+ t ? task_state_to_char(t) : '?', t ? t->rt_priority : 0xffU,
js, ja, jr, jw, (long)data_race(READ_ONCE(rcu_state.gp_wake_seq)),
(long)data_race(READ_ONCE(rcu_state.gp_seq)),
(long)data_race(READ_ONCE(rcu_get_root()->gp_seq_needed)),
--
2.40.1
^ permalink raw reply related [flat|nested] 28+ messages in thread* [PATCH RFC 02/16] rcu: Mark __rcu_access_pointer() as context_unsafe()
2026-07-31 1:01 ` [PATCH v2 0/16] Miscellaneous RCU updates for v7.3 Paul E. McKenney
2026-07-31 1:01 ` [PATCH RFC 01/16] rcu: Use task_state_to_char() in stall-warning prints Paul E. McKenney
@ 2026-07-31 1:01 ` Paul E. McKenney
2026-07-31 1:01 ` [PATCH RFC 03/16] doc: RCU: Adopt new coding style of type-aware kmalloc-family - part 2/2 Paul E. McKenney
` (13 subsequent siblings)
15 siblings, 0 replies; 28+ messages in thread
From: Paul E. McKenney @ 2026-07-31 1:01 UTC (permalink / raw)
To: rcu
Cc: linux-kernel, kernel-team, rostedt, Paul E. McKenney,
Christoph Hellwig, Marco Elver, Nilay Shroff
A simple comparison of a pointer returned by rcu_access_pointer() results
in a context-analysis warning for lockless inspection of the RCU-protected
(also known as __rcu-protected) pointer. This can be suppressed by
placing context_unsafe() calls around calls rcu_access_pointer(),
but this is messy and distracting. This commit therefore wraps the
underlying __rcu_access_pointer() macro with a call to context_unsafe(),
thereby informing the context-analysis code that rcu_access_pointer()
may safely be invoked outside of an RCU read-side critical section.
Reported-by: Christoph Hellwig <hch@lst.de>
Suggested-by: Marco Elver <elver@google.com>
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
Tested-by: Nilay Shroff <nilay@linux.ibm.com>
Reviewed-by: Marco Elver <elver@google.com>
---
include/linux/rcupdate.h | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/include/linux/rcupdate.h b/include/linux/rcupdate.h
index 5e95acc33989b6..e40dc2e20c5b1f 100644
--- a/include/linux/rcupdate.h
+++ b/include/linux/rcupdate.h
@@ -490,12 +490,12 @@ context_unsafe( \
*/
#define unrcu_pointer(p) __unrcu_pointer(p, __UNIQUE_ID(rcu))
-#define __rcu_access_pointer(p, local, space) \
+#define __rcu_access_pointer(p, local, space) context_unsafe( \
({ \
typeof(*p) *local = (typeof(*p) *__force)READ_ONCE(p); \
rcu_check_sparse(p, space); \
((typeof(*p) __force __kernel *)(local)); \
-})
+}) )
#define __rcu_dereference_check(p, local, c, space) \
({ \
/* Dependency order vs. p above. */ \
--
2.40.1
^ permalink raw reply related [flat|nested] 28+ messages in thread* [PATCH RFC 03/16] doc: RCU: Adopt new coding style of type-aware kmalloc-family - part 2/2
2026-07-31 1:01 ` [PATCH v2 0/16] Miscellaneous RCU updates for v7.3 Paul E. McKenney
2026-07-31 1:01 ` [PATCH RFC 01/16] rcu: Use task_state_to_char() in stall-warning prints Paul E. McKenney
2026-07-31 1:01 ` [PATCH RFC 02/16] rcu: Mark __rcu_access_pointer() as context_unsafe() Paul E. McKenney
@ 2026-07-31 1:01 ` Paul E. McKenney
2026-07-31 1:01 ` [PATCH RFC 04/16] doc: RCU: Fix brackets Paul E. McKenney
` (12 subsequent siblings)
15 siblings, 0 replies; 28+ messages in thread
From: Paul E. McKenney @ 2026-07-31 1:01 UTC (permalink / raw)
To: rcu; +Cc: linux-kernel, kernel-team, rostedt, Manuel Ebner,
Paul E . McKenney
From: Manuel Ebner <manuelebner@mailbox.org>
Update Documentation/RCU/* to suggest using the new type-aware
kmalloc_obj() per commit 2932ba8d9c99 ("slab: Introduce kmalloc_obj()
and family")
p = kmalloc(...);
-> p = kmalloc_obj(...);
Signed-off-by: Manuel Ebner <manuelebner@mailbox.org>
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
---
Documentation/RCU/rcu_dereference.rst | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/Documentation/RCU/rcu_dereference.rst b/Documentation/RCU/rcu_dereference.rst
index 2524dcdadde2b8..5bc3785ebfc2ae 100644
--- a/Documentation/RCU/rcu_dereference.rst
+++ b/Documentation/RCU/rcu_dereference.rst
@@ -236,7 +236,7 @@ precautions. To see this, consider the following code fragment::
{
struct foo *p;
- p = kmalloc(...);
+ p = kmalloc_obj(*p);
if (p == NULL)
deal_with_it();
p->a = 42; /* Each field in its own cache line. */
@@ -293,7 +293,7 @@ Then one approach is to use locking, for example, as follows::
{
struct foo *p;
- p = kmalloc(...);
+ p = kmalloc_obj(*p);
if (p == NULL)
deal_with_it();
spin_lock(&p->lock);
--
2.40.1
^ permalink raw reply related [flat|nested] 28+ messages in thread* [PATCH RFC 04/16] doc: RCU: Fix brackets
2026-07-31 1:01 ` [PATCH v2 0/16] Miscellaneous RCU updates for v7.3 Paul E. McKenney
` (2 preceding siblings ...)
2026-07-31 1:01 ` [PATCH RFC 03/16] doc: RCU: Adopt new coding style of type-aware kmalloc-family - part 2/2 Paul E. McKenney
@ 2026-07-31 1:01 ` Paul E. McKenney
2026-07-31 1:01 ` [PATCH RFC 05/16] rcu: introduce rcu_defer_qs_clear() helper Paul E. McKenney
` (11 subsequent siblings)
15 siblings, 0 replies; 28+ messages in thread
From: Paul E. McKenney @ 2026-07-31 1:01 UTC (permalink / raw)
To: rcu; +Cc: linux-kernel, kernel-team, rostedt, Manuel Ebner,
Paul E . McKenney
From: Manuel Ebner <manuelebner@mailbox.org>
Remove needless brackets and add missing brackets.
Signed-off-by: Manuel Ebner <manuelebner@mailbox.org>
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
---
.../Design/Expedited-Grace-Periods/Expedited-Grace-Periods.rst | 2 +-
Documentation/RCU/Design/Memory-Ordering/TreeRCU-gp.svg | 2 +-
Documentation/RCU/Design/Memory-Ordering/TreeRCU-qs.svg | 2 +-
Documentation/RCU/Design/Requirements/Requirements.rst | 2 +-
4 files changed, 4 insertions(+), 4 deletions(-)
diff --git a/Documentation/RCU/Design/Expedited-Grace-Periods/Expedited-Grace-Periods.rst b/Documentation/RCU/Design/Expedited-Grace-Periods/Expedited-Grace-Periods.rst
index 414f8a2012d684..cf0f9cdca7e8cb 100644
--- a/Documentation/RCU/Design/Expedited-Grace-Periods/Expedited-Grace-Periods.rst
+++ b/Documentation/RCU/Design/Expedited-Grace-Periods/Expedited-Grace-Periods.rst
@@ -410,7 +410,7 @@ workqueues (see Documentation/core-api/workqueue.rst).
The requesting task still does counter snapshotting and funnel-lock
processing, but the task reaching the top of the funnel lock does a
-``schedule_work()`` (from ``_synchronize_rcu_expedited()`` so that a
+``schedule_work()`` (from ``_synchronize_rcu_expedited()``) so that a
workqueue kthread does the actual grace-period processing. Because
workqueue kthreads do not accept POSIX signals, grace-period-wait
processing need not allow for POSIX signals. In addition, this approach
diff --git a/Documentation/RCU/Design/Memory-Ordering/TreeRCU-gp.svg b/Documentation/RCU/Design/Memory-Ordering/TreeRCU-gp.svg
index d05bc7b27edb7a..95a66de40ca5a2 100644
--- a/Documentation/RCU/Design/Memory-Ordering/TreeRCU-gp.svg
+++ b/Documentation/RCU/Design/Memory-Ordering/TreeRCU-gp.svg
@@ -3933,7 +3933,7 @@
font-style="normal"
y="-3914.085"
x="3745.7725"
- xml:space="preserve">rcu__report_qs_rdp())</text>
+ xml:space="preserve">rcu__report_qs_rdp()</text>
</g>
<g
id="g4504-3"
diff --git a/Documentation/RCU/Design/Memory-Ordering/TreeRCU-qs.svg b/Documentation/RCU/Design/Memory-Ordering/TreeRCU-qs.svg
index 7d6c5f7e505c68..882132680308ef 100644
--- a/Documentation/RCU/Design/Memory-Ordering/TreeRCU-qs.svg
+++ b/Documentation/RCU/Design/Memory-Ordering/TreeRCU-qs.svg
@@ -815,7 +815,7 @@
font-style="normal"
y="-3914.085"
x="3745.7725"
- xml:space="preserve">rcu__report_qs_rdp())</text>
+ xml:space="preserve">rcu__report_qs_rdp()</text>
</g>
<g
id="g4504-3"
diff --git a/Documentation/RCU/Design/Requirements/Requirements.rst b/Documentation/RCU/Design/Requirements/Requirements.rst
index 8a216e4a46a7df..8101fe6229d579 100644
--- a/Documentation/RCU/Design/Requirements/Requirements.rst
+++ b/Documentation/RCU/Design/Requirements/Requirements.rst
@@ -2785,7 +2785,7 @@ both srcu_read_lock() and srcu_read_unlock(). This need is handled by
a Tasks Trace RCU API implemented as thin wrappers around SRCU-fast,
which avoids the read-side memory barriers, at least for architectures
that apply noinstr to kernel entry/exit code (or that build with
-``CONFIG_TASKS_TRACE_RCU_NO_MB=y``.
+``CONFIG_TASKS_TRACE_RCU_NO_MB=y``).
Now that the implementation is based on SRCU-fast, a call
to synchronize_rcu_tasks_trace() implies at least one call to
--
2.40.1
^ permalink raw reply related [flat|nested] 28+ messages in thread* [PATCH RFC 05/16] rcu: introduce rcu_defer_qs_clear() helper
2026-07-31 1:01 ` [PATCH v2 0/16] Miscellaneous RCU updates for v7.3 Paul E. McKenney
` (3 preceding siblings ...)
2026-07-31 1:01 ` [PATCH RFC 04/16] doc: RCU: Fix brackets Paul E. McKenney
@ 2026-07-31 1:01 ` Paul E. McKenney
2026-07-31 1:01 ` [PATCH RFC 06/16] rcu: clear defer_qs_pending in deferred-QS bail when nesting > 0 Paul E. McKenney
` (10 subsequent siblings)
15 siblings, 0 replies; 28+ messages in thread
From: Paul E. McKenney @ 2026-07-31 1:01 UTC (permalink / raw)
To: rcu; +Cc: linux-kernel, kernel-team, rostedt, Joel Fernandes,
Paul E . McKenney
From: Joel Fernandes <joelagnelf@nvidia.com>
Currently rdp->defer_qs_pending transitions from DEFER_QS_PENDING to
DEFER_QS_IDLE at two sites: rcu_preempt_deferred_qs_irqrestore() and
rcu_preempt_deferred_qs_handler() (depth>0 reset). Both write the
IDLE value directly.
Introduce a single inline helper rcu_defer_qs_clear() in tree.h and
route both sites through it. This becomes the single
PENDING->IDLE transition point for upcoming work.
Signed-off-by: Joel Fernandes <joelagnelf@nvidia.com>
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
---
kernel/rcu/tree.h | 5 +++++
kernel/rcu/tree_plugin.h | 4 ++--
2 files changed, 7 insertions(+), 2 deletions(-)
diff --git a/kernel/rcu/tree.h b/kernel/rcu/tree.h
index 7dfc57e9adb18e..4069132f9d4446 100644
--- a/kernel/rcu/tree.h
+++ b/kernel/rcu/tree.h
@@ -296,6 +296,11 @@ struct rcu_data {
int cpu;
};
+static inline void rcu_defer_qs_clear(struct rcu_data *rdp)
+{
+ WRITE_ONCE(rdp->defer_qs_pending, DEFER_QS_IDLE);
+}
+
/* Values for nocb_defer_wakeup field in struct rcu_data. */
#define RCU_NOCB_WAKE_NOT 0
#define RCU_NOCB_WAKE_BYPASS 1
diff --git a/kernel/rcu/tree_plugin.h b/kernel/rcu/tree_plugin.h
index 95ad967adcf3cb..8637f405cb472f 100644
--- a/kernel/rcu/tree_plugin.h
+++ b/kernel/rcu/tree_plugin.h
@@ -488,7 +488,7 @@ rcu_preempt_deferred_qs_irqrestore(struct task_struct *t, unsigned long flags)
rdp = this_cpu_ptr(&rcu_data);
if (rdp->defer_qs_pending == DEFER_QS_PENDING)
- rdp->defer_qs_pending = DEFER_QS_IDLE;
+ rcu_defer_qs_clear(rdp);
/*
* If RCU core is waiting for this CPU to exit its critical section,
@@ -645,7 +645,7 @@ static void rcu_preempt_deferred_qs_handler(struct irq_work *iwp)
* 5. Deferred QS reporting does not happen.
*/
if (rcu_preempt_depth() > 0)
- WRITE_ONCE(rdp->defer_qs_pending, DEFER_QS_IDLE);
+ rcu_defer_qs_clear(rdp);
}
/*
--
2.40.1
^ permalink raw reply related [flat|nested] 28+ messages in thread* [PATCH RFC 06/16] rcu: clear defer_qs_pending in deferred-QS bail when nesting > 0
2026-07-31 1:01 ` [PATCH v2 0/16] Miscellaneous RCU updates for v7.3 Paul E. McKenney
` (4 preceding siblings ...)
2026-07-31 1:01 ` [PATCH RFC 05/16] rcu: introduce rcu_defer_qs_clear() helper Paul E. McKenney
@ 2026-07-31 1:01 ` Paul E. McKenney
2026-07-31 1:01 ` [PATCH RFC 07/16] rcu: Use this_cpu_{read,write}() for ->cpu_no_qs.b.exp Paul E. McKenney
` (9 subsequent siblings)
15 siblings, 0 replies; 28+ messages in thread
From: Paul E. McKenney @ 2026-07-31 1:01 UTC (permalink / raw)
To: rcu; +Cc: linux-kernel, kernel-team, rostedt, Joel Fernandes,
Paul E . McKenney
From: Joel Fernandes <joelagnelf@nvidia.com>
Paul McKenney noted that a softirq (or irq_work) handler arming for a
deferred QS can fire and find rcu_preempt_depth() > 0 -- the task is
still inside its outer reader, so rcu_preempt_need_deferred_qs() bails
without reporting the QS. At that point the queued mechanism has been
consumed but ->defer_qs_pending stays in DEFER_QS_PENDING.
In the meantime, the only remaining path back to a quiescent state on
this CPU may be a local_irq_disable()/_enable() pair that does not
call preempt_check_resched() (it is just `sti`/`cli`). patch 6's
unconditional set_need_resched_current() makes need_resched true, but
without an irq_work being raised the next outer rcu_read_unlock_special()
hits the P-gate at the arming code:
if (rdp->defer_qs_pending != DEFER_QS_PENDING) {
rdp->defer_qs_pending = DEFER_QS_PENDING;
irq_work_queue_on(...); // <-- skipped
}
so no irq_work is queued for the hardirq-exit preempt_schedule_irq()
path either. The deferred QS now waits until the next timer tick (or
similar preempt-safe boundary), needlessly extending expedited grace
period latency.
Clear ->defer_qs_pending in the bail-out path of rcu_preempt_deferred_qs()
when rcu_preempt_depth() > 0. The recursion guard semantics introduced
by commit b41642c87716 ("rcu: Fix rcu_read_unlock() deadloop due to IRQ
work").
The clear is also safe against fresh recursion at this exact program
point: rcu_preempt_depth() > 0 guarantees we are still inside an outer
reader, so any inner rcu_read_unlock() from tracing infrastructure
brings nesting back to outer (>0), never to 0. The slow path of
rcu_read_unlock_special() is structurally unreachable under that
condition, so no recursive raise_softirq_irqoff()/irq_work_queue_on()
can be triggered by the clear. Essentially, the mechanism will work to
prevent the following recursion which Xiongfeng had previously reported:
irq_exit() -> __irq_exit_rcu()
-> tick_irq_exit() -> tick_nohz_irq_exit() -> tick_nohz_stop_sched_tick()
-> trace_tick_stop() // BPF prog hooked here
-> rcu_read_unlock_special()
-> irq_work_queue_on(&rdp->defer_qs_iw, rdp->cpu) // self-IPI re-enters irq_exit
Reported-by: Paul E. McKenney <paulmck@kernel.org>
Signed-off-by: Joel Fernandes <joelagnelf@nvidia.com>
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
---
kernel/rcu/tree_plugin.h | 28 +++++++++++++++++++++++++++-
1 file changed, 27 insertions(+), 1 deletion(-)
diff --git a/kernel/rcu/tree_plugin.h b/kernel/rcu/tree_plugin.h
index 8637f405cb472f..9ba136a4233a26 100644
--- a/kernel/rcu/tree_plugin.h
+++ b/kernel/rcu/tree_plugin.h
@@ -614,9 +614,35 @@ static notrace bool rcu_preempt_need_deferred_qs(struct task_struct *t)
notrace void rcu_preempt_deferred_qs(struct task_struct *t)
{
unsigned long flags;
+ struct rcu_data *rdp;
- if (!rcu_preempt_need_deferred_qs(t))
+ if (!rcu_preempt_need_deferred_qs(t)) {
+ /*
+ * If we got here from a softirq/irq_work that fired while
+ * rcu_preempt_depth() > 0, the deferred-QS mechanism has been
+ * consumed without doing any work: rcu_preempt_need_deferred_qs()
+ * just returned false because the task is still in a reader, so
+ * the actual QS report has to wait for the next
+ * rcu_read_unlock().
+ *
+ * Clear ->defer_qs_pending here so the next outer
+ * rcu_read_unlock_special() can re-arm a fresh mechanism (in
+ * particular the irq_work path, which the local_irq_enable()
+ * recovery boundary cannot itself reschedule from).
+ *
+ * Recursion safety: rcu_preempt_depth() > 0 means we are inside
+ * an outer reader, so any inner rcu_read_unlock() reached via
+ * tracing (bpf programs attached to trace points) brings
+ * nesting to outer (> 0), never to 0, so no recursive
+ * raise_softirq_irqoff()/irq_work_queue_on() can be triggered
+ * by this clear.
+ */
+ if (rcu_preempt_depth() > 0) {
+ rdp = this_cpu_ptr(&rcu_data);
+ rcu_defer_qs_clear(rdp);
+ }
return;
+ }
local_irq_save(flags);
rcu_preempt_deferred_qs_irqrestore(t, flags);
}
--
2.40.1
^ permalink raw reply related [flat|nested] 28+ messages in thread* [PATCH RFC 07/16] rcu: Use this_cpu_{read,write}() for ->cpu_no_qs.b.exp
2026-07-31 1:01 ` [PATCH v2 0/16] Miscellaneous RCU updates for v7.3 Paul E. McKenney
` (5 preceding siblings ...)
2026-07-31 1:01 ` [PATCH RFC 06/16] rcu: clear defer_qs_pending in deferred-QS bail when nesting > 0 Paul E. McKenney
@ 2026-07-31 1:01 ` Paul E. McKenney
2026-07-31 1:01 ` [PATCH RFC 08/16] rcu: Use WRITE_ONCE() for ->rcu_need_heavy_qs Paul E. McKenney
` (8 subsequent siblings)
15 siblings, 0 replies; 28+ messages in thread
From: Paul E. McKenney @ 2026-07-31 1:01 UTC (permalink / raw)
To: rcu; +Cc: linux-kernel, kernel-team, rostedt, Paul E. McKenney
Currently __this_cpu_read() and __this_cpu_write() is used to access the
->cpu_no_qs.b.exp field of the per-CPU rcu_data structure. However,
this can fail when the accesses can happen in interrupt handlers, as
recently started being exercised by rcutorture. This commit therefore
upgrades the uses of __this_cpu_read() and __this_cpu_write() to their
interrupt-safe counterparts this_cpu_read() and this_cpu_write().
KCSAN located this issue.
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
---
kernel/rcu/tree_exp.h | 4 ++--
kernel/rcu/tree_plugin.h | 4 ++--
2 files changed, 4 insertions(+), 4 deletions(-)
diff --git a/kernel/rcu/tree_exp.h b/kernel/rcu/tree_exp.h
index 82cada459e5d0c..e03cb3d447a57d 100644
--- a/kernel/rcu/tree_exp.h
+++ b/kernel/rcu/tree_exp.h
@@ -731,7 +731,7 @@ static void rcu_exp_need_qs(void)
{
lockdep_assert_irqs_disabled();
ASSERT_EXCLUSIVE_WRITER_SCOPED(*this_cpu_ptr(&rcu_data.cpu_no_qs.b.exp));
- __this_cpu_write(rcu_data.cpu_no_qs.b.exp, true);
+ this_cpu_write(rcu_data.cpu_no_qs.b.exp, true);
/* Store .exp before .rcu_urgent_qs. */
smp_store_release(this_cpu_ptr(&rcu_data.rcu_urgent_qs), true);
set_need_resched_current();
@@ -870,7 +870,7 @@ static void rcu_exp_handler(void *unused)
ASSERT_EXCLUSIVE_WRITER_SCOPED(rdp->cpu_no_qs.b.exp);
if (!(READ_ONCE(rnp->expmask) & rdp->grpmask) ||
- __this_cpu_read(rcu_data.cpu_no_qs.b.exp))
+ this_cpu_read(rcu_data.cpu_no_qs.b.exp))
return;
if (rcu_is_cpu_rrupt_from_idle() ||
(IS_ENABLED(CONFIG_PREEMPT_COUNT) && preempt_bh_enabled)) {
diff --git a/kernel/rcu/tree_plugin.h b/kernel/rcu/tree_plugin.h
index 9ba136a4233a26..c3db1ad4a2807a 100644
--- a/kernel/rcu/tree_plugin.h
+++ b/kernel/rcu/tree_plugin.h
@@ -599,7 +599,7 @@ rcu_preempt_deferred_qs_irqrestore(struct task_struct *t, unsigned long flags)
*/
static notrace bool rcu_preempt_need_deferred_qs(struct task_struct *t)
{
- return (__this_cpu_read(rcu_data.cpu_no_qs.b.exp) ||
+ return (this_cpu_read(rcu_data.cpu_no_qs.b.exp) ||
READ_ONCE(t->rcu_read_unlock_special.s)) &&
rcu_preempt_depth() == 0;
}
@@ -981,7 +981,7 @@ static void rcu_qs(void)
trace_rcu_grace_period(TPS("rcu_sched"),
__this_cpu_read(rcu_data.gp_seq), TPS("cpuqs"));
__this_cpu_write(rcu_data.cpu_no_qs.b.norm, false);
- if (__this_cpu_read(rcu_data.cpu_no_qs.b.exp))
+ if (this_cpu_read(rcu_data.cpu_no_qs.b.exp))
rcu_report_exp_rdp(this_cpu_ptr(&rcu_data));
}
--
2.40.1
^ permalink raw reply related [flat|nested] 28+ messages in thread* [PATCH RFC 08/16] rcu: Use WRITE_ONCE() for ->rcu_need_heavy_qs
2026-07-31 1:01 ` [PATCH v2 0/16] Miscellaneous RCU updates for v7.3 Paul E. McKenney
` (6 preceding siblings ...)
2026-07-31 1:01 ` [PATCH RFC 07/16] rcu: Use this_cpu_{read,write}() for ->cpu_no_qs.b.exp Paul E. McKenney
@ 2026-07-31 1:01 ` Paul E. McKenney
2026-07-31 1:01 ` [PATCH RFC 09/16] rcu: Remove unused expedited_need_qs field from rcu_state Paul E. McKenney
` (7 subsequent siblings)
15 siblings, 0 replies; 28+ messages in thread
From: Paul E. McKenney @ 2026-07-31 1:01 UTC (permalink / raw)
To: rcu; +Cc: linux-kernel, kernel-team, rostedt, Paul E. McKenney
Currently raw_cpu_write() is used to clear the ->rcu_need_heavy_qs
field of the per-CPU rcu_data structure. However, on x86 this is a
normal assignment, which does not play well with concurrent accesses.
This commit therefore upgrades the uses of raw_cpu_write() to its
concurrency-safe counterpart WRITE_ONCE() of an rdp pointer obtained
from this_cpu_ptr(&rcu_data).
KCSAN located this issue.
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
---
kernel/rcu/tree.c | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)
diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c
index 03a43d3d261607..91d5b4dd08bf5c 100644
--- a/kernel/rcu/tree.c
+++ b/kernel/rcu/tree.c
@@ -357,9 +357,10 @@ bool rcu_watching_zero_in_eqs(int cpu, int *vp)
*/
notrace void rcu_momentary_eqs(void)
{
+ struct rcu_data *rdp = this_cpu_ptr(&rcu_data);
int seq;
- raw_cpu_write(rcu_data.rcu_need_heavy_qs, false);
+ WRITE_ONCE(rdp->rcu_need_heavy_qs, false);
seq = ct_state_inc(2 * CT_RCU_WATCHING);
/* It is illegal to call this from idle state. */
WARN_ON_ONCE(!(seq & CT_RCU_WATCHING));
--
2.40.1
^ permalink raw reply related [flat|nested] 28+ messages in thread* [PATCH RFC 09/16] rcu: Remove unused expedited_need_qs field from rcu_state
2026-07-31 1:01 ` [PATCH v2 0/16] Miscellaneous RCU updates for v7.3 Paul E. McKenney
` (7 preceding siblings ...)
2026-07-31 1:01 ` [PATCH RFC 08/16] rcu: Use WRITE_ONCE() for ->rcu_need_heavy_qs Paul E. McKenney
@ 2026-07-31 1:01 ` Paul E. McKenney
2026-07-31 1:01 ` [PATCH RFC 10/16] rcu: Remove unused func parameter from callback-enqueue functions Paul E. McKenney
` (6 subsequent siblings)
15 siblings, 0 replies; 28+ messages in thread
From: Paul E. McKenney @ 2026-07-31 1:01 UTC (permalink / raw)
To: rcu; +Cc: linux-kernel, kernel-team, rostedt, Joel Fernandes,
Paul E . McKenney
From: Joel Fernandes <joelagnelf@nvidia.com>
The ->expedited_need_qs counter was part of the old expedited
grace-period machinery that spun waiting for each CPU to check in.
The current implementation instead tracks holdout CPUs via the
rcu_node structures' ->expmask fields and waits on the ->exp_wq[]
wait queues, so nothing reads or writes ->expedited_need_qs any
longer.
Remove the field to avoid wasting space in rcu_state and to keep
readers of the expedited code from searching for nonexistent users.
Signed-off-by: Joel Fernandes <joelagnelf@nvidia.com>
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
---
kernel/rcu/tree.h | 1 -
1 file changed, 1 deletion(-)
diff --git a/kernel/rcu/tree.h b/kernel/rcu/tree.h
index 4069132f9d4446..587bd71396eaa3 100644
--- a/kernel/rcu/tree.h
+++ b/kernel/rcu/tree.h
@@ -391,7 +391,6 @@ struct rcu_state {
struct mutex exp_mutex; /* Serialize expedited GP. */
struct mutex exp_wake_mutex; /* Serialize wakeup. */
unsigned long expedited_sequence; /* Take a ticket. */
- atomic_t expedited_need_qs; /* # CPUs left to check in. */
struct swait_queue_head expedited_wq; /* Wait for check-ins. */
int ncpus_snap; /* # CPUs seen last time. */
u8 cbovld; /* Callback overload now? */
--
2.40.1
^ permalink raw reply related [flat|nested] 28+ messages in thread* [PATCH RFC 10/16] rcu: Remove unused func parameter from callback-enqueue functions
2026-07-31 1:01 ` [PATCH v2 0/16] Miscellaneous RCU updates for v7.3 Paul E. McKenney
` (8 preceding siblings ...)
2026-07-31 1:01 ` [PATCH RFC 09/16] rcu: Remove unused expedited_need_qs field from rcu_state Paul E. McKenney
@ 2026-07-31 1:01 ` Paul E. McKenney
2026-07-31 1:01 ` [PATCH RFC 11/16] rcu: Mark accesses to rdp->rcu_cpu_has_work Paul E. McKenney
` (5 subsequent siblings)
15 siblings, 0 replies; 28+ messages in thread
From: Paul E. McKenney @ 2026-07-31 1:01 UTC (permalink / raw)
To: rcu; +Cc: linux-kernel, kernel-team, rostedt, Joel Fernandes,
Paul E . McKenney
From: Joel Fernandes <joelagnelf@nvidia.com>
Ever since the kvfree_rcu() tracing moved out of the callback-enqueue
path, rcutree_enqueue() no longer looks at the callback function
pointer: By the time it is invoked, __call_rcu_common() has already
stored the function into rhp->func, and the enqueue path only adds
the rcu_head to the segmented callback list and emits tracepoints
that do not take the function pointer.
Nevertheless, the function pointer is still threaded through
call_rcu_core(), call_rcu_nocb(), and rcutree_enqueue(), forcing
each level to carry a dead argument.
Remove the parameter from all three functions, from the no-CBs stub,
and from the corresponding declarations. Anything needing the
callback function can still get it from rhp->func.
No functional change.
Signed-off-by: Joel Fernandes <joelagnelf@nvidia.com>
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
---
kernel/rcu/tree.c | 10 +++++-----
kernel/rcu/tree.h | 2 +-
kernel/rcu/tree_nocb.h | 6 +++---
3 files changed, 9 insertions(+), 9 deletions(-)
diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c
index 91d5b4dd08bf5c..390dad82675db5 100644
--- a/kernel/rcu/tree.c
+++ b/kernel/rcu/tree.c
@@ -3024,7 +3024,7 @@ static int __init rcu_spawn_core_kthreads(void)
return 0;
}
-static void rcutree_enqueue(struct rcu_data *rdp, struct rcu_head *head, rcu_callback_t func)
+static void rcutree_enqueue(struct rcu_data *rdp, struct rcu_head *head)
{
rcu_segcblist_enqueue(&rdp->cblist, head);
trace_rcu_callback(rcu_state.name, head,
@@ -3036,9 +3036,9 @@ static void rcutree_enqueue(struct rcu_data *rdp, struct rcu_head *head, rcu_cal
* Handle any core-RCU processing required by a call_rcu() invocation.
*/
static void call_rcu_core(struct rcu_data *rdp, struct rcu_head *head,
- rcu_callback_t func, unsigned long flags)
+ unsigned long flags)
{
- rcutree_enqueue(rdp, head, func);
+ rcutree_enqueue(rdp, head);
/*
* If called from an extended quiescent state, invoke the RCU
* core in order to force a re-evaluation of RCU's idleness.
@@ -3179,9 +3179,9 @@ __call_rcu_common(struct rcu_head *head, rcu_callback_t func, bool lazy_in)
check_cb_ovld(rdp);
if (unlikely(rcu_rdp_is_offloaded(rdp)))
- call_rcu_nocb(rdp, head, func, flags, lazy);
+ call_rcu_nocb(rdp, head, flags, lazy);
else
- call_rcu_core(rdp, head, func, flags);
+ call_rcu_core(rdp, head, flags);
local_irq_restore(flags);
}
diff --git a/kernel/rcu/tree.h b/kernel/rcu/tree.h
index 587bd71396eaa3..ca88aaa29d615e 100644
--- a/kernel/rcu/tree.h
+++ b/kernel/rcu/tree.h
@@ -507,7 +507,7 @@ static bool wake_nocb_gp(struct rcu_data *rdp);
static bool rcu_nocb_flush_bypass(struct rcu_data *rdp, struct rcu_head *rhp,
unsigned long j, bool lazy);
static void call_rcu_nocb(struct rcu_data *rdp, struct rcu_head *head,
- rcu_callback_t func, unsigned long flags, bool lazy);
+ unsigned long flags, bool lazy);
static void __maybe_unused __call_rcu_nocb_wake(struct rcu_data *rdp, bool was_empty,
unsigned long flags);
static int rcu_nocb_need_deferred_wakeup(struct rcu_data *rdp, int level);
diff --git a/kernel/rcu/tree_nocb.h b/kernel/rcu/tree_nocb.h
index 373b877cf171d3..95dbf33462a2ff 100644
--- a/kernel/rcu/tree_nocb.h
+++ b/kernel/rcu/tree_nocb.h
@@ -603,13 +603,13 @@ static void __call_rcu_nocb_wake(struct rcu_data *rdp, bool was_alldone,
}
static void call_rcu_nocb(struct rcu_data *rdp, struct rcu_head *head,
- rcu_callback_t func, unsigned long flags, bool lazy)
+ unsigned long flags, bool lazy)
{
bool was_alldone;
if (!rcu_nocb_try_bypass(rdp, head, &was_alldone, flags, lazy)) {
/* Not enqueued on bypass but locked, do regular enqueue */
- rcutree_enqueue(rdp, head, func);
+ rcutree_enqueue(rdp, head);
__call_rcu_nocb_wake(rdp, was_alldone, flags); /* unlocks */
}
}
@@ -1666,7 +1666,7 @@ static bool rcu_nocb_flush_bypass(struct rcu_data *rdp, struct rcu_head *rhp,
}
static void call_rcu_nocb(struct rcu_data *rdp, struct rcu_head *head,
- rcu_callback_t func, unsigned long flags, bool lazy)
+ unsigned long flags, bool lazy)
{
WARN_ON_ONCE(1); /* Should be dead code! */
}
--
2.40.1
^ permalink raw reply related [flat|nested] 28+ messages in thread* [PATCH RFC 11/16] rcu: Mark accesses to rdp->rcu_cpu_has_work
2026-07-31 1:01 ` [PATCH v2 0/16] Miscellaneous RCU updates for v7.3 Paul E. McKenney
` (9 preceding siblings ...)
2026-07-31 1:01 ` [PATCH RFC 10/16] rcu: Remove unused func parameter from callback-enqueue functions Paul E. McKenney
@ 2026-07-31 1:01 ` Paul E. McKenney
2026-07-31 1:01 ` [PATCH RFC 12/16] rcu: Mark interrupts-enabled accesses to rdp->cpu_no_qs.b.norm Paul E. McKenney
` (4 subsequent siblings)
15 siblings, 0 replies; 28+ messages in thread
From: Paul E. McKenney @ 2026-07-31 1:01 UTC (permalink / raw)
To: rcu; +Cc: linux-kernel, kernel-team, rostedt, Paul E. McKenney
Although the rdp->rcu_cpu_has_work field is accessed only by the
corresponding CPU, it can be accessed by both interrupt handlers via
invoke_rcu_core_kthread() and at task level via rcu_cpu_kthread().
This means that we need this_cpu_read() rather than __this_cpu_read(),
this_cpu_write() rather than __this_cpu_write(), and READ_ONCE()
rather than plain C-language loads. The exception is the boot-time
rcu_spawn_core_kthreads(), which cannot race with kthreads that have
not yet been spawned.
This commit therefore makes it so.
KCSAN located this issue.
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
---
kernel/rcu/tree.c | 8 ++++----
1 file changed, 4 insertions(+), 4 deletions(-)
diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c
index 390dad82675db5..c9780e7c0e2afa 100644
--- a/kernel/rcu/tree.c
+++ b/kernel/rcu/tree.c
@@ -2671,7 +2671,7 @@ static void rcu_do_batch(struct rcu_data *rdp)
// reporting, so check time limits for them.
if (rdp->rcu_cpu_kthread_status == RCU_KTHREAD_RUNNING &&
rcu_do_batch_check_time(count, tlimit, jlimit_check, jlimit)) {
- rdp->rcu_cpu_has_work = 1;
+ WRITE_ONCE(rdp->rcu_cpu_has_work, 1);
break;
}
}
@@ -2931,7 +2931,7 @@ static void invoke_rcu_core_kthread(void)
unsigned long flags;
local_irq_save(flags);
- __this_cpu_write(rcu_data.rcu_cpu_has_work, 1);
+ this_cpu_write(rcu_data.rcu_cpu_has_work, 1);
t = __this_cpu_read(rcu_data.rcu_cpu_kthread_task);
if (t != NULL && t != current)
rcu_wake_cond(t, __this_cpu_read(rcu_data.rcu_cpu_kthread_status));
@@ -2958,7 +2958,7 @@ static void rcu_cpu_kthread_park(unsigned int cpu)
static int rcu_cpu_kthread_should_run(unsigned int cpu)
{
- return __this_cpu_read(rcu_data.rcu_cpu_has_work);
+ return this_cpu_read(rcu_data.rcu_cpu_has_work);
}
/*
@@ -2979,7 +2979,7 @@ static void rcu_cpu_kthread(unsigned int cpu)
local_bh_disable();
*statusp = RCU_KTHREAD_RUNNING;
local_irq_disable();
- work = *workp;
+ work = READ_ONCE(*workp);
WRITE_ONCE(*workp, 0);
local_irq_enable();
if (work)
--
2.40.1
^ permalink raw reply related [flat|nested] 28+ messages in thread* [PATCH RFC 12/16] rcu: Mark interrupts-enabled accesses to rdp->cpu_no_qs.b.norm
2026-07-31 1:01 ` [PATCH v2 0/16] Miscellaneous RCU updates for v7.3 Paul E. McKenney
` (10 preceding siblings ...)
2026-07-31 1:01 ` [PATCH RFC 11/16] rcu: Mark accesses to rdp->rcu_cpu_has_work Paul E. McKenney
@ 2026-07-31 1:01 ` Paul E. McKenney
2026-07-31 1:01 ` [PATCH RFC 13/16] rcu: Remove unused rdp parameter from rcu_check_gp_start_stall() Paul E. McKenney
` (3 subsequent siblings)
15 siblings, 0 replies; 28+ messages in thread
From: Paul E. McKenney @ 2026-07-31 1:01 UTC (permalink / raw)
To: rcu; +Cc: linux-kernel, kernel-team, rostedt, Paul E. McKenney
The rdp->cpu_no_qs.b.norm field is accessed only by the current CPU,
but can be accessed both at task level and from interrupt handlers.
All accesses from interrupts-enabled code must therefore be marked.
This commit therefore converts from __this_cpu_read() to this_cpu_read(),
from __this_cpu_write() to this_cpu_write(), and plain C-language accesses
to READ_ONCE() and WRITE_ONCE(), but only in interrupts-enabled code.
KCSAN located this issue.
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
---
kernel/rcu/tree.c | 2 +-
kernel/rcu/tree_plugin.h | 8 ++++----
2 files changed, 5 insertions(+), 5 deletions(-)
diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c
index c9780e7c0e2afa..377153e0d93440 100644
--- a/kernel/rcu/tree.c
+++ b/kernel/rcu/tree.c
@@ -2540,7 +2540,7 @@ rcu_check_quiescent_state(struct rcu_data *rdp)
* Was there a quiescent state since the beginning of the grace
* period? If no, then exit and wait for the next call.
*/
- if (rdp->cpu_no_qs.b.norm)
+ if (READ_ONCE(rdp->cpu_no_qs.b.norm))
return;
/*
diff --git a/kernel/rcu/tree_plugin.h b/kernel/rcu/tree_plugin.h
index c3db1ad4a2807a..353effac5a37f1 100644
--- a/kernel/rcu/tree_plugin.h
+++ b/kernel/rcu/tree_plugin.h
@@ -298,11 +298,11 @@ static void rcu_preempt_ctxt_queue(struct rcu_node *rnp, struct rcu_data *rdp)
static void rcu_qs(void)
{
RCU_LOCKDEP_WARN(preemptible(), "rcu_qs() invoked with preemption enabled!!!\n");
- if (__this_cpu_read(rcu_data.cpu_no_qs.b.norm)) {
+ if (this_cpu_read(rcu_data.cpu_no_qs.b.norm)) {
trace_rcu_grace_period(TPS("rcu_preempt"),
__this_cpu_read(rcu_data.gp_seq),
TPS("cpuqs"));
- __this_cpu_write(rcu_data.cpu_no_qs.b.norm, false);
+ this_cpu_write(rcu_data.cpu_no_qs.b.norm, false);
barrier(); /* Coordinate with rcu_flavor_sched_clock_irq(). */
WRITE_ONCE(current->rcu_read_unlock_special.b.need_qs, false);
}
@@ -952,7 +952,7 @@ void rcu_read_unlock_strict(void)
* __rcu_read_unlock().
*/
rdp = this_cpu_ptr(&rcu_data);
- rdp->cpu_no_qs.b.norm = false;
+ WRITE_ONCE(rdp->cpu_no_qs.b.norm, false);
rcu_report_qs_rdp(rdp);
udelay(rcu_unlock_delay);
}
@@ -980,7 +980,7 @@ static void rcu_qs(void)
return;
trace_rcu_grace_period(TPS("rcu_sched"),
__this_cpu_read(rcu_data.gp_seq), TPS("cpuqs"));
- __this_cpu_write(rcu_data.cpu_no_qs.b.norm, false);
+ this_cpu_write(rcu_data.cpu_no_qs.b.norm, false);
if (this_cpu_read(rcu_data.cpu_no_qs.b.exp))
rcu_report_exp_rdp(this_cpu_ptr(&rcu_data));
}
--
2.40.1
^ permalink raw reply related [flat|nested] 28+ messages in thread* [PATCH RFC 13/16] rcu: Remove unused rdp parameter from rcu_check_gp_start_stall()
2026-07-31 1:01 ` [PATCH v2 0/16] Miscellaneous RCU updates for v7.3 Paul E. McKenney
` (11 preceding siblings ...)
2026-07-31 1:01 ` [PATCH RFC 12/16] rcu: Mark interrupts-enabled accesses to rdp->cpu_no_qs.b.norm Paul E. McKenney
@ 2026-07-31 1:01 ` Paul E. McKenney
2026-07-31 1:01 ` [PATCH RFC 14/16] rcu: Mark accesses to ->rcu_urgent_qs and ->rcu_need_heavy_qs Paul E. McKenney
` (2 subsequent siblings)
15 siblings, 0 replies; 28+ messages in thread
From: Paul E. McKenney @ 2026-07-31 1:01 UTC (permalink / raw)
To: rcu; +Cc: linux-kernel, kernel-team, rostedt, Zqiang, Paul E . McKenney
From: Zqiang <qiang.zhang@linux.dev>
The rcu_check_gp_start_stall() works entirely on rnp parameter,
and never uses it's rdp parameter. this commit therefore drop it,
updating both callers and the declaration.
No functional change.
Signed-off-by: Zqiang <qiang.zhang@linux.dev>
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
---
kernel/rcu/tree.c | 2 +-
kernel/rcu/tree.h | 3 +--
kernel/rcu/tree_stall.h | 5 ++---
3 files changed, 4 insertions(+), 6 deletions(-)
diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c
index 377153e0d93440..5dbc7506c0c0c0 100644
--- a/kernel/rcu/tree.c
+++ b/kernel/rcu/tree.c
@@ -2890,7 +2890,7 @@ static __latent_entropy void rcu_core(void)
rcu_accelerate_cbs_unlocked(rnp, rdp);
}
- rcu_check_gp_start_stall(rnp, rdp, rcu_jiffies_till_stall_check());
+ rcu_check_gp_start_stall(rnp, rcu_jiffies_till_stall_check());
/* If there are callbacks ready, invoke them. */
if (!rcu_rdp_is_offloaded(rdp) && rcu_segcblist_ready_cbs(&rdp->cblist) &&
diff --git a/kernel/rcu/tree.h b/kernel/rcu/tree.h
index ca88aaa29d615e..c3662c6ba39549 100644
--- a/kernel/rcu/tree.h
+++ b/kernel/rcu/tree.h
@@ -544,8 +544,7 @@ static bool rcu_nohz_full_cpu(void);
static void record_gp_stall_check_time(void);
static void rcu_iw_handler(struct irq_work *iwp);
static void check_cpu_stall(struct rcu_data *rdp);
-static void rcu_check_gp_start_stall(struct rcu_node *rnp, struct rcu_data *rdp,
- const unsigned long gpssdelay);
+static void rcu_check_gp_start_stall(struct rcu_node *rnp, const unsigned long gpssdelay);
/* Forward declarations for tree_exp.h. */
static void sync_rcu_do_polled_gp(struct work_struct *wp);
diff --git a/kernel/rcu/tree_stall.h b/kernel/rcu/tree_stall.h
index 45b9856ccd2b23..02684dadb4ee08 100644
--- a/kernel/rcu/tree_stall.h
+++ b/kernel/rcu/tree_stall.h
@@ -998,8 +998,7 @@ EXPORT_SYMBOL_GPL(show_rcu_gp_kthreads);
* This function checks for grace-period requests that fail to motivate
* RCU to come out of its idle mode.
*/
-static void rcu_check_gp_start_stall(struct rcu_node *rnp, struct rcu_data *rdp,
- const unsigned long gpssdelay)
+static void rcu_check_gp_start_stall(struct rcu_node *rnp, const unsigned long gpssdelay)
{
unsigned long flags;
unsigned long j;
@@ -1074,7 +1073,7 @@ void rcu_fwd_progress_check(unsigned long j)
__func__, jiffies - data_race(READ_ONCE(rcu_state.gp_end)));
preempt_disable();
rdp = this_cpu_ptr(&rcu_data);
- rcu_check_gp_start_stall(rdp->mynode, rdp, j);
+ rcu_check_gp_start_stall(rdp->mynode, j);
preempt_enable();
}
for_each_possible_cpu(cpu) {
--
2.40.1
^ permalink raw reply related [flat|nested] 28+ messages in thread* [PATCH RFC 14/16] rcu: Mark accesses to ->rcu_urgent_qs and ->rcu_need_heavy_qs
2026-07-31 1:01 ` [PATCH v2 0/16] Miscellaneous RCU updates for v7.3 Paul E. McKenney
` (12 preceding siblings ...)
2026-07-31 1:01 ` [PATCH RFC 13/16] rcu: Remove unused rdp parameter from rcu_check_gp_start_stall() Paul E. McKenney
@ 2026-07-31 1:01 ` Paul E. McKenney
2026-07-31 1:01 ` [PATCH RFC 15/16] rcu: Reduce stack usage in show_rcu_gp_kthreads() Paul E. McKenney
2026-07-31 1:01 ` [PATCH RFC 16/16] rcu: Mark interrupts-enabled accesses to rdp->cpu_no_qs.s Paul E. McKenney
15 siblings, 0 replies; 28+ messages in thread
From: Paul E. McKenney @ 2026-07-31 1:01 UTC (permalink / raw)
To: rcu; +Cc: linux-kernel, kernel-team, rostedt, Itai Handler,
Paul E . McKenney
From: Itai Handler <itai.handler@gmail.com>
rcu_all_qs() and rcu_note_context_switch() read/clear the per-CPU
->rcu_urgent_qs and ->rcu_need_heavy_qs flags with plain raw_cpu_read()
and this_cpu_write(), while the RCU core clears them with WRITE_ONCE() in
rcu_disable_urgency_upon_qs(). KCSAN flags the resulting same-CPU race:
BUG: KCSAN: data-race in rcu_all_qs / rcu_disable_urgency_upon_qs
It is benign -- the flags are advisory and rcu_all_qs() re-reads
->rcu_urgent_qs with smp_load_acquire() before acting on it -- but these
are the last unmarked accesses to the two flags; every other access
already uses READ_ONCE()/WRITE_ONCE()/smp_*. Mark them to match. No
functional change.
Reproduced on a PREEMPT_NONE, CONFIG_KCSAN_INTERRUPT_WATCHER=y kernel with
a pthreads program whose threads (two per CPU) loop reading a large file:
for (;;) {
int fd = open("/proc/kallsyms", O_RDONLY);
while (read(fd, buf, sizeof(buf)) > 0)
;
close(fd);
}
The read()s drive cond_resched() -> rcu_all_qs() while the busy CPUs keep
the grace period urgent, so the RCU core clears the flags concurrently.
Fixes: 2dba13f0b6c2 ("rcu: Switch urgent quiescent-state requests to rcu_data structure")
Signed-off-by: Itai Handler <itai.handler@gmail.com>
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
---
kernel/rcu/tree_plugin.h | 10 +++++-----
1 file changed, 5 insertions(+), 5 deletions(-)
diff --git a/kernel/rcu/tree_plugin.h b/kernel/rcu/tree_plugin.h
index 353effac5a37f1..844a9ced67cc0e 100644
--- a/kernel/rcu/tree_plugin.h
+++ b/kernel/rcu/tree_plugin.h
@@ -996,7 +996,7 @@ void rcu_all_qs(void)
{
unsigned long flags;
- if (!raw_cpu_read(rcu_data.rcu_urgent_qs))
+ if (!READ_ONCE(*raw_cpu_ptr(&rcu_data.rcu_urgent_qs)))
return;
preempt_disable(); // For CONFIG_PREEMPT_COUNT=y kernels
/* Load rcu_urgent_qs before other flags. */
@@ -1004,8 +1004,8 @@ void rcu_all_qs(void)
preempt_enable();
return;
}
- this_cpu_write(rcu_data.rcu_urgent_qs, false);
- if (unlikely(raw_cpu_read(rcu_data.rcu_need_heavy_qs))) {
+ WRITE_ONCE(*this_cpu_ptr(&rcu_data.rcu_urgent_qs), false);
+ if (unlikely(READ_ONCE(*this_cpu_ptr(&rcu_data.rcu_need_heavy_qs)))) {
local_irq_save(flags);
rcu_momentary_eqs();
local_irq_restore(flags);
@@ -1025,8 +1025,8 @@ void rcu_note_context_switch(bool preempt)
/* Load rcu_urgent_qs before other flags. */
if (!smp_load_acquire(this_cpu_ptr(&rcu_data.rcu_urgent_qs)))
goto out;
- this_cpu_write(rcu_data.rcu_urgent_qs, false);
- if (unlikely(raw_cpu_read(rcu_data.rcu_need_heavy_qs)))
+ WRITE_ONCE(*this_cpu_ptr(&rcu_data.rcu_urgent_qs), false);
+ if (unlikely(READ_ONCE(*this_cpu_ptr(&rcu_data.rcu_need_heavy_qs))))
rcu_momentary_eqs();
out:
rcu_tasks_qs(current, preempt);
--
2.40.1
^ permalink raw reply related [flat|nested] 28+ messages in thread* [PATCH RFC 15/16] rcu: Reduce stack usage in show_rcu_gp_kthreads()
2026-07-31 1:01 ` [PATCH v2 0/16] Miscellaneous RCU updates for v7.3 Paul E. McKenney
` (13 preceding siblings ...)
2026-07-31 1:01 ` [PATCH RFC 14/16] rcu: Mark accesses to ->rcu_urgent_qs and ->rcu_need_heavy_qs Paul E. McKenney
@ 2026-07-31 1:01 ` Paul E. McKenney
2026-07-31 1:01 ` [PATCH RFC 16/16] rcu: Mark interrupts-enabled accesses to rdp->cpu_no_qs.s Paul E. McKenney
15 siblings, 0 replies; 28+ messages in thread
From: Paul E. McKenney @ 2026-07-31 1:01 UTC (permalink / raw)
To: rcu; +Cc: linux-kernel, kernel-team, rostedt, Zqiang, Paul E . McKenney
From: Zqiang <qiang.zhang@linux.dev>
When CONFIG_KASAN=y and CONFIG_KASAN_STACK=y builds, the
show_rcu_gp_kthreads() exceeds the 1024-byte frame-size limit:
make kernel/rcu/tree.o KCFLAGS="-fstack-usage"
DESCEND objtool
DESCEND bpf/resolve_btfids
INSTALL libsubcmd_headers
CC kernel/rcu/tree.o
In file included from kernel/rcu/tree.c:4998:
kernel/rcu/tree_stall.h: In function 'show_rcu_gp_kthreads':
kernel/rcu/tree_stall.h:994:1: warning: the frame size of 1656 bytes is larger than 1024 bytes [-Wframe-larger-than=]
grep show_rcu kernel/rcu/tree.su
tree_nocb.h:1622:13:show_rcu_nocb_state 896 dynamic,bounded
tree_stall.h:933:6:show_rcu_gp_kthreads 1784 dynamic,bounded
tree_stall.h:1102:13:sysrq_show_rcu 16 static
Wrap the pr_info() into two noinline_for_stack helpers function:
show_rcu_state() print rcu_state status, and show_rcu_node()
print single rcu_node status.
After apply this change:
grep show_rcu kernel/rcu/tree.su
tree_stall.h:955:22:show_rcu_node 696 dynamic,bounded
tree_stall.h:930:22:show_rcu_state 872 dynamic,bounded
tree_nocb.h:1622:13:show_rcu_nocb_state 896 dynamic,bounded
tree_stall.h:972:6:show_rcu_gp_kthreads 544 static
tree_stall.h:1113:13:sysrq_show_rcu 16 static
Signed-off-by: Zqiang <qiang.zhang@linux.dev>
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
---
kernel/rcu/tree_stall.h | 47 +++++++++++++++++++++++++----------------
1 file changed, 29 insertions(+), 18 deletions(-)
diff --git a/kernel/rcu/tree_stall.h b/kernel/rcu/tree_stall.h
index 02684dadb4ee08..20634edfb44d8a 100644
--- a/kernel/rcu/tree_stall.h
+++ b/kernel/rcu/tree_stall.h
@@ -927,20 +927,13 @@ bool rcu_check_boost_fail(unsigned long gp_state, int *cpup)
}
EXPORT_SYMBOL_GPL(rcu_check_boost_fail);
-/*
- * Show the state of the grace-period kthreads.
- */
-void show_rcu_gp_kthreads(void)
+static noinline_for_stack void show_rcu_state(void)
{
- unsigned long cbs = 0;
- int cpu;
unsigned long j;
unsigned long ja;
unsigned long jr;
unsigned long js;
unsigned long jw;
- struct rcu_data *rdp;
- struct rcu_node *rnp;
struct task_struct *t = READ_ONCE(rcu_state.gp_kthread);
j = jiffies;
@@ -957,21 +950,39 @@ void show_rcu_gp_kthreads(void)
(long)data_race(READ_ONCE(rcu_get_root()->gp_seq_needed)),
data_race(READ_ONCE(rcu_state.gp_max)),
data_race(READ_ONCE(rcu_state.gp_flags)));
+}
+
+static noinline_for_stack void show_rcu_node(struct rcu_node *rnp)
+{
+ pr_info("\trcu_node %d:%d ->gp_seq %ld ->gp_seq_needed %ld ->qsmask %#lx %c%c%c%c ->n_boosts %ld\n",
+ rnp->grplo, rnp->grphi,
+ (long)data_race(READ_ONCE(rnp->gp_seq)),
+ (long)data_race(READ_ONCE(rnp->gp_seq_needed)),
+ data_race(READ_ONCE(rnp->qsmask)),
+ ".b"[!!data_race(READ_ONCE(rnp->boost_kthread_task))],
+ ".B"[!!data_race(READ_ONCE(rnp->boost_tasks))],
+ ".E"[!!data_race(READ_ONCE(rnp->exp_tasks))],
+ ".G"[!!data_race(READ_ONCE(rnp->gp_tasks))],
+ data_race(READ_ONCE(rnp->n_boosts)));
+}
+
+/*
+ * Show the state of the grace-period kthreads.
+ */
+void show_rcu_gp_kthreads(void)
+{
+ unsigned long cbs = 0;
+ int cpu;
+ struct rcu_data *rdp;
+ struct rcu_node *rnp;
+
+ show_rcu_state();
rcu_for_each_node_breadth_first(rnp) {
if (ULONG_CMP_GE(READ_ONCE(rcu_state.gp_seq), READ_ONCE(rnp->gp_seq_needed)) &&
!data_race(READ_ONCE(rnp->qsmask)) && !data_race(READ_ONCE(rnp->boost_tasks)) &&
!data_race(READ_ONCE(rnp->exp_tasks)) && !data_race(READ_ONCE(rnp->gp_tasks)))
continue;
- pr_info("\trcu_node %d:%d ->gp_seq %ld ->gp_seq_needed %ld ->qsmask %#lx %c%c%c%c ->n_boosts %ld\n",
- rnp->grplo, rnp->grphi,
- (long)data_race(READ_ONCE(rnp->gp_seq)),
- (long)data_race(READ_ONCE(rnp->gp_seq_needed)),
- data_race(READ_ONCE(rnp->qsmask)),
- ".b"[!!data_race(READ_ONCE(rnp->boost_kthread_task))],
- ".B"[!!data_race(READ_ONCE(rnp->boost_tasks))],
- ".E"[!!data_race(READ_ONCE(rnp->exp_tasks))],
- ".G"[!!data_race(READ_ONCE(rnp->gp_tasks))],
- data_race(READ_ONCE(rnp->n_boosts)));
+ show_rcu_node(rnp);
if (!rcu_is_leaf_node(rnp))
continue;
for_each_leaf_node_possible_cpu(rnp, cpu) {
--
2.40.1
^ permalink raw reply related [flat|nested] 28+ messages in thread* [PATCH RFC 16/16] rcu: Mark interrupts-enabled accesses to rdp->cpu_no_qs.s
2026-07-31 1:01 ` [PATCH v2 0/16] Miscellaneous RCU updates for v7.3 Paul E. McKenney
` (14 preceding siblings ...)
2026-07-31 1:01 ` [PATCH RFC 15/16] rcu: Reduce stack usage in show_rcu_gp_kthreads() Paul E. McKenney
@ 2026-07-31 1:01 ` Paul E. McKenney
15 siblings, 0 replies; 28+ messages in thread
From: Paul E. McKenney @ 2026-07-31 1:01 UTC (permalink / raw)
To: rcu; +Cc: linux-kernel, kernel-team, rostedt, Paul E. McKenney
The rdp->cpu_no_qs.s field is accessed only by the current CPU,
but can be accessed both at task level and from interrupt handlers.
All accesses from interrupts-enabled code must therefore be marked.
This commit therefore converts from __this_cpu_read() to this_cpu_read(),
but only in interrupts-enabled code, that is, the rcu_qs() function.
KCSAN located this issue.
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
---
kernel/rcu/tree_plugin.h | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/kernel/rcu/tree_plugin.h b/kernel/rcu/tree_plugin.h
index 844a9ced67cc0e..6904a4f826d1ea 100644
--- a/kernel/rcu/tree_plugin.h
+++ b/kernel/rcu/tree_plugin.h
@@ -976,7 +976,7 @@ static void __init rcu_bootup_announce(void)
static void rcu_qs(void)
{
RCU_LOCKDEP_WARN(preemptible(), "rcu_qs() invoked with preemption enabled!!!");
- if (!__this_cpu_read(rcu_data.cpu_no_qs.s))
+ if (!this_cpu_read(rcu_data.cpu_no_qs.s))
return;
trace_rcu_grace_period(TPS("rcu_sched"),
__this_cpu_read(rcu_data.gp_seq), TPS("cpuqs"));
--
2.40.1
^ permalink raw reply related [flat|nested] 28+ messages in thread
