* Re: [RFC PATCH] Documentation/rv: Explain epoll and aborted sleeps
From: Gabriele Monaco @ 2026-07-24 7:20 UTC (permalink / raw)
To: Chao Liu, Nam Cao, Steven Rostedt, Jonathan Corbet
Cc: lianux.mm, lianux.wang, Shuah Khan, linux-trace-kernel, linux-doc,
linux-kernel
In-Reply-To: <20260722045706.85633-1-chao.liu@processmission.com>
On Wed, 2026-07-22 at 12:57 +0800, Chao Liu wrote:
> The rtapp sleep monitor accepts epoll_wait() as a valid sleeping
> reason, but the prose only discusses clock_nanosleep and futexes. Add
> epoll_wait to the list of valid sleeping reasons.
>
> ABORT_SLEEP represents a task restoring TASK_RUNNING before entering
> the scheduler, but its meaning is not described. Explain why the
> aborted sleep attempt is valid.
>
> This RFC is based on Nam Cao's pending "rv: rtapp monitor update" v2
> series:
>
> https://lore.kernel.org/r/cover.1781852967.git.namcao@linutronix.de
>
> Signed-off-by: Chao Liu <chao.liu@processmission.com>
> ---
Thanks for the contribution! I have a couple of comments, we want to be
assertive in this docs: in general, let's not say something like "this is valid
because the monitor thinks it's valid" but "this is valid because it is rt-safe
for reason X".
> Documentation/trace/rv/monitor_rtapp.rst | 6 ++++++
> 1 file changed, 6 insertions(+)
>
> diff --git a/Documentation/trace/rv/monitor_rtapp.rst
> b/Documentation/trace/rv/monitor_rtapp.rst
> index 238b59395ff5..234ed4d581ac 100644
> --- a/Documentation/trace/rv/monitor_rtapp.rst
> +++ b/Documentation/trace/rv/monitor_rtapp.rst
> @@ -67,6 +67,8 @@ thread to sleep for one of the following reasons:
> variables as safe for real-time. As an alternative, the librtpi library
> exists to provide a conditional variable implementation that is correct
> for
> real-time applications in Linux.
> + - Real-time thread waiting for events using `epoll_wait`,
> + which the monitor accepts as a valid sleeping reason for real-time tasks.
Not adding much value, try:
which is a real-time-safe syscall for sleeping as it uses PI-aware locking.
>
> Beside the reason for sleeping, the eventual waker should also be
> real-time-safe. Namely, one of:
> @@ -114,6 +116,10 @@ The monitor's specification is::
> ALLOWLIST = BLOCK_ON_RT_MUTEX
> or FUTEX_LOCK_PI
>
> +`ABORT_SLEEP` represents a task restoring its state to `TASK_RUNNING` before
> +entering the scheduler. In this case, the task does not actually block,
> + so the monitor treats the aborted sleep attempt as valid.
This sentence is a bit vague. Indeed an /aborted/ sleep is a valid wakeup, as if
the task woke up itself without even sleeping, saying "the monitor treats the
attempt as valid" adds no value.
You could rephrase it to (full paragraph for clarity):
`ABORT_SLEEP` represents a task restoring its state to `TASK_RUNNING` before
entering the scheduler. In this case, the task does not actually block, so it is
back to runnable without any wakeup sequence unsafe for real-time.
What do you think?
Thanks,
Gabriele
^ permalink raw reply
* Re: [PATCH v1 00/11] RCU: Enable callbacks to benefit from expedited grace periods
From: Frederic Weisbecker @ 2026-07-24 12:44 UTC (permalink / raw)
To: Paul E. McKenney
Cc: Puranjay Mohan, rcu, linux-kernel, linux-trace-kernel,
Neeraj Upadhyay, Joel Fernandes, Josh Triplett, Boqun Feng,
Uladzislau Rezki, Steven Rostedt, Mathieu Desnoyers,
Lai Jiangshan, Zqiang, Masami Hiramatsu, Davidlohr Bueso,
Breno Leitao
In-Reply-To: <062ddbcd-dac1-4ee1-9d3f-ef115a667d6a@paulmck-laptop>
Le Thu, Jul 23, 2026 at 05:31:17PM -0700, Paul E. McKenney a écrit :
> On Wed, Jun 24, 2026 at 06:23:42AM -0700, Puranjay Mohan wrote:
> > This series lets call_rcu() callbacks be reclaimed as soon as either a
> > normal or an expedited grace period that covers them has elapsed, rather
> > than always waiting for a normal grace period.
> >
> > Motivation
> > ==========
> > Today there is an asymmetry: synchronize_rcu_expedited() callers get fast
> > reclaim, but call_rcu() callers never benefit from those same expedited
> > grace periods, even though an expedited GP proves exactly the same thing
> > as a normal one -- all pre-existing readers are done. When expedited GPs
> > are running on the system (driven by other subsystems), call_rcu()
> > callbacks that could already be freed instead sit in RCU_WAIT_TAIL until
> > the next normal GP. This series treats a grace period as a grace period
> > regardless of how it was driven, so memory is reclaimed sooner.
> >
> > Design
> > ======
> > Callback segments now record both the normal and expedited grace-period
> > sequence in struct rcu_gp_seq, and rcu_segcblist_advance() releases a
> > segment as soon as poll_state_synchronize_rcu_full() reports that either
> > has completed. Three notification paths are taught about expedited
> > completion so the advance actually happens: the NOCB rcuog kthreads,
> > the rcu_pending() tick gate, and rcu_core().
> >
> > Changelog:
> > RFC: https://lore.kernel.org/all/20260417231203.785172-1-puranjay@kernel.org/
> > Changes in v1:
> > - New prep patch 1 renames struct rcu_gp_oldstate to struct rcu_gp_seq
> > and its fields rgos_norm/rgos_exp to norm/exp tree-wide (Frederic).
> > - The rcu_segcblist segment field stays named gp_seq; only its type
> > changes (Frederic).
> > - Patch 8 (NOCB wake) is reworked. v1 woke the wrong waitqueue
> > (rdp_gp->nocb_gp_wq via wake_nocb_gp() rather than the leaf
> > rnp->nocb_gp_wq[] that an rcuog kthread waiting for a GP sleeps on),
> > and the wait condition only checked the normal ->gp_seq. The rcuog
> > grace-period wait now tracks a struct rcu_gp_seq and is released via
> > poll_state_synchronize_rcu_full(); rcu_exp_wait_wake() wakes the leaf
> > node through the new rcu_nocb_exp_cleanup() (Frederic).
> > - rcu_pending() uses a new memory-ordering-free
> > poll_state_synchronize_rcu_full_unordered() to avoid memory barriers
> > on every tick, leaving the ordering duty to rcu_core() (Frederic).
> >
> > Still open: Frederic asked whether the first smp_mb() in
> > poll_state_synchronize_rcu_full() is needed on the callback-advance path
> > (patch 6). That path still uses the fully ordered helper; only
> > rcu_pending() was switched to the unordered variant. Happy to revisit.
> >
> > Puranjay Mohan (11):
> > rcu: Rename struct rcu_gp_oldstate to rcu_gp_seq
> > rcu/segcblist: Add SRCU and Tasks RCU wrapper functions
> > rcu/segcblist: Factor out rcu_segcblist_advance_compact() helper
> > rcu/segcblist: Track segment grace periods with struct rcu_gp_seq
> > rcu: Add RCU_GET_STATE_NOT_TRACKED for subsystems without expedited
> > GPs
> > rcu: Enable RCU callbacks to benefit from expedited grace periods
> > rcu: Update comments for gp_seq and expedited GP tracking
> > rcu: Wake NOCB rcuog kthreads on expedited grace period completion
> > rcu: Detect expedited grace period completion in rcu_pending()
> > rcu: Advance callbacks for expedited GP completion in rcu_core()
> > rcuscale: Add concurrent expedited GP threads for callback scaling
> > tests
>
> I do see the occasional failure when running TREE01, as in a failure or
> four every couple hundred hours of testing. This is the only rcutorture
> scenario that exercises callback (de)offloading, which might (or might
> not) be a clue.
>
> The console log of a failing run is attached, or you can find it here:
>
> https://drive.google.com/file/d/1EH2P4SDy2a-SJndkN3GbCTGAXxI2xevm/view?usp=sharing
>
> This is a "??? Writer stall state" failure, where rcutorture never learns
> of grace-period completion. Possibly because no grace period completed.
> But there are no RCU CPU stall warnings. In addition, these two lines
>
> [ 631.032657] ??? Writer stall state RTWS_EXP_SYNC(4) g152813 f0x0 ->state D cpu 1
> . . .
> [ 3610.863114] ??? Writer stall state RTWS_EXP_SYNC(4) g736160 f0x0 ->state D cpu 1
>
> show that the grace-period sequence number advanced from 152813 to
> 736160 in about 50 minutes. So grace periods really are completing,
> lots of them.
>
> This rcutorture scenario has rcu_nocbs CPUs, and CPU 5 is in trouble:
>
> [ 631.066109] rcu: CB 5^4->4 KblSW F21044 L21044 C116 .W120(148380/168068).N295B101 q516 S CPU 16
> . . .
> [ 3610.869914] rcu: CB 5^4->4 KblSW F3000848 L3000848 C116 .W120(148380/168068).N295B101 q516 S CPU 16
>
> There are 120 callbacks on the WAIT segment, which are waiting on
> grace-period sequence number 148380, which was long gone even back at
> 631.032657 seconds in (g152813). There are 295 more callbacks in the NEXT
> segment, which long ago should have been assigned a grace period and moved
> into the currently empty NEXT_READY segment (which is the "." right before
> the "N295"). There are yet another 101 callbacks on the bypass list.
>
> This suggests that the rcuog kthread isn't being awakened when it
> should be. And that something is failing to advance callbacks out of
> NEXT in a timely fashion.
>
> Thoughts?
If a grace period has elapsed while in the middle of nocb_gp_wait(),
after the call to rcu_advance_cbs() and before the following polling which
determines if we a grace period must be waited upon, then the kthread
will wait for new callbacks only (and will ignore waiting callbacks that
can now be advanced).
Does the following help?
diff --git a/kernel/rcu/tree_nocb.h b/kernel/rcu/tree_nocb.h
index 6da1b8f52476..96d76eb6259d 100644
--- a/kernel/rcu/tree_nocb.h
+++ b/kernel/rcu/tree_nocb.h
@@ -792,22 +792,12 @@ static noinline_for_stack void nocb_gp_wait(struct rcu_data *my_rdp)
RCU_NEXT_READY_TAIL);
raw_spin_unlock_rcu_node(rnp); /* irqs disabled. */
}
- // Need to wait on some grace period?
+
WARN_ON_ONCE(wasempty &&
!rcu_segcblist_restempty(&rdp->cblist,
RCU_NEXT_READY_TAIL));
- /*
- * Only request a GP wait if the next pending callback's
- * GP has not already completed (normal or expedited).
- * If poll_state_synchronize_rcu_full() says it completed,
- * then rcu_advance_cbs() above already moved those
- * callbacks to RCU_DONE_TAIL, so there is no GP to wait
- * for. Any remaining callbacks got new (future) GP
- * numbers from rcu_accelerate_cbs() inside
- * rcu_advance_cbs() and will be handled on the next pass.
- */
- if (rcu_segcblist_nextgp(&rdp->cblist, &cur_gp_seq) &&
- !poll_state_synchronize_rcu_full(&cur_gp_seq)) {
+ // Need to wait on some grace period?
+ if (rcu_segcblist_nextgp(&rdp->cblist, &cur_gp_seq)) {
/*
* Track the earliest pending normal and expedited GP
* across the group so the wait below can be released by
^ permalink raw reply related
* Re: [PATCH v1 00/11] RCU: Enable callbacks to benefit from expedited grace periods
From: Puranjay Mohan @ 2026-07-24 12:52 UTC (permalink / raw)
To: Frederic Weisbecker
Cc: Paul E. McKenney, rcu, linux-kernel, linux-trace-kernel,
Neeraj Upadhyay, Joel Fernandes, Josh Triplett, Boqun Feng,
Uladzislau Rezki, Steven Rostedt, Mathieu Desnoyers,
Lai Jiangshan, Zqiang, Masami Hiramatsu, Davidlohr Bueso,
Breno Leitao
In-Reply-To: <amNeS_ahHPLqXIGT@localhost.localdomain>
On Fri, Jul 24, 2026 at 1:45 PM Frederic Weisbecker <frederic@kernel.org> wrote:
>
> Le Thu, Jul 23, 2026 at 05:31:17PM -0700, Paul E. McKenney a écrit :
> > On Wed, Jun 24, 2026 at 06:23:42AM -0700, Puranjay Mohan wrote:
> > > This series lets call_rcu() callbacks be reclaimed as soon as either a
> > > normal or an expedited grace period that covers them has elapsed, rather
> > > than always waiting for a normal grace period.
> > >
> > > Motivation
> > > ==========
> > > Today there is an asymmetry: synchronize_rcu_expedited() callers get fast
> > > reclaim, but call_rcu() callers never benefit from those same expedited
> > > grace periods, even though an expedited GP proves exactly the same thing
> > > as a normal one -- all pre-existing readers are done. When expedited GPs
> > > are running on the system (driven by other subsystems), call_rcu()
> > > callbacks that could already be freed instead sit in RCU_WAIT_TAIL until
> > > the next normal GP. This series treats a grace period as a grace period
> > > regardless of how it was driven, so memory is reclaimed sooner.
> > >
> > > Design
> > > ======
> > > Callback segments now record both the normal and expedited grace-period
> > > sequence in struct rcu_gp_seq, and rcu_segcblist_advance() releases a
> > > segment as soon as poll_state_synchronize_rcu_full() reports that either
> > > has completed. Three notification paths are taught about expedited
> > > completion so the advance actually happens: the NOCB rcuog kthreads,
> > > the rcu_pending() tick gate, and rcu_core().
> > >
> > > Changelog:
> > > RFC: https://lore.kernel.org/all/20260417231203.785172-1-puranjay@kernel.org/
> > > Changes in v1:
> > > - New prep patch 1 renames struct rcu_gp_oldstate to struct rcu_gp_seq
> > > and its fields rgos_norm/rgos_exp to norm/exp tree-wide (Frederic).
> > > - The rcu_segcblist segment field stays named gp_seq; only its type
> > > changes (Frederic).
> > > - Patch 8 (NOCB wake) is reworked. v1 woke the wrong waitqueue
> > > (rdp_gp->nocb_gp_wq via wake_nocb_gp() rather than the leaf
> > > rnp->nocb_gp_wq[] that an rcuog kthread waiting for a GP sleeps on),
> > > and the wait condition only checked the normal ->gp_seq. The rcuog
> > > grace-period wait now tracks a struct rcu_gp_seq and is released via
> > > poll_state_synchronize_rcu_full(); rcu_exp_wait_wake() wakes the leaf
> > > node through the new rcu_nocb_exp_cleanup() (Frederic).
> > > - rcu_pending() uses a new memory-ordering-free
> > > poll_state_synchronize_rcu_full_unordered() to avoid memory barriers
> > > on every tick, leaving the ordering duty to rcu_core() (Frederic).
> > >
> > > Still open: Frederic asked whether the first smp_mb() in
> > > poll_state_synchronize_rcu_full() is needed on the callback-advance path
> > > (patch 6). That path still uses the fully ordered helper; only
> > > rcu_pending() was switched to the unordered variant. Happy to revisit.
> > >
> > > Puranjay Mohan (11):
> > > rcu: Rename struct rcu_gp_oldstate to rcu_gp_seq
> > > rcu/segcblist: Add SRCU and Tasks RCU wrapper functions
> > > rcu/segcblist: Factor out rcu_segcblist_advance_compact() helper
> > > rcu/segcblist: Track segment grace periods with struct rcu_gp_seq
> > > rcu: Add RCU_GET_STATE_NOT_TRACKED for subsystems without expedited
> > > GPs
> > > rcu: Enable RCU callbacks to benefit from expedited grace periods
> > > rcu: Update comments for gp_seq and expedited GP tracking
> > > rcu: Wake NOCB rcuog kthreads on expedited grace period completion
> > > rcu: Detect expedited grace period completion in rcu_pending()
> > > rcu: Advance callbacks for expedited GP completion in rcu_core()
> > > rcuscale: Add concurrent expedited GP threads for callback scaling
> > > tests
> >
> > I do see the occasional failure when running TREE01, as in a failure or
> > four every couple hundred hours of testing. This is the only rcutorture
> > scenario that exercises callback (de)offloading, which might (or might
> > not) be a clue.
> >
> > The console log of a failing run is attached, or you can find it here:
> >
> > https://drive.google.com/file/d/1EH2P4SDy2a-SJndkN3GbCTGAXxI2xevm/view?usp=sharing
> >
> > This is a "??? Writer stall state" failure, where rcutorture never learns
> > of grace-period completion. Possibly because no grace period completed.
> > But there are no RCU CPU stall warnings. In addition, these two lines
> >
> > [ 631.032657] ??? Writer stall state RTWS_EXP_SYNC(4) g152813 f0x0 ->state D cpu 1
> > . . .
> > [ 3610.863114] ??? Writer stall state RTWS_EXP_SYNC(4) g736160 f0x0 ->state D cpu 1
> >
> > show that the grace-period sequence number advanced from 152813 to
> > 736160 in about 50 minutes. So grace periods really are completing,
> > lots of them.
> >
> > This rcutorture scenario has rcu_nocbs CPUs, and CPU 5 is in trouble:
> >
> > [ 631.066109] rcu: CB 5^4->4 KblSW F21044 L21044 C116 .W120(148380/168068).N295B101 q516 S CPU 16
> > . . .
> > [ 3610.869914] rcu: CB 5^4->4 KblSW F3000848 L3000848 C116 .W120(148380/168068).N295B101 q516 S CPU 16
> >
> > There are 120 callbacks on the WAIT segment, which are waiting on
> > grace-period sequence number 148380, which was long gone even back at
> > 631.032657 seconds in (g152813). There are 295 more callbacks in the NEXT
> > segment, which long ago should have been assigned a grace period and moved
> > into the currently empty NEXT_READY segment (which is the "." right before
> > the "N295"). There are yet another 101 callbacks on the bypass list.
> >
> > This suggests that the rcuog kthread isn't being awakened when it
> > should be. And that something is failing to advance callbacks out of
> > NEXT in a timely fashion.
> >
> > Thoughts?
>
> If a grace period has elapsed while in the middle of nocb_gp_wait(),
> after the call to rcu_advance_cbs() and before the following polling which
> determines if we a grace period must be waited upon, then the kthread
> will wait for new callbacks only (and will ignore waiting callbacks that
> can now be advanced).
>
> Does the following help?
>
> diff --git a/kernel/rcu/tree_nocb.h b/kernel/rcu/tree_nocb.h
> index 6da1b8f52476..96d76eb6259d 100644
> --- a/kernel/rcu/tree_nocb.h
> +++ b/kernel/rcu/tree_nocb.h
> @@ -792,22 +792,12 @@ static noinline_for_stack void nocb_gp_wait(struct rcu_data *my_rdp)
> RCU_NEXT_READY_TAIL);
> raw_spin_unlock_rcu_node(rnp); /* irqs disabled. */
> }
> - // Need to wait on some grace period?
> +
> WARN_ON_ONCE(wasempty &&
> !rcu_segcblist_restempty(&rdp->cblist,
> RCU_NEXT_READY_TAIL));
> - /*
> - * Only request a GP wait if the next pending callback's
> - * GP has not already completed (normal or expedited).
> - * If poll_state_synchronize_rcu_full() says it completed,
> - * then rcu_advance_cbs() above already moved those
> - * callbacks to RCU_DONE_TAIL, so there is no GP to wait
> - * for. Any remaining callbacks got new (future) GP
> - * numbers from rcu_accelerate_cbs() inside
> - * rcu_advance_cbs() and will be handled on the next pass.
> - */
> - if (rcu_segcblist_nextgp(&rdp->cblist, &cur_gp_seq) &&
> - !poll_state_synchronize_rcu_full(&cur_gp_seq)) {
> + // Need to wait on some grace period?
> + if (rcu_segcblist_nextgp(&rdp->cblist, &cur_gp_seq)) {
> /*
> * Track the earliest pending normal and expedited GP
> * across the group so the wait below can be released by
This looks correct to me, this optimization with
!poll_state_synchronize_rcu_full(&cur_gp_seq) opened up this race as
you correctly explained.
Thanks,
Puranjay
^ permalink raw reply
* Re: [PATCH v4] tracing: Fix use-after-free freeing trigger private data
From: Steven Rostedt @ 2026-07-24 13:42 UTC (permalink / raw)
To: sashiko-bot@kernel.org, sashiko-reviews@lists.linux.dev
Cc: David Carlier, Masami Hiramatsu, Mathieu Desnoyers,
linux-trace-kernel, linux-kernel
In-Reply-To: <20260724030523.19081-1-devnexen@gmail.com>
[ Replying to Sashiko ]
> --- Patch [1]: [PATCH v4] tracing: Fix use-after-free freeing trigger private data ---
> commit cd29772ccd2634c1b696094501e9c7fd0bcd1c25
> Author: David Carlier <devnexen@gmail.com>
>
> > diff --git a/kernel/trace/trace_events_hist.c b/kernel/trace/trace_events_hist.c
> > index 82ce492ab2681..58d28cd1afa3b 100644
> > --- a/kernel/trace/trace_events_hist.c
> > +++ b/kernel/trace/trace_events_hist.c
> > @@ -6349,6 +6349,7 @@ static void event_hist_trigger_free(struct event_trigger_data *data)
> >
> > trigger_data_free(data);
> >
> > + tracepoint_synchronize_unregister();
> > remove_hist_vars(hist_data);
> >
> > unregister_field_var_hists(hist_data);
>
> This isn't a bug introduced by this patch, but since we are fixing the teardown
> paths here, do the error paths in event_hist_trigger_parse() and
> event_enable_trigger_parse() need similar synchronization?
>
> For example, in event_hist_trigger_parse():
>
> out_free:
> remove_hist_vars(hist_data);
> trigger_data_free(trigger_data);
> destroy_hist_data(hist_data);
>
> If trigger registration fails after the trigger was briefly published to
> file->triggers, could a concurrent tracepoint that observed the trigger
> dereference the synchronously freed hist_data because we didn't wait for an RCU
> grace period?
Where was it published to file->triggers?
I see it happening as the last operation that can fail. If that fails, then
we hit the error path where it is not registered. Is there someplace it can
become visible and then fail that I'm not seeing?
From what I see, the last command before "success" is:
ret = hist_trigger_enable(trigger_data, file);
And that does:
static int hist_trigger_enable(struct event_trigger_data *data,
struct trace_event_file *file)
{
int ret = 0;
list_add_tail_rcu(&data->list, &file->triggers);
update_cond_flag(file);
if (trace_event_trigger_enable_disable(file, 1) < 0) {
list_del_rcu(&data->list);
update_cond_flag(file);
ret--;
}
return ret;
}
Now it adds to the file trigger and then tries to enable that event. If it
fails to enable it, it removes it from the list. But if the event is not
enabled, it can't be racing with this.
>
> [ ... ]
>
> > diff --git a/kernel/trace/trace_events_trigger.c b/kernel/trace/trace_events_trigger.c
> > index 655db2e825134..46e60b70a4bbc 100644
> > --- a/kernel/trace/trace_events_trigger.c
> > +++ b/kernel/trace/trace_events_trigger.c
> > @@ -1728,9 +1740,9 @@ void event_enable_trigger_free(struct event_trigger_data *data)
> > if (!data->ref) {
> > /* Remove the SOFT_MODE flag */
> > trace_event_enable_disable(enable_data->file, 0, 1);
> > + data->private_data_free = enable_trigger_private_data_free;
> > trace_event_put_ref(enable_data->file->event_call);
> > trigger_data_free(data);
> > - kfree(enable_data);
> > }
> > }
>
> This is a pre-existing issue, but by deferring the kfree of enable_data while
> keeping trace_event_put_ref() synchronous, could we still face a Use-After-Free
> issue during module unload?
>
> If trace_event_put_ref(enable_data->file->event_call) allows the target module
> to be unloaded and its trace event files to be freed immediately, could a
> concurrent tracepoint RCU reader dereference the freed enable_data->file in
> event_enable_trigger() when attempting to clear
> EVENT_FILE_FL_SOFT_DISABLED_BIT?
Hmm, this does look like it can be an issue. I'll write up a patch on top
of this one.
-- Steve
^ permalink raw reply
* Re: [PATCH v1 00/11] RCU: Enable callbacks to benefit from expedited grace periods
From: Puranjay Mohan @ 2026-07-24 13:47 UTC (permalink / raw)
To: Frederic Weisbecker
Cc: Paul E. McKenney, rcu, linux-kernel, linux-trace-kernel,
Neeraj Upadhyay, Joel Fernandes, Josh Triplett, Boqun Feng,
Uladzislau Rezki, Steven Rostedt, Mathieu Desnoyers,
Lai Jiangshan, Zqiang, Masami Hiramatsu, Davidlohr Bueso,
Breno Leitao
In-Reply-To: <CANk7y0hQQ3oDKt+_8oh4t=8NfMRSa80a9jGr+DG+zbrcuUskTA@mail.gmail.com>
On Fri, Jul 24, 2026 at 1:52 PM Puranjay Mohan <puranjay12@gmail.com> wrote:
>
> On Fri, Jul 24, 2026 at 1:45 PM Frederic Weisbecker <frederic@kernel.org> wrote:
> >
> > Le Thu, Jul 23, 2026 at 05:31:17PM -0700, Paul E. McKenney a écrit :
> > > On Wed, Jun 24, 2026 at 06:23:42AM -0700, Puranjay Mohan wrote:
> > > > This series lets call_rcu() callbacks be reclaimed as soon as either a
> > > > normal or an expedited grace period that covers them has elapsed, rather
> > > > than always waiting for a normal grace period.
> > > >
> > > > Motivation
> > > > ==========
> > > > Today there is an asymmetry: synchronize_rcu_expedited() callers get fast
> > > > reclaim, but call_rcu() callers never benefit from those same expedited
> > > > grace periods, even though an expedited GP proves exactly the same thing
> > > > as a normal one -- all pre-existing readers are done. When expedited GPs
> > > > are running on the system (driven by other subsystems), call_rcu()
> > > > callbacks that could already be freed instead sit in RCU_WAIT_TAIL until
> > > > the next normal GP. This series treats a grace period as a grace period
> > > > regardless of how it was driven, so memory is reclaimed sooner.
> > > >
> > > > Design
> > > > ======
> > > > Callback segments now record both the normal and expedited grace-period
> > > > sequence in struct rcu_gp_seq, and rcu_segcblist_advance() releases a
> > > > segment as soon as poll_state_synchronize_rcu_full() reports that either
> > > > has completed. Three notification paths are taught about expedited
> > > > completion so the advance actually happens: the NOCB rcuog kthreads,
> > > > the rcu_pending() tick gate, and rcu_core().
> > > >
> > > > Changelog:
> > > > RFC: https://lore.kernel.org/all/20260417231203.785172-1-puranjay@kernel.org/
> > > > Changes in v1:
> > > > - New prep patch 1 renames struct rcu_gp_oldstate to struct rcu_gp_seq
> > > > and its fields rgos_norm/rgos_exp to norm/exp tree-wide (Frederic).
> > > > - The rcu_segcblist segment field stays named gp_seq; only its type
> > > > changes (Frederic).
> > > > - Patch 8 (NOCB wake) is reworked. v1 woke the wrong waitqueue
> > > > (rdp_gp->nocb_gp_wq via wake_nocb_gp() rather than the leaf
> > > > rnp->nocb_gp_wq[] that an rcuog kthread waiting for a GP sleeps on),
> > > > and the wait condition only checked the normal ->gp_seq. The rcuog
> > > > grace-period wait now tracks a struct rcu_gp_seq and is released via
> > > > poll_state_synchronize_rcu_full(); rcu_exp_wait_wake() wakes the leaf
> > > > node through the new rcu_nocb_exp_cleanup() (Frederic).
> > > > - rcu_pending() uses a new memory-ordering-free
> > > > poll_state_synchronize_rcu_full_unordered() to avoid memory barriers
> > > > on every tick, leaving the ordering duty to rcu_core() (Frederic).
> > > >
> > > > Still open: Frederic asked whether the first smp_mb() in
> > > > poll_state_synchronize_rcu_full() is needed on the callback-advance path
> > > > (patch 6). That path still uses the fully ordered helper; only
> > > > rcu_pending() was switched to the unordered variant. Happy to revisit.
> > > >
> > > > Puranjay Mohan (11):
> > > > rcu: Rename struct rcu_gp_oldstate to rcu_gp_seq
> > > > rcu/segcblist: Add SRCU and Tasks RCU wrapper functions
> > > > rcu/segcblist: Factor out rcu_segcblist_advance_compact() helper
> > > > rcu/segcblist: Track segment grace periods with struct rcu_gp_seq
> > > > rcu: Add RCU_GET_STATE_NOT_TRACKED for subsystems without expedited
> > > > GPs
> > > > rcu: Enable RCU callbacks to benefit from expedited grace periods
> > > > rcu: Update comments for gp_seq and expedited GP tracking
> > > > rcu: Wake NOCB rcuog kthreads on expedited grace period completion
> > > > rcu: Detect expedited grace period completion in rcu_pending()
> > > > rcu: Advance callbacks for expedited GP completion in rcu_core()
> > > > rcuscale: Add concurrent expedited GP threads for callback scaling
> > > > tests
> > >
> > > I do see the occasional failure when running TREE01, as in a failure or
> > > four every couple hundred hours of testing. This is the only rcutorture
> > > scenario that exercises callback (de)offloading, which might (or might
> > > not) be a clue.
> > >
> > > The console log of a failing run is attached, or you can find it here:
> > >
> > > https://drive.google.com/file/d/1EH2P4SDy2a-SJndkN3GbCTGAXxI2xevm/view?usp=sharing
> > >
> > > This is a "??? Writer stall state" failure, where rcutorture never learns
> > > of grace-period completion. Possibly because no grace period completed.
> > > But there are no RCU CPU stall warnings. In addition, these two lines
> > >
> > > [ 631.032657] ??? Writer stall state RTWS_EXP_SYNC(4) g152813 f0x0 ->state D cpu 1
> > > . . .
> > > [ 3610.863114] ??? Writer stall state RTWS_EXP_SYNC(4) g736160 f0x0 ->state D cpu 1
> > >
> > > show that the grace-period sequence number advanced from 152813 to
> > > 736160 in about 50 minutes. So grace periods really are completing,
> > > lots of them.
> > >
> > > This rcutorture scenario has rcu_nocbs CPUs, and CPU 5 is in trouble:
> > >
> > > [ 631.066109] rcu: CB 5^4->4 KblSW F21044 L21044 C116 .W120(148380/168068).N295B101 q516 S CPU 16
> > > . . .
> > > [ 3610.869914] rcu: CB 5^4->4 KblSW F3000848 L3000848 C116 .W120(148380/168068).N295B101 q516 S CPU 16
> > >
> > > There are 120 callbacks on the WAIT segment, which are waiting on
> > > grace-period sequence number 148380, which was long gone even back at
> > > 631.032657 seconds in (g152813). There are 295 more callbacks in the NEXT
> > > segment, which long ago should have been assigned a grace period and moved
> > > into the currently empty NEXT_READY segment (which is the "." right before
> > > the "N295"). There are yet another 101 callbacks on the bypass list.
> > >
> > > This suggests that the rcuog kthread isn't being awakened when it
> > > should be. And that something is failing to advance callbacks out of
> > > NEXT in a timely fashion.
> > >
> > > Thoughts?
> >
> > If a grace period has elapsed while in the middle of nocb_gp_wait(),
> > after the call to rcu_advance_cbs() and before the following polling which
> > determines if we a grace period must be waited upon, then the kthread
> > will wait for new callbacks only (and will ignore waiting callbacks that
> > can now be advanced).
> >
> > Does the following help?
> >
> > diff --git a/kernel/rcu/tree_nocb.h b/kernel/rcu/tree_nocb.h
> > index 6da1b8f52476..96d76eb6259d 100644
> > --- a/kernel/rcu/tree_nocb.h
> > +++ b/kernel/rcu/tree_nocb.h
> > @@ -792,22 +792,12 @@ static noinline_for_stack void nocb_gp_wait(struct rcu_data *my_rdp)
> > RCU_NEXT_READY_TAIL);
> > raw_spin_unlock_rcu_node(rnp); /* irqs disabled. */
> > }
> > - // Need to wait on some grace period?
> > +
> > WARN_ON_ONCE(wasempty &&
> > !rcu_segcblist_restempty(&rdp->cblist,
> > RCU_NEXT_READY_TAIL));
> > - /*
> > - * Only request a GP wait if the next pending callback's
> > - * GP has not already completed (normal or expedited).
> > - * If poll_state_synchronize_rcu_full() says it completed,
> > - * then rcu_advance_cbs() above already moved those
> > - * callbacks to RCU_DONE_TAIL, so there is no GP to wait
> > - * for. Any remaining callbacks got new (future) GP
> > - * numbers from rcu_accelerate_cbs() inside
> > - * rcu_advance_cbs() and will be handled on the next pass.
> > - */
> > - if (rcu_segcblist_nextgp(&rdp->cblist, &cur_gp_seq) &&
> > - !poll_state_synchronize_rcu_full(&cur_gp_seq)) {
> > + // Need to wait on some grace period?
> > + if (rcu_segcblist_nextgp(&rdp->cblist, &cur_gp_seq)) {
> > /*
> > * Track the earliest pending normal and expedited GP
> > * across the group so the wait below can be released by
>
>
> This looks correct to me, this optimization with
> !poll_state_synchronize_rcu_full(&cur_gp_seq) opened up this race as
> you correctly explained.
My local reproducer says that the problem is gone, but let's see what
Paul's 100s of hours of testing says.
^ permalink raw reply
* Re: [PATCH v1 00/11] RCU: Enable callbacks to benefit from expedited grace periods
From: Paul E. McKenney @ 2026-07-24 14:06 UTC (permalink / raw)
To: Puranjay Mohan
Cc: Frederic Weisbecker, rcu, linux-kernel, linux-trace-kernel,
Neeraj Upadhyay, Joel Fernandes, Josh Triplett, Boqun Feng,
Uladzislau Rezki, Steven Rostedt, Mathieu Desnoyers,
Lai Jiangshan, Zqiang, Masami Hiramatsu, Davidlohr Bueso,
Breno Leitao
In-Reply-To: <CANk7y0isSWC0mARzqhn_4OWb+oHkrAt9k_83j_V3eDzKGbsxjA@mail.gmail.com>
On Fri, Jul 24, 2026 at 02:47:22PM +0100, Puranjay Mohan wrote:
> On Fri, Jul 24, 2026 at 1:52 PM Puranjay Mohan <puranjay12@gmail.com> wrote:
> >
> > On Fri, Jul 24, 2026 at 1:45 PM Frederic Weisbecker <frederic@kernel.org> wrote:
> > >
> > > Le Thu, Jul 23, 2026 at 05:31:17PM -0700, Paul E. McKenney a écrit :
> > > > On Wed, Jun 24, 2026 at 06:23:42AM -0700, Puranjay Mohan wrote:
> > > > > This series lets call_rcu() callbacks be reclaimed as soon as either a
> > > > > normal or an expedited grace period that covers them has elapsed, rather
> > > > > than always waiting for a normal grace period.
> > > > >
> > > > > Motivation
> > > > > ==========
> > > > > Today there is an asymmetry: synchronize_rcu_expedited() callers get fast
> > > > > reclaim, but call_rcu() callers never benefit from those same expedited
> > > > > grace periods, even though an expedited GP proves exactly the same thing
> > > > > as a normal one -- all pre-existing readers are done. When expedited GPs
> > > > > are running on the system (driven by other subsystems), call_rcu()
> > > > > callbacks that could already be freed instead sit in RCU_WAIT_TAIL until
> > > > > the next normal GP. This series treats a grace period as a grace period
> > > > > regardless of how it was driven, so memory is reclaimed sooner.
> > > > >
> > > > > Design
> > > > > ======
> > > > > Callback segments now record both the normal and expedited grace-period
> > > > > sequence in struct rcu_gp_seq, and rcu_segcblist_advance() releases a
> > > > > segment as soon as poll_state_synchronize_rcu_full() reports that either
> > > > > has completed. Three notification paths are taught about expedited
> > > > > completion so the advance actually happens: the NOCB rcuog kthreads,
> > > > > the rcu_pending() tick gate, and rcu_core().
> > > > >
> > > > > Changelog:
> > > > > RFC: https://lore.kernel.org/all/20260417231203.785172-1-puranjay@kernel.org/
> > > > > Changes in v1:
> > > > > - New prep patch 1 renames struct rcu_gp_oldstate to struct rcu_gp_seq
> > > > > and its fields rgos_norm/rgos_exp to norm/exp tree-wide (Frederic).
> > > > > - The rcu_segcblist segment field stays named gp_seq; only its type
> > > > > changes (Frederic).
> > > > > - Patch 8 (NOCB wake) is reworked. v1 woke the wrong waitqueue
> > > > > (rdp_gp->nocb_gp_wq via wake_nocb_gp() rather than the leaf
> > > > > rnp->nocb_gp_wq[] that an rcuog kthread waiting for a GP sleeps on),
> > > > > and the wait condition only checked the normal ->gp_seq. The rcuog
> > > > > grace-period wait now tracks a struct rcu_gp_seq and is released via
> > > > > poll_state_synchronize_rcu_full(); rcu_exp_wait_wake() wakes the leaf
> > > > > node through the new rcu_nocb_exp_cleanup() (Frederic).
> > > > > - rcu_pending() uses a new memory-ordering-free
> > > > > poll_state_synchronize_rcu_full_unordered() to avoid memory barriers
> > > > > on every tick, leaving the ordering duty to rcu_core() (Frederic).
> > > > >
> > > > > Still open: Frederic asked whether the first smp_mb() in
> > > > > poll_state_synchronize_rcu_full() is needed on the callback-advance path
> > > > > (patch 6). That path still uses the fully ordered helper; only
> > > > > rcu_pending() was switched to the unordered variant. Happy to revisit.
> > > > >
> > > > > Puranjay Mohan (11):
> > > > > rcu: Rename struct rcu_gp_oldstate to rcu_gp_seq
> > > > > rcu/segcblist: Add SRCU and Tasks RCU wrapper functions
> > > > > rcu/segcblist: Factor out rcu_segcblist_advance_compact() helper
> > > > > rcu/segcblist: Track segment grace periods with struct rcu_gp_seq
> > > > > rcu: Add RCU_GET_STATE_NOT_TRACKED for subsystems without expedited
> > > > > GPs
> > > > > rcu: Enable RCU callbacks to benefit from expedited grace periods
> > > > > rcu: Update comments for gp_seq and expedited GP tracking
> > > > > rcu: Wake NOCB rcuog kthreads on expedited grace period completion
> > > > > rcu: Detect expedited grace period completion in rcu_pending()
> > > > > rcu: Advance callbacks for expedited GP completion in rcu_core()
> > > > > rcuscale: Add concurrent expedited GP threads for callback scaling
> > > > > tests
> > > >
> > > > I do see the occasional failure when running TREE01, as in a failure or
> > > > four every couple hundred hours of testing. This is the only rcutorture
> > > > scenario that exercises callback (de)offloading, which might (or might
> > > > not) be a clue.
> > > >
> > > > The console log of a failing run is attached, or you can find it here:
> > > >
> > > > https://drive.google.com/file/d/1EH2P4SDy2a-SJndkN3GbCTGAXxI2xevm/view?usp=sharing
> > > >
> > > > This is a "??? Writer stall state" failure, where rcutorture never learns
> > > > of grace-period completion. Possibly because no grace period completed.
> > > > But there are no RCU CPU stall warnings. In addition, these two lines
> > > >
> > > > [ 631.032657] ??? Writer stall state RTWS_EXP_SYNC(4) g152813 f0x0 ->state D cpu 1
> > > > . . .
> > > > [ 3610.863114] ??? Writer stall state RTWS_EXP_SYNC(4) g736160 f0x0 ->state D cpu 1
> > > >
> > > > show that the grace-period sequence number advanced from 152813 to
> > > > 736160 in about 50 minutes. So grace periods really are completing,
> > > > lots of them.
> > > >
> > > > This rcutorture scenario has rcu_nocbs CPUs, and CPU 5 is in trouble:
> > > >
> > > > [ 631.066109] rcu: CB 5^4->4 KblSW F21044 L21044 C116 .W120(148380/168068).N295B101 q516 S CPU 16
> > > > . . .
> > > > [ 3610.869914] rcu: CB 5^4->4 KblSW F3000848 L3000848 C116 .W120(148380/168068).N295B101 q516 S CPU 16
> > > >
> > > > There are 120 callbacks on the WAIT segment, which are waiting on
> > > > grace-period sequence number 148380, which was long gone even back at
> > > > 631.032657 seconds in (g152813). There are 295 more callbacks in the NEXT
> > > > segment, which long ago should have been assigned a grace period and moved
> > > > into the currently empty NEXT_READY segment (which is the "." right before
> > > > the "N295"). There are yet another 101 callbacks on the bypass list.
> > > >
> > > > This suggests that the rcuog kthread isn't being awakened when it
> > > > should be. And that something is failing to advance callbacks out of
> > > > NEXT in a timely fashion.
> > > >
> > > > Thoughts?
> > >
> > > If a grace period has elapsed while in the middle of nocb_gp_wait(),
> > > after the call to rcu_advance_cbs() and before the following polling which
> > > determines if we a grace period must be waited upon, then the kthread
> > > will wait for new callbacks only (and will ignore waiting callbacks that
> > > can now be advanced).
> > >
> > > Does the following help?
> > >
> > > diff --git a/kernel/rcu/tree_nocb.h b/kernel/rcu/tree_nocb.h
> > > index 6da1b8f52476..96d76eb6259d 100644
> > > --- a/kernel/rcu/tree_nocb.h
> > > +++ b/kernel/rcu/tree_nocb.h
> > > @@ -792,22 +792,12 @@ static noinline_for_stack void nocb_gp_wait(struct rcu_data *my_rdp)
> > > RCU_NEXT_READY_TAIL);
> > > raw_spin_unlock_rcu_node(rnp); /* irqs disabled. */
> > > }
> > > - // Need to wait on some grace period?
> > > +
> > > WARN_ON_ONCE(wasempty &&
> > > !rcu_segcblist_restempty(&rdp->cblist,
> > > RCU_NEXT_READY_TAIL));
> > > - /*
> > > - * Only request a GP wait if the next pending callback's
> > > - * GP has not already completed (normal or expedited).
> > > - * If poll_state_synchronize_rcu_full() says it completed,
> > > - * then rcu_advance_cbs() above already moved those
> > > - * callbacks to RCU_DONE_TAIL, so there is no GP to wait
> > > - * for. Any remaining callbacks got new (future) GP
> > > - * numbers from rcu_accelerate_cbs() inside
> > > - * rcu_advance_cbs() and will be handled on the next pass.
> > > - */
> > > - if (rcu_segcblist_nextgp(&rdp->cblist, &cur_gp_seq) &&
> > > - !poll_state_synchronize_rcu_full(&cur_gp_seq)) {
> > > + // Need to wait on some grace period?
> > > + if (rcu_segcblist_nextgp(&rdp->cblist, &cur_gp_seq)) {
> > > /*
> > > * Track the earliest pending normal and expedited GP
> > > * across the group so the wait below can be released by
> >
> >
> > This looks correct to me, this optimization with
> > !poll_state_synchronize_rcu_full(&cur_gp_seq) opened up this race as
> > you correctly explained.
>
> My local reproducer says that the problem is gone, but let's see what
> Paul's 100s of hours of testing says.
I will fire it up, and thank you both!!!
And if it passes the 100s of hours (about 600), it will get 1000s of
hours over the weekend. ;-)
Thanx, Paul
^ permalink raw reply
* Re: [PATCH v1 10/11] rcu: Advance callbacks for expedited GP completion in rcu_core()
From: Puranjay Mohan @ 2026-07-24 14:54 UTC (permalink / raw)
To: Frederic Weisbecker
Cc: rcu, linux-kernel, linux-trace-kernel, Paul E. McKenney,
Neeraj Upadhyay, Joel Fernandes, Josh Triplett, Boqun Feng,
Uladzislau Rezki, Steven Rostedt, Mathieu Desnoyers,
Lai Jiangshan, Zqiang, Masami Hiramatsu, Davidlohr Bueso,
Breno Leitao
In-Reply-To: <amE7NrJRm0YDYroX@pavilion.home>
On Wed, Jul 22, 2026 at 10:50 PM Frederic Weisbecker
<frederic@kernel.org> wrote:
>
> Le Tue, Jul 21, 2026 at 04:06:24PM +0100, Puranjay Mohan a écrit :
> > On Tue, Jul 21, 2026 at 3:35 PM Frederic Weisbecker <frederic@kernel.org> wrote:
> > >
> > > Le Wed, Jun 24, 2026 at 06:23:52AM -0700, Puranjay Mohan a écrit :
> > > > Even when rcu_pending() triggers rcu_core(), the normal callback
> > > > advancement path through note_gp_changes() -> __note_gp_changes() bails
> > > > out when rdp->gp_seq == rnp->gp_seq (no normal GP change). Since
> > > > expedited GPs do not update rnp->gp_seq, rcu_advance_cbs() is never
> > > > called and callbacks remain stuck in RCU_WAIT_TAIL.
> > > >
> > > > Add a direct callback advancement block in rcu_core() that checks for GP
> > > > completion via rcu_segcblist_nextgp() combined with
> > > > poll_state_synchronize_rcu_full(). When detected, trylock rnp and call
> > > > rcu_advance_cbs() to move completed callbacks to RCU_DONE_TAIL. Wake the
> > > > GP kthread if rcu_advance_cbs() requests a new grace period.
> > > >
> > > > Uses trylock to avoid adding contention on rnp->lock. If the lock is
> > > > contended, callbacks will be advanced on the next tick.
> > > >
> > > > Reviewed-by: Paul E. McKenney <paulmck@kernel.org>
> > > > Signed-off-by: Puranjay Mohan <puranjay@kernel.org>
> > > > ---
> > > > kernel/rcu/tree.c | 17 +++++++++++++++++
> > > > 1 file changed, 17 insertions(+)
> > > >
> > > > diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c
> > > > index b01d7bf6b57b1..f42e01ef479c4 100644
> > > > --- a/kernel/rcu/tree.c
> > > > +++ b/kernel/rcu/tree.c
> > > > @@ -2891,6 +2891,23 @@ static __latent_entropy void rcu_core(void)
> > > > /* Update RCU state based on any recent quiescent states. */
> > > > rcu_check_quiescent_state(rdp);
> > > >
> > > > + /* Advance callbacks if an expedited GP has completed. */
> > > > + if (!rcu_rdp_is_offloaded(rdp) && rcu_segcblist_is_enabled(&rdp->cblist)) {
> > > > + struct rcu_gp_seq gp_state;
> > > > +
> > > > + if (rcu_segcblist_nextgp(&rdp->cblist, &gp_state) &&
> > > > + poll_state_synchronize_rcu_full(&gp_state)) {
> > > > + guard(irqsave)();
> > > > + if (raw_spin_trylock_rcu_node(rnp)) {
> > > > + bool needwake = rcu_advance_cbs(rnp, rdp);
> > > > +
> > > > + raw_spin_unlock_rcu_node(rnp);
> > > > + if (needwake)
> > > > + rcu_gp_kthread_wake();
> > > > + }
> > > > + }
> > > > + }
> > >
> > > Should that go as an improvement to note_gp_changes() instead?
> >
> > note_gp_changes() only reconciles rdp->gp_seq against rnp->gp_seq, and
> > the expedited path never advances rnp->gp_seq. So the gap this closes
> > is exactly rdp->gp_seq == rnp->gp_seq, where note_gp_changes() and
> > __note_gp_changes() both short-circuit, the expedited completion isn't
> > visible there at all. It's detected from the cblist's stored gp_seq
> > (rcu_segcblist_nextgp()) confirmed with
> > poll_state_synchronize_rcu_full(), so hosting it in note_gp_changes()
> > would mean running that in the lockless preamble for every caller,
> > including the off-tick call_rcu_core() path. In rcu_core() it's
> > already gated by rcu_pending(), which does the barrier-free detection.
>
> Let's take a step back. note_gp_changes() is for the CPU to ackowledge
> a grace period change, either start or completion, and react upon with:
>
> _ Making the callback progress through the state machine if a grace period
> has changed.
>
> _ Starting to chase quiescent states.
>
> And now callback advancing/acceleration don't even refer anymore to the
> leaf node state but to the global one. So why not proceed with that
> logic?
>
> Also other callers of note_gp_changes() may want to benefit from expedited
> grace periods as well.
>
> Would the following (untested) work?
>
> diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c
> index ff6601411a89..96bf7fe03be8 100644
> --- a/kernel/rcu/tree.c
> +++ b/kernel/rcu/tree.c
> @@ -1271,27 +1271,29 @@ static bool __note_gp_changes(struct rcu_node *rnp, struct rcu_data *rdp)
> {
> bool ret = false;
> bool need_qs;
> + struct rcu_gp_seq gp_state;
> const bool offloaded = rcu_rdp_is_offloaded(rdp);
>
> raw_lockdep_assert_held_rcu_node(rnp);
>
> - if (rdp->gp_seq == rnp->gp_seq)
> - return false; /* Nothing to do. */
> -
> /* Handle the ends of any preceding grace periods first. */
> - if (rcu_seq_completed_gp(rdp->gp_seq, rnp->gp_seq) ||
> + if ((rcu_segcblist_nextgp(&rdp->cblist, &gp_state) &&
> + poll_state_synchronize_rcu_full_unordered(&gp_state)) ||
> unlikely(rdp->gpwrap)) {
> if (!offloaded)
> ret = rcu_advance_cbs(rnp, rdp); /* Advance CBs. */
> rdp->core_needs_qs = false;
> trace_rcu_grace_period(rcu_state.name, rdp->gp_seq, TPS("cpuend"));
> - } else {
> + } else if (rdp->gp_seq != rnp->gp_seq) {
> if (!offloaded)
> ret = rcu_accelerate_cbs(rnp, rdp); /* Recent CBs. */
> if (rdp->core_needs_qs)
> rdp->core_needs_qs = !!(rnp->qsmask & rdp->grpmask);
> }
>
> + if (rdp->gp_seq == rnp->gp_seq)
> + return ret; /* Nothing else to do. */
> +
> /* Now handle the beginnings of any new-to-this-CPU grace periods. */
> if (rcu_seq_new_gp(rdp->gp_seq, rnp->gp_seq) ||
> unlikely(rdp->gpwrap)) {
> @@ -1316,6 +1318,27 @@ static bool __note_gp_changes(struct rcu_node *rnp, struct rcu_data *rdp)
> return ret;
> }
>
> +static bool need_note_gp_changes(struct rcu_data *rdp)
> +{
> + struct rcu_gp_seq gp_state;
> + struct rcu_node *rnp = rdp->mynode;
> +
> + /* Need to chase QS or accelerate? */
> + if (rdp->gp_seq != rcu_seq_current(&rnp->gp_seq))
> + return true;
> +
> + /* Waited upon GP has ended, need to advance CBs ? */
> + if (rcu_segcblist_nextgp(&rdp->cblist, &gp_state) &&
> + poll_state_synchronize_rcu_full_unordered(&gp_state))
> + return true;
> +
> + /* Wrapped? */
> + if (unlikely(READ_ONCE(rdp->gpwrap)))
> + return true;
> +
> + return false;
> +}
> +
> static void note_gp_changes(struct rcu_data *rdp)
> {
> unsigned long flags;
> @@ -1324,8 +1347,7 @@ static void note_gp_changes(struct rcu_data *rdp)
>
> local_irq_save(flags);
> rnp = rdp->mynode;
> - if ((rdp->gp_seq == rcu_seq_current(&rnp->gp_seq) &&
> - !unlikely(READ_ONCE(rdp->gpwrap))) || /* w/out lock. */
> + if (!need_note_gp_changes(rdp) || /* w/out lock. */
> !raw_spin_trylock_rcu_node(rnp)) { /* irqs already off, so later. */
> local_irq_restore(flags);
> return;
> @@ -2888,23 +2910,6 @@ static __latent_entropy void rcu_core(void)
> /* Update RCU state based on any recent quiescent states. */
> rcu_check_quiescent_state(rdp);
>
> - /* Advance callbacks if an expedited GP has completed. */
> - if (!rcu_rdp_is_offloaded(rdp) && rcu_segcblist_is_enabled(&rdp->cblist)) {
> - struct rcu_gp_seq gp_state;
> -
> - if (rcu_segcblist_nextgp(&rdp->cblist, &gp_state) &&
> - poll_state_synchronize_rcu_full(&gp_state)) {
> - guard(irqsave)();
> - if (raw_spin_trylock_rcu_node(rnp)) {
> - bool needwake = rcu_advance_cbs(rnp, rdp);
> -
> - raw_spin_unlock_rcu_node(rnp);
> - if (needwake)
> - rcu_gp_kthread_wake();
> - }
> - }
> - }
> -
> /* No grace period and unregistered callbacks? */
> if (!rcu_gp_in_progress() &&
> rcu_segcblist_is_enabled(&rdp->cblist) && !rcu_rdp_is_offloaded(rdp)) {
> diff --git a/kernel/rcu/tree.h b/kernel/rcu/tree.h
> index 01a1b2985abd..6b9b058d138e 100644
> --- a/kernel/rcu/tree.h
> +++ b/kernel/rcu/tree.h
> @@ -517,6 +517,7 @@ static void rcu_nocb_unlock(struct rcu_data *rdp);
> static void rcu_nocb_unlock_irqrestore(struct rcu_data *rdp,
> unsigned long flags);
> static void rcu_lockdep_assert_cblist_protected(struct rcu_data *rdp);
> +static bool poll_state_synchronize_rcu_full_unordered(struct rcu_gp_seq *gsp);
> #ifdef CONFIG_RCU_NOCB_CPU
> static void __init rcu_organize_nocb_kthreads(void);
>
>
Hi Frederic.
I took your approach and created this commit with minor changes to your diff:
A few points I'd like a second opinion on. I kept the unordered
poll_state_synchronize_rcu_full_unordered() for the advance check (the
old rcu_core() block used the ordered variant): this looks safe
because rcu_segcblist_advance() re-checks each segment with the
ordered poll_state_synchronize_rcu_full() before moving callbacks to
RCU_DONE_TAIL, so the check here is only a gate and the barriers still
apply where callbacks are actually advanced, please confirm that
reasoning. need_note_gp_changes() runs the callback-list poll on the
lockless preamble for both callers, including offloaded rdps where
__note_gp_changes() won't advance anything; I left it ungated since it
only leads to a trylock, but it could take a
!rcu_rdp_is_offloaded(rdp) guard. I also dropped the
rcu_segcblist_is_enabled() guard the rcu_core() block had, relying on
__note_gp_changes() already operating on the cblist unconditionally
for non-offloaded rdps.
-- >8 --
From c38c5f599d699e2c40d3459fdf3ef383a99c0097 Mon Sep 17 00:00:00 2001
From: Puranjay Mohan <puranjay@kernel.org>
Date: Fri, 24 Jul 2026 07:26:25 -0700
Subject: [PATCH] rcu: Advance callbacks for expedited GP completion in
note_gp_changes()
When rcu_pending() triggers rcu_core(), the callback advancement path
through note_gp_changes() -> __note_gp_changes() bails out when
rdp->gp_seq == rnp->gp_seq (no normal GP change). Since expedited GPs do
not update rnp->gp_seq, rcu_advance_cbs() is never reached from there and
callbacks satisfied by an expedited GP would otherwise remain stuck in
RCU_WAIT_TAIL until the next normal GP.
This is currently handled by a dedicated advancement block in rcu_core()
that polls the callback list and advances under a trylock. But callback
advancement no longer depends on the leaf-node grace-period delta; it is
driven by the grace-period state stored in the callback list, which
tracks both normal and expedited GPs. __note_gp_changes() is the natural
home for it, and hosting it there lets every note_gp_changes() caller
benefit from expedited completions rather than just rcu_core().
Move the advancement into __note_gp_changes(): trigger rcu_advance_cbs()
whenever rcu_segcblist_nextgp() confirmed with
poll_state_synchronize_rcu_full_unordered() reports a completed grace
period, and add need_note_gp_changes() so the lockless preamble takes the
lock for an expedited-only completion instead of short-circuiting on
rdp->gp_seq == rnp->gp_seq. Remove the now-redundant rcu_core() block.
The quiescent-state bookkeeping stays keyed to an actual rnp->gp_seq
change, so an expedited completion never clears a still-pending
core_needs_qs and stalls the normal grace period.
Signed-off-by: Puranjay Mohan <puranjay@kernel.org>
---
kernel/rcu/tree.c | 56 ++++++++++++++++++++++++-----------------------
kernel/rcu/tree.h | 1 +
2 files changed, 30 insertions(+), 27 deletions(-)
diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c
index 21b6ce1dffb63..0e700d0ecf27e 100644
--- a/kernel/rcu/tree.c
+++ b/kernel/rcu/tree.c
@@ -1270,27 +1270,33 @@ static bool __note_gp_changes(struct rcu_node
*rnp, struct rcu_data *rdp)
{
bool ret = false;
bool need_qs;
+ struct rcu_gp_seq gp_state;
const bool offloaded = rcu_rdp_is_offloaded(rdp);
+ bool completed = rcu_seq_completed_gp(rdp->gp_seq, rnp->gp_seq) ||
+ unlikely(rdp->gpwrap);
raw_lockdep_assert_held_rcu_node(rnp);
- if (rdp->gp_seq == rnp->gp_seq)
- return false; /* Nothing to do. */
-
/* Handle the ends of any preceding grace periods first. */
- if (rcu_seq_completed_gp(rdp->gp_seq, rnp->gp_seq) ||
- unlikely(rdp->gpwrap)) {
+ if (completed ||
+ (rcu_segcblist_nextgp(&rdp->cblist, &gp_state) &&
+ poll_state_synchronize_rcu_full_unordered(&gp_state))) {
if (!offloaded)
ret = rcu_advance_cbs(rnp, rdp); /* Advance CBs. */
- rdp->core_needs_qs = false;
- trace_rcu_grace_period(rcu_state.name, rdp->gp_seq,
TPS("cpuend"));
- } else {
+ if (completed) {
+ rdp->core_needs_qs = false;
+ trace_rcu_grace_period(rcu_state.name,
rdp->gp_seq, TPS("cpuend"));
+ }
+ } else if (rdp->gp_seq != rnp->gp_seq) {
if (!offloaded)
ret = rcu_accelerate_cbs(rnp, rdp); /* Recent CBs. */
if (rdp->core_needs_qs)
rdp->core_needs_qs = !!(rnp->qsmask & rdp->grpmask);
}
+ if (rdp->gp_seq == rnp->gp_seq)
+ return ret; /* Nothing else to do. */
+
/* Now handle the beginnings of any new-to-this-CPU grace periods. */
if (rcu_seq_new_gp(rdp->gp_seq, rnp->gp_seq) ||
unlikely(rdp->gpwrap)) {
@@ -1315,6 +1321,20 @@ static bool __note_gp_changes(struct rcu_node
*rnp, struct rcu_data *rdp)
return ret;
}
+static bool need_note_gp_changes(struct rcu_data *rdp)
+{
+ struct rcu_gp_seq gp_state;
+ struct rcu_node *rnp = rdp->mynode;
+
+ if (rdp->gp_seq != rcu_seq_current(&rnp->gp_seq) ||
+ unlikely(READ_ONCE(rdp->gpwrap)))
+ return true;
+
+ /* Has a grace period a callback is waiting on completed? */
+ return rcu_segcblist_nextgp(&rdp->cblist, &gp_state) &&
+ poll_state_synchronize_rcu_full_unordered(&gp_state);
+}
+
static void note_gp_changes(struct rcu_data *rdp)
{
unsigned long flags;
@@ -1323,8 +1343,7 @@ static void note_gp_changes(struct rcu_data *rdp)
local_irq_save(flags);
rnp = rdp->mynode;
- if ((rdp->gp_seq == rcu_seq_current(&rnp->gp_seq) &&
- !unlikely(READ_ONCE(rdp->gpwrap))) || /* w/out lock. */
+ if (!need_note_gp_changes(rdp) || /* w/out lock. */
!raw_spin_trylock_rcu_node(rnp)) { /* irqs already off, so later. */
local_irq_restore(flags);
return;
@@ -2886,23 +2905,6 @@ static __latent_entropy void rcu_core(void)
/* Update RCU state based on any recent quiescent states. */
rcu_check_quiescent_state(rdp);
- /* Advance callbacks if an expedited GP has completed. */
- if (!rcu_rdp_is_offloaded(rdp) &&
rcu_segcblist_is_enabled(&rdp->cblist)) {
- struct rcu_gp_seq gp_state;
-
- if (rcu_segcblist_nextgp(&rdp->cblist, &gp_state) &&
- poll_state_synchronize_rcu_full(&gp_state)) {
- guard(irqsave)();
- if (raw_spin_trylock_rcu_node(rnp)) {
- bool needwake = rcu_advance_cbs(rnp, rdp);
-
- raw_spin_unlock_rcu_node(rnp);
- if (needwake)
- rcu_gp_kthread_wake();
- }
- }
- }
-
/* No grace period and unregistered callbacks? */
if (!rcu_gp_in_progress() &&
rcu_segcblist_is_enabled(&rdp->cblist) &&
!rcu_rdp_is_offloaded(rdp)) {
diff --git a/kernel/rcu/tree.h b/kernel/rcu/tree.h
index eedfa43059e80..962f86afc3c9a 100644
--- a/kernel/rcu/tree.h
+++ b/kernel/rcu/tree.h
@@ -521,6 +521,7 @@ static void rcu_nocb_unlock(struct rcu_data *rdp);
static void rcu_nocb_unlock_irqrestore(struct rcu_data *rdp,
unsigned long flags);
static void rcu_lockdep_assert_cblist_protected(struct rcu_data *rdp);
+static bool poll_state_synchronize_rcu_full_unordered(struct rcu_gp_seq *gsp);
#ifdef CONFIG_RCU_NOCB_CPU
static void __init rcu_organize_nocb_kthreads(void);
base-commit: dc597bcabf31a58f8c1aca01677a6af0b3307398
--
2.53.0-Meta
-- 8< --
Thanks,
Puranjay
^ permalink raw reply related
* Re: [RFC PATCH] Documentation/rv: Explain epoll and aborted sleeps
From: Chao Liu @ 2026-07-24 15:59 UTC (permalink / raw)
To: Gabriele Monaco
Cc: Nam Cao, Steven Rostedt, Jonathan Corbet, lianux.mm, lianux.wang,
Shuah Khan, linux-trace-kernel, linux-doc, linux-kernel
In-Reply-To: <f2679e058fa3904db3534a5f75d1339a4065018f.camel@redhat.com>
Hi Gabriele,
On Fri, Jul 24, 2026 at 09:20:44AM +0800, Gabriele Monaco wrote:
> On Wed, 2026-07-22 at 12:57 +0800, Chao Liu wrote:
> > The rtapp sleep monitor accepts epoll_wait() as a valid sleeping
> > reason, but the prose only discusses clock_nanosleep and futexes. Add
> > epoll_wait to the list of valid sleeping reasons.
> >
> > ABORT_SLEEP represents a task restoring TASK_RUNNING before entering
> > the scheduler, but its meaning is not described. Explain why the
> > aborted sleep attempt is valid.
> >
> > This RFC is based on Nam Cao's pending "rv: rtapp monitor update" v2
> > series:
> >
> > https://lore.kernel.org/r/cover.1781852967.git.namcao@linutronix.de
> >
> > Signed-off-by: Chao Liu <chao.liu@processmission.com>
> > ---
>
> Thanks for the contribution! I have a couple of comments, we want to be
> assertive in this docs: in general, let's not say something like "this is valid
> because the monitor thinks it's valid" but "this is valid because it is rt-safe
> for reason X".
Thanks for the review. :)
Agreed. In v2, I will update the documentation and the commit message
body to say directly why each case is safe for real-time use.
>
> > Documentation/trace/rv/monitor_rtapp.rst | 6 ++++++
> > 1 file changed, 6 insertions(+)
> >
> > diff --git a/Documentation/trace/rv/monitor_rtapp.rst
> > b/Documentation/trace/rv/monitor_rtapp.rst
> > index 238b59395ff5..234ed4d581ac 100644
> > --- a/Documentation/trace/rv/monitor_rtapp.rst
> > +++ b/Documentation/trace/rv/monitor_rtapp.rst
> > @@ -67,6 +67,8 @@ thread to sleep for one of the following reasons:
> > variables as safe for real-time. As an alternative, the librtpi library
> > exists to provide a conditional variable implementation that is correct
> > for
> > real-time applications in Linux.
> > + - Real-time thread waiting for events using `epoll_wait`,
>
> > + which the monitor accepts as a valid sleeping reason for real-time tasks.
>
> Not adding much value, try:
>
> which is a real-time-safe syscall for sleeping as it uses PI-aware locking.
>
Agreed. I will use this wording in v2.
> >
> > Beside the reason for sleeping, the eventual waker should also be
> > real-time-safe. Namely, one of:
> > @@ -114,6 +116,10 @@ The monitor's specification is::
> > ALLOWLIST = BLOCK_ON_RT_MUTEX
> > or FUTEX_LOCK_PI
> >
> > +`ABORT_SLEEP` represents a task restoring its state to `TASK_RUNNING` before
> > +entering the scheduler. In this case, the task does not actually block,
>
>
> > + so the monitor treats the aborted sleep attempt as valid.
>
> This sentence is a bit vague. Indeed an /aborted/ sleep is a valid wakeup, as if
> the task woke up itself without even sleeping, saying "the monitor treats the
> attempt as valid" adds no value.
>
> You could rephrase it to (full paragraph for clarity):
>
> `ABORT_SLEEP` represents a task restoring its state to `TASK_RUNNING` before
> entering the scheduler. In this case, the task does not actually block, so it is
> back to runnable without any wakeup sequence unsafe for real-time.
>
> What do you think?
>
Yes, this is clearer. I will use the proposed paragraph in v2.
To check my understanding, I also put together a small eventfd/epoll test
and ran it on arm64 and x86_64. The test code, configurations, patches,
serial traces, and reproduction steps are available here:
https://github.com/zevorn/rtapp-sleep-monitor-lab
Test environments
-----------------
+---------+--------------------------+---------------+------+------------+
| Guest | Host CPU | QEMU/accel | vCPU | Kernel |
+---------+--------------------------+---------------+------+------------+
| arm64 | Apple M5 Pro | 11.0.0/HVF | 2 | PREEMPT_RT |
| x86_64 | Intel Xeon Platinum 8163 | 11.0.0/KVM | 2 | PREEMPT_RT |
+---------+--------------------------+---------------+------+------------+
All three guest kernels used the same v7.2-rc4-based source baseline.
Test design
-----------
Phase 1 (P1):
P1 runs one epoll wait per iteration, for 2048 iterations, on
two vCPUs:
- vCPU 0 runs the SCHED_FIFO/50 waiter (`PID_WAITER`). It starts each
iteration, waits on epoll, reads the eventfd when epoll returns, and
acknowledges completion.
- vCPU 1 runs the SCHED_FIFO/60 writer (`PID_WRITER`). It waits for the
start signal, uses a different short delay in each iteration, writes
the eventfd, and waits for the acknowledgement.
The per-iteration flow is:
vCPU 0: waiter (PID_WAITER) vCPU 1: writer (PID_WRITER)
-------------------------------- ---------------------------------
set go = i
|------------------------> wait for go == i
enter epoll wait apply delay(i)
|<------------------------ write(eventfd)
epoll_wait() returns
read(eventfd)
set ack = i
|------------------------> wait for ack == i
Because the unmodified arm64 monitor does not recognize the
`epoll_pwait` entry, each wait that reaches the monitored sleep path
produces a P1 error. The test makes 2048 wait attempts, but an iteration
that sees readiness before setting `TASK_INTERRUPTIBLE` does not produce
a `SLEEP` event.
The PIDs are assigned dynamically. During P1, the `event_sleep` and
`error_sleep` filters use `pid == PID_WAITER`; `PID_WRITER` only supplies
the event.
Separate vCPUs let the event arrive while the waiter is preparing to
sleep, covering both normal blocking and ABORT_SLEEP. The writer has the
higher priority, so the normal wakeup is RT-safe.
Phase 2 (P2):
P2 is the negative control. A fresh task (`PID_NEGATIVE`) on
vCPU 0 performs one relative clock_nanosleep() without TIMER_ABSTIME. The
trace filters are switched to `pid == PID_NEGATIVE`. The expected
error_sleep confirms that the monitor is active.
The RV monitor tracks tasks, not vCPUs. PID 1 enables the sleep monitor
and the `event_sleep` and `error_sleep` events. After each phase, PID 1
stops tracing and prints the trace buffer; `scripts/summarize.sh` counts
the records in the serial log.
For example, the tests can be run and summarized with:
$ make initramfs-arm64
$ KERNEL_IMAGE=/path/to/Image ACCEL=hvf \
./scripts/run-qemu.sh arm64
$ ./scripts/summarize.sh serial-arm64.log
$ make initramfs-x86_64
$ KERNEL_IMAGE=/path/to/bzImage ACCEL=kvm \
./scripts/run-qemu.sh x86_64
$ ./scripts/summarize.sh serial-x86_64.log
The complete build steps, including the arm64 mapping variant, are in
docs/build-and-run.md.
Results
-------
The first two rows compare arm64 without and with the experimental
mapping. The x86_64 row is the unmodified cross-check using its native
`__NR_epoll_wait`. The arm64 rows enter through `epoll_pwait`.
Each case was run five times. `M [L-H]` means median
`[minimum-maximum]`; a single value means all five runs had that result.
+----------------+------------------+------------+---------------+--------+
| Case | P1 err | Ep abort | PI block | P2 err |
+----------------+------------------+------------+---------------+--------+
| arm64 baseline | 2043 [2031-2047] | 0 | 0 | 1 |
| arm64 mapping | 0 | 26 [20-28] | 355 [308-399] | 1 |
| x86_64 native | 0 | 31 [25-92] | 201 [147-204] | 1 |
+----------------+------------------+------------+---------------+--------+
The columns mean:
- `P1 err` counts `error_sleep` records from the Phase 1 epoll test.
- `Ep abort` counts records containing both `ep_wa` (`EPOLL_WAIT`) and
`ab_sl` (`ABORT_SLEEP`).
- `PI block` counts records containing both `ep_wa` and
`bl_on_rt_mu` (`BLOCK_ON_RT_MUTEX`).
- `P2 err` counts `error_sleep` records from the Phase 2 negative
control.
These are records retained in the trace buffer, not syscall totals or
rates. The P1/P2 error outcome was the same in all five runs. The
`Ep abort` and `PI block` counts vary with timing.
The per-run summaries and validation checks are in
results/revalidation-2026-07-24.md in the repository.
From the traces, I see the following path for a correctly recognized
epoll wait:
sys_enter(epoll_wait or epoll_pwait)
|
| native x86_64 entry, or arm64 experiment mapping
v
EPOLL_WAIT = true
|
v
ep_poll()
|
v
TASK_INTERRUPTIBLE -> SLEEP
|
+-- no event
| |
| v
| schedule() -> RT-safe wakeup -> SCHEDULE_IN
|
`-- event found by the locked recheck
|
v
skip schedule()
|
v
TASK_RUNNING -> ABORT_SLEEP
In the abort case, `ep_wa + sle` (`EPOLL_WAIT + SLEEP`) is followed by
`ep_wa + ab_sl` (`EPOLL_WAIT + ABORT_SLEEP`), with no `sch_in`
(`SCHEDULE_IN`) in between. This matches your wording: the task becomes
runnable again without blocking or going through a wakeup sequence.
I also observed `ep_wa + bl_on_rt_mu` while epoll wait was active, so the
test did reach the PI-aware locking path on PREEMPT_RT.
arm64 syscall-coverage gap
--------------------------
The arm64 run also turned up what looks like a syscall-coverage gap. The
current epoll support checks only `__NR_epoll_wait`. arm64 does not define
that syscall; its epoll wait enters the common implementation through
`__NR_epoll_pwait`. I went back to the review of the original epoll
support:
https://lore.kernel.org/r/58674d7f10c260369f5cb78599ba6ecb3804358f.camel@redhat.com
That reply discusses the `ltl_atom_update()` detail. I could not find any
discussion of cross-architecture syscall entry coverage there. In all
five unmodified arm64 runs, every retained `SLEEP` record has a matching
P1 error. Mapping `__NR_epoll_pwait` to the same atom removes those
errors, while P2 still produces the expected error.
This points to an instrumentation coverage issue rather than a difference
in epoll's real-time-safety. I kept the mapping out of this documentation
patch. Once the intended scope for `epoll_pwait`, `epoll_pwait2`, and
compat syscalls is clear, I can send a separate functional patch.
For reference, this is the experiment-only change:
diff --git a/kernel/trace/rv/monitors/sleep/sleep.c b/kernel/trace/rv/monitors/sleep/sleep.c
index aa5a984853b5..bf9ca4b5414e 100644
--- a/kernel/trace/rv/monitors/sleep/sleep.c
+++ b/kernel/trace/rv/monitors/sleep/sleep.c
@@ -136,8 +136,11 @@ static void handle_sys_enter(void *data, struct pt_regs *regs, long id)
break;
#ifdef __NR_epoll_wait
case __NR_epoll_wait:
+#endif
+#ifdef __NR_epoll_pwait
+ case __NR_epoll_pwait:
+#endif
ltl_atom_update(current, LTL_EPOLL_WAIT, true);
break;
-#endif
}
}
Thanks,
Chao
^ permalink raw reply related
* Re: [RFC PATCH] Documentation/rv: Explain epoll and aborted sleeps
From: Chao Liu @ 2026-07-24 16:17 UTC (permalink / raw)
To: Gabriele Monaco
Cc: Nam Cao, Steven Rostedt, Jonathan Corbet, lianux.mm, lianux.wang,
Shuah Khan, linux-trace-kernel, linux-doc, linux-kernel
In-Reply-To: <amNlh-GZO-RcbpiL@ChaodeMacBook-Pro.local>
On Sat, Jul 25, 2026 at 12:00:06AM +0800, Chao Liu wrote:
> Results
> -------
>
> The first two rows compare arm64 without and with the experimental
> mapping. The x86_64 row is the unmodified cross-check using its native
> `__NR_epoll_wait`. The arm64 rows enter through `epoll_pwait`.
>
> Each case was run five times. `M [L-H]` means median
To correct the ambiguity of my last email (sorry for the trouble):
the data in each entry is formatted as M[L-H].
For example, in the second column of "Ep abort": 26 [20-28].
> `[minimum-maximum]`; a single value means all five runs had that result.
>
> +----------------+------------------+------------+---------------+--------+
> | Case | P1 err | Ep abort | PI block | P2 err |
> +----------------+------------------+------------+---------------+--------+
> | arm64 baseline | 2043 [2031-2047] | 0 | 0 | 1 |
> | arm64 mapping | 0 | 26 [20-28] | 355 [308-399] | 1 |
> | x86_64 native | 0 | 31 [25-92] | 201 [147-204] | 1 |
> +----------------+------------------+------------+---------------+--------+
>
Thanks,
Chao
^ permalink raw reply
* [PATCH] tracing: Delay module ref count for "enable_event" trigger
From: Steven Rostedt @ 2026-07-24 17:24 UTC (permalink / raw)
To: LKML, Linux Trace Kernel; +Cc: Masami Hiramatsu, Mathieu Desnoyers
From: Steven Rostedt <rostedt@goodmis.org>
Triggers are now delayed from freeing, but can still be triggered until
after the RCU grace period has ended. The freeing of the enable_event data
is put into the private_data_free() callback, but the put of the module
refcount is done immediately.
It is possible that if a module is removed that has an event that would
enable (or disable) it is still active, it can read the data of the module
after it is removed causing a use-after-free bug.
Move the trace_event_put_ref() that releases the module into the delayed
callback so that the module can not be removed until any reference to its
events are finished.
Cc: stable@vger.kernel.org
Reported-by: sashiko-bot@kernel.org
Link: https://sashiko.dev/#/patchset/20260724030523.19081-1-devnexen%40gmail.com
Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
---
kernel/trace/trace_events_trigger.c | 6 ++++--
1 file changed, 4 insertions(+), 2 deletions(-)
diff --git a/kernel/trace/trace_events_trigger.c b/kernel/trace/trace_events_trigger.c
index 46e60b70a4bb..ad83419cb420 100644
--- a/kernel/trace/trace_events_trigger.c
+++ b/kernel/trace/trace_events_trigger.c
@@ -1726,7 +1726,10 @@ int event_enable_trigger_print(struct seq_file *m,
static void enable_trigger_private_data_free(struct event_trigger_data *data)
{
- kfree(data->private_data);
+ struct enable_trigger_data *enable_data = data->private_data;
+
+ trace_event_put_ref(enable_data->file->event_call);
+ kfree(enable_data);
}
void event_enable_trigger_free(struct event_trigger_data *data)
@@ -1741,7 +1744,6 @@ void event_enable_trigger_free(struct event_trigger_data *data)
/* Remove the SOFT_MODE flag */
trace_event_enable_disable(enable_data->file, 0, 1);
data->private_data_free = enable_trigger_private_data_free;
- trace_event_put_ref(enable_data->file->event_call);
trigger_data_free(data);
}
}
--
2.53.0
^ permalink raw reply related
* Re: [PATCH v1 00/11] RCU: Enable callbacks to benefit from expedited grace periods
From: Paul E. McKenney @ 2026-07-24 17:41 UTC (permalink / raw)
To: Puranjay Mohan
Cc: Frederic Weisbecker, rcu, linux-kernel, linux-trace-kernel,
Neeraj Upadhyay, Joel Fernandes, Josh Triplett, Boqun Feng,
Uladzislau Rezki, Steven Rostedt, Mathieu Desnoyers,
Lai Jiangshan, Zqiang, Masami Hiramatsu, Davidlohr Bueso,
Breno Leitao
In-Reply-To: <88e5e2e4-8043-4d23-a362-cd864f9d4145@paulmck-laptop>
On Fri, Jul 24, 2026 at 07:06:42AM -0700, Paul E. McKenney wrote:
> On Fri, Jul 24, 2026 at 02:47:22PM +0100, Puranjay Mohan wrote:
> > On Fri, Jul 24, 2026 at 1:52 PM Puranjay Mohan <puranjay12@gmail.com> wrote:
> > >
> > > On Fri, Jul 24, 2026 at 1:45 PM Frederic Weisbecker <frederic@kernel.org> wrote:
> > > >
> > > > Le Thu, Jul 23, 2026 at 05:31:17PM -0700, Paul E. McKenney a écrit :
> > > > > On Wed, Jun 24, 2026 at 06:23:42AM -0700, Puranjay Mohan wrote:
> > > > > > This series lets call_rcu() callbacks be reclaimed as soon as either a
> > > > > > normal or an expedited grace period that covers them has elapsed, rather
> > > > > > than always waiting for a normal grace period.
> > > > > >
> > > > > > Motivation
> > > > > > ==========
> > > > > > Today there is an asymmetry: synchronize_rcu_expedited() callers get fast
> > > > > > reclaim, but call_rcu() callers never benefit from those same expedited
> > > > > > grace periods, even though an expedited GP proves exactly the same thing
> > > > > > as a normal one -- all pre-existing readers are done. When expedited GPs
> > > > > > are running on the system (driven by other subsystems), call_rcu()
> > > > > > callbacks that could already be freed instead sit in RCU_WAIT_TAIL until
> > > > > > the next normal GP. This series treats a grace period as a grace period
> > > > > > regardless of how it was driven, so memory is reclaimed sooner.
> > > > > >
> > > > > > Design
> > > > > > ======
> > > > > > Callback segments now record both the normal and expedited grace-period
> > > > > > sequence in struct rcu_gp_seq, and rcu_segcblist_advance() releases a
> > > > > > segment as soon as poll_state_synchronize_rcu_full() reports that either
> > > > > > has completed. Three notification paths are taught about expedited
> > > > > > completion so the advance actually happens: the NOCB rcuog kthreads,
> > > > > > the rcu_pending() tick gate, and rcu_core().
> > > > > >
> > > > > > Changelog:
> > > > > > RFC: https://lore.kernel.org/all/20260417231203.785172-1-puranjay@kernel.org/
> > > > > > Changes in v1:
> > > > > > - New prep patch 1 renames struct rcu_gp_oldstate to struct rcu_gp_seq
> > > > > > and its fields rgos_norm/rgos_exp to norm/exp tree-wide (Frederic).
> > > > > > - The rcu_segcblist segment field stays named gp_seq; only its type
> > > > > > changes (Frederic).
> > > > > > - Patch 8 (NOCB wake) is reworked. v1 woke the wrong waitqueue
> > > > > > (rdp_gp->nocb_gp_wq via wake_nocb_gp() rather than the leaf
> > > > > > rnp->nocb_gp_wq[] that an rcuog kthread waiting for a GP sleeps on),
> > > > > > and the wait condition only checked the normal ->gp_seq. The rcuog
> > > > > > grace-period wait now tracks a struct rcu_gp_seq and is released via
> > > > > > poll_state_synchronize_rcu_full(); rcu_exp_wait_wake() wakes the leaf
> > > > > > node through the new rcu_nocb_exp_cleanup() (Frederic).
> > > > > > - rcu_pending() uses a new memory-ordering-free
> > > > > > poll_state_synchronize_rcu_full_unordered() to avoid memory barriers
> > > > > > on every tick, leaving the ordering duty to rcu_core() (Frederic).
> > > > > >
> > > > > > Still open: Frederic asked whether the first smp_mb() in
> > > > > > poll_state_synchronize_rcu_full() is needed on the callback-advance path
> > > > > > (patch 6). That path still uses the fully ordered helper; only
> > > > > > rcu_pending() was switched to the unordered variant. Happy to revisit.
> > > > > >
> > > > > > Puranjay Mohan (11):
> > > > > > rcu: Rename struct rcu_gp_oldstate to rcu_gp_seq
> > > > > > rcu/segcblist: Add SRCU and Tasks RCU wrapper functions
> > > > > > rcu/segcblist: Factor out rcu_segcblist_advance_compact() helper
> > > > > > rcu/segcblist: Track segment grace periods with struct rcu_gp_seq
> > > > > > rcu: Add RCU_GET_STATE_NOT_TRACKED for subsystems without expedited
> > > > > > GPs
> > > > > > rcu: Enable RCU callbacks to benefit from expedited grace periods
> > > > > > rcu: Update comments for gp_seq and expedited GP tracking
> > > > > > rcu: Wake NOCB rcuog kthreads on expedited grace period completion
> > > > > > rcu: Detect expedited grace period completion in rcu_pending()
> > > > > > rcu: Advance callbacks for expedited GP completion in rcu_core()
> > > > > > rcuscale: Add concurrent expedited GP threads for callback scaling
> > > > > > tests
> > > > >
> > > > > I do see the occasional failure when running TREE01, as in a failure or
> > > > > four every couple hundred hours of testing. This is the only rcutorture
> > > > > scenario that exercises callback (de)offloading, which might (or might
> > > > > not) be a clue.
> > > > >
> > > > > The console log of a failing run is attached, or you can find it here:
> > > > >
> > > > > https://drive.google.com/file/d/1EH2P4SDy2a-SJndkN3GbCTGAXxI2xevm/view?usp=sharing
> > > > >
> > > > > This is a "??? Writer stall state" failure, where rcutorture never learns
> > > > > of grace-period completion. Possibly because no grace period completed.
> > > > > But there are no RCU CPU stall warnings. In addition, these two lines
> > > > >
> > > > > [ 631.032657] ??? Writer stall state RTWS_EXP_SYNC(4) g152813 f0x0 ->state D cpu 1
> > > > > . . .
> > > > > [ 3610.863114] ??? Writer stall state RTWS_EXP_SYNC(4) g736160 f0x0 ->state D cpu 1
> > > > >
> > > > > show that the grace-period sequence number advanced from 152813 to
> > > > > 736160 in about 50 minutes. So grace periods really are completing,
> > > > > lots of them.
> > > > >
> > > > > This rcutorture scenario has rcu_nocbs CPUs, and CPU 5 is in trouble:
> > > > >
> > > > > [ 631.066109] rcu: CB 5^4->4 KblSW F21044 L21044 C116 .W120(148380/168068).N295B101 q516 S CPU 16
> > > > > . . .
> > > > > [ 3610.869914] rcu: CB 5^4->4 KblSW F3000848 L3000848 C116 .W120(148380/168068).N295B101 q516 S CPU 16
> > > > >
> > > > > There are 120 callbacks on the WAIT segment, which are waiting on
> > > > > grace-period sequence number 148380, which was long gone even back at
> > > > > 631.032657 seconds in (g152813). There are 295 more callbacks in the NEXT
> > > > > segment, which long ago should have been assigned a grace period and moved
> > > > > into the currently empty NEXT_READY segment (which is the "." right before
> > > > > the "N295"). There are yet another 101 callbacks on the bypass list.
> > > > >
> > > > > This suggests that the rcuog kthread isn't being awakened when it
> > > > > should be. And that something is failing to advance callbacks out of
> > > > > NEXT in a timely fashion.
> > > > >
> > > > > Thoughts?
> > > >
> > > > If a grace period has elapsed while in the middle of nocb_gp_wait(),
> > > > after the call to rcu_advance_cbs() and before the following polling which
> > > > determines if we a grace period must be waited upon, then the kthread
> > > > will wait for new callbacks only (and will ignore waiting callbacks that
> > > > can now be advanced).
> > > >
> > > > Does the following help?
> > > >
> > > > diff --git a/kernel/rcu/tree_nocb.h b/kernel/rcu/tree_nocb.h
> > > > index 6da1b8f52476..96d76eb6259d 100644
> > > > --- a/kernel/rcu/tree_nocb.h
> > > > +++ b/kernel/rcu/tree_nocb.h
> > > > @@ -792,22 +792,12 @@ static noinline_for_stack void nocb_gp_wait(struct rcu_data *my_rdp)
> > > > RCU_NEXT_READY_TAIL);
> > > > raw_spin_unlock_rcu_node(rnp); /* irqs disabled. */
> > > > }
> > > > - // Need to wait on some grace period?
> > > > +
> > > > WARN_ON_ONCE(wasempty &&
> > > > !rcu_segcblist_restempty(&rdp->cblist,
> > > > RCU_NEXT_READY_TAIL));
> > > > - /*
> > > > - * Only request a GP wait if the next pending callback's
> > > > - * GP has not already completed (normal or expedited).
> > > > - * If poll_state_synchronize_rcu_full() says it completed,
> > > > - * then rcu_advance_cbs() above already moved those
> > > > - * callbacks to RCU_DONE_TAIL, so there is no GP to wait
> > > > - * for. Any remaining callbacks got new (future) GP
> > > > - * numbers from rcu_accelerate_cbs() inside
> > > > - * rcu_advance_cbs() and will be handled on the next pass.
> > > > - */
> > > > - if (rcu_segcblist_nextgp(&rdp->cblist, &cur_gp_seq) &&
> > > > - !poll_state_synchronize_rcu_full(&cur_gp_seq)) {
> > > > + // Need to wait on some grace period?
> > > > + if (rcu_segcblist_nextgp(&rdp->cblist, &cur_gp_seq)) {
> > > > /*
> > > > * Track the earliest pending normal and expedited GP
> > > > * across the group so the wait below can be released by
> > >
> > >
> > > This looks correct to me, this optimization with
> > > !poll_state_synchronize_rcu_full(&cur_gp_seq) opened up this race as
> > > you correctly explained.
> >
> > My local reproducer says that the problem is gone, but let's see what
> > Paul's 100s of hours of testing says.
>
> I will fire it up, and thank you both!!!
>
> And if it passes the 100s of hours (about 600), it will get 1000s of
> hours over the weekend. ;-)
And error-free!!! (Must be some mistake?)
I am starting an eight-hour run for 1600 hours of TREE01-only test time.
Puranjay is also experimenting with ways of reproducing this faster,
which just might find their way into the normal rcutorture testing. ;-)
I will be running a few longer runs of the full set of scenarios over
the weekend.
Thanx, Paul
^ permalink raw reply
* [PATCH v2 6/11] rcu: Enable RCU callbacks to benefit from expedited grace periods
From: Puranjay Mohan @ 2026-07-24 18:31 UTC (permalink / raw)
To: rcu, linux-kernel, linux-trace-kernel
Cc: Paul E. McKenney, Frederic Weisbecker, Neeraj Upadhyay,
Joel Fernandes, Josh Triplett, Boqun Feng, Uladzislau Rezki,
Steven Rostedt, Mathieu Desnoyers, Lai Jiangshan, Zqiang,
Masami Hiramatsu, Davidlohr Bueso, Breno Leitao, Puranjay Mohan
In-Reply-To: <20260624132356.516959-7-puranjay@kernel.org>
Currently, RCU callbacks only track normal grace-period sequence
numbers. This means callbacks must wait for normal grace periods to
complete even when expedited grace periods have already elapsed.
Use the full struct rcu_gp_seq (which tracks both the normal and
expedited grace-period sequences) throughout the callback
infrastructure.
rcu_segcblist_advance() now checks both normal and expedited GP
completion via poll_state_synchronize_rcu_full(), and becomes
parameterless since it reads the grace-period state internally.
rcu_segcblist_accelerate() stores the full state (both sequences)
instead of just the normal one. rcu_accelerate_cbs() and
rcu_accelerate_cbs_unlocked() use get_state_synchronize_rcu_full() to
capture both sequences, and the NOCB advance checks use
poll_state_synchronize_rcu_full() instead of comparing only the normal
sequence.
srcu_segcblist_advance() becomes a standalone implementation because it
compares SRCU sequences directly and cannot use
poll_state_synchronize_rcu_full(), which reads RCU-specific globals.
srcu_segcblist_accelerate() sets the ->exp field to
RCU_GET_STATE_NOT_TRACKED so that poll_state_synchronize_rcu_full()
compares only ->norm and ignores ->exp.
Signed-off-by: Puranjay Mohan <puranjay@kernel.org>
---
Paul, please take this in place of the original 6/11. The only change
from v1 is dropping the premature !poll_state_synchronize_rcu_full() gate
in nocb_gp_wait(): if a grace period completed between rcu_advance_cbs()
and that poll, the rcuog slept grace-period-deaf in nocb_gp_sleep() with
advanceable callbacks still queued -- the lost wakeup Frederic diagnosed.
Verified by widening that window with a delay: the original reproduced the
writer stall within ~75s, this version stayed clean.
kernel/rcu/rcu_segcblist.c | 30 +++++++++++++++++++++++-------
kernel/rcu/rcu_segcblist.h | 2 +-
kernel/rcu/tree.c | 9 +++------
kernel/rcu/tree_nocb.h | 22 ++++++++++++----------
4 files changed, 39 insertions(+), 24 deletions(-)
diff --git a/kernel/rcu/rcu_segcblist.c b/kernel/rcu/rcu_segcblist.c
index 4e3dfe42bc097..cf8951d33e767 100644
--- a/kernel/rcu/rcu_segcblist.c
+++ b/kernel/rcu/rcu_segcblist.c
@@ -12,6 +12,7 @@
#include <linux/kernel.h>
#include <linux/types.h>
+#include "rcu.h"
#include "rcu_segcblist.h"
/* Initialize simple callback list. */
@@ -494,9 +495,9 @@ static void rcu_segcblist_advance_compact(struct rcu_segcblist *rsclp, int i)
/*
* Advance the callbacks in the specified rcu_segcblist structure based
- * on the current value passed in for the grace-period counter.
+ * on the current value of the grace-period counter.
*/
-void rcu_segcblist_advance(struct rcu_segcblist *rsclp, struct rcu_gp_seq *gsp)
+void rcu_segcblist_advance(struct rcu_segcblist *rsclp)
{
int i;
@@ -509,7 +510,7 @@ void rcu_segcblist_advance(struct rcu_segcblist *rsclp, struct rcu_gp_seq *gsp)
* are ready to invoke, and put them into the RCU_DONE_TAIL segment.
*/
for (i = RCU_WAIT_TAIL; i < RCU_NEXT_TAIL; i++) {
- if (ULONG_CMP_LT(gsp->norm, rsclp->gp_seq[i].norm))
+ if (!poll_state_synchronize_rcu_full(&rsclp->gp_seq[i]))
break;
WRITE_ONCE(rsclp->tails[RCU_DONE_TAIL], rsclp->tails[i]);
rcu_segcblist_move_seglen(rsclp, i, RCU_DONE_TAIL);
@@ -595,7 +596,7 @@ bool rcu_segcblist_accelerate(struct rcu_segcblist *rsclp, struct rcu_gp_seq *gs
*/
for (; i < RCU_NEXT_TAIL; i++) {
WRITE_ONCE(rsclp->tails[i], rsclp->tails[RCU_NEXT_TAIL]);
- rsclp->gp_seq[i].norm = gsp->norm;
+ rsclp->gp_seq[i] = *gsp;
}
return true;
}
@@ -637,14 +638,29 @@ void rcu_segcblist_merge(struct rcu_segcblist *dst_rsclp,
void srcu_segcblist_advance(struct rcu_segcblist *rsclp, unsigned long seq)
{
- struct rcu_gp_seq gs = { .norm = seq };
+ int i;
+
+ WARN_ON_ONCE(!rcu_segcblist_is_enabled(rsclp));
+ if (rcu_segcblist_restempty(rsclp, RCU_DONE_TAIL))
+ return;
+
+ for (i = RCU_WAIT_TAIL; i < RCU_NEXT_TAIL; i++) {
+ if (ULONG_CMP_LT(seq, rsclp->gp_seq[i].norm))
+ break;
+ WRITE_ONCE(rsclp->tails[RCU_DONE_TAIL], rsclp->tails[i]);
+ rcu_segcblist_move_seglen(rsclp, i, RCU_DONE_TAIL);
+ }
+
+ /* If no callbacks moved, nothing more need be done. */
+ if (i == RCU_WAIT_TAIL)
+ return;
- rcu_segcblist_advance(rsclp, &gs);
+ rcu_segcblist_advance_compact(rsclp, i);
}
bool srcu_segcblist_accelerate(struct rcu_segcblist *rsclp, unsigned long seq)
{
- struct rcu_gp_seq gs = { .norm = seq };
+ struct rcu_gp_seq gs = { .norm = seq, .exp = RCU_GET_STATE_NOT_TRACKED };
return rcu_segcblist_accelerate(rsclp, &gs);
}
diff --git a/kernel/rcu/rcu_segcblist.h b/kernel/rcu/rcu_segcblist.h
index 16b0cb6b32507..431c4466b8898 100644
--- a/kernel/rcu/rcu_segcblist.h
+++ b/kernel/rcu/rcu_segcblist.h
@@ -139,7 +139,7 @@ void rcu_segcblist_insert_done_cbs(struct rcu_segcblist *rsclp,
struct rcu_cblist *rclp);
void rcu_segcblist_insert_pend_cbs(struct rcu_segcblist *rsclp,
struct rcu_cblist *rclp);
-void rcu_segcblist_advance(struct rcu_segcblist *rsclp, struct rcu_gp_seq *gsp);
+void rcu_segcblist_advance(struct rcu_segcblist *rsclp);
bool rcu_segcblist_accelerate(struct rcu_segcblist *rsclp, struct rcu_gp_seq *gsp);
void rcu_segcblist_merge(struct rcu_segcblist *dst_rsclp,
struct rcu_segcblist *src_rsclp);
diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c
index 640ebe055a246..f63ce75c5b4f5 100644
--- a/kernel/rcu/tree.c
+++ b/kernel/rcu/tree.c
@@ -1159,7 +1159,7 @@ static bool rcu_accelerate_cbs(struct rcu_node *rnp, struct rcu_data *rdp)
* accelerating callback invocation to an earlier grace-period
* number.
*/
- gs.norm = rcu_seq_snap(&rcu_state.gp_seq);
+ get_state_synchronize_rcu_full(&gs);
if (rcu_segcblist_accelerate(&rdp->cblist, &gs))
ret = rcu_start_this_gp(rnp, rdp, gs.norm);
@@ -1188,7 +1188,7 @@ static void rcu_accelerate_cbs_unlocked(struct rcu_node *rnp,
bool needwake;
rcu_lockdep_assert_cblist_protected(rdp);
- gs.norm = rcu_seq_snap(&rcu_state.gp_seq);
+ get_state_synchronize_rcu_full(&gs);
if (!READ_ONCE(rdp->gpwrap) && ULONG_CMP_GE(rdp->gp_seq_needed, gs.norm)) {
/* Old request still live, so mark recent callbacks. */
(void)rcu_segcblist_accelerate(&rdp->cblist, &gs);
@@ -1213,8 +1213,6 @@ static void rcu_accelerate_cbs_unlocked(struct rcu_node *rnp,
*/
static bool rcu_advance_cbs(struct rcu_node *rnp, struct rcu_data *rdp)
{
- struct rcu_gp_seq gs;
-
rcu_lockdep_assert_cblist_protected(rdp);
raw_lockdep_assert_held_rcu_node(rnp);
@@ -1226,8 +1224,7 @@ static bool rcu_advance_cbs(struct rcu_node *rnp, struct rcu_data *rdp)
* Find all callbacks whose ->gp_seq numbers indicate that they
* are ready to invoke, and put them into the RCU_DONE_TAIL sublist.
*/
- gs.norm = rnp->gp_seq;
- rcu_segcblist_advance(&rdp->cblist, &gs);
+ rcu_segcblist_advance(&rdp->cblist);
/* Classify any remaining callbacks. */
return rcu_accelerate_cbs(rnp, rdp);
diff --git a/kernel/rcu/tree_nocb.h b/kernel/rcu/tree_nocb.h
index eaa1ad5297e29..876dd897f18be 100644
--- a/kernel/rcu/tree_nocb.h
+++ b/kernel/rcu/tree_nocb.h
@@ -502,7 +502,7 @@ static bool rcu_nocb_try_bypass(struct rcu_data *rdp, struct rcu_head *rhp,
}
if (j != rdp->nocb_gp_adv_time &&
rcu_segcblist_nextgp(&rdp->cblist, &cur_gp_seq) &&
- rcu_seq_done(&rdp->mynode->gp_seq, cur_gp_seq.norm)) {
+ poll_state_synchronize_rcu_full(&cur_gp_seq)) {
rcu_advance_cbs_nowake(rdp->mynode, rdp);
rdp->nocb_gp_adv_time = j;
}
@@ -731,17 +731,17 @@ static noinline_for_stack void nocb_gp_wait(struct rcu_data *my_rdp)
if (!rcu_segcblist_restempty(&rdp->cblist,
RCU_NEXT_READY_TAIL) ||
(rcu_segcblist_nextgp(&rdp->cblist, &cur_gp_seq) &&
- rcu_seq_done(&rnp->gp_seq, cur_gp_seq.norm))) {
+ poll_state_synchronize_rcu_full(&cur_gp_seq))) {
raw_spin_lock_rcu_node(rnp); /* irqs disabled. */
needwake_gp = rcu_advance_cbs(rnp, rdp);
wasempty = rcu_segcblist_restempty(&rdp->cblist,
RCU_NEXT_READY_TAIL);
raw_spin_unlock_rcu_node(rnp); /* irqs disabled. */
}
- // Need to wait on some grace period?
WARN_ON_ONCE(wasempty &&
!rcu_segcblist_restempty(&rdp->cblist,
RCU_NEXT_READY_TAIL));
+ // Need to wait on some grace period?
if (rcu_segcblist_nextgp(&rdp->cblist, &cur_gp_seq)) {
if (!needwait_gp ||
ULONG_CMP_LT(cur_gp_seq.norm, wait_gp_seq))
@@ -919,7 +919,7 @@ static void nocb_cb_wait(struct rcu_data *rdp)
lockdep_assert_irqs_enabled();
rcu_nocb_lock_irqsave(rdp, flags);
if (rcu_segcblist_nextgp(cblist, &cur_gp_seq) &&
- rcu_seq_done(&rnp->gp_seq, cur_gp_seq.norm) &&
+ poll_state_synchronize_rcu_full(&cur_gp_seq) &&
raw_spin_trylock_rcu_node(rnp)) { /* irqs already disabled. */
needwake_gp = rcu_advance_cbs(rdp->mynode, rdp);
raw_spin_unlock_rcu_node(rnp); /* irqs remain disabled. */
@@ -1548,8 +1548,8 @@ static void show_rcu_nocb_gp_state(struct rcu_data *rdp)
static void show_rcu_nocb_state(struct rcu_data *rdp)
{
char bufd[22];
- char bufw[45];
- char bufr[45];
+ char bufw[64];
+ char bufr[64];
char bufn[22];
char bufb[22];
struct rcu_data *nocb_next_rdp;
@@ -1569,10 +1569,12 @@ static void show_rcu_nocb_state(struct rcu_data *rdp)
nocb_entry_rdp);
sprintf(bufd, "%ld", rsclp->seglen[RCU_DONE_TAIL]);
- sprintf(bufw, "%ld(%ld)", rsclp->seglen[RCU_WAIT_TAIL],
- rsclp->gp_seq[RCU_WAIT_TAIL].norm);
- sprintf(bufr, "%ld(%ld)", rsclp->seglen[RCU_NEXT_READY_TAIL],
- rsclp->gp_seq[RCU_NEXT_READY_TAIL].norm);
+ sprintf(bufw, "%ld(%ld/%ld)", rsclp->seglen[RCU_WAIT_TAIL],
+ rsclp->gp_seq[RCU_WAIT_TAIL].norm,
+ rsclp->gp_seq[RCU_WAIT_TAIL].exp);
+ sprintf(bufr, "%ld(%ld/%ld)", rsclp->seglen[RCU_NEXT_READY_TAIL],
+ rsclp->gp_seq[RCU_NEXT_READY_TAIL].norm,
+ rsclp->gp_seq[RCU_NEXT_READY_TAIL].exp);
sprintf(bufn, "%ld", rsclp->seglen[RCU_NEXT_TAIL]);
sprintf(bufb, "%ld", rcu_cblist_n_cbs(&rdp->nocb_bypass));
pr_info(" CB %d^%d->%d %c%c%c%c%c F%ld L%ld C%d %c%s%c%s%c%s%c%s%c%s q%ld %c CPU %d%s\n",
--
2.53.0-Meta
^ permalink raw reply related
page: | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox