Possible enhancement to RCU CPU stall warning messages

Possible enhancement to RCU CPU stall warning messages

[Paul E. McKenney]

Hello!

While chasing an "interesting" bug [1], I modified the RCU CPU
stall-warning code so that when a task (as opposed to a CPU) is
blocking the grace period due to being preempted, the stall-warning
message includes the task that is currently running on the CPU that the
preempted task wants to run on.  (In the case of the "interesting" bug,
I kid you not, that CPU was idle.)

Would this stall-warning addition be useful for PREEMPT_RT users?

If it is not useful, I am going to keep it out of tree.  Something about
it needing a way to dig into scheduler internals.  ;-)

But if you want it, please let me know.

							Thanx, Paul

[1] https://lore.kernel.org/all/8fa7f795-518c-4507-b684-60b0c1cac2c5@paulmck-laptop/

------------------------------------------------------------------------

commit 24853cb1cfb363ce608e2098ef22c272d475b84c
Author: Paul E. McKenney <paulmck@kernel.org>
Date:   Mon Sep 28 13:35:01 2026 -0700

    EXP rcu: Print CPU that preempted stalled reader wants to run on
    
    This commit uses the dirty trick of defining an rcu_cpu_curr() within
    kernel/sched/core.c in order to allow the RCU CPU stall warning call out
    the CPU that a task preempted within an RCU reader would like to run on.
    In addition, print out the task (if any) currently running on that CPU,
    and add an exclamation mark ("!") after the CPU number if that CPU is
    currently offline.
    
    This is a potential step towards allowing people (or LLMs, for that
    matter) to locate any tasks that might be preventing the preempted RCU
    reader task from running.
    
    Signed-off-by: Paul E. McKenney <paulmck@kernel.org>

diff --git a/include/linux/sched.h b/include/linux/sched.h
index 8b3d47a325cca..a8b3434edb2b4 100644
--- a/include/linux/sched.h
+++ b/include/linux/sched.h
@@ -2532,3 +2532,5 @@ extern void migrate_enable(void);
 DEFINE_LOCK_GUARD_0(migrate, migrate_disable(), migrate_enable())
 
 #endif
+
+struct task_struct *rcu_cpu_curr(int cpu);
diff --git a/kernel/rcu/tree_stall.h b/kernel/rcu/tree_stall.h
index 803a56af3259f..7826c4d905248 100644
--- a/kernel/rcu/tree_stall.h
+++ b/kernel/rcu/tree_stall.h
@@ -309,6 +309,9 @@ struct rcu_stall_chk_rdr {
 	bool on_blkd_list;
 	bool rcu_rdr_running;
 	int rcu_rdr_boosted;
+	int rcu_rdr_cpu;
+	int rcu_rdr_cpu_offline;
+	struct task_struct *rcu_rdr_cpu_task;
 };
 
 /*
@@ -317,8 +320,10 @@ struct rcu_stall_chk_rdr {
  */
 static int check_slow_task(struct task_struct *t, void *arg)
 {
+	int cpu;
 	struct rcu_node *rnp;
 	struct rcu_stall_chk_rdr *rscrp = arg;
+	struct task_struct *th;
 
 	if (task_curr(t))
 		return -EBUSY; // It is running, so decline to inspect it.
@@ -327,6 +332,9 @@ static int check_slow_task(struct task_struct *t, void *arg)
 	rscrp->on_blkd_list = !list_empty(&t->rcu_node_entry);
 	rscrp->rcu_rdr_running = task_curr(t);
 	rscrp->rcu_rdr_boosted = 0;
+	rscrp->rcu_rdr_cpu = cpu = task_cpu(t);
+	rscrp->rcu_rdr_cpu_offline = cpu_is_offline(cpu);
+	rscrp->rcu_rdr_cpu_task = th = rcu_cpu_curr(cpu);
 	if (rscrp->on_blkd_list) {
 		rnp = READ_ONCE(t->rcu_blocked_node);
 		raw_spin_lock_rcu_node(rnp); /* irqs already disabled. */
@@ -352,6 +360,7 @@ static int rcu_print_task_stall(struct rcu_node *rnp, unsigned long flags)
 	int ndetected = 0;
 	struct rcu_stall_chk_rdr rscr;
 	struct task_struct *t;
+	struct task_struct *tc;
 	struct task_struct *ts[8];
 
 	lockdep_assert_irqs_disabled();
@@ -372,17 +381,24 @@ static int rcu_print_task_stall(struct rcu_node *rnp, unsigned long flags)
 	raw_spin_unlock_irqrestore_rcu_node(rnp, flags);
 	while (i) {
 		t = ts[--i];
-		if (task_call_func(t, check_slow_task, &rscr))
+		if (task_call_func(t, check_slow_task, &rscr)) {
 			pr_cont(" P%d", t->pid);
-		else
-			pr_cont(" P%d/%d:%c%c%c%c%c%c",
+		} else {
+			tc = rscr.rcu_rdr_cpu_task;
+			pr_cont(" P%d/%d:%c%c%c%c%c%c CPU %d%s %s (%d)%s",
 				t->pid, rscr.nesting,
 				".b"[rscr.rs.b.blocked],
 				".q"[rscr.rs.b.need_qs],
 				".e"[rscr.rs.b.exp_hint],
 				".l"[rscr.on_blkd_list],
 				".R"[rscr.rcu_rdr_running],
-				".B?"[rscr.rcu_rdr_boosted]);
+				".B?"[rscr.rcu_rdr_boosted],
+				rscr.rcu_rdr_cpu,
+				rscr.rcu_rdr_cpu_offline ? "!" : "",
+				tc ? tc->comm : "???",
+				tc ? tc->pid : -1,
+				tc ? (is_idle_task(tc) ? " (idle!)" : "") : " (no task)");
+		}
 		lockdep_assert_irqs_disabled();
 		put_task_struct(t);
 		ndetected++;
diff --git a/kernel/sched/core.c b/kernel/sched/core.c
index b77152edafd92..e137cba24fc92 100644
--- a/kernel/sched/core.c
+++ b/kernel/sched/core.c
@@ -11321,3 +11321,10 @@ void sched_change_end(struct sched_change_ctx *ctx)
 		p->sched_class->prio_changed(rq, p, ctx->prio);
 	}
 }
+
+// Not for mainline, but hopefully someone has a trick that gets this
+// job done.
+struct task_struct *rcu_cpu_curr(int cpu)
+{
+	return cpu_curr(cpu);
+}

Re: Possible enhancement to RCU CPU stall warning messages

[Clark Williams]

On Tue, Oct 13, 2026 at 05:41:41PM -0700, Paul E. McKenney wrote:
> Hello!
> 
> While chasing an "interesting" bug [1], I modified the RCU CPU
> stall-warning code so that when a task (as opposed to a CPU) is
> blocking the grace period due to being preempted, the stall-warning
> message includes the task that is currently running on the CPU that the
> preempted task wants to run on.  (In the case of the "interesting" bug,
> I kid you not, that CPU was idle.)
> 
> Would this stall-warning addition be useful for PREEMPT_RT users?
> 
> If it is not useful, I am going to keep it out of tree.  Something about
> it needing a way to dig into scheduler internals.  ;-)
> 
> But if you want it, please let me know.
> 
> 							Thanx, Paul
> 
> [1] https://lore.kernel.org/all/8fa7f795-518c-4507-b684-60b0c1cac2c5@paulmck-laptop/
> 

Paul,

While RCU isn't one of the common reasons for a latency spike, if it
*is*, then it's not one that's easily tracked down.  For that reason I'm
in favor of using your patch. 

That said, if it's going to be a pain to maintain, I can live without it. 

Clark

Re: Possible enhancement to RCU CPU stall warning messages

[Paul E. McKenney]

On Thu, Oct 15, 2026 at 10:27:24AM -0500, Clark Williams wrote:
> On Tue, Oct 13, 2026 at 05:41:41PM -0700, Paul E. McKenney wrote:
> > Hello!
> > 
> > While chasing an "interesting" bug [1], I modified the RCU CPU
> > stall-warning code so that when a task (as opposed to a CPU) is
> > blocking the grace period due to being preempted, the stall-warning
> > message includes the task that is currently running on the CPU that the
> > preempted task wants to run on.  (In the case of the "interesting" bug,
> > I kid you not, that CPU was idle.)
> > 
> > Would this stall-warning addition be useful for PREEMPT_RT users?
> > 
> > If it is not useful, I am going to keep it out of tree.  Something about
> > it needing a way to dig into scheduler internals.  ;-)
> > 
> > But if you want it, please let me know.
> > 
> > 							Thanx, Paul
> > 
> > [1] https://lore.kernel.org/all/8fa7f795-518c-4507-b684-60b0c1cac2c5@paulmck-laptop/
> > 
> 
> Paul,
> 
> While RCU isn't one of the common reasons for a latency spike, if it
> *is*, then it's not one that's easily tracked down.  For that reason I'm
> in favor of using your patch. 
> 
> That said, if it's going to be a pain to maintain, I can live without it. 

No problem maintaining it, just a small up-front cost of convincing the
scheduler guys to give me access to cpu_curr() or some other API that
allows RCU to find out what is currently running on the CPU that a given
task is trying to run on.

Given that it would be helpful to you, that is enough to motivate me to
go hassle the scheduler guys.  ;-)

							Thanx, Paul

Re: Possible enhancement to RCU CPU stall warning messages

[Crystal Wood]

On Thu, 2026-10-15 at 10:27 -0500, Clark Williams wrote:
> On Tue, Oct 13, 2026 at 05:41:41PM -0700, Paul E. McKenney wrote:
> > Hello!
> > 
> > While chasing an "interesting" bug [1], I modified the RCU CPU
> > stall-warning code so that when a task (as opposed to a CPU) is
> > blocking the grace period due to being preempted, the stall-warning
> > message includes the task that is currently running on the CPU that the
> > preempted task wants to run on.  (In the case of the "interesting" bug,
> > I kid you not, that CPU was idle.)
> > 
> > Would this stall-warning addition be useful for PREEMPT_RT users?
> > 
> > If it is not useful, I am going to keep it out of tree.  Something about
> > it needing a way to dig into scheduler internals.  ;-)
> > 
> > But if you want it, please let me know.
> > 
> > 							Thanx, Paul
> > 
> > [1] https://lore.kernel.org/all/8fa7f795-518c-4507-b684-60b0c1cac2c5@paulmck-laptop/
> > 
> 
> Paul,
> 
> While RCU isn't one of the common reasons for a latency spike, if it
> *is*, then it's not one that's easily tracked down.  For that reason I'm
> in favor of using your patch. 
> 
> That said, if it's going to be a pain to maintain, I can live without it. 

RCU isn't usually the reason, but it's often the thing that actually
speaks up and says "something's wrong here", and being able to print the
task that has a good chance of being the problem (in cases where the
stalling thread is runnable) sounds quite useful.


-Crystal

Re: Possible enhancement to RCU CPU stall warning messages

[Paul E. McKenney]

On Fri, Oct 16, 2026 at 02:13:15AM -0500, Crystal Wood wrote:
> On Thu, 2026-10-15 at 10:27 -0500, Clark Williams wrote:
> > On Tue, Oct 13, 2026 at 05:41:41PM -0700, Paul E. McKenney wrote:
> > > Hello!
> > > 
> > > While chasing an "interesting" bug [1], I modified the RCU CPU
> > > stall-warning code so that when a task (as opposed to a CPU) is
> > > blocking the grace period due to being preempted, the stall-warning
> > > message includes the task that is currently running on the CPU that the
> > > preempted task wants to run on.  (In the case of the "interesting" bug,
> > > I kid you not, that CPU was idle.)
> > > 
> > > Would this stall-warning addition be useful for PREEMPT_RT users?
> > > 
> > > If it is not useful, I am going to keep it out of tree.  Something about
> > > it needing a way to dig into scheduler internals.  ;-)
> > > 
> > > But if you want it, please let me know.
> > > 
> > > 							Thanx, Paul
> > > 
> > > [1] https://lore.kernel.org/all/8fa7f795-518c-4507-b684-60b0c1cac2c5@paulmck-laptop/
> > > 
> > 
> > Paul,
> > 
> > While RCU isn't one of the common reasons for a latency spike, if it
> > *is*, then it's not one that's easily tracked down.  For that reason I'm
> > in favor of using your patch. 
> > 
> > That said, if it's going to be a pain to maintain, I can live without it. 
> 
> RCU isn't usually the reason, but it's often the thing that actually
> speaks up and says "something's wrong here", and being able to print the
> task that has a good chance of being the problem (in cases where the
> stalling thread is runnable) sounds quite useful.

Thank you!  Thus far, we have two in favor and none against.

Plus the scheduler guys gave me a way forward that I am testing now.  ;-)

							Thanx, Paul

Re: Possible enhancement to RCU CPU stall warning messages

[Steven Rostedt]

On Tue, 13 Oct 2026 17:41:41 -0700
"Paul E. McKenney" <paulmck@kernel.org> wrote:

> diff --git a/include/linux/sched.h b/include/linux/sched.h
> index 8b3d47a325cca..a8b3434edb2b4 100644
> --- a/include/linux/sched.h
> +++ b/include/linux/sched.h
> @@ -2532,3 +2532,5 @@ extern void migrate_enable(void);
>  DEFINE_LOCK_GUARD_0(migrate, migrate_disable(), migrate_enable())
>  
>  #endif
> +
> +struct task_struct *rcu_cpu_curr(int cpu);
> diff --git a/kernel/rcu/tree_stall.h b/kernel/rcu/tree_stall.h
> index 803a56af3259f..7826c4d905248 100644
> --- a/kernel/rcu/tree_stall.h
> +++ b/kernel/rcu/tree_stall.h
> @@ -309,6 +309,9 @@ struct rcu_stall_chk_rdr {
>  	bool on_blkd_list;
>  	bool rcu_rdr_running;
>  	int rcu_rdr_boosted;
> +	int rcu_rdr_cpu;
> +	int rcu_rdr_cpu_offline;
> +	struct task_struct *rcu_rdr_cpu_task;
>  };
>  
>  /*
> @@ -317,8 +320,10 @@ struct rcu_stall_chk_rdr {
>   */
>  static int check_slow_task(struct task_struct *t, void *arg)
>  {
> +	int cpu;
>  	struct rcu_node *rnp;
>  	struct rcu_stall_chk_rdr *rscrp = arg;
> +	struct task_struct *th;
>  
>  	if (task_curr(t))
>  		return -EBUSY; // It is running, so decline to inspect it.
> @@ -327,6 +332,9 @@ static int check_slow_task(struct task_struct *t, void *arg)
>  	rscrp->on_blkd_list = !list_empty(&t->rcu_node_entry);
>  	rscrp->rcu_rdr_running = task_curr(t);
>  	rscrp->rcu_rdr_boosted = 0;
> +	rscrp->rcu_rdr_cpu = cpu = task_cpu(t);
> +	rscrp->rcu_rdr_cpu_offline = cpu_is_offline(cpu);
> +	rscrp->rcu_rdr_cpu_task = th = rcu_cpu_curr(cpu);
>  	if (rscrp->on_blkd_list) {
>  		rnp = READ_ONCE(t->rcu_blocked_node);
>  		raw_spin_lock_rcu_node(rnp); /* irqs already disabled. */
> @@ -352,6 +360,7 @@ static int rcu_print_task_stall(struct rcu_node *rnp, unsigned long flags)
>  	int ndetected = 0;
>  	struct rcu_stall_chk_rdr rscr;
>  	struct task_struct *t;
> +	struct task_struct *tc;
>  	struct task_struct *ts[8];
>  
>  	lockdep_assert_irqs_disabled();
> @@ -372,17 +381,24 @@ static int rcu_print_task_stall(struct rcu_node *rnp, unsigned long flags)
>  	raw_spin_unlock_irqrestore_rcu_node(rnp, flags);
>  	while (i) {
>  		t = ts[--i];
> -		if (task_call_func(t, check_slow_task, &rscr))
> +		if (task_call_func(t, check_slow_task, &rscr)) {
>  			pr_cont(" P%d", t->pid);
> -		else
> -			pr_cont(" P%d/%d:%c%c%c%c%c%c",
> +		} else {
> +			tc = rscr.rcu_rdr_cpu_task;

Is there a possible race between the call to rcu_cpu_curr() and here, where
that task could have exited and is freed?

There's no get_task_struct() on that task.

-- Steve


> +			pr_cont(" P%d/%d:%c%c%c%c%c%c CPU %d%s %s (%d)%s",
>  				t->pid, rscr.nesting,
>  				".b"[rscr.rs.b.blocked],
>  				".q"[rscr.rs.b.need_qs],
>  				".e"[rscr.rs.b.exp_hint],
>  				".l"[rscr.on_blkd_list],
>  				".R"[rscr.rcu_rdr_running],
> -				".B?"[rscr.rcu_rdr_boosted]);
> +				".B?"[rscr.rcu_rdr_boosted],
> +				rscr.rcu_rdr_cpu,
> +				rscr.rcu_rdr_cpu_offline ? "!" : "",
> +				tc ? tc->comm : "???",
> +				tc ? tc->pid : -1,
> +				tc ? (is_idle_task(tc) ? " (idle!)" : "") : " (no task)");
> +		}
>  		lockdep_assert_irqs_disabled();
>  		put_task_struct(t);
>  		ndetected++;
> diff --git a/kernel/sched/core.c b/kernel/sched/core.c
> index b77152edafd92..e137cba24fc92 100644
> --- a/kernel/sched/core.c
> +++ b/kernel/sched/core.c
> @@ -11321,3 +11321,10 @@ void sched_change_end(struct sched_change_ctx *ctx)
>  		p->sched_class->prio_changed(rq, p, ctx->prio);
>  	}
>  }
> +
> +// Not for mainline, but hopefully someone has a trick that gets this
> +// job done.
> +struct task_struct *rcu_cpu_curr(int cpu)
> +{
> +	return cpu_curr(cpu);
> +}

Re: Possible enhancement to RCU CPU stall warning messages

[Paul E. McKenney]

On Fri, Oct 16, 2026 at 11:23:39AM -0400, Steven Rostedt wrote:
> On Tue, 13 Oct 2026 17:41:41 -0700
> "Paul E. McKenney" <paulmck@kernel.org> wrote:
> 
> > diff --git a/include/linux/sched.h b/include/linux/sched.h
> > index 8b3d47a325cca..a8b3434edb2b4 100644
> > --- a/include/linux/sched.h
> > +++ b/include/linux/sched.h
> > @@ -2532,3 +2532,5 @@ extern void migrate_enable(void);
> >  DEFINE_LOCK_GUARD_0(migrate, migrate_disable(), migrate_enable())
> >  
> >  #endif
> > +
> > +struct task_struct *rcu_cpu_curr(int cpu);
> > diff --git a/kernel/rcu/tree_stall.h b/kernel/rcu/tree_stall.h
> > index 803a56af3259f..7826c4d905248 100644
> > --- a/kernel/rcu/tree_stall.h
> > +++ b/kernel/rcu/tree_stall.h
> > @@ -309,6 +309,9 @@ struct rcu_stall_chk_rdr {
> >  	bool on_blkd_list;
> >  	bool rcu_rdr_running;
> >  	int rcu_rdr_boosted;
> > +	int rcu_rdr_cpu;
> > +	int rcu_rdr_cpu_offline;
> > +	struct task_struct *rcu_rdr_cpu_task;
> >  };
> >  
> >  /*
> > @@ -317,8 +320,10 @@ struct rcu_stall_chk_rdr {
> >   */
> >  static int check_slow_task(struct task_struct *t, void *arg)
> >  {
> > +	int cpu;
> >  	struct rcu_node *rnp;
> >  	struct rcu_stall_chk_rdr *rscrp = arg;
> > +	struct task_struct *th;
> >  
> >  	if (task_curr(t))
> >  		return -EBUSY; // It is running, so decline to inspect it.
> > @@ -327,6 +332,9 @@ static int check_slow_task(struct task_struct *t, void *arg)
> >  	rscrp->on_blkd_list = !list_empty(&t->rcu_node_entry);
> >  	rscrp->rcu_rdr_running = task_curr(t);
> >  	rscrp->rcu_rdr_boosted = 0;
> > +	rscrp->rcu_rdr_cpu = cpu = task_cpu(t);
> > +	rscrp->rcu_rdr_cpu_offline = cpu_is_offline(cpu);
> > +	rscrp->rcu_rdr_cpu_task = th = rcu_cpu_curr(cpu);
> >  	if (rscrp->on_blkd_list) {
> >  		rnp = READ_ONCE(t->rcu_blocked_node);
> >  		raw_spin_lock_rcu_node(rnp); /* irqs already disabled. */
> > @@ -352,6 +360,7 @@ static int rcu_print_task_stall(struct rcu_node *rnp, unsigned long flags)
> >  	int ndetected = 0;
> >  	struct rcu_stall_chk_rdr rscr;
> >  	struct task_struct *t;
> > +	struct task_struct *tc;
> >  	struct task_struct *ts[8];
> >  
> >  	lockdep_assert_irqs_disabled();
> > @@ -372,17 +381,24 @@ static int rcu_print_task_stall(struct rcu_node *rnp, unsigned long flags)
> >  	raw_spin_unlock_irqrestore_rcu_node(rnp, flags);
> >  	while (i) {
> >  		t = ts[--i];
> > -		if (task_call_func(t, check_slow_task, &rscr))
> > +		if (task_call_func(t, check_slow_task, &rscr)) {
> >  			pr_cont(" P%d", t->pid);
> > -		else
> > -			pr_cont(" P%d/%d:%c%c%c%c%c%c",
> > +		} else {
> > +			tc = rscr.rcu_rdr_cpu_task;
> 
> Is there a possible race between the call to rcu_cpu_curr() and here, where
> that task could have exited and is freed?
> 
> There's no get_task_struct() on that task.

The get_task_struct() happens in the list_for_each_entry_continue()
loop that it right before the raw_spin_unlock_irqrestore_rcu_node().

Or do I have an off-by-one bug here?

							Thanx, Paul

> -- Steve
> 
> 
> > +			pr_cont(" P%d/%d:%c%c%c%c%c%c CPU %d%s %s (%d)%s",
> >  				t->pid, rscr.nesting,
> >  				".b"[rscr.rs.b.blocked],
> >  				".q"[rscr.rs.b.need_qs],
> >  				".e"[rscr.rs.b.exp_hint],
> >  				".l"[rscr.on_blkd_list],
> >  				".R"[rscr.rcu_rdr_running],
> > -				".B?"[rscr.rcu_rdr_boosted]);
> > +				".B?"[rscr.rcu_rdr_boosted],
> > +				rscr.rcu_rdr_cpu,
> > +				rscr.rcu_rdr_cpu_offline ? "!" : "",
> > +				tc ? tc->comm : "???",
> > +				tc ? tc->pid : -1,
> > +				tc ? (is_idle_task(tc) ? " (idle!)" : "") : " (no task)");
> > +		}
> >  		lockdep_assert_irqs_disabled();
> >  		put_task_struct(t);
> >  		ndetected++;
> > diff --git a/kernel/sched/core.c b/kernel/sched/core.c
> > index b77152edafd92..e137cba24fc92 100644
> > --- a/kernel/sched/core.c
> > +++ b/kernel/sched/core.c
> > @@ -11321,3 +11321,10 @@ void sched_change_end(struct sched_change_ctx *ctx)
> >  		p->sched_class->prio_changed(rq, p, ctx->prio);
> >  	}
> >  }
> > +
> > +// Not for mainline, but hopefully someone has a trick that gets this
> > +// job done.
> > +struct task_struct *rcu_cpu_curr(int cpu)
> > +{
> > +	return cpu_curr(cpu);
> > +}
> 

Re: Possible enhancement to RCU CPU stall warning messages

[Steven Rostedt]

On Fri, 16 Oct 2026 09:00:31 -0700
"Paul E. McKenney" <paulmck@kernel.org> wrote:

> > > @@ -372,17 +381,24 @@ static int rcu_print_task_stall(struct rcu_node *rnp, unsigned long flags)
> > >  	raw_spin_unlock_irqrestore_rcu_node(rnp, flags);
> > >  	while (i) {
> > >  		t = ts[--i];
> > > -		if (task_call_func(t, check_slow_task, &rscr))
> > > +		if (task_call_func(t, check_slow_task, &rscr)) {
> > >  			pr_cont(" P%d", t->pid);
> > > -		else
> > > -			pr_cont(" P%d/%d:%c%c%c%c%c%c",
> > > +		} else {
> > > +			tc = rscr.rcu_rdr_cpu_task;  
> > 
> > Is there a possible race between the call to rcu_cpu_curr() and here, where
> > that task could have exited and is freed?
> > 
> > There's no get_task_struct() on that task.  
> 
> The get_task_struct() happens in the list_for_each_entry_continue()
> loop that it right before the raw_spin_unlock_irqrestore_rcu_node().
> 
> Or do I have an off-by-one bug here?
> 

But doesn't that loop just cover the tasks in &rnp->blkd_tasks?

Does that include the task returned by cpu_curr(cpu) ?

-- Steve

Re: Possible enhancement to RCU CPU stall warning messages

[Paul E. McKenney]

On Fri, Oct 16, 2026 at 12:17:53PM -0400, Steven Rostedt wrote:
> On Fri, 16 Oct 2026 09:00:31 -0700
> "Paul E. McKenney" <paulmck@kernel.org> wrote:
> 
> > > > @@ -372,17 +381,24 @@ static int rcu_print_task_stall(struct rcu_node *rnp, unsigned long flags)
> > > >  	raw_spin_unlock_irqrestore_rcu_node(rnp, flags);
> > > >  	while (i) {
> > > >  		t = ts[--i];
> > > > -		if (task_call_func(t, check_slow_task, &rscr))
> > > > +		if (task_call_func(t, check_slow_task, &rscr)) {
> > > >  			pr_cont(" P%d", t->pid);
> > > > -		else
> > > > -			pr_cont(" P%d/%d:%c%c%c%c%c%c",
> > > > +		} else {
> > > > +			tc = rscr.rcu_rdr_cpu_task;  
> > > 
> > > Is there a possible race between the call to rcu_cpu_curr() and here, where
> > > that task could have exited and is freed?
> > > 
> > > There's no get_task_struct() on that task.  
> > 
> > The get_task_struct() happens in the list_for_each_entry_continue()
> > loop that it right before the raw_spin_unlock_irqrestore_rcu_node().
> > 
> > Or do I have an off-by-one bug here?
> > 
> 
> But doesn't that loop just cover the tasks in &rnp->blkd_tasks?
> 
> Does that include the task returned by cpu_curr(cpu) ?

Here is the full function:

------------------------------------------------------------------------

static int rcu_print_task_stall(struct rcu_node *rnp, unsigned long flags)
	__releases(rnp->lock)
{
	int i = 0;
	int ndetected = 0;
	struct rcu_stall_chk_rdr rscr;
	struct task_struct *t;
	struct task_struct *tc;
	struct task_struct *ts[8];

	lockdep_assert_irqs_disabled();
	if (!rcu_preempt_blocked_readers_cgp(rnp)) {
		raw_spin_unlock_irqrestore_rcu_node(rnp, flags);
		return 0;
	}
	pr_err("\tTasks blocked on level-%d rcu_node (CPUs %d-%d):",
	       rnp->level, rnp->grplo, rnp->grphi);
	t = list_entry(rnp->gp_tasks->prev,
		       struct task_struct, rcu_node_entry);
	list_for_each_entry_continue(t, &rnp->blkd_tasks, rcu_node_entry) {
		get_task_struct(t);
		ts[i++] = t;
		if (i >= ARRAY_SIZE(ts))
			break;
	}
	raw_spin_unlock_irqrestore_rcu_node(rnp, flags);
	while (i) {
		t = ts[--i];
		if (task_call_func(t, check_slow_task, &rscr)) {
			pr_cont(" P%d", t->pid);
		} else {
			tc = rscr.rcu_rdr_cpu_task;
			pr_cont(" P%d/%d:%c%c%c%c%c%c CPU %d%s %s (%d)%s",
				t->pid, rscr.nesting,
				".b"[rscr.rs.b.blocked],
				".q"[rscr.rs.b.need_qs],
				".e"[rscr.rs.b.exp_hint],
				".l"[rscr.on_blkd_list],
				".R"[rscr.rcu_rdr_running],
				".B?"[rscr.rcu_rdr_boosted],
				rscr.rcu_rdr_cpu,
				rscr.rcu_rdr_cpu_offline ? "!" : "",
				tc ? tc->comm : "???",
				tc ? tc->pid : -1,
				tc ? (is_idle_task(tc) ? " (idle!)" : "") : " (no task)");
		}
		lockdep_assert_irqs_disabled();
		put_task_struct(t);
		ndetected++;
	}
	pr_cont("\n");
	return ndetected;
}

------------------------------------------------------------------------

The ts[] array keeps track of up to eight tasks, the number indicated
by local variable "i", and the list_for_each_entry_continue() has
done get_task_struct() on each of those tasks.  Then the "while" loop
iterates over those tasks, printing each one and then doing the matching
put_task_struct().

Does that work, or am I missing something subtle?

							Thanx, Paul

Re: Possible enhancement to RCU CPU stall warning messages

[Steven Rostedt]

On Fri, 16 Oct 2026 09:48:12 -0700
"Paul E. McKenney" <paulmck@kernel.org> wrote:

> ------------------------------------------------------------------------
> 
> static int rcu_print_task_stall(struct rcu_node *rnp, unsigned long flags)
> 	__releases(rnp->lock)
> {
> 	int i = 0;
> 	int ndetected = 0;
> 	struct rcu_stall_chk_rdr rscr;
> 	struct task_struct *t;
> 	struct task_struct *tc;
> 	struct task_struct *ts[8];
> 
> 	lockdep_assert_irqs_disabled();
> 	if (!rcu_preempt_blocked_readers_cgp(rnp)) {
> 		raw_spin_unlock_irqrestore_rcu_node(rnp, flags);
> 		return 0;
> 	}
> 	pr_err("\tTasks blocked on level-%d rcu_node (CPUs %d-%d):",
> 	       rnp->level, rnp->grplo, rnp->grphi);
> 	t = list_entry(rnp->gp_tasks->prev,
> 		       struct task_struct, rcu_node_entry);
> 	list_for_each_entry_continue(t, &rnp->blkd_tasks, rcu_node_entry) {
> 		get_task_struct(t);

This gets the ts[i]

> 		ts[i++] = t;
> 		if (i >= ARRAY_SIZE(ts))
> 			break;
> 	}
> 	raw_spin_unlock_irqrestore_rcu_node(rnp, flags);
> 	while (i) {
> 		t = ts[--i];
> 		if (task_call_func(t, check_slow_task, &rscr)) {
> 			pr_cont(" P%d", t->pid);
> 		} else {
> 			tc = rscr.rcu_rdr_cpu_task;

But what gets the tc?

> 			pr_cont(" P%d/%d:%c%c%c%c%c%c CPU %d%s %s (%d)%s",
> 				t->pid, rscr.nesting,
> 				".b"[rscr.rs.b.blocked],
> 				".q"[rscr.rs.b.need_qs],
> 				".e"[rscr.rs.b.exp_hint],
> 				".l"[rscr.on_blkd_list],
> 				".R"[rscr.rcu_rdr_running],
> 				".B?"[rscr.rcu_rdr_boosted],
> 				rscr.rcu_rdr_cpu,
> 				rscr.rcu_rdr_cpu_offline ? "!" : "",
> 				tc ? tc->comm : "???",
> 				tc ? tc->pid : -1,
> 				tc ? (is_idle_task(tc) ? " (idle!)" : "") : " (no task)");

Here tc is dereferenced.

> 		}
> 		lockdep_assert_irqs_disabled();
> 		put_task_struct(t);

There's a put_task_struct(t), but I don't see a put_task_struct(tc) :-/

-- Steve

> 		ndetected++;
> 	}
> 	pr_cont("\n");
> 	return ndetected;
> }
> 
> ------------------------------------------------------------------------
> 
> The ts[] array keeps track of up to eight tasks, the number indicated
> by local variable "i", and the list_for_each_entry_continue() has
> done get_task_struct() on each of those tasks.  Then the "while" loop
> iterates over those tasks, printing each one and then doing the matching
> put_task_struct().
> 
> Does that work, or am I missing something subtle?

Re: Possible enhancement to RCU CPU stall warning messages

[Paul E. McKenney]

On Fri, Oct 16, 2026 at 12:54:53PM -0400, Steven Rostedt wrote:
> On Fri, 16 Oct 2026 09:48:12 -0700
> "Paul E. McKenney" <paulmck@kernel.org> wrote:
> 
> > ------------------------------------------------------------------------
> > 
> > static int rcu_print_task_stall(struct rcu_node *rnp, unsigned long flags)
> > 	__releases(rnp->lock)
> > {
> > 	int i = 0;
> > 	int ndetected = 0;
> > 	struct rcu_stall_chk_rdr rscr;
> > 	struct task_struct *t;
> > 	struct task_struct *tc;
> > 	struct task_struct *ts[8];
> > 
> > 	lockdep_assert_irqs_disabled();
> > 	if (!rcu_preempt_blocked_readers_cgp(rnp)) {
> > 		raw_spin_unlock_irqrestore_rcu_node(rnp, flags);
> > 		return 0;
> > 	}
> > 	pr_err("\tTasks blocked on level-%d rcu_node (CPUs %d-%d):",
> > 	       rnp->level, rnp->grplo, rnp->grphi);
> > 	t = list_entry(rnp->gp_tasks->prev,
> > 		       struct task_struct, rcu_node_entry);
> > 	list_for_each_entry_continue(t, &rnp->blkd_tasks, rcu_node_entry) {
> > 		get_task_struct(t);
> 
> This gets the ts[i]
> 
> > 		ts[i++] = t;
> > 		if (i >= ARRAY_SIZE(ts))
> > 			break;
> > 	}
> > 	raw_spin_unlock_irqrestore_rcu_node(rnp, flags);
> > 	while (i) {
> > 		t = ts[--i];
> > 		if (task_call_func(t, check_slow_task, &rscr)) {
> > 			pr_cont(" P%d", t->pid);
> > 		} else {
> > 			tc = rscr.rcu_rdr_cpu_task;
> 
> But what gets the tc?
> 
> > 			pr_cont(" P%d/%d:%c%c%c%c%c%c CPU %d%s %s (%d)%s",
> > 				t->pid, rscr.nesting,
> > 				".b"[rscr.rs.b.blocked],
> > 				".q"[rscr.rs.b.need_qs],
> > 				".e"[rscr.rs.b.exp_hint],
> > 				".l"[rscr.on_blkd_list],
> > 				".R"[rscr.rcu_rdr_running],
> > 				".B?"[rscr.rcu_rdr_boosted],
> > 				rscr.rcu_rdr_cpu,
> > 				rscr.rcu_rdr_cpu_offline ? "!" : "",
> > 				tc ? tc->comm : "???",
> > 				tc ? tc->pid : -1,
> > 				tc ? (is_idle_task(tc) ? " (idle!)" : "") : " (no task)");
> 
> Here tc is dereferenced.
> 
> > 		}
> > 		lockdep_assert_irqs_disabled();
> > 		put_task_struct(t);
> 
> There's a put_task_struct(t), but I don't see a put_task_struct(tc) :-/

You are absolutely right, and thank you!  I need that get_task_struct()
in check_slow_task(), then an additional put_task_struct() here.

Which might mean holding the rq lock in check_slow_task().  Except that
task_call_func() already does that for me, right?

Like this, to be folded into the existing patch?  Or am I still missing
something?

							Thanx, Paul

------------------------------------------------------------------------

diff --git a/kernel/rcu/tree_stall.h b/kernel/rcu/tree_stall.h
index 33e953bb40b33..b314d79c07840 100644
--- a/kernel/rcu/tree_stall.h
+++ b/kernel/rcu/tree_stall.h
@@ -336,6 +336,8 @@ static int check_slow_task(struct task_struct *t, void *arg)
 	rscrp->rcu_rdr_cpu_offline = cpu_is_offline(cpu);
 	// Interrupt are disabled for RCU read-side critical section. 
 	rscrp->rcu_rdr_cpu_task = th = cpu_curr_snapshot_rcu(cpu);
+	if (th)
+		get_task_struct(th);
 	if (rscrp->on_blkd_list) {
 		rnp = READ_ONCE(t->rcu_blocked_node);
 		raw_spin_lock_rcu_node(rnp); /* irqs already disabled. */
@@ -402,6 +404,8 @@ static int rcu_print_task_stall(struct rcu_node *rnp, unsigned long flags)
 		}
 		lockdep_assert_irqs_disabled();
 		put_task_struct(t);
+		if (tc)
+			put_task_struct(tc);
 		ndetected++;
 	}
 	pr_cont("\n");

Re: Possible enhancement to RCU CPU stall warning messages

[Steven Rostedt]

On Fri, 16 Oct 2026 10:05:02 -0700
"Paul E. McKenney" <paulmck@kernel.org> wrote:

> Which might mean holding the rq lock in check_slow_task().  Except that
> task_call_func() already does that for me, right?

Does it? It looks to only take the run queue lock if the task passed in is
running. But isn't this task usually blocked?

Maybe we can add a special get_cpu_curr(cpu) function that could not only
return the task but also get a reference for it too? Or NULL if idle.

-- Steve

> 
> Like this, to be folded into the existing patch?  Or am I still missing
> something?
> 
> 							Thanx, Paul
> 
> ------------------------------------------------------------------------
> 
> diff --git a/kernel/rcu/tree_stall.h b/kernel/rcu/tree_stall.h
> index 33e953bb40b33..b314d79c07840 100644
> --- a/kernel/rcu/tree_stall.h
> +++ b/kernel/rcu/tree_stall.h
> @@ -336,6 +336,8 @@ static int check_slow_task(struct task_struct *t,
> void *arg) rscrp->rcu_rdr_cpu_offline = cpu_is_offline(cpu);
>  	// Interrupt are disabled for RCU read-side critical section. 
>  	rscrp->rcu_rdr_cpu_task = th = cpu_curr_snapshot_rcu(cpu);
> +	if (th)
> +		get_task_struct(th);
>  	if (rscrp->on_blkd_list) {
>  		rnp = READ_ONCE(t->rcu_blocked_node);
>  		raw_spin_lock_rcu_node(rnp); /* irqs already disabled. */
> @@ -402,6 +404,8 @@ static int rcu_print_task_stall(struct rcu_node *rnp,
> unsigned long flags) }
>  		lockdep_assert_irqs_disabled();
>  		put_task_struct(t);
> +		if (tc)
> +			put_task_struct(tc);
>  		ndetected++;
>  	}
>  	pr_cont("\n");

Re: Possible enhancement to RCU CPU stall warning messages

[Paul E. McKenney]

On Fri, Oct 16, 2026 at 01:40:30PM -0400, Steven Rostedt wrote:
> On Fri, 16 Oct 2026 10:05:02 -0700
> "Paul E. McKenney" <paulmck@kernel.org> wrote:
> 
> > Which might mean holding the rq lock in check_slow_task().  Except that
> > task_call_func() already does that for me, right?
> 
> Does it? It looks to only take the run queue lock if the task passed in is
> running. But isn't this task usually blocked?

You are again quite right, it does not always acquire the run-queue lock.
However...

> Maybe we can add a special get_cpu_curr(cpu) function that could not only
> return the task but also get a reference for it too? Or NULL if idle.

Or we can rely on interrupts being disabled across the full call to
rcu_print_task_stall().  This interrupts-disabled region acts as an RCU
read-side critical section, so that the call_rcu() in put_task_struct()
cannot invoke its callback (and thus cannot free the task_struct
structure) until some time after rcu_print_task_stall().

Does that work, or am I still missing something?

(And yes, even if it doesn't work, comments might be nice.)

							Thanx, Paul

> -- Steve
> 
> > 
> > Like this, to be folded into the existing patch?  Or am I still missing
> > something?
> > 
> > 							Thanx, Paul
> > 
> > ------------------------------------------------------------------------
> > 
> > diff --git a/kernel/rcu/tree_stall.h b/kernel/rcu/tree_stall.h
> > index 33e953bb40b33..b314d79c07840 100644
> > --- a/kernel/rcu/tree_stall.h
> > +++ b/kernel/rcu/tree_stall.h
> > @@ -336,6 +336,8 @@ static int check_slow_task(struct task_struct *t,
> > void *arg) rscrp->rcu_rdr_cpu_offline = cpu_is_offline(cpu);
> >  	// Interrupt are disabled for RCU read-side critical section. 
> >  	rscrp->rcu_rdr_cpu_task = th = cpu_curr_snapshot_rcu(cpu);
> > +	if (th)
> > +		get_task_struct(th);
> >  	if (rscrp->on_blkd_list) {
> >  		rnp = READ_ONCE(t->rcu_blocked_node);
> >  		raw_spin_lock_rcu_node(rnp); /* irqs already disabled. */
> > @@ -402,6 +404,8 @@ static int rcu_print_task_stall(struct rcu_node *rnp,
> > unsigned long flags) }
> >  		lockdep_assert_irqs_disabled();
> >  		put_task_struct(t);
> > +		if (tc)
> > +			put_task_struct(tc);
> >  		ndetected++;
> >  	}
> >  	pr_cont("\n");
> 

Re: Possible enhancement to RCU CPU stall warning messages

[Paul E. McKenney]

On Fri, Oct 16, 2026 at 04:45:06PM -0700, Paul E. McKenney wrote:
> On Fri, Oct 16, 2026 at 01:40:30PM -0400, Steven Rostedt wrote:
> > On Fri, 16 Oct 2026 10:05:02 -0700
> > "Paul E. McKenney" <paulmck@kernel.org> wrote:
> > 
> > > Which might mean holding the rq lock in check_slow_task().  Except that
> > > task_call_func() already does that for me, right?
> > 
> > Does it? It looks to only take the run queue lock if the task passed in is
> > running. But isn't this task usually blocked?
> 
> You are again quite right, it does not always acquire the run-queue lock.
> However...
> 
> > Maybe we can add a special get_cpu_curr(cpu) function that could not only
> > return the task but also get a reference for it too? Or NULL if idle.
> 
> Or we can rely on interrupts being disabled across the full call to
> rcu_print_task_stall().  This interrupts-disabled region acts as an RCU
> read-side critical section, so that the call_rcu() in put_task_struct()
> cannot invoke its callback (and thus cannot free the task_struct
> structure) until some time after rcu_print_task_stall().
> 
> Does that work, or am I still missing something?

Except that if this is the case, then I don't need get_task_struct()
in the first place for any of them, do I?  I can instead just rely on
the call_rcu() in put_task_struct() that had to have happened sometime
after picking up the pointer to each task structure.

I clearly need to go through this one step by step and upgrade it as
needed from a debugging hack to a proper patch.

> (And yes, even if it doesn't work, comments might be nice.)

But this point still holds.  ;-)

							Thanx, Paul

> > -- Steve
> > 
> > > 
> > > Like this, to be folded into the existing patch?  Or am I still missing
> > > something?
> > > 
> > > 							Thanx, Paul
> > > 
> > > ------------------------------------------------------------------------
> > > 
> > > diff --git a/kernel/rcu/tree_stall.h b/kernel/rcu/tree_stall.h
> > > index 33e953bb40b33..b314d79c07840 100644
> > > --- a/kernel/rcu/tree_stall.h
> > > +++ b/kernel/rcu/tree_stall.h
> > > @@ -336,6 +336,8 @@ static int check_slow_task(struct task_struct *t,
> > > void *arg) rscrp->rcu_rdr_cpu_offline = cpu_is_offline(cpu);
> > >  	// Interrupt are disabled for RCU read-side critical section. 
> > >  	rscrp->rcu_rdr_cpu_task = th = cpu_curr_snapshot_rcu(cpu);
> > > +	if (th)
> > > +		get_task_struct(th);
> > >  	if (rscrp->on_blkd_list) {
> > >  		rnp = READ_ONCE(t->rcu_blocked_node);
> > >  		raw_spin_lock_rcu_node(rnp); /* irqs already disabled. */
> > > @@ -402,6 +404,8 @@ static int rcu_print_task_stall(struct rcu_node *rnp,
> > > unsigned long flags) }
> > >  		lockdep_assert_irqs_disabled();
> > >  		put_task_struct(t);
> > > +		if (tc)
> > > +			put_task_struct(tc);
> > >  		ndetected++;
> > >  	}
> > >  	pr_cont("\n");
> > 

Re: Possible enhancement to RCU CPU stall warning messages

[Steven Rostedt]

On Sat, 17 Oct 2026 12:02:08 -0700
"Paul E. McKenney" <paulmck@kernel.org> wrote:

> > 
> > Or we can rely on interrupts being disabled across the full call to
> > rcu_print_task_stall().  This interrupts-disabled region acts as an RCU
> > read-side critical section, so that the call_rcu() in put_task_struct()
> > cannot invoke its callback (and thus cannot free the task_struct
> > structure) until some time after rcu_print_task_stall().
> > 
> > Does that work, or am I still missing something?  
> 
> Except that if this is the case, then I don't need get_task_struct()
> in the first place for any of them, do I?  I can instead just rely on
> the call_rcu() in put_task_struct() that had to have happened sometime
> after picking up the pointer to each task structure.

I was thinking that since this is called via a RCU stall, would RCU
still make progress? This is for task RCU right? But as you said, if
interrupts are disabled here, it becomes a RCU grace period itself. And
yeah, I don't think we need to call any get_task_struct() at all.

> 
> I clearly need to go through this one step by step and upgrade it as
> needed from a debugging hack to a proper patch.
> 
> > (And yes, even if it doesn't work, comments might be nice.)  
> 
> But this point still holds.  ;-)

Exactly. A nice big comment saying "we are creating a RCU grace period
here that would prevent put_task_struct() from freeing any tasks, so
any tasks we reference now will still exist when printing (as long as
we never re-enable interrupts)." Or something like that.

Cheers,

-- Steve