Possible enhancement to RCU CPU stall warning messages

Possible enhancement to RCU CPU stall warning messages

[Paul E. McKenney]

Hello!

While chasing an "interesting" bug [1], I modified the RCU CPU
stall-warning code so that when a task (as opposed to a CPU) is
blocking the grace period due to being preempted, the stall-warning
message includes the task that is currently running on the CPU that the
preempted task wants to run on.  (In the case of the "interesting" bug,
I kid you not, that CPU was idle.)

Would this stall-warning addition be useful for PREEMPT_RT users?

If it is not useful, I am going to keep it out of tree.  Something about
it needing a way to dig into scheduler internals.  ;-)

But if you want it, please let me know.

							Thanx, Paul

[1] https://lore.kernel.org/all/8fa7f795-518c-4507-b684-60b0c1cac2c5@paulmck-laptop/

------------------------------------------------------------------------

commit 24853cb1cfb363ce608e2098ef22c272d475b84c
Author: Paul E. McKenney <paulmck@kernel.org>
Date:   Mon Sep 28 13:35:01 2026 -0700

    EXP rcu: Print CPU that preempted stalled reader wants to run on
    
    This commit uses the dirty trick of defining an rcu_cpu_curr() within
    kernel/sched/core.c in order to allow the RCU CPU stall warning call out
    the CPU that a task preempted within an RCU reader would like to run on.
    In addition, print out the task (if any) currently running on that CPU,
    and add an exclamation mark ("!") after the CPU number if that CPU is
    currently offline.
    
    This is a potential step towards allowing people (or LLMs, for that
    matter) to locate any tasks that might be preventing the preempted RCU
    reader task from running.
    
    Signed-off-by: Paul E. McKenney <paulmck@kernel.org>

diff --git a/include/linux/sched.h b/include/linux/sched.h
index 8b3d47a325cca..a8b3434edb2b4 100644
--- a/include/linux/sched.h
+++ b/include/linux/sched.h
@@ -2532,3 +2532,5 @@ extern void migrate_enable(void);
 DEFINE_LOCK_GUARD_0(migrate, migrate_disable(), migrate_enable())
 
 #endif
+
+struct task_struct *rcu_cpu_curr(int cpu);
diff --git a/kernel/rcu/tree_stall.h b/kernel/rcu/tree_stall.h
index 803a56af3259f..7826c4d905248 100644
--- a/kernel/rcu/tree_stall.h
+++ b/kernel/rcu/tree_stall.h
@@ -309,6 +309,9 @@ struct rcu_stall_chk_rdr {
 	bool on_blkd_list;
 	bool rcu_rdr_running;
 	int rcu_rdr_boosted;
+	int rcu_rdr_cpu;
+	int rcu_rdr_cpu_offline;
+	struct task_struct *rcu_rdr_cpu_task;
 };
 
 /*
@@ -317,8 +320,10 @@ struct rcu_stall_chk_rdr {
  */
 static int check_slow_task(struct task_struct *t, void *arg)
 {
+	int cpu;
 	struct rcu_node *rnp;
 	struct rcu_stall_chk_rdr *rscrp = arg;
+	struct task_struct *th;
 
 	if (task_curr(t))
 		return -EBUSY; // It is running, so decline to inspect it.
@@ -327,6 +332,9 @@ static int check_slow_task(struct task_struct *t, void *arg)
 	rscrp->on_blkd_list = !list_empty(&t->rcu_node_entry);
 	rscrp->rcu_rdr_running = task_curr(t);
 	rscrp->rcu_rdr_boosted = 0;
+	rscrp->rcu_rdr_cpu = cpu = task_cpu(t);
+	rscrp->rcu_rdr_cpu_offline = cpu_is_offline(cpu);
+	rscrp->rcu_rdr_cpu_task = th = rcu_cpu_curr(cpu);
 	if (rscrp->on_blkd_list) {
 		rnp = READ_ONCE(t->rcu_blocked_node);
 		raw_spin_lock_rcu_node(rnp); /* irqs already disabled. */
@@ -352,6 +360,7 @@ static int rcu_print_task_stall(struct rcu_node *rnp, unsigned long flags)
 	int ndetected = 0;
 	struct rcu_stall_chk_rdr rscr;
 	struct task_struct *t;
+	struct task_struct *tc;
 	struct task_struct *ts[8];
 
 	lockdep_assert_irqs_disabled();
@@ -372,17 +381,24 @@ static int rcu_print_task_stall(struct rcu_node *rnp, unsigned long flags)
 	raw_spin_unlock_irqrestore_rcu_node(rnp, flags);
 	while (i) {
 		t = ts[--i];
-		if (task_call_func(t, check_slow_task, &rscr))
+		if (task_call_func(t, check_slow_task, &rscr)) {
 			pr_cont(" P%d", t->pid);
-		else
-			pr_cont(" P%d/%d:%c%c%c%c%c%c",
+		} else {
+			tc = rscr.rcu_rdr_cpu_task;
+			pr_cont(" P%d/%d:%c%c%c%c%c%c CPU %d%s %s (%d)%s",
 				t->pid, rscr.nesting,
 				".b"[rscr.rs.b.blocked],
 				".q"[rscr.rs.b.need_qs],
 				".e"[rscr.rs.b.exp_hint],
 				".l"[rscr.on_blkd_list],
 				".R"[rscr.rcu_rdr_running],
-				".B?"[rscr.rcu_rdr_boosted]);
+				".B?"[rscr.rcu_rdr_boosted],
+				rscr.rcu_rdr_cpu,
+				rscr.rcu_rdr_cpu_offline ? "!" : "",
+				tc ? tc->comm : "???",
+				tc ? tc->pid : -1,
+				tc ? (is_idle_task(tc) ? " (idle!)" : "") : " (no task)");
+		}
 		lockdep_assert_irqs_disabled();
 		put_task_struct(t);
 		ndetected++;
diff --git a/kernel/sched/core.c b/kernel/sched/core.c
index b77152edafd92..e137cba24fc92 100644
--- a/kernel/sched/core.c
+++ b/kernel/sched/core.c
@@ -11321,3 +11321,10 @@ void sched_change_end(struct sched_change_ctx *ctx)
 		p->sched_class->prio_changed(rq, p, ctx->prio);
 	}
 }
+
+// Not for mainline, but hopefully someone has a trick that gets this
+// job done.
+struct task_struct *rcu_cpu_curr(int cpu)
+{
+	return cpu_curr(cpu);
+}

Re: Possible enhancement to RCU CPU stall warning messages

[Clark Williams]

On Tue, Oct 13, 2026 at 05:41:41PM -0700, Paul E. McKenney wrote:
> Hello!
> 
> While chasing an "interesting" bug [1], I modified the RCU CPU
> stall-warning code so that when a task (as opposed to a CPU) is
> blocking the grace period due to being preempted, the stall-warning
> message includes the task that is currently running on the CPU that the
> preempted task wants to run on.  (In the case of the "interesting" bug,
> I kid you not, that CPU was idle.)
> 
> Would this stall-warning addition be useful for PREEMPT_RT users?
> 
> If it is not useful, I am going to keep it out of tree.  Something about
> it needing a way to dig into scheduler internals.  ;-)
> 
> But if you want it, please let me know.
> 
> 							Thanx, Paul
> 
> [1] https://lore.kernel.org/all/8fa7f795-518c-4507-b684-60b0c1cac2c5@paulmck-laptop/
> 

Paul,

While RCU isn't one of the common reasons for a latency spike, if it
*is*, then it's not one that's easily tracked down.  For that reason I'm
in favor of using your patch. 

That said, if it's going to be a pain to maintain, I can live without it. 

Clark

Re: Possible enhancement to RCU CPU stall warning messages

[Paul E. McKenney]

On Thu, Oct 15, 2026 at 10:27:24AM -0500, Clark Williams wrote:
> On Tue, Oct 13, 2026 at 05:41:41PM -0700, Paul E. McKenney wrote:
> > Hello!
> > 
> > While chasing an "interesting" bug [1], I modified the RCU CPU
> > stall-warning code so that when a task (as opposed to a CPU) is
> > blocking the grace period due to being preempted, the stall-warning
> > message includes the task that is currently running on the CPU that the
> > preempted task wants to run on.  (In the case of the "interesting" bug,
> > I kid you not, that CPU was idle.)
> > 
> > Would this stall-warning addition be useful for PREEMPT_RT users?
> > 
> > If it is not useful, I am going to keep it out of tree.  Something about
> > it needing a way to dig into scheduler internals.  ;-)
> > 
> > But if you want it, please let me know.
> > 
> > 							Thanx, Paul
> > 
> > [1] https://lore.kernel.org/all/8fa7f795-518c-4507-b684-60b0c1cac2c5@paulmck-laptop/
> > 
> 
> Paul,
> 
> While RCU isn't one of the common reasons for a latency spike, if it
> *is*, then it's not one that's easily tracked down.  For that reason I'm
> in favor of using your patch. 
> 
> That said, if it's going to be a pain to maintain, I can live without it. 

No problem maintaining it, just a small up-front cost of convincing the
scheduler guys to give me access to cpu_curr() or some other API that
allows RCU to find out what is currently running on the CPU that a given
task is trying to run on.

Given that it would be helpful to you, that is enough to motivate me to
go hassle the scheduler guys.  ;-)

							Thanx, Paul