The Linux Kernel Mailing List
 help / color / mirror / Atom feed
* sched_ext/lavd hard lockup in old call_rcu_tasks_generic needadjust path
@ 2026-06-09 10:47 Matt Fleming
  2026-06-09 11:23 ` Paul E. McKenney
  0 siblings, 1 reply; 10+ messages in thread
From: Matt Fleming @ 2026-06-09 10:47 UTC (permalink / raw)
  To: Tejun Heo, Andrea Righi
  Cc: Paul E . McKenney, sched-ext, linux-kernel, kernel-team

Hi there,

We're investigating a hard lockup on a 6.18.33-based kernel with scx_lavd
running.

The vmcore shows CPU#67 stuck in:

  native_queued_spin_lock_slowpath
  _raw_spin_lock
  task_rq_lock
  sched_ext_free
  __put_task_struct
  rcu_core
  handle_softirqs
  irq_exit_rcu
  sysvec_apic_timer_interrupt

The rq lock being waited on is for CPU#66.

Another CPU in the same dump is in:

  sched_ext_free
  scx_exit_task
  bpf_task_storage_delete
  call_rcu_tasks_generic+547
  _printk
  console_unlock
  wake_up_q
  try_to_wake_up
  __task_rq_lock

The `call_rcu_tasks_generic+547` site is the old needadjust path that
prints:

  Switching RCU Tasks Trace to per-CPU callback queuing.

So the current theory is that a task teardown under rq lock enters
bpf_task_storage_delete(), hits the callback-queue adjustment printk,
then console wakeup tries to acquire the same rq lock again.

We found related upstream changes:

  3063b33a347c ("rcu-tasks: Avoid raw-spinlocked wakeups from call_rcu_tasks_generic()")
  d245698d727a ("cgroup: Defer task cgroup unlink until after the task is done switching out")
  7900aa699c34 ("sched_ext: Fix cgroup exit ordering by moving sched_ext_free() to finish_task_switch()")
  7c405fb3279b ("rcu: Use an intermediate irq_work to start process_srcu()")

But none appears to directly fix the old 6.18 needadjust printk path.

Would backporting d245698d727a and 7900aa699c34 be useful, or should
the needadjust printk path itself be deferred away from rq-locked
callers?

Thanks,
Matt

^ permalink raw reply	[flat|nested] 10+ messages in thread

* Re: sched_ext/lavd hard lockup in old call_rcu_tasks_generic needadjust path
  2026-06-09 10:47 sched_ext/lavd hard lockup in old call_rcu_tasks_generic needadjust path Matt Fleming
@ 2026-06-09 11:23 ` Paul E. McKenney
  2026-06-11 13:02   ` Matt Fleming
  0 siblings, 1 reply; 10+ messages in thread
From: Paul E. McKenney @ 2026-06-09 11:23 UTC (permalink / raw)
  To: Matt Fleming
  Cc: Tejun Heo, Andrea Righi, sched-ext, linux-kernel, kernel-team

On Tue, Jun 09, 2026 at 11:47:33AM +0100, Matt Fleming wrote:
> Hi there,
> 
> We're investigating a hard lockup on a 6.18.33-based kernel with scx_lavd
> running.
> 
> The vmcore shows CPU#67 stuck in:
> 
>   native_queued_spin_lock_slowpath
>   _raw_spin_lock
>   task_rq_lock
>   sched_ext_free
>   __put_task_struct
>   rcu_core
>   handle_softirqs
>   irq_exit_rcu
>   sysvec_apic_timer_interrupt
> 
> The rq lock being waited on is for CPU#66.
> 
> Another CPU in the same dump is in:
> 
>   sched_ext_free
>   scx_exit_task
>   bpf_task_storage_delete
>   call_rcu_tasks_generic+547
>   _printk
>   console_unlock
>   wake_up_q
>   try_to_wake_up
>   __task_rq_lock
> 
> The `call_rcu_tasks_generic+547` site is the old needadjust path that
> prints:
> 
>   Switching RCU Tasks Trace to per-CPU callback queuing.
> 
> So the current theory is that a task teardown under rq lock enters
> bpf_task_storage_delete(), hits the callback-queue adjustment printk,
> then console wakeup tries to acquire the same rq lock again.
> 
> We found related upstream changes:
> 
>   3063b33a347c ("rcu-tasks: Avoid raw-spinlocked wakeups from call_rcu_tasks_generic()")
>   d245698d727a ("cgroup: Defer task cgroup unlink until after the task is done switching out")
>   7900aa699c34 ("sched_ext: Fix cgroup exit ordering by moving sched_ext_free() to finish_task_switch()")
>   7c405fb3279b ("rcu: Use an intermediate irq_work to start process_srcu()")
> 
> But none appears to directly fix the old 6.18 needadjust printk path.
> 
> Would backporting d245698d727a and 7900aa699c34 be useful, or should
> the needadjust printk path itself be deferred away from rq-locked
> callers?

Does commenting out the 'call_rcu_tasks_generic+547' printk() avoid the
issue?  If so, that printk() might be deferred or some such.

"But if you cannot trust printk(), what *can* you trust?"  ;-)

							Thanx, Paul

^ permalink raw reply	[flat|nested] 10+ messages in thread

* Re: sched_ext/lavd hard lockup in old call_rcu_tasks_generic needadjust path
  2026-06-09 11:23 ` Paul E. McKenney
@ 2026-06-11 13:02   ` Matt Fleming
  2026-06-11 13:45     ` Paul E. McKenney
  0 siblings, 1 reply; 10+ messages in thread
From: Matt Fleming @ 2026-06-11 13:02 UTC (permalink / raw)
  To: Paul E. McKenney
  Cc: Tejun Heo, Andrea Righi, sched-ext, linux-kernel, kernel-team

On Tue, Jun 09, 2026 at 04:23:23AM -0700, Paul E. McKenney wrote:
> 
> Does commenting out the 'call_rcu_tasks_generic+547' printk() avoid the
> issue?  If so, that printk() might be deferred or some such.
> 
> "But if you cannot trust printk(), what *can* you trust?"  ;-)

I tried this and it still crashes so apparently not! I'll keep digging
to find the real cause (it's somewhat cumbersome to reproduce this hard
lockup).

^ permalink raw reply	[flat|nested] 10+ messages in thread

* Re: sched_ext/lavd hard lockup in old call_rcu_tasks_generic needadjust path
  2026-06-11 13:02   ` Matt Fleming
@ 2026-06-11 13:45     ` Paul E. McKenney
  2026-06-12 10:55       ` Matt Fleming
  0 siblings, 1 reply; 10+ messages in thread
From: Paul E. McKenney @ 2026-06-11 13:45 UTC (permalink / raw)
  To: Matt Fleming
  Cc: Tejun Heo, Andrea Righi, sched-ext, linux-kernel, kernel-team

On Thu, Jun 11, 2026 at 02:02:58PM +0100, Matt Fleming wrote:
> On Tue, Jun 09, 2026 at 04:23:23AM -0700, Paul E. McKenney wrote:
> > 
> > Does commenting out the 'call_rcu_tasks_generic+547' printk() avoid the
> > issue?  If so, that printk() might be deferred or some such.
> > 
> > "But if you cannot trust printk(), what *can* you trust?"  ;-)
> 
> I tried this and it still crashes so apparently not! I'll keep digging
> to find the real cause (it's somewhat cumbersome to reproduce this hard
> lockup).

If this code path is nevertheless involved, one thing that might speed
things up would be to do bursts of call_rcu_tasks() from lots of CPUs,
then avoid doing any call_rcu_tasks() for some time, then do a single
isolated call_rcu_tasks().  Or maybe you are already doing this.

							Thanx, Paul

^ permalink raw reply	[flat|nested] 10+ messages in thread

* Re: sched_ext/lavd hard lockup in old call_rcu_tasks_generic needadjust path
  2026-06-11 13:45     ` Paul E. McKenney
@ 2026-06-12 10:55       ` Matt Fleming
  2026-06-12 14:00         ` Paul E. McKenney
  0 siblings, 1 reply; 10+ messages in thread
From: Matt Fleming @ 2026-06-12 10:55 UTC (permalink / raw)
  To: Paul E. McKenney
  Cc: Tejun Heo, Andrea Righi, sched-ext, linux-kernel, kernel-team

On Thu, Jun 11, 2026 at 06:45:14AM -0700, Paul E. McKenney wrote:
> On Thu, Jun 11, 2026 at 02:02:58PM +0100, Matt Fleming wrote:
> > On Tue, Jun 09, 2026 at 04:23:23AM -0700, Paul E. McKenney wrote:
> > > 
> > > Does commenting out the 'call_rcu_tasks_generic+547' printk() avoid the
> > > issue?  If so, that printk() might be deferred or some such.
> > > 
> > > "But if you cannot trust printk(), what *can* you trust?"  ;-)
> > 
> > I tried this and it still crashes so apparently not! I'll keep digging
> > to find the real cause (it's somewhat cumbersome to reproduce this hard
> > lockup).
> 
> If this code path is nevertheless involved, one thing that might speed
> things up would be to do bursts of call_rcu_tasks() from lots of CPUs,
> then avoid doing any call_rcu_tasks() for some time, then do a single
> isolated call_rcu_tasks().  Or maybe you are already doing this.

Thanks, I managed to shrink the time to reproduce the lockup and it's
now clear that the bug is an ABBA deadlock on cbs_gbl_lock.

   CPU #1
   ==========
     sched_ext_free()
       task_rq_lock()                               // acquires rq->lock
       scx_exit_task()
         SCX_CALL_OP_TASK(.exit_task)
           bpf_task_storage_delete()
             bpf_selem_unlink()
               bpf_selem_unlink_storage()
                 bpf_selem_free()
                   call_rcu_tasks_trace()
                     call_rcu_tasks_generic()
                       raw_spin_lock(cbs_gbl_lock)  // BLOCKS: CPU #2 owns it
   CPU #2
   ==========
     rcu_tasks_kthread()
       rcu_tasks_one_gp()
         raw_spin_lock(cbs_gbl_lock)                // acquired
         pr_info("Starting switch ...")             // still under cbs_gbl_lock
           console_unlock()
             wake_up_q()
               try_to_wake_up(repro)
                 raw_spin_lock(rq->lock)            // BLOCKS: CPU #1 owns it

Given this, I can see why removing the single printk() didn't fix
anything, and we can expect any code path under cbs_gbl_lock that wakes
a task could trigger this hard lockup. Right now in 6.18 that's a few
printk()s and a WARN_ON_ONCE().

This issue doesn't exist in v7.0-rc1 because of commit c27cea4416a3
("rcu: Re-implement RCU Tasks Trace in terms of SRCU-fast").

Is there an easy way to defer call_rcu_tasks_generic() work so it gets
executed without rq->lock being held? I'm assuming backporting the SRCU
patches would be too invasive for LTS?

Thanks,
Matt

^ permalink raw reply	[flat|nested] 10+ messages in thread

* Re: sched_ext/lavd hard lockup in old call_rcu_tasks_generic needadjust path
  2026-06-12 10:55       ` Matt Fleming
@ 2026-06-12 14:00         ` Paul E. McKenney
  2026-06-16 10:56           ` Matt Fleming
  0 siblings, 1 reply; 10+ messages in thread
From: Paul E. McKenney @ 2026-06-12 14:00 UTC (permalink / raw)
  To: Matt Fleming
  Cc: Tejun Heo, Andrea Righi, sched-ext, linux-kernel, kernel-team

On Fri, Jun 12, 2026 at 11:55:57AM +0100, Matt Fleming wrote:
> On Thu, Jun 11, 2026 at 06:45:14AM -0700, Paul E. McKenney wrote:
> > On Thu, Jun 11, 2026 at 02:02:58PM +0100, Matt Fleming wrote:
> > > On Tue, Jun 09, 2026 at 04:23:23AM -0700, Paul E. McKenney wrote:
> > > > 
> > > > Does commenting out the 'call_rcu_tasks_generic+547' printk() avoid the
> > > > issue?  If so, that printk() might be deferred or some such.
> > > > 
> > > > "But if you cannot trust printk(), what *can* you trust?"  ;-)
> > > 
> > > I tried this and it still crashes so apparently not! I'll keep digging
> > > to find the real cause (it's somewhat cumbersome to reproduce this hard
> > > lockup).
> > 
> > If this code path is nevertheless involved, one thing that might speed
> > things up would be to do bursts of call_rcu_tasks() from lots of CPUs,
> > then avoid doing any call_rcu_tasks() for some time, then do a single
> > isolated call_rcu_tasks().  Or maybe you are already doing this.
> 
> Thanks, I managed to shrink the time to reproduce the lockup and it's
> now clear that the bug is an ABBA deadlock on cbs_gbl_lock.
> 
>    CPU #1
>    ==========
>      sched_ext_free()
>        task_rq_lock()                               // acquires rq->lock
>        scx_exit_task()
>          SCX_CALL_OP_TASK(.exit_task)
>            bpf_task_storage_delete()
>              bpf_selem_unlink()
>                bpf_selem_unlink_storage()
>                  bpf_selem_free()
>                    call_rcu_tasks_trace()
>                      call_rcu_tasks_generic()
>                        raw_spin_lock(cbs_gbl_lock)  // BLOCKS: CPU #2 owns it
>    CPU #2
>    ==========
>      rcu_tasks_kthread()
>        rcu_tasks_one_gp()
>          raw_spin_lock(cbs_gbl_lock)                // acquired
>          pr_info("Starting switch ...")             // still under cbs_gbl_lock
>            console_unlock()
>              wake_up_q()
>                try_to_wake_up(repro)
>                  raw_spin_lock(rq->lock)            // BLOCKS: CPU #1 owns it
> 
> Given this, I can see why removing the single printk() didn't fix
> anything, and we can expect any code path under cbs_gbl_lock that wakes
> a task could trigger this hard lockup. Right now in 6.18 that's a few
> printk()s and a WARN_ON_ONCE().
> 
> This issue doesn't exist in v7.0-rc1 because of commit c27cea4416a3
> ("rcu: Re-implement RCU Tasks Trace in terms of SRCU-fast").
> 
> Is there an easy way to defer call_rcu_tasks_generic() work so it gets
> executed without rq->lock being held? I'm assuming backporting the SRCU
> patches would be too invasive for LTS?

Huh.  Looks like I did not implement RCU Tasks Trace in terms of SRCU
any time too soon.  But there is also RCU Tasks and RCU Tasks Rude.

If we don't backport the SRCU patches, then the obvious alternative is
for call_rcu_tasks*() to defer to IRQ work when invoked with interrupts
disabled.  Or is there a better way?

							Thanx, Paul

^ permalink raw reply	[flat|nested] 10+ messages in thread

* Re: sched_ext/lavd hard lockup in old call_rcu_tasks_generic needadjust path
  2026-06-12 14:00         ` Paul E. McKenney
@ 2026-06-16 10:56           ` Matt Fleming
  2026-06-17  3:24             ` Paul E. McKenney
  0 siblings, 1 reply; 10+ messages in thread
From: Matt Fleming @ 2026-06-16 10:56 UTC (permalink / raw)
  To: Paul E. McKenney
  Cc: Tejun Heo, Andrea Righi, sched-ext, linux-kernel, kernel-team

On Fri, Jun 12, 2026 at 07:00:31AM -0700, Paul E. McKenney wrote:
> 
> Huh.  Looks like I did not implement RCU Tasks Trace in terms of SRCU
> any time too soon.  But there is also RCU Tasks and RCU Tasks Rude.
> 
> If we don't backport the SRCU patches, then the obvious alternative is
> for call_rcu_tasks*() to defer to IRQ work when invoked with interrupts
> disabled.  Or is there a better way?

What about something like this?

----8<----

From 6b2dc5002f3413f4f89eb1735259d065b7003a52 Mon Sep 17 00:00:00 2001
From: Matt Fleming <mfleming@cloudflare.com>
Date: Mon, 15 Jun 2026 11:19:43 +0100
Subject: [PATCH] rcu-tasks: Defer callback queue adjustment to irq_work

call_rcu_tasks_generic() can run from BPF task-storage teardown while
sched_ext still holds rq->lock. The RCU Tasks kthread can concurrently
hold cbs_gbl_lock while printing under it, then wake a task on the same
rq while the caller waits for cbs_gbl_lock.

Queue the adjustment through irq_work instead. This keeps callback
enqueueing synchronous while moving cbs_gbl_lock acquisition out of the
caller context.

Signed-off-by: Matt Fleming <mfleming@cloudflare.com>
---
 kernel/rcu/tasks.h | 33 +++++++++++++++++++++++----------
 1 file changed, 23 insertions(+), 10 deletions(-)

diff --git a/kernel/rcu/tasks.h b/kernel/rcu/tasks.h
index 2dc044fd126e..92aead9fc200 100644
--- a/kernel/rcu/tasks.h
+++ b/kernel/rcu/tasks.h
@@ -104,6 +104,7 @@ struct rcu_tasks {
 	unsigned long n_ipis;
 	unsigned long n_ipis_fails;
 	struct task_struct *kthread_ptr;
+	struct irq_work cbs_adjust_irq_work;
 	unsigned long lazy_jiffies;
 	rcu_tasks_gp_func_t gp_func;
 	pregp_func_t pregp_func;
@@ -129,6 +130,7 @@ struct rcu_tasks {
 };
 
 static void call_rcu_tasks_iw_wakeup(struct irq_work *iwp);
+static void call_rcu_tasks_iw_adjust(struct irq_work *iwp);
 
 #define DEFINE_RCU_TASKS(rt_name, gp, call, n)						\
 static DEFINE_PER_CPU(struct rcu_tasks_percpu, rt_name ## __percpu) = {			\
@@ -144,6 +146,7 @@ static struct rcu_tasks rt_name =							\
 	.call_func = call,								\
 	.wait_state = TASK_UNINTERRUPTIBLE,						\
 	.rtpcpu = &rt_name ## __percpu,							\
+	.cbs_adjust_irq_work = IRQ_WORK_INIT_HARD(call_rcu_tasks_iw_adjust),	\
 	.lazy_jiffies = DIV_ROUND_UP(HZ, 4),						\
 	.name = n,									\
 	.percpu_enqueue_shift = order_base_2(CONFIG_NR_CPUS),				\
@@ -342,6 +345,24 @@ static void call_rcu_tasks_iw_wakeup(struct irq_work *iwp)
 	rcuwait_wake_up(&rtp->cbs_wait);
 }
 
+static void call_rcu_tasks_iw_adjust(struct irq_work *iwp)
+{
+	unsigned long flags;
+	bool expanded = false;
+	struct rcu_tasks *rtp = container_of(iwp, struct rcu_tasks, cbs_adjust_irq_work);
+
+	raw_spin_lock_irqsave(&rtp->cbs_gbl_lock, flags);
+	if (rtp->percpu_enqueue_lim != rcu_task_cpu_ids) {
+		WRITE_ONCE(rtp->percpu_enqueue_shift, 0);
+		WRITE_ONCE(rtp->percpu_dequeue_lim, rcu_task_cpu_ids);
+		smp_store_release(&rtp->percpu_enqueue_lim, rcu_task_cpu_ids);
+		expanded = true;
+	}
+	raw_spin_unlock_irqrestore(&rtp->cbs_gbl_lock, flags);
+	if (expanded)
+		pr_info("Switching %s to per-CPU callback queuing.\n", rtp->name);
+}
+
 // Enqueue a callback for the specified flavor of Tasks RCU.
 static void call_rcu_tasks_generic(struct rcu_head *rhp, rcu_callback_t func,
 				   struct rcu_tasks *rtp)
@@ -389,16 +410,8 @@ static void call_rcu_tasks_generic(struct rcu_head *rhp, rcu_callback_t func,
 		rtpcp->urgent_gp = 3;
 	rcu_segcblist_enqueue(&rtpcp->cblist, rhp);
 	raw_spin_unlock_irqrestore_rcu_node(rtpcp, flags);
-	if (unlikely(needadjust)) {
-		raw_spin_lock_irqsave(&rtp->cbs_gbl_lock, flags);
-		if (rtp->percpu_enqueue_lim != rcu_task_cpu_ids) {
-			WRITE_ONCE(rtp->percpu_enqueue_shift, 0);
-			WRITE_ONCE(rtp->percpu_dequeue_lim, rcu_task_cpu_ids);
-			smp_store_release(&rtp->percpu_enqueue_lim, rcu_task_cpu_ids);
-			pr_info("Switching %s to per-CPU callback queuing.\n", rtp->name);
-		}
-		raw_spin_unlock_irqrestore(&rtp->cbs_gbl_lock, flags);
-	}
+	if (unlikely(needadjust))
+		irq_work_queue(&rtp->cbs_adjust_irq_work);
 	rcu_read_unlock();
 	/* We can't create the thread unless interrupts are enabled. */
 	if (needwake && READ_ONCE(rtp->kthread_ptr))
-- 
2.43.0


^ permalink raw reply related	[flat|nested] 10+ messages in thread

* Re: sched_ext/lavd hard lockup in old call_rcu_tasks_generic needadjust path
  2026-06-16 10:56           ` Matt Fleming
@ 2026-06-17  3:24             ` Paul E. McKenney
  2026-06-18  6:56               ` Matt Fleming
  0 siblings, 1 reply; 10+ messages in thread
From: Paul E. McKenney @ 2026-06-17  3:24 UTC (permalink / raw)
  To: Matt Fleming
  Cc: Tejun Heo, Andrea Righi, sched-ext, linux-kernel, kernel-team

On Tue, Jun 16, 2026 at 11:56:19AM +0100, Matt Fleming wrote:
> On Fri, Jun 12, 2026 at 07:00:31AM -0700, Paul E. McKenney wrote:
> > 
> > Huh.  Looks like I did not implement RCU Tasks Trace in terms of SRCU
> > any time too soon.  But there is also RCU Tasks and RCU Tasks Rude.
> > 
> > If we don't backport the SRCU patches, then the obvious alternative is
> > for call_rcu_tasks*() to defer to IRQ work when invoked with interrupts
> > disabled.  Or is there a better way?
> 
> What about something like this?

Not bad, actually.

But we are going to have to similarly bypass call_rcu_tasks_generic()
soon enough, though only when interrupts are disabled at the time
of the call.  One complication is that it would need to interact
with rcu_barrier_tasks_generic(), which would need to remove any
pending callbacks from bypassed call_rcu_tasks_generic() invocations,
as in llist_del_all() and enqueue the callbacks.  A well-placed
raw_spin_trylock_rcu_node(rtpcp) should suffice for the serialization.
The trick is that rcu_barrier_tasks_generic() is not obligated to
wait on later calls to call_rcu_tasks_generic().

Is this something that you would be interested in digging into?

						Thanx, Paul

> ----8<----
> 
> >From 6b2dc5002f3413f4f89eb1735259d065b7003a52 Mon Sep 17 00:00:00 2001
> From: Matt Fleming <mfleming@cloudflare.com>
> Date: Mon, 15 Jun 2026 11:19:43 +0100
> Subject: [PATCH] rcu-tasks: Defer callback queue adjustment to irq_work
> 
> call_rcu_tasks_generic() can run from BPF task-storage teardown while
> sched_ext still holds rq->lock. The RCU Tasks kthread can concurrently
> hold cbs_gbl_lock while printing under it, then wake a task on the same
> rq while the caller waits for cbs_gbl_lock.
> 
> Queue the adjustment through irq_work instead. This keeps callback
> enqueueing synchronous while moving cbs_gbl_lock acquisition out of the
> caller context.
> 
> Signed-off-by: Matt Fleming <mfleming@cloudflare.com>
> ---
>  kernel/rcu/tasks.h | 33 +++++++++++++++++++++++----------
>  1 file changed, 23 insertions(+), 10 deletions(-)
> 
> diff --git a/kernel/rcu/tasks.h b/kernel/rcu/tasks.h
> index 2dc044fd126e..92aead9fc200 100644
> --- a/kernel/rcu/tasks.h
> +++ b/kernel/rcu/tasks.h
> @@ -104,6 +104,7 @@ struct rcu_tasks {
>  	unsigned long n_ipis;
>  	unsigned long n_ipis_fails;
>  	struct task_struct *kthread_ptr;
> +	struct irq_work cbs_adjust_irq_work;
>  	unsigned long lazy_jiffies;
>  	rcu_tasks_gp_func_t gp_func;
>  	pregp_func_t pregp_func;
> @@ -129,6 +130,7 @@ struct rcu_tasks {
>  };
>  
>  static void call_rcu_tasks_iw_wakeup(struct irq_work *iwp);
> +static void call_rcu_tasks_iw_adjust(struct irq_work *iwp);
>  
>  #define DEFINE_RCU_TASKS(rt_name, gp, call, n)						\
>  static DEFINE_PER_CPU(struct rcu_tasks_percpu, rt_name ## __percpu) = {			\
> @@ -144,6 +146,7 @@ static struct rcu_tasks rt_name =							\
>  	.call_func = call,								\
>  	.wait_state = TASK_UNINTERRUPTIBLE,						\
>  	.rtpcpu = &rt_name ## __percpu,							\
> +	.cbs_adjust_irq_work = IRQ_WORK_INIT_HARD(call_rcu_tasks_iw_adjust),	\
>  	.lazy_jiffies = DIV_ROUND_UP(HZ, 4),						\
>  	.name = n,									\
>  	.percpu_enqueue_shift = order_base_2(CONFIG_NR_CPUS),				\
> @@ -342,6 +345,24 @@ static void call_rcu_tasks_iw_wakeup(struct irq_work *iwp)
>  	rcuwait_wake_up(&rtp->cbs_wait);
>  }
>  
> +static void call_rcu_tasks_iw_adjust(struct irq_work *iwp)
> +{
> +	unsigned long flags;
> +	bool expanded = false;
> +	struct rcu_tasks *rtp = container_of(iwp, struct rcu_tasks, cbs_adjust_irq_work);
> +
> +	raw_spin_lock_irqsave(&rtp->cbs_gbl_lock, flags);
> +	if (rtp->percpu_enqueue_lim != rcu_task_cpu_ids) {
> +		WRITE_ONCE(rtp->percpu_enqueue_shift, 0);
> +		WRITE_ONCE(rtp->percpu_dequeue_lim, rcu_task_cpu_ids);
> +		smp_store_release(&rtp->percpu_enqueue_lim, rcu_task_cpu_ids);
> +		expanded = true;
> +	}
> +	raw_spin_unlock_irqrestore(&rtp->cbs_gbl_lock, flags);
> +	if (expanded)
> +		pr_info("Switching %s to per-CPU callback queuing.\n", rtp->name);
> +}
> +
>  // Enqueue a callback for the specified flavor of Tasks RCU.
>  static void call_rcu_tasks_generic(struct rcu_head *rhp, rcu_callback_t func,
>  				   struct rcu_tasks *rtp)
> @@ -389,16 +410,8 @@ static void call_rcu_tasks_generic(struct rcu_head *rhp, rcu_callback_t func,
>  		rtpcp->urgent_gp = 3;
>  	rcu_segcblist_enqueue(&rtpcp->cblist, rhp);
>  	raw_spin_unlock_irqrestore_rcu_node(rtpcp, flags);
> -	if (unlikely(needadjust)) {
> -		raw_spin_lock_irqsave(&rtp->cbs_gbl_lock, flags);
> -		if (rtp->percpu_enqueue_lim != rcu_task_cpu_ids) {
> -			WRITE_ONCE(rtp->percpu_enqueue_shift, 0);
> -			WRITE_ONCE(rtp->percpu_dequeue_lim, rcu_task_cpu_ids);
> -			smp_store_release(&rtp->percpu_enqueue_lim, rcu_task_cpu_ids);
> -			pr_info("Switching %s to per-CPU callback queuing.\n", rtp->name);
> -		}
> -		raw_spin_unlock_irqrestore(&rtp->cbs_gbl_lock, flags);
> -	}
> +	if (unlikely(needadjust))
> +		irq_work_queue(&rtp->cbs_adjust_irq_work);
>  	rcu_read_unlock();
>  	/* We can't create the thread unless interrupts are enabled. */
>  	if (needwake && READ_ONCE(rtp->kthread_ptr))
> -- 
> 2.43.0
> 

^ permalink raw reply	[flat|nested] 10+ messages in thread

* Re: sched_ext/lavd hard lockup in old call_rcu_tasks_generic needadjust path
  2026-06-17  3:24             ` Paul E. McKenney
@ 2026-06-18  6:56               ` Matt Fleming
  2026-06-18 15:55                 ` Paul E. McKenney
  0 siblings, 1 reply; 10+ messages in thread
From: Matt Fleming @ 2026-06-18  6:56 UTC (permalink / raw)
  To: Paul E. McKenney
  Cc: Tejun Heo, Andrea Righi, sched-ext, linux-kernel, kernel-team

On Tue, Jun 16, 2026 at 08:24:55PM -0700, Paul E. McKenney wrote:
> On Tue, Jun 16, 2026 at 11:56:19AM +0100, Matt Fleming wrote:
> > 
> > What about something like this?
> 
> Not bad, actually.
> 
> But we are going to have to similarly bypass call_rcu_tasks_generic()
> soon enough, though only when interrupts are disabled at the time
> of the call.  One complication is that it would need to interact
> with rcu_barrier_tasks_generic(), which would need to remove any
> pending callbacks from bypassed call_rcu_tasks_generic() invocations,
> as in llist_del_all() and enqueue the callbacks.  A well-placed
> raw_spin_trylock_rcu_node(rtpcp) should suffice for the serialization.
> The trick is that rcu_barrier_tasks_generic() is not obligated to
> wait on later calls to call_rcu_tasks_generic().
> 
> Is this something that you would be interested in digging into?

Sure, I'll take a look.

Thanks,
Matt

^ permalink raw reply	[flat|nested] 10+ messages in thread

* Re: sched_ext/lavd hard lockup in old call_rcu_tasks_generic needadjust path
  2026-06-18  6:56               ` Matt Fleming
@ 2026-06-18 15:55                 ` Paul E. McKenney
  0 siblings, 0 replies; 10+ messages in thread
From: Paul E. McKenney @ 2026-06-18 15:55 UTC (permalink / raw)
  To: Matt Fleming
  Cc: Tejun Heo, Andrea Righi, sched-ext, linux-kernel, kernel-team

On Thu, Jun 18, 2026 at 07:56:05AM +0100, Matt Fleming wrote:
> On Tue, Jun 16, 2026 at 08:24:55PM -0700, Paul E. McKenney wrote:
> > On Tue, Jun 16, 2026 at 11:56:19AM +0100, Matt Fleming wrote:
> > > 
> > > What about something like this?
> > 
> > Not bad, actually.
> > 
> > But we are going to have to similarly bypass call_rcu_tasks_generic()
> > soon enough, though only when interrupts are disabled at the time
> > of the call.  One complication is that it would need to interact
> > with rcu_barrier_tasks_generic(), which would need to remove any
> > pending callbacks from bypassed call_rcu_tasks_generic() invocations,
> > as in llist_del_all() and enqueue the callbacks.  A well-placed
> > raw_spin_trylock_rcu_node(rtpcp) should suffice for the serialization.
> > The trick is that rcu_barrier_tasks_generic() is not obligated to
> > wait on later calls to call_rcu_tasks_generic().
> > 
> > Is this something that you would be interested in digging into?
> 
> Sure, I'll take a look.

If you have questions or suggestions for the surrounding code, you know
where to find me!  ;-)

							Thanx, Paul

^ permalink raw reply	[flat|nested] 10+ messages in thread

end of thread, other threads:[~2026-06-18 15:55 UTC | newest]

Thread overview: 10+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2026-06-09 10:47 sched_ext/lavd hard lockup in old call_rcu_tasks_generic needadjust path Matt Fleming
2026-06-09 11:23 ` Paul E. McKenney
2026-06-11 13:02   ` Matt Fleming
2026-06-11 13:45     ` Paul E. McKenney
2026-06-12 10:55       ` Matt Fleming
2026-06-12 14:00         ` Paul E. McKenney
2026-06-16 10:56           ` Matt Fleming
2026-06-17  3:24             ` Paul E. McKenney
2026-06-18  6:56               ` Matt Fleming
2026-06-18 15:55                 ` Paul E. McKenney

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox