Re: [mainline][BUG] Observed Workqueue lockups on offline CPUs.

[Boqun Feng <boqun@kernel.org>]

On Mon, Apr 27, 2026 at 05:00:10PM +0530, Samir M wrote:
> 

Hi Samir,

> On 27/04/26 3:32 pm, Samir M wrote:
> > Hi Paul,
> > 
> > I've been testing the latest upstream kernel on a PowerPC system and
> > encountered workqueue lockup issues that I've bisected to commit
> > 61bbcfb50514 ("srcu: Push srcu_node allocation to GP when
> > non-preemptible").
> > After booting, I'm seeing workqueue lockup warnings for CPUs 81-96,
> > which are offline on my system. The workqueues remain stuck for over 237
> > seconds:
> > 
> > [  243.309302][    C0] BUG: workqueue lockup - pool cpus=81 node=0
> > flags=0x4 nice=0 stuck for 237s!
> > [  243.309311][    C0] BUG: workqueue lockup - pool cpus=82 node=0
> > flags=0x4 nice=0 stuck for 237s!
> > [  243.309318][    C0] BUG: workqueue lockup - pool cpus=83 node=0
> > flags=0x4 nice=0 stuck for 237s!
> > [  243.309326][    C0] BUG: workqueue lockup - pool cpus=84 node=0
> > flags=0x4 nice=0 stuck for 237s!
> > [  243.309333][    C0] BUG: workqueue lockup - pool cpus=85 node=0
> > flags=0x4 nice=0 stuck for 237s!
> > [  243.309341][    C0] BUG: workqueue lockup - pool cpus=86 node=0
> > flags=0x4 nice=0 stuck for 237s!
> > [  243.309348][    C0] BUG: workqueue lockup - pool cpus=87 node=0
> > flags=0x4 nice=0 stuck for 237s!
> > [  243.309355][    C0] BUG: workqueue lockup - pool cpus=88 node=0
> > flags=0x4 nice=0 stuck for 237s!
> > [  243.309363][    C0] BUG: workqueue lockup - pool cpus=89 node=0
> > flags=0x4 nice=0 stuck for 237s!
> > [  243.309370][    C0] BUG: workqueue lockup - pool cpus=90 node=0
> > flags=0x4 nice=0 stuck for 237s!
> > [  243.309377][    C0] BUG: workqueue lockup - pool cpus=91 node=0
> > flags=0x4 nice=0 stuck for 237s!
> > [  243.309384][    C0] BUG: workqueue lockup - pool cpus=92 node=0
> > flags=0x4 nice=0 stuck for 237s!
> > [  243.309392][    C0] BUG: workqueue lockup - pool cpus=93 node=0
> > flags=0x4 nice=0 stuck for 237s!
> > [  243.309399][    C0] BUG: workqueue lockup - pool cpus=94 node=0
> > flags=0x4 nice=0 stuck for 237s!
> > [  243.309406][    C0] BUG: workqueue lockup - pool cpus=95 node=0
> > flags=0x4 nice=0 stuck for 237s!
> > [  243.309413][    C0] BUG: workqueue lockup - pool cpus=96 node=0
> > flags=0x4 nice=0 stuck for 237s!
> > 
> > Git bisect identified this as the first bad commit:
> > 
> > commit 61bbcfb50514a8a94e035a7349697a3790ab4783
> > Author: Paul E. McKenney <paulmck@kernel.org>
> > Date:   Fri Mar 20 20:29:20 2026 -0700
> > 
> >     srcu: Push srcu_node allocation to GP when non-preemptible
> > 
> >     When the srcutree.convert_to_big and srcutree.big_cpu_lim kernel boot
> >     parameters specify initialization-time allocation of the srcu_node
> >     tree for statically allocated srcu_struct structures (for example, in
> >     DEFINE_SRCU() at build time instead of init_srcu_struct() at
> > runtime),
> >     init_srcu_struct_nodes() will attempt to dynamically allocate this
> > tree
> >     at the first run-time update-side use of this srcu_struct structure,
> >     but while holding a raw spinlock. Because the memory allocator can
> >     acquire non-raw spinlocks, this can result in lockdep splats.
> > 
> >     This commit therefore uses the same SRCU_SIZE_ALLOC trick that is
> > used
> >     when the first run-time update-side use of this srcu_struct structure
> >     happens before srcu_init() is called. The actual allocation then
> > takes
> >     place from workqueue context at the ends of upcoming SRCU grace
> > periods.
> > 
> >     [boqun: Adjust the sha1 of the Fixes tag]
> > 
> >     Fixes: 175b45ed343a ("srcu: Use raw spinlocks so call_srcu() can be
> > used under preempt_disable()")
> >     Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
> >     Signed-off-by: Boqun Feng <boqun@kernel.org>
> > 
> >  kernel/rcu/srcutree.c | 7 +++++--
> >  1 file changed, 5 insertions(+), 2 deletions(-)
> > 
> > Reverting this commit resolves the issue.
> > 
> > The problem appears to be that the workqueue is attempting to execute on
> > offline CPUs. The commit moves SRCU node allocation to workqueue context
> > to avoid lockdep issues with memory allocation under raw spinlocks,
> > which makes sense. However, it seems the workqueue scheduling doesn't
> > properly account for CPU online/offline state in this code path.
> > 
> > My test environment:
> > - Architecture: PowerPC
> > - Kernel version: Latest upstream (7.1-rc1)
> > - CPUs 81-96 are offline at boot time
> > 
> > I suspect the issue might be related to:
> > 1. Workqueue not checking CPU online status before scheduling SRCU
> > allocation work
> > 2. Missing CPU hotplug awareness in the new workqueue-based allocation
> > path
> > 3. Possible race condition with CPU hotplug events
> > 
> > Would it make sense to use queue_work_on() with explicit online CPU
> > selection, or add CPU hotplug handlers for this workqueue? I'm not
> > deeply familiar with the workqueue internals, so I might be missing
> > something.
> > Please let me know if you need any additional details or if you'd like
> > me to test any patches.
> > 
> > If you happen to fix the above issue, then please add below tag.
> > Reported-by: Samir M <samir@linux.ibm.com>
> > 
> > 
> > Thanks,
> > Samir
> 
> Hi Paul,
> 
> 
> I worked on fixing the issue and introduced the changes below. With these
> updates, I no longer observe any workqueue lockup messages for offline CPUs.
> Could you please review the changes and share your feedback?
> 
> The commit 61bbcfb50514 ("srcu: Push srcu_node allocation to GP when
> non-preemptible") introduced workqueue lockups on systems with offline
> CPUs. The issue occurs because srcu_queue_delayed_work_on() calls
> queue_work_on() with sdp->cpu, which may be offline, causing the
> workqueue to spin indefinitely on that CPU.
> 
> This patch fixes the issue by checking if the target CPU is online
> before queuing work on it. If the CPU is offline, we fall back to
> using queue_work() which will schedule the work on any available
> online CPU.
> 
> Fixes: 61bbcfb50514 ("srcu: Push srcu_node allocation to GP when
> non-preemptible")
> 
> Signed-off-by: Samir <samir@linux.ibm.com>

Thanks for the patch, but I wonder: have you checked this email thread:

	https://lore.kernel.org/rcu/ttd89ul@ub.hpns/	

Paul had a fix [1], and TJ had a "fix" [2] on workqueue side.

In general I think we discovered that as long as a CPU has been onlined
once, it's OK to queue the work on that CPU (which may be offlined) even
with our TJ's patch (whether we should do that is a different problem
;-)). Please do check whether Paul's fix works for your case, thanks!

[1]: https://lore.kernel.org/rcu/ed1fa6cd-7343-4ca3-8b9d-d699ca496f83@paulmck-laptop/
[2]: https://lore.kernel.org/rcu/adlHKowvhn8AGXCc@slm.duckdns.org/

Regards,
Boqun

> ---
>  kernel/rcu/srcutree.c | 7 ++++++-
>  1 file changed, 6 insertions(+), 1 deletion(-)
> 
> diff --git a/kernel/rcu/srcutree.c b/kernel/rcu/srcutree.c
> index 0d01cd8c4b4a..55a90dd4a030 100644
> --- a/kernel/rcu/srcutree.c
> +++ b/kernel/rcu/srcutree.c
> @@ -869,10 +869,15 @@ static void srcu_delay_timer(struct timer_list *t)
>  static void srcu_queue_delayed_work_on(struct srcu_data *sdp,
>  unsigned long delay)
>  {
> -       if (!delay) {
> +       if (!delay && cpu_online(sdp->cpu)) {
>                 queue_work_on(sdp->cpu, rcu_gp_wq, &sdp->work);
>                 return;
> +       } else if (!delay) {
> +               /* CPU is offline, queue on any available CPU */
> +               queue_work(rcu_gp_wq, &sdp->work);
> +               return;
> +       }
> 
>         timer_reduce(&sdp->delay_work, jiffies + delay);
>  }
> --
> 
> 
> Thanks,
> Samir