Re: BUG: Stall on adding/removing wokers into workqueue pool

[Tejun Heo <tj@kernel.org>]

Hello, Tim.

On Tue, Oct 29, 2024 at 03:03:33PM -0700, Tim Chen wrote:
> Hi Tejun,
> 
> Forwarding this task hung seen by my colleague Doug Nelson. He tested
> the 6.12-rc4 kernel with an OLTP workload running on a 2 socket with
> Granite Rapids CPU that has 86 cores per socket. The traces 
> seem to indicate that the acquisition 
> of wq_pool_attach_mutex stalled in idle_cull_fn() when removing worker from
> the pool. Doug hit this problem occasionally in his tests.
> 
> Searching through the bug reports, there's a similar report by szybot on the
> 6.12-rc2 kernel. Szybot reported similar task hung when attaching workers to
> the pool: https://lore.kernel.org/all/6706c4ba.050a0220.1139e6.0008.GAE@google.com/T/
> So we suspect that the problem is not GNR CPU specific.
> 
> Wonder if this problem is a known one?

First time I see it. The trace doesn't show who's holding the mutex. There
doesn't seem to be any place where that mutex should leak at least on a
glance, so hopefully it shouldn't be too difficult to find who's holding it.
Can you trigger sysrq-d and sysrq-t and post the output?

Thanks.

-- 
tejun