Re: [PATCH 0/4] workqueue: Detect stalled in-flight workers

[Breno Leitao <leitao@debian.org>]

Hello Tejun,

On Wed, Feb 11, 2026 at 08:56:11AM -1000, Tejun Heo wrote:
> On Wed, Feb 11, 2026 at 04:29:14AM -0800, Breno Leitao wrote:
> > The workqueue watchdog detects pools that haven't made forward progress
> > by checking whether pending work items on the worklist have been waiting
> > too long. However, this approach has a blind spot: if a pool has only
> > one work item and that item has already been dequeued and is executing on
> > a worker, the worklist is empty and the watchdog skips the pool entirely.
> > This means a single hogged worker with no other pending work is invisible
> > to the stall detector.
> > 
> > I was able to come up with the following example that shows this blind
> > spot:
> > 
> > 	static void stall_work_fn(struct work_struct *work)
> > 	{
> > 		for (;;) {
> > 			mdelay(1000);
> > 			cond_resched();
> > 		}
> > 	}
> 
> Workqueue doesn't require users to limit execution time. As long as there is
> enough supply of concurrency to avoid stalling of pending work items, work
> items can run as long as they want, including indefinitely. Workqueue stall
> is there to indicate that there is insufficient supply of concurrency.

Thank you for the clarification. Let me share more context about the
actual problem I am observing so we can think through it together.

On some production hosts, I am seeing a workqueue stall where no
backtraces are printed:

	BUG: workqueue lockup - pool cpus=4 node=0 flags=0x0 nice=0 stuck for 132s!
	Showing busy workqueues and worker pools:
	workqueue events: flags=0x100
		pwq 18: cpus=4 node=0 flags=0x0 nice=0 active=4 refcnt=5
		in-flight: 178:stall_work1_fn [wq_stall]
		pending: stall_work2_fn [wq_stall], free_obj_work, psi_avgs_work
	workqueue mm_percpu_wq: flags=0x108
		pwq 18: cpus=4 node=0 flags=0x0 nice=0 active=1 refcnt=2
		pending: vmstat_update
		pool 18: cpus=4 node=0 flags=0x0 nice=0 hung=132s workers=2 idle: 45
	Showing backtraces of running workers in stalled
	CPU-bound worker pools:
		<nothing here>

We initially suspected a TOCTOU issue, and Omar put together a patch to
address that, but it did not identify anything.

After digging deeper, I believe I have found the root cause along with
a reproducer[1]:

  1) kfence executes toggle_allocation_gate() as a delayed workqueue
     item (kfence_timer) on the system WQ.

  2) toggle_allocation_gate() enables a static key, which IPIs every
     CPU to patch code:
          static_branch_enable(&kfence_allocation_key);

  3) toggle_allocation_gate() then sleeps in TASK_IDLE waiting for a
     kfence allocation to occur:
          wait_event_idle(allocation_wait,
                  atomic_read(&kfence_allocation_gate) > 0 || ...);

     This can last indefinitely if no allocation goes through the
     kfence path. The worker remains in the pool's busy_hash
     (in-flight) but is no longer task_is_running().

  4) The workqueue watchdog detects the stall and calls
     show_cpu_pool_hog(), which only prints backtraces for workers
     that are actively running on CPU:

          static void show_cpu_pool_hog(struct worker_pool *pool) {
                  ...
                  if (task_is_running(worker->task))
                          sched_show_task(worker->task);
          }

  5) Nothing is printed because the offending worker is in TASK_IDLE
     state. The output shows "Showing backtraces of running workers in
     stalled CPU-bound worker pools:" followed by nothing, effectively
     hiding the actual culprit.

The fix I am considering is to remove the task_is_running() filter in
show_cpu_pool_hog() so that all in-flight workers in stalled pools have
their backtraces printed, regardless of whether they are running or
sleeping. This would make sleeping culprits like toggle_allocation_gate()
visible in the watchdog output.

When I test without the task_runinng, then I see the culprit.

Fix I am testing:

	diff --git a/kernel/workqueue.c b/kernel/workqueue.c
	index aeaec79bc09c4..3f5ee08f99313 100644
	--- a/kernel/workqueue.c
	+++ b/kernel/workqueue.c
	@@ -7593,19 +7593,17 @@ static void show_cpu_pool_hog(struct worker_pool *pool)
		raw_spin_lock_irqsave(&pool->lock, irq_flags);

		hash_for_each(pool->busy_hash, bkt, worker, hentry) {
	-               if (task_is_running(worker->task)) {
	-                       /*
	-                        * Defer printing to avoid deadlocks in console
	-                        * drivers that queue work while holding locks
	-                        * also taken in their write paths.
	-                        */
	-                       printk_deferred_enter();
	+               /*
	+                * Defer printing to avoid deadlocks in console
	+                * drivers that queue work while holding locks
	+                * also taken in their write paths.
	+                */
	+               printk_deferred_enter();

	-                       pr_info("pool %d:\n", pool->id);
	-                       sched_show_task(worker->task);
	+               pr_info("pool %d:\n", pool->id);
	+               sched_show_task(worker->task);

	-                       printk_deferred_exit();
	-               }
	+               printk_deferred_exit();
		}

		raw_spin_unlock_irqrestore(&pool->lock, irq_flags);
	@@ -7616,7 +7614,7 @@ static void show_cpu_pools_hogs(void)
		struct worker_pool *pool;
		int pi;

	-       pr_info("Showing backtraces of running workers in stalled CPU-bound worker pools:\n");
	+       pr_info("Showing backtraces of in-flight workers in stalled CPU-bound worker pools:\n");

		rcu_read_lock();

Then I see:

	BUG: workqueue lockup - pool cpus=6 node=0 flags=0x0 nice=0 stuck for 34s!
	 Showing busy workqueues and worker pools:
	 workqueue events: flags=0x100
	   pwq 26: cpus=6 node=0 flags=0x0 nice=0 active=3 refcnt=4
	     in-flight: 161:stall_work1_fn [wq_stall]
	     pending: stall_work2_fn [wq_stall], psi_avgs_work
	 workqueue mm_percpu_wq: flags=0x108
	   pwq 26: cpus=6 node=0 flags=0x0 nice=0 active=1 refcnt=2
	     pending: vmstat_update
	 pool 26: cpus=6 node=0 flags=0x0 nice=0 hung=34s workers=3 idle: 210 57
	 Showing backtraces of in-flight workers in stalled CPU-bound worker pools:
	 pool 26:
	 task:kworker/6:1     state:I stack:0     pid:161   tgid:161   ppid:2      task_flags:0x4208040 flags:0x00080000
	 Call Trace:
	  <TASK>
	  __schedule+0x1521/0x5360
	  ? console_trylock+0x40/0x40
	  ? preempt_count_add+0x92/0x1a0
	  ? do_raw_spin_lock+0x12c/0x2f0
	  ? is_mmconf_reserved+0x390/0x390
	  ? schedule+0x91/0x350
	  ? schedule+0x91/0x350
	  schedule+0x165/0x350
	  stall_work1_fn+0x17f/0x250 [wq_stall]


Link: https://github.com/leitao/debug/blob/main/workqueue_stall/wq_stall.c [1]