Re: [PATCH] blk-mq: dispatch kworker to housekeeping CPU when hctx cpumask is empty

[Ming Lei <tom.leiming@gmail.com>]

On Tue, Oct 13, 2026 at 09:33:42AM -0400, Aaron Tomlin wrote:
> On Tue, Oct 13, 2026 at 07:28:45AM -0500, Ming Lei wrote:
> > No.
> > 
> > For example, schedule_work() is queue_work_on(WORK_CPU_UNBOUND, system_percpu_wq),
> > a WQ_PERCPU workqueue, so every schedule_work() from an isolated CPU runs
> > on that CPU. blk-mq isn't special here; this is the general tradeoff of
> > submitting IO from an isolated CPU.
> 
> Fair point regarding schedule_work(). I concede that mechanically,
> kblockd_workqueue is not unique in accepting WORK_CPU_UNBOUND on a per-CPU
> workqueue. schedule_work() indeed directs WORK_CPU_UNBOUND to
> system_percpu_wq, which resolves to raw_smp_processor_id() within
> __queue_work().
> 
> However, the crucial distinction lies in the semantic contract and intent.
> Callers of schedule_work() seek local (or best-effort local) execution;
> there is neither an assumption nor a guarantee that schedule_work() avoids
> isolated CPUs.

Then you admit the same generic problem, nothing different with blk-mq.

> 
> By contrast, blk-mq has an explicit architectural contract to avoid running
> block kworkers on isolated CPUs. In commit a46c27026da1 ("blk-mq: don't
> schedule block kworker on isolated CPUs"), the intent was to deliberately
> excluded isolated CPUs from hctx->cpumask precisely because:
> 
>     "blk-mq kworker shouldn't be scheduled on isolated CPUs."

I am pretty sure that the pre-condition is that IO isn't submitted from isolated
CPU. You can send a patch to make the point explicitly.

> 
> Furthermore, blk_mq_run_hw_queue() explicitly honours this policy:
> 
>     if (async || !cpumask_test_cpu(raw_smp_processor_id(), hctx->cpumask)) {
>         blk_mq_delay_run_hw_queue(hctx, 0);
>         return;
>     }
> 
> When executing on an isolated core (!cpumask_test_cpu(...)), blk-mq
> deliberately refrains from inline dispatch specifically to offload the work
> asynchronously via blk_mq_delay_run_hw_queue().
> 
> When all CPUs mapped to that hardware queue are isolated
> (blk_mq_hctx_empty_cpumask(hctx)), commit a46c27026da1 added:
> 
>     /* Switch to unbound if no allowable CPUs in this hctx */
>     if (hctx->queue->nr_hw_queues == 1 || blk_mq_hctx_empty_cpumask(hctx))
>         return WORK_CPU_UNBOUND;
> 
> The express intention of that change was to switch away from the isolated
> CPUs. However, because kblockd_workqueue is WQ_PERCPU, returning
> WORK_CPU_UNBOUND inadvertently resolves to raw_smp_processor_id(), queuing
> the work right back onto the isolated CPU that blk-mq just went out of its
> way to avoid.
> 
> That is why blk-mq is in an unusual position; it intended to avoid isolated
> cores, but relied on WORK_CPU_UNBOUND to achieve that on a bound per-CPU
> workqueue where that mechanism cannot take effect.
> 
> > You already disturb isolated CPUs by submitting IO via syscall in this
> > case, so why does it matter here to dispatch on same isolated CPU one more
> > time?
> 
> There is a fundamental difference between synchronous syscall execution and
> asynchronous background kworker execution:
> 
>     1.  When an isolated user task issues a synchronous I/O request (e.g.
>         via write(2)), execution occurs synchronously within the task's own
>         context and CPU accounting. The application remains in control of
>         when this overhead is incurred.
> 
>     2.  When dispatch must be deferred (e.g. BLK_STS_RESOURCE), blk-mq
>         queues &hctx->run_work. A background kworker runs under
>         SCHED_NORMAL. On isolated cores, workloads typically execute under
>         real-time scheduling classes (SCHED_FIFO, SCHED_RR, SCHED_DEADLINE)
>         or tight polling loops. A SCHED_FIFO task executing on that core
>         will completely starve the local kworker. Consequently,
>         blk_mq_run_work_fn() never runs, hardware dispatch stalls, pending
>         completions cannot occur, and the I/O pipeline hangs indefinitely.
> 
>     3.  Even where starvation is avoided (e.g. via the EEVDF scheduler's
>         Fair server or cooperative yielding), queuing bound work on a
>         nohz_full isolated core forces the scheduler tick to re-activate,
>         injecting unwanted jitter and defeating deterministic isolation
>         guarantees.
> 
> Steering deferred queue dispatch to an online housekeeping CPU via
> housekeeping_any_cpu(HK_TYPE_TIMER) sidesteps these pitfalls entirely.
> The kworker executes promptly on a housekeeping core without priority
> inversion, hardware dispatch proceeds unhindered, and the isolated CPU's
> low-latency domain is preserved.

Any user visible effect? Both are a piece of code running on isolated CPU,
the choice is in application. If they want to run IO on isolated CPU, we
do it in same way. Otherwise, we won't schedule work on isolated CPU for
respecting the kernel interface.


Thanks,
Ming