[PATCH] blk-mq: dispatch kworker to housekeeping CPU when hctx cpumask is empty

[PATCH] blk-mq: dispatch kworker to housekeeping CPU when hctx cpumask is empty

[Aaron Tomlin]

Commit a46c27026da1 ("blk-mq: don't schedule block kworker on isolated
CPUs") excluded isolated CPUs from hctx->cpumask to prevent worker
threads from executing block work on isolated CPUs. When all CPUs mapped
to a hardware queue are isolated (i.e. blk_mq_hctx_empty_cpumask(hctx)
evaluates to true), blk_mq_hctx_next_cpu() returns WORK_CPU_UNBOUND with
the intention of offloading execution.

However, kblockd_workqueue is allocated as a per-CPU workqueue
(WQ_PERCPU). When WORK_CPU_UNBOUND is queued on a per-CPU workqueue, the
workqueue core resolves the target CPU via raw_smp_processor_id().
Consequently, if a task executing on an isolated CPU submits I/O and
triggers an asynchronous queue dispatch with zero delay, &hctx->run_work
is queued directly onto the local isolated CPU's worker pool. This
causes the local per-CPU high-priority kworker
(e.g. "kworker/6:1H-kblockd") to be scheduled on the isolated CPU,
introducing unwanted latency jitter and execution delays, thereby
defeating the original objective of commit a46c27026da1.

Rectify this by selecting an online housekeeping CPU via
housekeeping_any_cpu(HK_TYPE_TIMER) when blk_mq_hctx_empty_cpumask(hctx)
is true. This ensures that:
    1.  The delayed work is cleanly offloaded to a housekeeping CPU.

    2.  The per-CPU high-priority kworker on the isolated CPU is not
        scheduled.

    3.  Hardware queues with valid mapped CPUs preserve their per-CPU
        locality and batch round-robin behaviour.

Fixes: a46c27026da1 ("blk-mq: don't schedule block kworker on isolated CPUs")
Cc: Ming Lei <ming.lei@redhat.com>
Cc: Jens Axboe <axboe@kernel.dk>
Cc: stable@vger.kernel.org
Signed-off-by: Aaron Tomlin <atomlin@atomlin.com>
---
 block/blk-mq.c | 7 +++++--
 1 file changed, 5 insertions(+), 2 deletions(-)

diff --git a/block/blk-mq.c b/block/blk-mq.c
index 025a799f35b9..9480c3aac898 100644
--- a/block/blk-mq.c
+++ b/block/blk-mq.c
@@ -2252,8 +2252,11 @@ static int blk_mq_hctx_next_cpu(struct blk_mq_hw_ctx *hctx)
 	bool tried = false;
 	int next_cpu = hctx->next_cpu;
 
-	/* Switch to unbound if no allowable CPUs in this hctx */
-	if (hctx->queue->nr_hw_queues == 1 || blk_mq_hctx_empty_cpumask(hctx))
+	/* Switch to housekeeping CPU if no allowable CPUs in this hctx */
+	if (blk_mq_hctx_empty_cpumask(hctx))
+		return housekeeping_any_cpu(HK_TYPE_TIMER);
+
+	if (hctx->queue->nr_hw_queues == 1)
 		return WORK_CPU_UNBOUND;
 
 	if (--hctx->next_cpu_batch <= 0) {
-- 
2.55.0

Re: [PATCH] blk-mq: dispatch kworker to housekeeping CPU when hctx cpumask is empty

[Ming Lei]

On Sun, Oct 11, 2026 at 8:29 PM Aaron Tomlin <atomlin@atomlin.com> wrote:
>
> Commit a46c27026da1 ("blk-mq: don't schedule block kworker on isolated
> CPUs") excluded isolated CPUs from hctx->cpumask to prevent worker
> threads from executing block work on isolated CPUs. When all CPUs mapped
> to a hardware queue are isolated (i.e. blk_mq_hctx_empty_cpumask(hctx)
> evaluates to true), blk_mq_hctx_next_cpu() returns WORK_CPU_UNBOUND with
> the intention of offloading execution.
>
> However, kblockd_workqueue is allocated as a per-CPU workqueue
> (WQ_PERCPU). When WORK_CPU_UNBOUND is queued on a per-CPU workqueue, the
> workqueue core resolves the target CPU via raw_smp_processor_id().

Then it looks like you need to fix WORK_CPU_UNBOUND, which has more wide uses,
if you think it is one problem.

> Consequently, if a task executing on an isolated CPU submits I/O and
> triggers an asynchronous queue dispatch with zero delay, &hctx->run_work
> is queued directly onto the local isolated CPU's worker pool. This

If your theory is correct, why do botter to change to
housekeeping_any_cpu(HK_TYPE_TIMER)
if the user is submitting IO from an isolated CPU?

> causes the local per-CPU high-priority kworker
> (e.g. "kworker/6:1H-kblockd") to be scheduled on the isolated CPU,
> introducing unwanted latency jitter and execution delays, thereby
> defeating the original objective of commit a46c27026da1.
>
> Rectify this by selecting an online housekeeping CPU via
> housekeeping_any_cpu(HK_TYPE_TIMER) when blk_mq_hctx_empty_cpumask(hctx)
> is true. This ensures that:
>     1.  The delayed work is cleanly offloaded to a housekeeping CPU.
>
>     2.  The per-CPU high-priority kworker on the isolated CPU is not
>         scheduled.
>
>     3.  Hardware queues with valid mapped CPUs preserve their per-CPU
>         locality and batch round-robin behaviour.
>
> Fixes: a46c27026da1 ("blk-mq: don't schedule block kworker on isolated CPUs")
> Cc: Ming Lei <ming.lei@redhat.com>
> Cc: Jens Axboe <axboe@kernel.dk>
> Cc: stable@vger.kernel.org
> Signed-off-by: Aaron Tomlin <atomlin@atomlin.com>
> ---
>  block/blk-mq.c | 7 +++++--
>  1 file changed, 5 insertions(+), 2 deletions(-)
>
> diff --git a/block/blk-mq.c b/block/blk-mq.c
> index 025a799f35b9..9480c3aac898 100644
> --- a/block/blk-mq.c
> +++ b/block/blk-mq.c
> @@ -2252,8 +2252,11 @@ static int blk_mq_hctx_next_cpu(struct blk_mq_hw_ctx *hctx)
>         bool tried = false;
>         int next_cpu = hctx->next_cpu;
>
> -       /* Switch to unbound if no allowable CPUs in this hctx */
> -       if (hctx->queue->nr_hw_queues == 1 || blk_mq_hctx_empty_cpumask(hctx))
> +       /* Switch to housekeeping CPU if no allowable CPUs in this hctx */
> +       if (blk_mq_hctx_empty_cpumask(hctx))
> +               return housekeeping_any_cpu(HK_TYPE_TIMER);
> +
> +       if (hctx->queue->nr_hw_queues == 1)
>                 return WORK_CPU_UNBOUND;

Why is SQ so special here to keep WORK_CPU_UNBOUND?


Thanks,
Ming Lei

Re: [PATCH] blk-mq: dispatch kworker to housekeeping CPU when hctx cpumask is empty

[Aaron Tomlin]

On Sun, Oct 11, 2026 at 08:56:07PM -0500, Ming Lei wrote:
> On Sun, Oct 11, 2026 at 8:29 PM Aaron Tomlin <atomlin@atomlin.com> wrote:
> >
> > Commit a46c27026da1 ("blk-mq: don't schedule block kworker on isolated
> > CPUs") excluded isolated CPUs from hctx->cpumask to prevent worker
> > threads from executing block work on isolated CPUs. When all CPUs mapped
> > to a hardware queue are isolated (i.e. blk_mq_hctx_empty_cpumask(hctx)
> > evaluates to true), blk_mq_hctx_next_cpu() returns WORK_CPU_UNBOUND with
> > the intention of offloading execution.
> >
> > However, kblockd_workqueue is allocated as a per-CPU workqueue
> > (WQ_PERCPU). When WORK_CPU_UNBOUND is queued on a per-CPU workqueue, the
> > workqueue core resolves the target CPU via raw_smp_processor_id().
> 
> Then it looks like you need to fix WORK_CPU_UNBOUND, which has more wide uses,
> if you think it is one problem.

Hi Ming,

Since kblockd_workqueue is WQ_PERCPU, the responsibility rests with blk-mq
to provide a non-isolated CPU when dispatching &hctx->run_work, no?

> > Consequently, if a task executing on an isolated CPU submits I/O and
> > triggers an asynchronous queue dispatch with zero delay, &hctx->run_work
> > is queued directly onto the local isolated CPU's worker pool. This
> 
> If your theory is correct, why do botter to change to
> housekeeping_any_cpu(HK_TYPE_TIMER)
> if the user is submitting IO from an isolated CPU?

I suspect, steering the asynchronous dispatch to a housekeeping CPU is both
architecturally necessary and fully aligned with the rest of the kernel,
no? When a queue is plugged, throttled, or returns BLK_STS_RESOURCE, blk-mq
defers execution to a background workqueue. That work runs asynchronously
at some indeterminate time in the future, long after the thread has
returned to its time-critical user loop. Having a kworker suddenly preempt
the thread later is likely not desired.


> > diff --git a/block/blk-mq.c b/block/blk-mq.c
> > index 025a799f35b9..9480c3aac898 100644
> > --- a/block/blk-mq.c
> > +++ b/block/blk-mq.c
> > @@ -2252,8 +2252,11 @@ static int blk_mq_hctx_next_cpu(struct blk_mq_hw_ctx *hctx)
> >         bool tried = false;
> >         int next_cpu = hctx->next_cpu;
> >
> > -       /* Switch to unbound if no allowable CPUs in this hctx */
> > -       if (hctx->queue->nr_hw_queues == 1 || blk_mq_hctx_empty_cpumask(hctx))
> > +       /* Switch to housekeeping CPU if no allowable CPUs in this hctx */
> > +       if (blk_mq_hctx_empty_cpumask(hctx))
> > +               return housekeeping_any_cpu(HK_TYPE_TIMER);
> > +
> > +       if (hctx->queue->nr_hw_queues == 1)
> >                 return WORK_CPU_UNBOUND;
> 
> Why is SQ so special here to keep WORK_CPU_UNBOUND?

Indeed, it is not special here, and neither is the offline CPU fallback
path (!cpu_online(next_cpu)). Both currently return WORK_CPU_UNBOUND and
suffer from the exact same issue if triggered from an isolated CPU.

To address this consistently without penalising non-isolated CPUs with
unnecessary cross-CPU bounces, we can define a small helper. Furthermore,
the following will address the valid issue raised by Sashiko [1]
comprehensively, using the new helper:
    1.  When executing on an isolated CPU
        cpu_is_isolated(raw_smp_processor_id()) evaluates to true.
        It redirects to an online housekeeping CPU
        (housekeeping_any_cpu(HK_TYPE_TIMER)), preventing kworkers from
        running on isolated cores across all three cases:
            - Single-queue devices (nr_hw_queues == 1)
            - Hardware queues with empty cpumasks
              (blk_mq_hctx_empty_cpumask(hctx))
            - Offline CPU fallback paths (!cpu_online(next_cpu))

    2.  When executing on a housekeeping CPU cpu_is_isolated() is false.
        It returns WORK_CPU_UNBOUND, which kblockd_workqueue executes on
        the local CPU (raw_smp_processor_id()), preserving optimal cache
        locality with zero CPU bouncing.

    3.  When CPU isolation is not configured cpu_is_isolated() uses a
        static branch (static_branch_unlikely(&housekeeping_overridden)),
        compiling to a NOP that returns WORK_CPU_UNBOUND with zero
        overhead.

[1]: https://sashiko.dev/#/patchset/20261012012905.267920-1-atomlin%40atomlin.com

diff --git a/block/blk-mq.c b/block/blk-mq.c
index e28c467a18bb..9fb1a93b2bb1 100644
--- a/block/blk-mq.c
+++ b/block/blk-mq.c
@@ -2244,6 +2244,13 @@ static bool blk_mq_hctx_empty_cpumask(struct blk_mq_hw_ctx *hctx)
         return hctx->next_cpu >= nr_cpu_ids;
 }

+static inline int blk_mq_hctx_unbound_cpu(void)
+{
+    if (cpu_is_isolated(raw_smp_processor_id()))
+        return housekeeping_any_cpu(HK_TYPE_TIMER);
+    return WORK_CPU_UNBOUND;
+}
+
 /*
  * It'd be great if the workqueue API had a way to pass
  * in a mask and had some smarts for more clever placement.
@@ -2256,7 +2263,7 @@ static int blk_mq_hctx_next_cpu(struct blk_mq_hw_ctx *hctx)

     /* Switch to unbound if no allowable CPUs in this hctx */
     if (hctx->queue->nr_hw_queues == 1 || blk_mq_hctx_empty_cpumask(hctx))
-        return WORK_CPU_UNBOUND;
+        return blk_mq_hctx_unbound_cpu();

     if (--hctx->next_cpu_batch <= 0) {
 select_cpu:
@@ -2281,7 +2288,7 @@ static int blk_mq_hctx_next_cpu(struct blk_mq_hw_ctx *hctx)
          */
         hctx->next_cpu = next_cpu;
         hctx->next_cpu_batch = 1;
-        return WORK_CPU_UNBOUND;
+        return blk_mq_hctx_unbound_cpu();
     }

         hctx->next_cpu = next_cpu;



Kind regards,
-- 
Aaron Tomlin

Re: [PATCH] blk-mq: dispatch kworker to housekeeping CPU when hctx cpumask is empty

[Ming Lei]

On Sun, Oct 11, 2026 at 10:21:01PM -0400, Aaron Tomlin wrote:
> On Sun, Oct 11, 2026 at 08:56:07PM -0500, Ming Lei wrote:
> > On Sun, Oct 11, 2026 at 8:29 PM Aaron Tomlin <atomlin@atomlin.com> wrote:
> > >
> > > Commit a46c27026da1 ("blk-mq: don't schedule block kworker on isolated
> > > CPUs") excluded isolated CPUs from hctx->cpumask to prevent worker
> > > threads from executing block work on isolated CPUs. When all CPUs mapped
> > > to a hardware queue are isolated (i.e. blk_mq_hctx_empty_cpumask(hctx)
> > > evaluates to true), blk_mq_hctx_next_cpu() returns WORK_CPU_UNBOUND with
> > > the intention of offloading execution.
> > >
> > > However, kblockd_workqueue is allocated as a per-CPU workqueue
> > > (WQ_PERCPU). When WORK_CPU_UNBOUND is queued on a per-CPU workqueue, the
> > > workqueue core resolves the target CPU via raw_smp_processor_id().
> > 
> > Then it looks like you need to fix WORK_CPU_UNBOUND, which has more wide uses,
> > if you think it is one problem.
> 
> Hi Ming,
> 
> Since kblockd_workqueue is WQ_PERCPU, the responsibility rests with blk-mq
> to provide a non-isolated CPU when dispatching &hctx->run_work, no?

Does every WORK_CPU_UNBOUND user have to handle it?

> 
> > > Consequently, if a task executing on an isolated CPU submits I/O and
> > > triggers an asynchronous queue dispatch with zero delay, &hctx->run_work
> > > is queued directly onto the local isolated CPU's worker pool. This
> > 
> > If your theory is correct, why do botter to change to
> > housekeeping_any_cpu(HK_TYPE_TIMER)
> > if the user is submitting IO from an isolated CPU?
> 
> I suspect, steering the asynchronous dispatch to a housekeeping CPU is both
> architecturally necessary and fully aligned with the rest of the kernel,
> no? When a queue is plugged, throttled, or returns BLK_STS_RESOURCE, blk-mq
> defers execution to a background workqueue. That work runs asynchronously
> at some indeterminate time in the future, long after the thread has
> returned to its time-critical user loop. Having a kworker suddenly preempt
> the thread later is likely not desired.

We are talking about `if a task executing on an isolated CPU submits I/O `.

If the task has been submitting IO on isolated CPU, why can't we keep
scheduling WQ on isolated CPU for running blk-mq queue? I am not sure your change
makes any difference from user viewpoint.


Thanks,
Ming

Re: [PATCH] blk-mq: dispatch kworker to housekeeping CPU when hctx cpumask is empty

[Aaron Tomlin]

On Sun, Oct 11, 2026 at 09:51:25PM -0500, Ming Lei wrote:
> > Since kblockd_workqueue is WQ_PERCPU, the responsibility rests with blk-mq
> > to provide a non-isolated CPU when dispatching &hctx->run_work, no?
> 
> Does every WORK_CPU_UNBOUND user have to handle it?

[Adding Tejun Heo]

No, because almost all other subsystems passing WORK_CPU_UNBOUND do so on
unbound workqueues (e.g. system_unbound_wq).

In kernel/workqueue.c, when WORK_CPU_UNBOUND is queued on a WQ_UNBOUND
workqueue, workqueue core already handles CPU isolation automatically via
wq_select_unbound_cpu(), which respects wq_unbound_cpumask and steers the
work item to a housekeeping CPU.

kblockd_workqueue is unique in that it is allocated as WQ_PERCPU to
preserve per-CPU locality for multi-queue devices. For WQ_PERCPU
workqueues, workqueue core cannot silently migrate WORK_CPU_UNBOUND to a
remote CPU because callers of bound per-CPU workqueues expect local per-CPU
execution. Thus, blk-mq is in the unusual position of passing
WORK_CPU_UNBOUND to a bound per-CPU workqueue with the expectation that it
avoids isolated cores.

> > I suspect, steering the asynchronous dispatch to a housekeeping CPU is both
> > architecturally necessary and fully aligned with the rest of the kernel,
> > no? When a queue is plugged, throttled, or returns BLK_STS_RESOURCE, blk-mq
> > defers execution to a background workqueue. That work runs asynchronously
> > at some indeterminate time in the future, long after the thread has
> > returned to its time-critical user loop. Having a kworker suddenly preempt
> > the thread later is likely not desired.
> 
> We are talking about `if a task executing on an isolated CPU submits I/O `.
> 
> If the task has been submitting IO on isolated CPU, why can't we keep
> scheduling WQ on isolated CPU for running blk-mq queue? I am not sure your change
> makes any difference from user viewpoint.

When a task submits I/O, synchronous submission occurs directly in syscall
context. Note, in the following example, IOCB_DIRECT is applied since the
file was opened with file->f_flags & O_DIRECT:

  __x64_sys_write
    ksys_write
      vfs_write
        else if (file->f_op->write_iter)
          new_sync_write(file, buf, count, pos)
            // filp->f_op->write_iter(&kiocb, &iter)
            ext4_file_write_iter
              if (iocb->ki_flags & IOCB_DIRECT)
               ext4_dio_write_iter(iocb, from) 
                 iomap_dio_rw(iocb, from, &ext4_iomap_ops,
                              &ext4_dio_write_ops, dio_flags, NULL, 0)
                   __iomap_dio_rw
                     ...
                     iomap_dio_submit_bio
                       blk_crypto_submit_bio(bio)
                         if (!bio_has_crypt_ctx(bio) ...)
                           submit_bio(bio)
                            ...
                             __submit_bio
                               blk_mq_submit_bio(bio)
                                 // blk_mq_run_dispatch_ops(q, blk_mq_try_issue_directly(hctx, rq))
                                 __blk_mq_run_dispatch_ops(q, true blk_mq_try_issue_directly(hctx, rq))
                                   blk_mq_try_issue_directly(hctx, rq)
                                     __blk_mq_issue_directly(hctx, rq, true)
                                       // q->mq_ops->queue_rq(hctx, &bd)


However, if dispatch must be deferred (e.g. BLK_STS_RESOURCE),
blk-mq queues &hctx->run_work. Now, if &hctx->run_work is queued on the
isolated CPU, it must be executed by the local kworker. Because that
kworker runs under SCHED_NORMAL, the user's SCHED_FIFO task completely
starves the kworker. As a result, blk_mq_run_work_fn() never executes,
hardware dispatch stalls, and the I/O pipeline hangs indefinitely.
Albeit, I believe an enabled Fair server, under the EEVDF scheduler, will
prevent this stall.

Anyhow, offloading deferred dispatch to an online housekeeping CPU allows
the kworker to run immediately and complete dispatch without priority
inversion. As noted in commit a46c27026da1:
    "it isn't expected to let block IO disturb these CPUs. blk-mq kworker
     shouldn't be scheduled on isolated CPUs."


Kind regards,
-- 
Aaron Tomlin

Re: [PATCH] blk-mq: dispatch kworker to housekeeping CPU when hctx cpumask is empty

[Ming Lei]

On Mon, Oct 12, 2026 at 12:06:25PM -0400, Aaron Tomlin wrote:
> On Sun, Oct 11, 2026 at 09:51:25PM -0500, Ming Lei wrote:
> > > Since kblockd_workqueue is WQ_PERCPU, the responsibility rests with blk-mq
> > > to provide a non-isolated CPU when dispatching &hctx->run_work, no?
> > 
> > Does every WORK_CPU_UNBOUND user have to handle it?
> 
> [Adding Tejun Heo]
> 
> No, because almost all other subsystems passing WORK_CPU_UNBOUND do so on
> unbound workqueues (e.g. system_unbound_wq).
> 
> In kernel/workqueue.c, when WORK_CPU_UNBOUND is queued on a WQ_UNBOUND
> workqueue, workqueue core already handles CPU isolation automatically via
> wq_select_unbound_cpu(), which respects wq_unbound_cpumask and steers the
> work item to a housekeeping CPU.
> 
> kblockd_workqueue is unique in that it is allocated as WQ_PERCPU to
> preserve per-CPU locality for multi-queue devices. For WQ_PERCPU
> workqueues, workqueue core cannot silently migrate WORK_CPU_UNBOUND to a
> remote CPU because callers of bound per-CPU workqueues expect local per-CPU
> execution. Thus, blk-mq is in the unusual position of passing
> WORK_CPU_UNBOUND to a bound per-CPU workqueue with the expectation that it
> avoids isolated cores.

No.

For example, schedule_work() is queue_work_on(WORK_CPU_UNBOUND, system_percpu_wq),
a WQ_PERCPU workqueue, so every schedule_work() from an isolated CPU runs
on that CPU. blk-mq isn't special here; this is the general tradeoff of
submitting IO from an isolated CPU.

> 
> > > I suspect, steering the asynchronous dispatch to a housekeeping CPU is both
> > > architecturally necessary and fully aligned with the rest of the kernel,
> > > no? When a queue is plugged, throttled, or returns BLK_STS_RESOURCE, blk-mq
> > > defers execution to a background workqueue. That work runs asynchronously
> > > at some indeterminate time in the future, long after the thread has
> > > returned to its time-critical user loop. Having a kworker suddenly preempt
> > > the thread later is likely not desired.
> > 
> > We are talking about `if a task executing on an isolated CPU submits I/O `.
> > 
> > If the task has been submitting IO on isolated CPU, why can't we keep
> > scheduling WQ on isolated CPU for running blk-mq queue? I am not sure your change
> > makes any difference from user viewpoint.
> 
> When a task submits I/O, synchronous submission occurs directly in syscall
> context. Note, in the following example, IOCB_DIRECT is applied since the
> file was opened with file->f_flags & O_DIRECT:
> 
>   __x64_sys_write
>     ksys_write
>       vfs_write
>         else if (file->f_op->write_iter)
>           new_sync_write(file, buf, count, pos)
>             // filp->f_op->write_iter(&kiocb, &iter)
>             ext4_file_write_iter
>               if (iocb->ki_flags & IOCB_DIRECT)
>                ext4_dio_write_iter(iocb, from) 
>                  iomap_dio_rw(iocb, from, &ext4_iomap_ops,
>                               &ext4_dio_write_ops, dio_flags, NULL, 0)
>                    __iomap_dio_rw
>                      ...
>                      iomap_dio_submit_bio
>                        blk_crypto_submit_bio(bio)
>                          if (!bio_has_crypt_ctx(bio) ...)
>                            submit_bio(bio)
>                             ...
>                              __submit_bio
>                                blk_mq_submit_bio(bio)
>                                  // blk_mq_run_dispatch_ops(q, blk_mq_try_issue_directly(hctx, rq))
>                                  __blk_mq_run_dispatch_ops(q, true blk_mq_try_issue_directly(hctx, rq))
>                                    blk_mq_try_issue_directly(hctx, rq)
>                                      __blk_mq_issue_directly(hctx, rq, true)
>                                        // q->mq_ops->queue_rq(hctx, &bd)
> 
> 
> However, if dispatch must be deferred (e.g. BLK_STS_RESOURCE),
> blk-mq queues &hctx->run_work. Now, if &hctx->run_work is queued on the
> isolated CPU, it must be executed by the local kworker. Because that
> kworker runs under SCHED_NORMAL, the user's SCHED_FIFO task completely
> starves the kworker. As a result, blk_mq_run_work_fn() never executes,
> hardware dispatch stalls, and the I/O pipeline hangs indefinitely.
> Albeit, I believe an enabled Fair server, under the EEVDF scheduler, will
> prevent this stall.
> 
> Anyhow, offloading deferred dispatch to an online housekeeping CPU allows
> the kworker to run immediately and complete dispatch without priority
> inversion. As noted in commit a46c27026da1:
>     "it isn't expected to let block IO disturb these CPUs. blk-mq kworker
>      shouldn't be scheduled on isolated CPUs."

You already disturb isolated CPUs by submitting IO via syscall in this
case, so why does it matter here to dispatch on same isolated CPU one more
time?


Thanks,
Ming

Re: [PATCH] blk-mq: dispatch kworker to housekeeping CPU when hctx cpumask is empty

[Aaron Tomlin]

On Tue, Oct 13, 2026 at 07:28:45AM -0500, Ming Lei wrote:
> No.
> 
> For example, schedule_work() is queue_work_on(WORK_CPU_UNBOUND, system_percpu_wq),
> a WQ_PERCPU workqueue, so every schedule_work() from an isolated CPU runs
> on that CPU. blk-mq isn't special here; this is the general tradeoff of
> submitting IO from an isolated CPU.

Fair point regarding schedule_work(). I concede that mechanically,
kblockd_workqueue is not unique in accepting WORK_CPU_UNBOUND on a per-CPU
workqueue. schedule_work() indeed directs WORK_CPU_UNBOUND to
system_percpu_wq, which resolves to raw_smp_processor_id() within
__queue_work().

However, the crucial distinction lies in the semantic contract and intent.
Callers of schedule_work() seek local (or best-effort local) execution;
there is neither an assumption nor a guarantee that schedule_work() avoids
isolated CPUs.

By contrast, blk-mq has an explicit architectural contract to avoid running
block kworkers on isolated CPUs. In commit a46c27026da1 ("blk-mq: don't
schedule block kworker on isolated CPUs"), the intent was to deliberately
excluded isolated CPUs from hctx->cpumask precisely because:

    "blk-mq kworker shouldn't be scheduled on isolated CPUs."

Furthermore, blk_mq_run_hw_queue() explicitly honours this policy:

    if (async || !cpumask_test_cpu(raw_smp_processor_id(), hctx->cpumask)) {
        blk_mq_delay_run_hw_queue(hctx, 0);
        return;
    }

When executing on an isolated core (!cpumask_test_cpu(...)), blk-mq
deliberately refrains from inline dispatch specifically to offload the work
asynchronously via blk_mq_delay_run_hw_queue().

When all CPUs mapped to that hardware queue are isolated
(blk_mq_hctx_empty_cpumask(hctx)), commit a46c27026da1 added:

    /* Switch to unbound if no allowable CPUs in this hctx */
    if (hctx->queue->nr_hw_queues == 1 || blk_mq_hctx_empty_cpumask(hctx))
        return WORK_CPU_UNBOUND;

The express intention of that change was to switch away from the isolated
CPUs. However, because kblockd_workqueue is WQ_PERCPU, returning
WORK_CPU_UNBOUND inadvertently resolves to raw_smp_processor_id(), queuing
the work right back onto the isolated CPU that blk-mq just went out of its
way to avoid.

That is why blk-mq is in an unusual position; it intended to avoid isolated
cores, but relied on WORK_CPU_UNBOUND to achieve that on a bound per-CPU
workqueue where that mechanism cannot take effect.

> You already disturb isolated CPUs by submitting IO via syscall in this
> case, so why does it matter here to dispatch on same isolated CPU one more
> time?

There is a fundamental difference between synchronous syscall execution and
asynchronous background kworker execution:

    1.  When an isolated user task issues a synchronous I/O request (e.g.
        via write(2)), execution occurs synchronously within the task's own
        context and CPU accounting. The application remains in control of
        when this overhead is incurred.

    2.  When dispatch must be deferred (e.g. BLK_STS_RESOURCE), blk-mq
        queues &hctx->run_work. A background kworker runs under
        SCHED_NORMAL. On isolated cores, workloads typically execute under
        real-time scheduling classes (SCHED_FIFO, SCHED_RR, SCHED_DEADLINE)
        or tight polling loops. A SCHED_FIFO task executing on that core
        will completely starve the local kworker. Consequently,
        blk_mq_run_work_fn() never runs, hardware dispatch stalls, pending
        completions cannot occur, and the I/O pipeline hangs indefinitely.

    3.  Even where starvation is avoided (e.g. via the EEVDF scheduler's
        Fair server or cooperative yielding), queuing bound work on a
        nohz_full isolated core forces the scheduler tick to re-activate,
        injecting unwanted jitter and defeating deterministic isolation
        guarantees.

Steering deferred queue dispatch to an online housekeeping CPU via
housekeeping_any_cpu(HK_TYPE_TIMER) sidesteps these pitfalls entirely.
The kworker executes promptly on a housekeeping core without priority
inversion, hardware dispatch proceeds unhindered, and the isolated CPU's
low-latency domain is preserved.


Kind regards,
-- 
Aaron Tomlin

Re: [PATCH] blk-mq: dispatch kworker to housekeeping CPU when hctx cpumask is empty

[Ming Lei]

On Tue, Oct 13, 2026 at 09:33:42AM -0400, Aaron Tomlin wrote:
> On Tue, Oct 13, 2026 at 07:28:45AM -0500, Ming Lei wrote:
> > No.
> > 
> > For example, schedule_work() is queue_work_on(WORK_CPU_UNBOUND, system_percpu_wq),
> > a WQ_PERCPU workqueue, so every schedule_work() from an isolated CPU runs
> > on that CPU. blk-mq isn't special here; this is the general tradeoff of
> > submitting IO from an isolated CPU.
> 
> Fair point regarding schedule_work(). I concede that mechanically,
> kblockd_workqueue is not unique in accepting WORK_CPU_UNBOUND on a per-CPU
> workqueue. schedule_work() indeed directs WORK_CPU_UNBOUND to
> system_percpu_wq, which resolves to raw_smp_processor_id() within
> __queue_work().
> 
> However, the crucial distinction lies in the semantic contract and intent.
> Callers of schedule_work() seek local (or best-effort local) execution;
> there is neither an assumption nor a guarantee that schedule_work() avoids
> isolated CPUs.

Then you admit the same generic problem, nothing different with blk-mq.

> 
> By contrast, blk-mq has an explicit architectural contract to avoid running
> block kworkers on isolated CPUs. In commit a46c27026da1 ("blk-mq: don't
> schedule block kworker on isolated CPUs"), the intent was to deliberately
> excluded isolated CPUs from hctx->cpumask precisely because:
> 
>     "blk-mq kworker shouldn't be scheduled on isolated CPUs."

I am pretty sure that the pre-condition is that IO isn't submitted from isolated
CPU. You can send a patch to make the point explicitly.

> 
> Furthermore, blk_mq_run_hw_queue() explicitly honours this policy:
> 
>     if (async || !cpumask_test_cpu(raw_smp_processor_id(), hctx->cpumask)) {
>         blk_mq_delay_run_hw_queue(hctx, 0);
>         return;
>     }
> 
> When executing on an isolated core (!cpumask_test_cpu(...)), blk-mq
> deliberately refrains from inline dispatch specifically to offload the work
> asynchronously via blk_mq_delay_run_hw_queue().
> 
> When all CPUs mapped to that hardware queue are isolated
> (blk_mq_hctx_empty_cpumask(hctx)), commit a46c27026da1 added:
> 
>     /* Switch to unbound if no allowable CPUs in this hctx */
>     if (hctx->queue->nr_hw_queues == 1 || blk_mq_hctx_empty_cpumask(hctx))
>         return WORK_CPU_UNBOUND;
> 
> The express intention of that change was to switch away from the isolated
> CPUs. However, because kblockd_workqueue is WQ_PERCPU, returning
> WORK_CPU_UNBOUND inadvertently resolves to raw_smp_processor_id(), queuing
> the work right back onto the isolated CPU that blk-mq just went out of its
> way to avoid.
> 
> That is why blk-mq is in an unusual position; it intended to avoid isolated
> cores, but relied on WORK_CPU_UNBOUND to achieve that on a bound per-CPU
> workqueue where that mechanism cannot take effect.
> 
> > You already disturb isolated CPUs by submitting IO via syscall in this
> > case, so why does it matter here to dispatch on same isolated CPU one more
> > time?
> 
> There is a fundamental difference between synchronous syscall execution and
> asynchronous background kworker execution:
> 
>     1.  When an isolated user task issues a synchronous I/O request (e.g.
>         via write(2)), execution occurs synchronously within the task's own
>         context and CPU accounting. The application remains in control of
>         when this overhead is incurred.
> 
>     2.  When dispatch must be deferred (e.g. BLK_STS_RESOURCE), blk-mq
>         queues &hctx->run_work. A background kworker runs under
>         SCHED_NORMAL. On isolated cores, workloads typically execute under
>         real-time scheduling classes (SCHED_FIFO, SCHED_RR, SCHED_DEADLINE)
>         or tight polling loops. A SCHED_FIFO task executing on that core
>         will completely starve the local kworker. Consequently,
>         blk_mq_run_work_fn() never runs, hardware dispatch stalls, pending
>         completions cannot occur, and the I/O pipeline hangs indefinitely.
> 
>     3.  Even where starvation is avoided (e.g. via the EEVDF scheduler's
>         Fair server or cooperative yielding), queuing bound work on a
>         nohz_full isolated core forces the scheduler tick to re-activate,
>         injecting unwanted jitter and defeating deterministic isolation
>         guarantees.
> 
> Steering deferred queue dispatch to an online housekeeping CPU via
> housekeeping_any_cpu(HK_TYPE_TIMER) sidesteps these pitfalls entirely.
> The kworker executes promptly on a housekeeping core without priority
> inversion, hardware dispatch proceeds unhindered, and the isolated CPU's
> low-latency domain is preserved.

Any user visible effect? Both are a piece of code running on isolated CPU,
the choice is in application. If they want to run IO on isolated CPU, we
do it in same way. Otherwise, we won't schedule work on isolated CPU for
respecting the kernel interface.


Thanks,
Ming