Re: [PATCH sched_ext/for-7.1] tools/sched_ext: Kick home CPU for stranded tasks in scx_qmap

Cheng-Yang Chou
2026-04-13T05:38:43Z

Hi Tejun,

On Sun, Apr 12, 2026 at 05:30:52PM -1000, Tejun Heo wrote:
> scx_qmap uses global BPF queue maps (BPF_MAP_TYPE_QUEUE) that any CPU's
> ops.dispatch() can pop from. When a CPU pops a task that can't run on it
> (e.g. a pinned per-CPU kthread), it inserts the task into SHARED_DSQ.
> consume_dispatch_q() then skips the task due to affinity mismatch, leaving it
> stranded until some CPU in its allowed mask calls ops.dispatch(). This doesn't
> cause indefinite stalls -- the periodic tick keeps firing (can_stop_idle_tick()
> returns false when softirq is pending) -- but can cause noticeable scheduling
> delays.
>
> After inserting to SHARED_DSQ, kick the task's home CPU if this CPU can't run
> it. There's a small race window where the home CPU can enter idle before the
> kick lands -- if a per-CPU kthread like ksoftirqd is the stranded task, this
> can trigger a "NOHZ tick-stop error" warning. The kick arrives shortly after
> and the home CPU drains the task.
>
> Rather than fully eliminating the warning by routing pinned tasks to local or
> global DSQs, the current code keeps them going through the normal BPF queue
> path and documents the race and the resulting warning in detail. scx_qmap is an
> example scheduler and having tasks go through the usual dispatch path is useful
> for testing. The detailed comment also serves as a reference for other
> schedulers that may encounter similar warnings.
>
> Signed-off-by: Tejun Heo <tj@kernel.org>
> ---
> v2: Replaced the previous enqueue-side fix which kicked when a pinned task was
> enqueued. That was based on the theory that ops.select_cpu() being skipped
> meant the home CPU wouldn't be woken, which wasn't quite right --
> wakeup_preempt() kicks the target CPU regardless. Moved the fix to
> ops.dispatch() where the stranding is actually observable.
>
> tools/sched_ext/scx_qmap.bpf.c | 40 ++++++++++++++++++++++++++++++++++
> 1 file changed, 40 insertions(+)
>
> diff --git a/tools/sched_ext/scx_qmap.bpf.c b/tools/sched_ext/scx_qmap.bpf.c
> index f3587fb709c9..a4543c7ab25d 100644
> --- a/tools/sched_ext/scx_qmap.bpf.c
> +++ b/tools/sched_ext/scx_qmap.bpf.c
> @@ -471,6 +471,46 @@ void BPF_STRUCT_OPS(qmap_dispatch, s32 cpu, struct task_struct *prev)
> __sync_fetch_and_add(&nr_dispatched, 1);
>
> scx_bpf_dsq_insert(p, SHARED_DSQ, slice_ns, 0);
> +
> + /*
> + * scx_qmap uses a global BPF queue that any CPU's
> + * dispatch can pop from. If this CPU popped a task that
> + * can't run here, it gets stranded on SHARED_DSQ after
> + * consume_dispatch_q() skips it. Kick the task's home
> + * CPU so it drains SHARED_DSQ.
> + *
> + * There's a race between the pop and the flush of the
> + * buffered dsq_insert:
> + *
> + * CPU 0 (dispatching) CPU 1 (home, idle)
> + * ~~~~~~~~~~~~~~~~~~~ ~~~~~~~~~~~~~~~~~~~
> + * pop from BPF queue
> + * dsq_insert(buffered)
> + * balance:
> + * SHARED_DSQ empty
> + * BPF queue empty
> + * -> goes idle
> + * flush -> on SHARED
> + * kick CPU 1
> + * wakes, drains task
> + *
> + * The kick prevents indefinite stalls but a per-CPU
> + * kthread like ksoftirqd can be briefly stranded when
> + * its home CPU enters idle with softirq pending,
> + * triggering:
> + *
> + * "NOHZ tick-stop error: local softirq work is pending, handler #N!!!"
> + *
> + * from report_idle_softirq(). The kick lands shortly
> + * after and the home CPU drains the task. This could be
> + * avoided by e.g. dispatching pinned tasks to local or
> + * global DSQs, but the current code is left as-is to
> + * document this class of issue -- other schedulers
> + * seeing similar warnings can use this as a reference.
> + */
> + if (!bpf_cpumask_test_cpu(cpu, p->cpus_ptr))
> + scx_bpf_kick_cpu(scx_bpf_task_cpu(p), 0);
> +
> bpf_task_release(p);
>
> batch--;
> --
> 2.53.0

This makes sense.

I also realized my previous patch for scx_userland was unnecessary, as
the global DSQ logic handles this automatically. Sorry for the nose on
that one.

Reviewed-by: Cheng-Yang Chou <yphbchou0911@gmail.com>

--
Thanks,
Cheng-Yang