Re: [PATCH v2 3/3] sched_ext: Allow scx_bpf_reenqueue_local() to be called from anywhere

[Peter Zijlstra <peterz@infradead.org>]

On Mon, Oct 27, 2025 at 08:19:40AM -1000, Tejun Heo wrote:
> The ops.cpu_acquire/release() callbacks are broken - they miss events under
> multiple conditions and can't be fixed without adding global sched core hooks
> that sched maintainers don't want.

I think I'll object to that statement just a wee bit. I think we can
make it work -- just not with the things proposed earlier.

Anyway, if you want to reduce the sched_ext interface and remove
cpu_acquire/release entirely, this is fine too.

I might still do that wakeup_preempt() change if I can merge / replace
the queue_mask RETRY_TASK logic -- I have vague memories the RT people
also wanted something like this a while ago and it isn't that big of a
change.

> There are two distinct task dispatch gaps that can cause cpu_released flag
> desynchronization:
> 
> 1. balance-to-pick_task gap: This is what was originally reported. balance_scx()
>    can enqueue a task, but during consume_remote_task() when the rq lock is
>    released, a higher priority task can be enqueued and ultimately picked while
>    cpu_released remains false. This gap is closeable via RETRY_TASK handling.
> 
> 2. ttwu-to-pick_task gap: ttwu() can directly dispatch a task to a CPU's local
>    DSQ. By the time the sched path runs on the target CPU, higher class tasks may
>    already be queued. In such cases, nothing on sched_ext side will be invoked,
>    and the only solution would be a hook invoked regardless of sched class, which
>    isn't desirable.
> 
> Rather than adding invasive core hooks, BPF schedulers can use generic BPF
> mechanisms like tracepoints. From SCX scheduler's perspective, this is congruent
> with other mechanisms it already uses and doesn't add further friction.
> 
> The main use case for cpu_release() was calling scx_bpf_reenqueue_local() when
> a CPU gets preempted by a higher priority scheduling class. However, the old
> scx_bpf_reenqueue_local() could only be called from cpu_release() context.
> 
> Add a new version of scx_bpf_reenqueue_local() that can be called from any
> context by deferring the actual re-enqueue operation. This eliminates the need
> for cpu_acquire/release() ops entirely. Schedulers can now use standard BPF
> mechanisms like the sched_switch tracepoint to detect and handle CPU preemption.
> 
> Update scx_qmap to demonstrate the new approach using sched_switch instead of
> cpu_release, with compat support for older kernels. Mark cpu_acquire/release()
> as deprecated. The old scx_bpf_reenqueue_local() variant will be removed in
> v6.23.
> 
> Reported-by: Wen-Fang Liu <liuwenfang@honor.com>
> Link: https://lore.kernel.org/all/8d64c74118c6440f81bcf5a4ac6b9f00@honor.com/
> Cc: Peter Zijlstra <peterz@infradead.org>
> Signed-off-by: Tejun Heo <tj@kernel.org>

Yeah, this Changelog is much better, thanks!

6.23 is a long time, can't we throw this out quicker? This thing wasn't
supposed to be an ABI after all. A 1 release cycle seems fine to me ;-)