Re: [PATCH 1/2] sched_ext: Fix idle state tracking across idle repicks

[Andrea Righi <arighi@nvidia.com>]

Hi Tejun,

On Mon, Sep 14, 2026 at 01:42:58PM -1000, Tejun Heo wrote:
> An idle CPU can be reserved and kicked without receiving a task. When it
> picks idle again, the builtin idle masks recover the reservation, but
> ops.update_idle() is not called. The BPF scheduler's own idle tracking can
> leave the CPU unavailable until another real idle transition. This is a hole
> in the notification interface: BPF schedulers cannot reliably recover unused
> idle reservations, which is required for cid-form schedulers to maintain
> their own idle tracking.
> 
> Fix this by notifying on idle-to-idle picks and making these notifications
> automatic for cid-form schedulers. Keep CPU-form schedulers opt-in because
> existing callbacks may restart idle accounting or repeat actions intended
> only for idle entry. The known cid-form users, scx_qmap in tools/sched_ext

But this would change the semantics of ops.update_idle() for cid-form
schedulers. A scheduler that maintains balanced busy/idle accounting may now
receive multiple idle=true notifications without an intervening idle=false
notification.

We discussed essentially the same behavior here:
https://lore.kernel.org/r/Zw5_FlXfbLXDLCPG@slm.duckdns.org

> and scx_nitosis in the scx repository, have idempotent update_idle() bodies.
> All sub-schedulers use the cid form, so this is a root property and can use
> a static key.
> 
> scx_root_enable_workfn() must initialize idle tracking from sch->ops, where
> the automatic cid flag has been set. The local ops pointer still refers to
> the caller-provided table. Using it would leave the static key disabled for
> cid-form schedulers that omit the flag.
> 
> Signed-off-by: Tejun Heo <tj@kernel.org>

...

> ---
>  kernel/sched/ext/ext.c                        |  3 +-
>  kernel/sched/ext/idle.c                       | 60 +++++++++++--------
>  kernel/sched/ext/internal.h                   | 21 ++++++-
>  tools/sched_ext/include/scx/compat.h          |  2 +
>  .../sched_ext/include/scx/enum_defs.autogen.h |  1 +
>  .../sched_ext/include/scx/enums_abi.autogen.h |  3 +-
>  6 files changed, 60 insertions(+), 30 deletions(-)
> 
> diff --git a/kernel/sched/ext/ext.c b/kernel/sched/ext/ext.c
> index 83999203a63a..934605dfd950 100644
> --- a/kernel/sched/ext/ext.c
> +++ b/kernel/sched/ext/ext.c
> @@ -7251,6 +7251,7 @@ struct scx_sched *scx_alloc_and_add_sched(struct scx_enable_cmd *cmd,
>  	 */
>  	if (cmd->is_cid_type) {
>  		sch->ops_cid = *cmd->ops_cid;
> +		sch->ops_cid.flags |= SCX_OPS_UPDATE_IDLE_TO_IDLE;
>  		sch->is_cid_type = true;
>  	} else {
>  		sch->ops = *cmd->ops;

The unused-reservation case can already be handled by restoring the cid's idle
state at the end of ops.dispatch() when no task was selected for dispatch. For
example, scx_cidland does this:
https://github.com/sched-ext/scx/blob/6c54f4f664bb8e9cc445c8333128b4ba17b93608/scheds/experimental/scx_cidland/src/bpf/main.bpf.c#L2960

That said, idle-to-idle notifications would simplify scx_cidland and allow the
special handling in ops.dispatch() to be removed. However, other schedulers may
rely on ops.update_idle() being called only for actual idle state transitions.

Would it make sense to leave SCX_OPS_UPDATE_IDLE_TO_IDLE opt-in for cid-form
schedulers too?

Thanks,
-Andrea