Re: [PATCH 2/3] sched_ext: Implement SCX_ENQ_IMMED

[Andrea Righi <arighi@nvidia.com>]

Hi Tejun,

On Fri, Mar 13, 2026 at 12:40:46AM -1000, Tejun Heo wrote:
> Hello,
> 
> On Mon, Mar 09, 2026 at 06:35:37PM +0100, Andrea Righi wrote:
> > > diff --git a/kernel/sched/ext_internal.h b/kernel/sched/ext_internal.h
> > > index f8df73044515..cd4272117be4 100644
> > > --- a/kernel/sched/ext_internal.h
> > > +++ b/kernel/sched/ext_internal.h
> > > @@ -31,6 +31,8 @@ enum scx_consts {
> > >  	SCX_BYPASS_LB_MIN_DELTA_DIV	= 4,
> > >  	SCX_BYPASS_LB_BATCH		= 256,
> > >  
> > > +	SCX_REENQ_LOCAL_MAX_REPEAT	= 256,
> > 
> > That's a lot of re-enqueues. What if we simply ignore SCX_ENQ_IMMED when
> > SCX_ENQ_REENQ is set?
> 
> It's meant to be a safety mechanism against system lockup, not a workaround
> for BPF scheduler misbehavior.
> 
> > This would solve the SCX_OPS_ALWAYS_ENQ_IMMED issue and naturally limit the
> > loop to a single retry:
> >  - first attempt (IMMED) fails -> task re-enqueued with REENQ flag,
> >  - second attempt sees REENQ -> ignores IMMED check -> queues normally on
> >    local DSQ.
> > 
> > This approach seems more robust and would avoid the latency overhead of
> > repeated failures (the re-enqueues were actually the reason of the latency
> > issues that I was experiencing). If I don't use SCX_OPS_ALWAYS_ENQ_IMMED
> > and I selectively use SCX_ENQ_IMMED with just one retry I can actually see
> > some small, but consistent, benefits with scx_cosmos running some latency
> > benchmarks.
> 
> The intention is making IMMED guarnatee immediate execution - if IMMED is
> set, the task will get on the CPU or get re-enqueued. On v2 patchset, this
> behavior is extended to staying on CPU. If an IMMED task is preempted for
> whatever reason, it gets fully reenqueued instead of e.g. silently put back
> on the local DSQ. The goal is giving the BPF controller full latency
> control.
> 
> I don't think it makes sense to paper over IMMED failures. The BPF scheduler
> shouldn't be doing that in the first place. If the CPU is not availalbe and
> the task keeps requesting IMMED dispatch of a task on that CPU, the
> scheduler is buggy. Is cosmos doing DSQ_LOCAL dispatch on single-CPU bound
> tasks? If so, it shouldn't use ALWAYS_IMMED. Instead, it should only mark
> dispatches that know the target CPU to be available (IOW, claimed idle) with
> SCX_ENQ_IMMED. I don't think that's too much of a burden.

Yeah, I think that's fair. I was trying to figure out an optimal "default
behavior", but ultimately the BPF scheduler can just avoid setting
SCX_OPS_ALWAYS_ENQ_IMMED and manage the IMMED dispatches directly, instead
of relying on the "always" behavior.

I also did more experiments using IMMED in scx_cosmos and it seems useful,
so you can add my:

Reviewed-by: Andrea Righi <arighi@nvidia.com>

BTW, did you send a v2 (in that case I missed it)?

Thanks,
-Andrea