Re: [PATCH 12/15] sched_ext: Delegate proxy donor admission to BPF schedulers

[Andrea Righi <arighi@nvidia.com>]

Hi Tejun,

On Mon, Aug 03, 2026 at 12:18:49PM -1000, Tejun Heo wrote:
> On Tue, Jul 28, 2026 at 05:43:30PM +0200, Andrea Righi wrote:
> ...
> > +/*
> > + * Called with @p's pi and rq locks held immediately before
> > + * sched_change_begin(). The caller must pass DEQUEUE_NOCLOCK so the rq clock
> > + * is updated only once.
> > + */
> > +void scx_prepare_task_sched_change(struct task_struct *p, struct scx_sched *sch)
> > +{
> > +	lockdep_assert_held(&p->pi_lock);
> > +	lockdep_assert_rq_held(task_rq(p));
> > +
> > +	update_rq_clock(task_rq(p));
> > +
> > +	/* Block retained donors that the incoming scheduler cannot manage. */
> > +	if (!(sch->ops.flags & SCX_OPS_ENQ_BLOCKED))
> > +		sched_proxy_block_task(task_rq(p), p);
> >  }
> 
> What are the cases that this one catches that scx_allow_proxy_exec() or
> prepare_switch_scx() doesn't?

scx_allow_proxy_exec() controls whether a task is retained when it first blocks
in __schedule(), it doesn't handle a donor that was already retained before its
scheduler ownership changes.

prepare_switch_scx() handles a scheduling-class transition into EXT, but it
isn't called for an EXT-to-EXT scheduler change. The helper was intended to
cover these same-class transitions, i.e., moving between parent and child
sub-schedulers.

Thinking more about this, retained proxy execution can be terminated on all
class changes centrally in sched_change_begin(). In this way we can remove the
.prepare_switch() class callback and prepare_switch_scx(). We would still need
to terminate retained proxy execution explicitly for EXT-to-EXT scheduler
ownership changes, but that shouldn't be an issue. I'll test this approach, it
should simplify the transition handling considerably.

> 
> > @@ -2299,11 +2351,24 @@ static void wakeup_preempt_scx(struct rq *rq, struct task_struct *p, int wake_fl
> >  {
> >  	/*
> >  	 * Preemption between SCX tasks is implemented by resetting the victim
> > -	 * task's slice to 0 and triggering reschedule on the target CPU.
> > -	 * Nothing to do.
> > +	 * task's slice to 0 and triggering reschedule on the target CPU. A
> > +	 * mutex-blocked task is kept queued for proxy execution, so its wakeup
> > +	 * doesn't go through enqueue_task_scx(). If the BPF scheduler manages
> > +	 * blocked donors, reschedule explicitly so that it can reconsider a
> > +	 * donor it declined to dispatch while blocked.
> 
> Can you make this a separate paragraph and is the comment uptodate? I'm
> having a difficulty understanding what "if the BPF scheduler manages blocked
> donors" mean.

"manages blocked donors" means the BPF scheduler sets SCX_OPS_ENQ_BLOCKED. I'll
split the comment and clarify it.

> 
> >  	 */
> > -	if (p->sched_class == &ext_sched_class)
> > +	if (p->sched_class == &ext_sched_class) {
> > +		bool enq_wakeup = p->scx.flags & SCX_TASK_ENQ_WAKEUP;
> > +
> > +		p->scx.flags &= ~SCX_TASK_ENQ_WAKEUP;
> > +		if (!enq_wakeup && p->is_blocked) {
> > +			struct scx_sched *sch = scx_task_sched(p);
> > +
> > +			if (sch && (sch->ops.flags & SCX_OPS_ENQ_BLOCKED))
> > +				resched_curr(rq);
> > +		}
> >  		return;
> > +	}
> 
> My understanding of what happens here is hazy. I suppose this is for the
> case of an active proxy execution being preempted by another SCX task? I'm
> not following why resched_curr() is needed here.

The relevant case is a mutex waiter receiving a wakeup while it's retained on
the rq as a proxy donor. Although the task is basically blocked and cannot
execute itself, its scheduling context remains on the rq, so that the mutex
owner can execute through it.

There are two wakeup paths when the mutex is released:

 1) If the donor was not proxy-migrated and is still on its callback rq,
    ttwu_runnable() handles the wakeup while the task remains on the rq. It
    calls wakeup_preempt() and then clears p->is_blocked, without calling
    enqueue_task_scx(). BPF doesn't receive any new ops.enqueue() notification.
    resched_curr() requests another scheduling cycle so that ops.dispatch() can
    reconsider the now-unblocked task (BPF scheduler may have kept the blocked
    donor in a BPF-managed queue).

 2) If the donor was proxy-migrated to the owner's rq, proxy_needs_return()
    removes it from that rq and the wakeup proceeds through the full activation
    path. That path calls enqueue_task_scx() before wakeup_preempt().
    SCX_TASK_ENQ_WAKEUP records that this enqueue already happened, preventing
    the additional resched_curr().

> 
> > @@ -3198,6 +3279,37 @@ static void put_prev_task_scx(struct rq *rq, struct task_struct *p,
> >  	if (p->scx.flags & SCX_TASK_QUEUED) {
> >  		set_task_runnable(rq, p);
> >  
> > +		/*
> > +		 * The rq lock has remained held since scx_allow_proxy_exec(), so
> > +		 * @p's scheduler association cannot have changed. An associated
> > +		 * donor stays queued only when its BPF scheduler enables
> > +		 * %SCX_OPS_ENQ_BLOCKED; delegate its admission to that scheduler.
> > +		 *
> > +		 * If @sch is NULL, @p is transitioning into the root scheduler. The
> > +		 * root is published before tasks enter EXT and cannot be cleared while
> > +		 * this rq is locked. Preserve generic proxy execution by placing the
> > +		 * donor directly on the local DSQ.
> > +		 */
> > +		if (p->is_blocked) {
> > +			/*
> > +			 * If the donor is the same and only the mutex owner
> > +			 * changes, avoid triggering another ops.enqueue(): the
> > +			 * BPF scheduler has already admitted the donor, so it
> > +			 * can continue running.
> > +			 */
> > +			if (next == p)
> > +				goto switch_class;
> > +
> > +			if (sch) {
> > +				WARN_ON_ONCE(!(sch->ops.flags & SCX_OPS_ENQ_BLOCKED));
> > +				scx_do_enqueue_task(rq, p, 0, -1);
> > +			} else {
> > +				scx_dispatch_enqueue(scx_root, rq, &rq->scx.local_dsq,
> > +						     p, 0);
> 
> Does this else arm actually happen? Can you describe the scenario? Oh, maybe
> below is the counterpart.

Right, this was intended for the root-enable transition where a task could
already be on the EXT class but not yet have an associated BPF scheduler. In
that window scx_allow_proxy_exec() permits generic proxy execution and sch can
be NULL.

IF we call sched_proxy_block_task() unconditionally, root enable will block any
retained donor before establishing the new scheduler ownership, so the
NULL-scheduler fallback is then unnecessary and we can remove it.

> 
> > @@ -7758,6 +7875,10 @@ static void scx_root_enable_workfn(struct kthread_work *work)
> >  
> >  		if (old_class != new_class)
> >  			queue_flags |= DEQUEUE_CLASS;
> > +		if (old_class == new_class && new_class == &ext_sched_class) {
> > +			scx_prepare_task_sched_change(p, sch);
> > +			queue_flags |= DEQUEUE_NOCLOCK;
> > +		}
> 
> I'd appreciate if there's more explanation of what happens during enable.
> Wouldn't it be simpler if we just do sched_proxy_block_task() on all
> transitions and start with a clean slate?

Agreed. I'll change the logic so that a proxy session never survives a
scheduling-class or BPF-scheduler ownership transition.

For scheduling-class changes, we can call sched_proxy_block_task() centrally
from sched_change_begin(), this makes the new .prepare_switch() sched-class
callback and prepare_switch_scx() unnecessary.

Thanks,
-Andrea