Re: [PATCH 17/17] sched_ext: Fix queued bypassed tasks stalling when the host loses SCX_CAP_BASE

[Andrea Righi <arighi@nvidia.com>]

Hi Tejun,

On Fri, Oct 23, 2026 at 07:56:08AM -1000, Tejun Heo wrote:
> A task already queued on a bypass DSQ stalls when the host loses
> SCX_CAP_BASE on that cpu. The cpu's dispatch no longer consumes the DSQ, and
> the task sits there until the host regains BASE or the bypass window ends.
> Sending a bypassed task to rescue at enqueue time covers only a task
> enqueued after the loss, and the balancer no longer donates to a capless
> cpu.
> 
> Have the bypass load balancer drain such a DSQ: a cpu where the host lacks
> BASE is a donor with a target of zero. The target already counts its tasks
> against the cpus that hold BASE, so the donees take them under their
> ordinary fill. The queued-duration threshold doesn't apply to a zero target.
> 
> The balancer reenqueues a task no donee can take at its home, under the home
> rq lock the move holds there anyway. The enqueue sends the task to rescue
> when the host lacks BASE at home and rescue is enabled, and to the home's
> bypass DSQ otherwise. A task that must run at home, migration-disabled or
> with a migration pending, is admitted to the home's local DSQ in either
> case. Every donee refuses a migration-disabled task, so it always takes this
> path.
> 
> The same reenqueue replaces the put-back for a task that can't run on the
> donee the walk picked for it. That is a cold path: the donee went inactive
> or the task is a proxy-migrated blocked donor. An affinity change while the
> move is in flight dequeues the task and the balancer loses the claim
> instead.
> 
> Without rescue, the reenqueue of a task homed on the drained cpu lands on
> the DSQ it came from, apart from the admit above. A task stuck that way is
> reenqueued once per pass and waits for the host to regain BASE or the bypass
> to end, or for the stall watchdog to eject the host, which it blames for a
> task stuck on its DSQ.
> 
> With a zero interval the stuck task is reenqueued once per BASE loss, and a
> task that lands on the drained DSQ after that pass waits the same way. These
> reenqueues, like those of a task refused pass after pass, are not counted
> against the reenqueue limit, as the task's scheduler is bypassing.
> 
> On a BASE loss, scx_bypass_lb_kick() sets the balancer's timer to expire
> now, so that the drain runs before the next interval, and runs at all when
> the interval is zero. A donee can lose BASE while the balancer is moving
> tasks onto it, and the kick from that loss covers it.
> 
> The balancer re-arms with timer_reduce(), which only brings the expiry
> forward, so a loss during a run leaves the timer at now and the next run
> sees it. The arming at bypass dispatch enable uses timer_reduce() too, so
> that a kick landing there is kept.
> 
> v2: The description drops an in-flight affinity change from the causes of a
> refused balancer move.
> 
> v3:
> - The description says that a stuck task's reenqueues are not counted
>   against the reenqueue limit (sashiko).
> - The description names the stall watchdog as the way out for a task stuck
>   without rescue, and the tasks a zero interval leaves on a drained DSQ.
> - bypass_lb_reenq_home()'s kerneldoc and the description give the placement
>   with rescue disabled and for a task that must run at home, and the other
>   two comments on that placement point to the kerneldoc.
> 
> Fixes: 147d1885f390 ("sched_ext: Add the SCX_CAP_ENQ_IMMED cap")
> Signed-off-by: Tejun Heo <tj@kernel.org>

...

> @@ -6579,20 +6602,21 @@ static s32 bypass_lb_move(struct scx_sched *sch, struct rq *donor_rq,
>  	/*
>  	 * @p's affinity is stable under its rq lock. A task whose donee is its
>  	 * own cpu can always run there, and task_can_run_on_remote_rq() warns
> -	 * when asked about a task's own cpu.
> +	 * when asked about a task's own cpu. A @donee of -1 means none, so
> +	 * don't pass it to cpu_rq().
>  	 */
> -	if (donee == task_cpu(p) || task_can_run_on_remote_rq(sch, p, cpu_rq(donee), false)) {
> -		dst_dsq = scx_bypass_dsq(sch, donee);
> -		enq_flags = 0;
> +	if (donee >= 0 && (donee == task_cpu(p) ||
> +			   task_can_run_on_remote_rq(sch, p, cpu_rq(donee), false))) {
> +		scx_dispatch_enqueue(sch, rq, scx_bypass_dsq(sch, donee), p, 0, 0, 0);
>  		ret = donee;
>  	} else {
> -		/* back where it was, at the head so that the walk skips it */
> -		dst_dsq = donor_dsq;
> -		enq_flags = SCX_ENQ_HEAD;
> -		ret = -1;
> +		/*
> +		 * No donee, or @p can't run on it. Reenqueue @p at home
> +		 * instead, see bypass_lb_reenq_home().
> +		 */
> +		bypass_lb_reenq_home(rq, p, cpu_of(donor_rq));
> +		ret = cpu_of(rq);

Not a blocker, but I was wondering if we should preserve the old head or
ordinary balancing when the donor still holds BASE?

If a selected donee refuses a task homed on the donor, this can put it
at the tail of the same DSQ. A proxy-migrated blocked donor is refused
by remote CPUs, so repeated balancing passes could keep pushing it back.

Thanks,
-Andrea