Re: [PATCH 08/12] sched_ext: Add bandwidth-limited rescue execution for stranded tasks

[Andrea Righi <arighi@nvidia.com>]

Hi Tejun,

On Sun, Aug 02, 2026 at 11:54:43AM -1000, Tejun Heo wrote:
> A local DSQ insert lacking the needed caps is diverted to the reject DSQ and
> bounced back through ops.enqueue() so the scheduler can re-decide. That
> recovery assumes the scheduler has somewhere legal to send the task. When it
> doesn't, e.g. when the task's affinity is restricted to cids delegated away,
> the task starves until the stall watchdog ejects the scheduler. An exiting
> task is worse - it skips ops.enqueue() and the rejection becomes a
> self-requeuing cycle that burns the CPU until the watchdog fires.
> 
> Add SCX_ENQ_RESCUE, a fallback modifier on local DSQ inserts. When the
> insert would be rejected for missing caps, the kernel takes over and runs
> the task on the target CPU without consulting the owning scheduler. The
> kernel sets the flag itself when enqueueing an exiting task.
> 
> Rescue is a last-resort forward-progress backstop with a persistent
> disadvantage, not a way around cap enforcement. A per-CPU token bucket
> accrues rescue_bandwidth_ppt (default 2%) of CPU time and rescues run one at
> a time in arrival order. Each is granted a slice of the rescue_quantum_us
> (default 5ms) quantum divided across the waiters, waits at the tail of the
> local DSQ claiming no priority, and rejoins its scheduler as a fresh arrival
> once the slice is served.
> 
> The schedulers keep their normal control over an admitted rescuee and may
> preempt or reslice it. Service is measured on CPU time actually received, so
> neither shortens the rescue. Prolonged denial escalates - the remaining
> slice turns into protected execution (SCX_TASK_PROTECTED) and the rescuee
> preempts the current task. Escalation is paced by the same bucket, and
> delivered service converges on the configured bandwidth no matter how
> aggressively the schedulers dispatch.
> 
> Both knobs are root-only and SCX_RESCUE_DISABLE turns rescue off, making
> SCX_ENQ_RESCUE inserts reject as usual.
> 
> Signed-off-by: Tejun Heo <tj@kernel.org>
> ---
...
> diff --git a/kernel/sched/ext/internal.h b/kernel/sched/ext/internal.h
> index d418935f1e6b..18983dbe81f4 100644
> --- a/kernel/sched/ext/internal.h
> +++ b/kernel/sched/ext/internal.h
> @@ -924,6 +924,37 @@ struct sched_ext_ops {
>  	 */
>  	u32 cid_shard_size;
>  
> +	/**
> +	 * @rescue_bandwidth_ppt: Rescue execution bandwidth in parts per thousand
> +	 *
> +	 * The fraction of each CPU's time that may be consumed running tasks
> +	 * from its rescue DSQ. A higher bandwidth admits and escalates rescues
> +	 * faster, see @rescue_quantum_us.
> +	 *
> +	 * Only the root scheduler's value is used. 0 means the default of 20
> +	 * (2%). May not exceed 250 (25%). %SCX_RESCUE_DISABLE disables rescue -
> +	 * %SCX_ENQ_RESCUE inserts are then rejected like any other insert
> +	 * lacking the caps.
> +	 */
> +	u32 rescue_bandwidth_ppt;
> +
> +	/**
> +	 * @rescue_quantum_us: Rescue execution quantum in microseconds
> +	 *
> +	 * How much CPU time each rescue gets. Rescues run one at a time per CPU
> +	 * and admissions are paced to keep rescue execution within
> +	 * @rescue_bandwidth_ppt - with the defaults, one 5ms rescue every
> +	 * 250ms. A crowded queue round-robins on the quantum divided across the
> +	 * waiters, floored at 1ms. A stuck rescue eventually escalates to
> +	 * forced execution. A larger quantum interrupts the CPU less often but
> +	 * for longer and spaces rescues further apart.
> +	 *
> +	 * Only the root scheduler's value is used. 0 means the default (5000).
> +	 * Non-zero values must be within [1000, 100000]. Values too short for
> +	 * the kernel to meter are lifted silently.
> +	 */
> +	u32 rescue_quantum_us;
> +

Not a blocker, but should we add compatibility handling for these two optional
ops fields in tools/sched_ext/include/scx/compat.h?

Otherwise the later scx_qmap patch would break the qmap build with older
kernels.

Thanks,
-Andrea