Re: [PATCH v5 sched_ext/for-7.3 16/33] sched_ext: Add sub_ecaps_updated() effective-cap change notifier

[Andrea Righi <arighi@nvidia.com>]

Hi Tejun,

On Thu, Jul 09, 2026 at 12:50:24PM -1000, Tejun Heo wrote:
> A sub-scheduler that gains or loses effective caps on a cpu may want to act
> on it right away - e.g. place or preempt on a newly usable cpu. The existing
> ops.sub_caps_updated() doesn't fit as it is delivered asynchronously to
> scheduling operations and can arrive before the per-cpu effective caps go
> live.
> 
> Add ops.sub_ecaps_updated(cid, before, after), a cid-form callback fired
> from scx_process_sync_ecaps() when a sub-sched's effective caps on a cid
> change. It runs in dispatch context so the sched can insert, kick or preempt
> on the cid directly. @before is the caps as of the last delivery.
> 
> Cpu hotplug rides the same machinery. Going down zeroes each sched's ecaps
> on the cpu's cid, with queued syncs discarded at consumption while the cpu
> is inactive. Coming back up queues a sync for every sched. reported_ecaps is
> kept across the down/up cycle, so the resync fires the callback only if
> ownership actually changed while the cpu was down.
> 
> v2: Compute cid below the active-cpu guard; discard queued syncs on !cpu_active(). (sashiko AI)
> 
> Signed-off-by: Tejun Heo <tj@kernel.org>
> ---
...
>  /*
>   * @pcpu's sched was unhashed before the grace period, so nothing new queues.
> - * Flush its pending sync so the pcpu can be freed. scx_process_sync_ecaps()
> - * takes nodes off the list before syncing and acquiring the rq lock waits for
> - * any in-flight walk.
> + * Flush its pending sync so the pcpu can be freed. If the cpu is online and
> + * scx is enabled, drain via balance_one(). Otherwise, discard under the rq
> + * lock.
>   */
>  void scx_discard_ecaps_to_sync(s32 cpu, struct scx_sched_pcpu *pcpu)
>  {
> -	scoped_guard (rq_lock_irqsave, cpu_rq(cpu))
> -		scx_process_sync_ecaps(cpu_rq(cpu));
> +	struct rq *rq = cpu_rq(cpu);
>  
> -	WARN_ON_ONCE(llist_on_list(&pcpu->ecaps_to_sync_node));
> +	while (true) {
> +		scoped_guard (rq_lock_irqsave, rq) {
> +			/*
> +			 * scx_process_sync_ecaps() takes the node off the list
> +			 * before it is done accessing @pcpu but does all of it
> +			 * under the rq lock. Off-list observed under the rq
> +			 * lock guarantees that the sync is complete.
> +			 */
> +			if (!llist_on_list(&pcpu->ecaps_to_sync_node))
> +				return;
> +			/*
> +			 * Discard only when the cpu is truly down. cpu_active()
> +			 * is already set when scx_online_ecaps() queues an online
> +			 * resync while SCX_RQ_ONLINE is not - so test cpu_active(),
> +			 * or that resync would be dropped.
> +			 */
> +			if (!scx_enabled() || !cpu_active(cpu)) {
> +				discard_queued_syncs(rq);
> +				return;
> +			}
> +		}
> +		resched_cpu(cpu);
> +		msleep(1);
> +	}

This is a nit, so feel free to ignore. I was wondering if we could wait
indefinitely in this loop if the target CPU never reaches balance_one().
However, I think the only way to never trigger balance_one() is a higher class
monopolizing the CPU, but this can't happen because of the ext dl_server.

In that case, should we document this "dependency" here? Something like:

/*
 * The active EXT deadline server guarantees that balance_one() eventually runs
 * even under sustained FAIR or RT load.
 */

Thanks,
-Andrea