Re: [PATCH 5/6] sched_ext: Report the ecaps loss when a cpu goes offline

[Andrea Righi <arighi@nvidia.com>]

On Sat, Oct 17, 2026 at 03:59:51PM -1000, Tejun Heo wrote:
> When a cpu goes offline, a sub-scheduler's effective caps on its cid are
> cleared, but neither the sub-scheduler nor its parent is notified. The
> sub-scheduler keeps treating the cid as usable, and the parent can't act on
> the loss, e.g. reset the cpuperf target.
> 
> Report the loss before the root's ops.cpu_offline() and the regain after its
> ops.cpu_online(). The zeroing at offline was the last change to a sched's
> ecaps outside a delivery, so they now change only with one. Syncs on an
> inactive cpu are processed instead of discarded, and the reseed reschedules
> the cpu itself, as they got no resched when queued. Suspend and hibernation
> now report their cpu downs and ups as well. This is temporary, and a later
> patch leaves them out.
> 
> Two things need care. A cpu going down marks a sync for each sub-scheduler,
> and its hotplug thread waits until each marked sync has been processed. The
> cpu's own ext picks process them, and the ext server is started in case a
> fair task holds the cpu. Only the marked syncs count, so a flood of changes
> on the cid can't hold up the hotplug.
> 
> The detach report reaches only active cpus. It holds off cpu hotplug, so a
> cpu it skips has reported its loss as it went down, and no unreported loss
> is left to drop.
> 
> Drop the cpuperf reset from scx_qmap's ops.sub_detach(), as its
> ops.sub_child_ecaps_updated() now resets the targets on the loss reports.
> 
> v2: Reschedule from the irq work and document the regain order.
> 
> Signed-off-by: Tejun Heo <tj@kernel.org>

...

> @@ -1313,40 +1347,69 @@ void scx_online_ecaps(struct rq *rq)
>  		guard(raw_spinlock)(&ps->lock);
>  		queue_sync_ecaps(pos, cid);
>  	}
> +
> +	resched_sync_ecaps(rq);
>  }

With a root-only scheduler the walk skips the root and queues nothing, but
resched_sync_ecaps() still runs on every online CPU, starting the ext_server and
queueing the resched irq work for no reason. Can we do it only if something was
actually queued?

Something like:

	bool queued = false;
	...
		queue_sync_ecaps(pos, cid);
		queued = true;
	}

	if (queued)
		resched_sync_ecaps(rq);

>  
>  /*
> - * A cpu is going down, and the root's ops.cpu_offline() follows. Stop the cpu's
> - * ecaps deliveries and zero each sub-sched's in-effect ecaps so cap checks
> - * treat the cpu as capless while offline. Pending and late-queued syncs are
> - * discarded at consumption by scx_process_sync_ecaps() while the cpu is
> - * inactive. Leave reported_ecaps. Ownership is unchanged, so the
> - * scx_online_ecaps() reseed reports only a genuine delta. No callback fires
> - * here.
> + * A cpu is going down, and the root's ops.cpu_offline() follows. Queue a marked
> + * sync for each sub-sched to report the loss and wait until each has been
> + * processed. Then stop the cpu's ecaps deliveries, so that none comes after
> + * ops.cpu_offline().
> + *
> + * This runs on the cpu's hotplug thread. While it sleeps, the cpu keeps
> + * scheduling, and a pick through the ext class processes the syncs, see
> + * resched_sync_ecaps(). Only the marked syncs count, so a flood of changes on
> + * the cid can't hold up the hotplug. After the wait, every sub-sched's ecaps on
> + * the cpu are zero, and the later syncs have nothing to deliver.
>   */
>  void scx_offline_ecaps(struct rq *rq)
>  {
>  	s32 cpu = cpu_of(rq);
>  	struct scx_sched *root, *pos;
> +	u32 nr = 0;
> +	s32 cid;
>  
> -	guard(rq_lock_irqsave)(rq);
> +	/*
> +	 * ecaps and the tables need a live root, see scx_online_ecaps(). The
> +	 * ext class isn't picked without one, so the syncs wouldn't be
> +	 * processed and the wait would never end. scx_enabled() can't change
> +	 * during hotplug.
> +	 */
> +	if (scx_enabled()) {
> +		scoped_guard (rq_lock_irqsave, rq) {
> +			root = scx_root_protected();
> +			cid = __scx_cpu_to_cid(cpu);
>  
> -	root = scx_root_protected();
> +			scx_for_each_descendant_pre(pos, root) {
> +				struct scx_sched_pcpu *pcpu = per_cpu_ptr(pos->pcpu, cpu);
> +				struct scx_pshard *ps;
>  
> -	scx_for_each_descendant_pre(pos, root) {
> -		struct scx_sched_pcpu *pcpu = per_cpu_ptr(pos->pcpu, cpu);
> +				/* root holds every cap and never uses ecaps */
> +				if (!pos->level)
> +					continue;
>  
> -		/* root holds every cap and never uses ecaps */
> -		if (!pos->level)
> -			continue;
> +				/*
> +				 * A batch that has yet to reach the sync sees
> +				 * the mark. A sync already being processed has
> +				 * left the list and is queued again below.
> +				 */
> +				pcpu->flags |= SCX_SCHED_PCPU_ECAPS_OFFLINE;
> +				nr++;
>  
> -		WRITE_ONCE(pcpu->ecaps, 0);
> +				ps = cid_pshard(pos, cid);
> +				guard(raw_spinlock)(&ps->lock);
> +				queue_sync_ecaps(pos, cid);
> +			}
>  
> -		/* deliveries have stopped on this cpu, clear its mark */
> -		if (pcpu->flags & SCX_SCHED_PCPU_ECAPS_REPORT)
> -			ecaps_report_done(pcpu);
> +			WRITE_ONCE(rq->scx.nr_ecaps_offline, nr);
> +			resched_sync_ecaps(rq);
> +		}
> +
> +		wait_event(ecaps_offline_waitq, !READ_ONCE(rq->scx.nr_ecaps_offline));
>  	}

ame on the offline side: with no sub-sched nr is 0, but we still do
update_rq_clock(), dl_server_start(&rq->ext_server) and the irq work on a CPU
that's going down and the ext_server stays enqueued there until
sched_cpu_dying() stops it. How about:

	WRITE_ONCE(rq->scx.nr_ecaps_offline, nr);
	if (nr)
		resched_sync_ecaps(rq);

The wait_event() returns right away when nr is 0, so nothing else changes for
root-only schedulers.

Thanks,
-Andrea