Re: [PATCH 01/10] sched/core: Drop mutex locks before proxy rescheduling

[Andrea Righi <arighi@nvidia.com>]

On Fri, Jul 10, 2026 at 05:21:45PM -0700, John Stultz wrote:
> On Fri, Jul 10, 2026 at 1:47 PM Andrea Righi <arighi@nvidia.com> wrote:
> > Inspecting the proxy chain is not strictly required. A BPF scheduler can just
> > accept a blocked donor, enqueue it on its current CPU and let the core
> > proxy-exec path resolve and "virtually" migrate it as needed to the owner's CPU.
> >
> > The motivation for providing a scx_bpf_task_proxy_cpu() kfunc is purely for
> > optimization reasons. By knowing the owner's CPU, a scheduler could make more
> > informed admission decisions. For example, it can inspect the task currently
> > running on the target CPU, determine whether the donor should take precedence
> > over the running task and kick that CPU to trigger a preemption. Without this
> > information, the scheduler can only enqueue the donor and wait until the task
> > running on the owner's CPU releases it (potentially adding up to one time slice
> > of latency).
> 
> So... I suspect I'm missing a subtlety of scx, but in the non-scx
> case, if a blocked task is important enough to be selected on the
> current cpu, the find_proxy_task() logic will walk the chain and do
> the proxy-migration if needed and mark resched, so then that cpu can
> evaluate which of the currently running task or the migrated donor is
> important enough to run.

With scx if a resched is triggered balance_one() does this:

if (prev is runnable and prev->scx.slice > 0)
    keep running prev;

We need to set the prev->scx.slice to 0 to select the new task. That means, if
the donor is "more important" than the task running in the owner's CPU we need
to zero it's time slice. And This can be done via
scx_bpf_kick_cpu(cpu, SCX_KICK_PREEMPT).

That's the reason why I was considering introducing the
scx_bpf_task_proxy_cpu() kfunc.

However, I agree that this is adding too much complexity and we can ignore this
for now.

Moreover, let's also consider the following:
 - if donor and owner are on the same CPU everything is already working as
   intended without scx_bpf_task_proxy_cpu()
 - if donor and owner are on different CPUs, something like this should happen
   (D = donor, O = owner, C = contending task):

  CPU0                              CPU1
  ----                              ----
  D blocks                          C runs
  ops.enqueue(D, SCX_ENQ_BLOCKED)
    [D inserted into CPU0 local DSQ]
  D briefly yields to idle
  D selected from local DSQ
  D migrates ---------------------> ops.enqueue(D, SCX_ENQ_BLOCKED)
                                      [D enters CPU1 local DSQ]
                                    resched, but C retains its slice
                                    C continues
                                    C's slice expires
				    ops.enqueue(C, 0)
				      [C is returned to the BPF policy]
                                    D selected as donor
                                    O proxy-runs using D's slice
                                    O releases mutex

If we want to handle the preemption when the donor's CPU != owner's CPU we can
just use SCX_ENQ_PREEMPT when scx_bpf_cpu_curr(scx_bpf_task_cpu(p)) is an SCX
task.

So the BPF scheduler could do something like this:

ops.enqueue(p, enq_flags) {
  if (enq_flags & SCX_ENQ_BLOCKED) { // p is a donor
      struct task_struct *curr = scx_bpf_cpu_curr(cpu);

      if ((curr && task_is_scx(curr) && is_more_important(p, curr))
          enq_flags |= SCX_ENQ_PREEMPT; // curr->scx.slice to 0 + resched

      scx_bpf_dsq_insert(p, SCX_DSQ_LOCAL_ON | cpu, 0, flags);
}

Therefore we don't really need scx_bpf_task_proxy_cpu().

> 
> I worry the difficulty (with my poor understanding of the
> optimization) is that it seems like its evaluting the preemption on
> admission (I assume this means the point when we keep the blocked_on
> task on the rq? I may be totally off base here), is that if a very
> important task was breifly running at that moment, you might not allow
> the blocked_on task to be enqueued. And at that point the donor is
> effectively sleeping and no proxying can happen until it wakes (which
> may be only when the lock owner eventually runs and releases the
> lock).  In this way the optimization might make a call in that instant
> that results in *many* time slices of latency (particularly if there
> are lots other unimportant tasks on that cpu).
> 
> By letting the proxy logic handle the proxy migration and
> rescheduling, the blocked_on tasks are just in the same pool of
> selectable tasks on that cpu and the scheduler on that cpu gets to
> decide what is the most important thing to run next.
> 
> Now, I can see the benefit of potentially saving the resched kick on
> the target cpu when we do a proxy-migration - it is an interesting
> idea I should think more on to see how we might do that better in the
> core logic.
> 
> > That said, I don't see a correctness dependency on exposing this information.
> > If the latency-policy use case is not strong enough to justify the API, I'm
> > happy to drop the query kfuncs and this preparatory lock-scope change for now,
> > the blocked-donor admission support can stand on its own.
> 
> Yeah. If that's the case I wonder if it might be worth waiting a bit
> on these optimizations until after the rest of the core proxy logic
> makes it upstream and settles a bit?

Agreed. I've already dropped that in my local tree. Let's keep it simple for
now. :)

> 
> It might also help make sure the demand for this
> optimization/interface is strong before we start adding interfaces
> prematurely.
> 
> Really, I'd like it to all be transparent to the class schedulers, but
> given the complexities, just having the ability for the scx bpf
> schedulers to disable keeping blocked_on tasks on the rq (effectively
> disabling scx proxy donation) seems like a good initial step.
> 
> Again, I am really excited about your work here! I've sadly really not
> had any time since Dec to look into sched_ext details, and I've had to
> tell a few folks who are interested in both sched_ext and proxy_exec
> that they should turn proxy_exec off if they want to use sched_ext.
> So I'm very eager to have a better answer there!
> 
> Thanks for all your great work here!
> -john

Thank you!
-Andrea