Re: [PATCH 08/11] sched_ext: Delegate proxy donor admission to BPF schedulers

[Andrea Righi <arighi@nvidia.com>]

Hi John,

On Sat, Jul 18, 2026 at 04:23:40PM +0200, Andrea Righi wrote:
> Hi John,
> 
> On Fri, Jul 17, 2026 at 11:50:20PM -0700, John Stultz wrote:
> > On Fri, Jul 17, 2026 at 11:16 PM John Stultz <jstultz@google.com> wrote:
> > >
> > > On Thu, Jul 16, 2026 at 6:23 AM Andrea Righi <arighi@nvidia.com> wrote:
> > > > @@ -3098,18 +3164,16 @@ static void put_prev_task_scx(struct rq *rq, struct task_struct *p,
> > > >                 set_task_runnable(rq, p);
> > > >
> > > >                 /*
> > > > -                * Mutex-blocked donors stay queued on the runqueue under proxy
> > > > -                * execution, but the donor never runs as itself, proxy-exec
> > > > -                * walks the blocked_on chain on the next __schedule() and runs
> > > > -                * the lock owner in its place.
> > > > +                * Mutex-blocked donors only stay queued when their BPF scheduler
> > > > +                * enables %SCX_OPS_ENQ_BLOCKED. The rq lock has remained held since
> > > > +                * scx_allow_proxy_exec(), so @p's scheduler association cannot have
> > > > +                * changed and @sch must be non-NULL with the flag set.
> > > >                  *
> > > > -                * Put the donor on the local DSQ directly so pick_next_task()
> > > > -                * can still see it. find_proxy_task() will either run the chain
> > > > -                * owner or deactivate the donor so the wakeup path can return it
> > > > -                * and let BPF make a new dispatch decision once it is unblocked.
> > > > +                * Delegate admission to the BPF scheduler.
> > > >                  */
> > > >                 if (p->is_blocked) {
> > > > -                       scx_dispatch_enqueue(sch, rq, &rq->scx.local_dsq, p, 0);
> > > > +                       WARN_ON_ONCE(!(sch->ops.flags & SCX_OPS_ENQ_BLOCKED));
> > > > +                       scx_do_enqueue_task(rq, p, 0, -1);
> > > >                         goto switch_class;
> > > >                 }
> > > >
> > >
> > > So this isn't a blocker for your patches, but just as a heads up: when
> > > applying my sleeping owner handling changes (even just patches 4-7
> > > from my v30 submission[1]), I managed to trip the above WARN_ON, when
> > > running the test-ww_mutex driver under the scx_pair (usually right as
> > > scx_pair loads).
> > >
> > > [ 5818.970324] WARNING: kernel/sched/ext/ext.c:3175 at
> > > put_prev_task_scx+0x527/0x550, CPU#54: kworker/u261:11/30527
> > > [ 5818.974927] CPU: 54 UID: 0 PID: 30527 Comm: kworker/u261:11
> > > Tainted: G        W           7.1.0-13303-gb5ecf3c9f881 #116
> > > PREEMPT(full)
> > > [ 5818.979484] Tainted: [W]=WARN
> > > [ 5818.980654] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996),
> > > BIOS 1.17.0-debian-1.17.0-1 04/01/2014
> > > [ 5818.984225] Workqueue: test-ww_mutex test_cycle_work
> > > [ 5818.986199] Sched_ext: pair (enabled+all), task: runnable_at=-369ms
> > > [ 5818.986202] RIP: 0010:put_prev_task_scx+0x527/0x550
> > > [ 5818.990473] Code: 48 2b 05 34 f3 21 03 75 3c 48 83 c4 30 48 89 de
> > > 48 89 ef b9 ff ff ff ff 5b 31 d2 5d 41 5c 41 5d 41 5e 41 5f e9 fa f0
> > > ff ff 90 <0f> 0b 90 e9 b8 fe ff ff 90 0f 0b 90 e9 35 fe ff ff b8 02 00
> > > 00 00
> > > [ 5818.997428] RSP: 0018:ffffc900088dfb28 EFLAGS: 00010046
> > > [ 5818.999426] RAX: 0000000000000036 RBX: ffff888101258000 RCX: ffff888101d64990
> > > [ 5819.002113] RDX: ffff8881b9bae638 RSI: ffff888101d64990 RDI: ffff888101258390
> > > [ 5819.004787] RBP: ffff8881b9badb00 R08: ffff8881b9badb00 R09: 0000000000000000
> > > [ 5819.007967] R10: 0000000000000036 R11: ffff888120a9b030 R12: ffff888100c30000
> > > [ 5819.010697] R13: ffff88810d163800 R14: ffff888101258000 R15: ffff888100c30000
> > > [ 5819.013378] FS:  0000000000000000(0000) GS:ffff8882355c0000(0000)
> > > knlGS:0000000000000000
> > > [ 5819.016354] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> > > [ 5819.018524] CR2: 00007ffe7b985348 CR3: 0000000123ec6004 CR4: 0000000000370ef0
> > > [ 5819.021163] Call Trace:
> > > [ 5819.022168]  <TASK>
> > > [ 5819.023010]  __schedule+0x15c4/0x2460
> > > [ 5819.024504]  ? find_held_lock+0x2b/0x80
> > > [ 5819.025981]  ? lock_release+0x191/0x310
> > > [ 5819.027503]  schedule+0x3d/0x130
> > > [ 5819.028772]  schedule_preempt_disabled+0x18/0x30
> > > [ 5819.030539]  __ww_mutex_lock.constprop.0+0xaea/0x19e0
> > > [ 5819.032492]  ? schedule_timeout+0xca/0x130
> > > [ 5819.034095]  ? test_cycle_work+0x15d/0x340
> > > [ 5819.035686]  ? ww_mutex_lock+0x3c/0xb0
> > > [ 5819.037094]  ww_mutex_lock+0x3c/0xb0
> > > [ 5819.038494]  test_cycle_work+0x15d/0x340
> > > [ 5819.039949]  process_one_work+0x20c/0x5e0
> > > [ 5819.041631]  ? lock_is_held_type+0xcd/0x130
> > > [ 5819.043277]  worker_thread+0x1a4/0x360
> > > [ 5819.044717]  ? __pfx_worker_thread+0x10/0x10
> > > [ 5819.046392]  kthread+0x103/0x130
> > > [ 5819.047650]  ? __pfx_kthread+0x10/0x10
> > > [ 5819.049074]  ret_from_fork+0x27c/0x340
> > > [ 5819.050525]  ? __pfx_kthread+0x10/0x10
> > > [ 5819.051990]  ret_from_fork_asm+0x1a/0x30
> > > [ 5819.053488]  </TASK>
> > > [ 5819.054371] irq event stamp: 38
> > > [ 5819.055583] hardirqs last  enabled at (37): [<ffffffff826a7100>]
> > > _raw_spin_unlock_irqrestore+0x50/0x60
> > > [ 5819.059054] hardirqs last disabled at (38): [<ffffffff82699205>]
> > > __schedule+0xb95/0x2460
> > > [ 5819.061998] softirqs last  enabled at (26): [<ffffffff813244d0>]
> > > __irq_exit_rcu+0xe0/0x150
> > > [ 5819.065015] softirqs last disabled at (21): [<ffffffff813244d0>]
> > > __irq_exit_rcu+0xe0/0x150
> > > [ 5819.068013] ---[ end trace 0000000000000000 ]---
> > >
> > > It seems to be coming from the proxy_resched_idle() call in the "if
> > > (!READ_ONCE(owner->on_rq) || owner->se.sched_delayed) {" case in
> > > find_proxy_task(), prior to proxy_enqueue_on_owner() calling
> > > block_task().
> > >
> > > I'll have to dig a bit more next week on this, as I'm not yet seeing
> > > whats going wrong here.
> > 
> > Ok, my tired theory is something like:
> > 
> > 1) You have a task A that that is_blocked waiting on a sleeping owner
> > B. It gets enqueued onto that owner and waits.
> > 
> > 2) We start scx_pair, and scx_prepare_task_sched_change() calls
> > sched_proxy_block_task(), which bails because !task_on_rq_queued()
> > 
> > 3) B wakes up, and that causes us to activate_blocked_waiters(), which
> > re-adds A (with is_blocked still set) to the rq
> >    - this is problematic because now we have is_blocked tasks on the
> > sched_ext DSQ where its not allowed.
> > 
> > 4) A is selected as a donor, and maybe B is sleeping again, so we call
> > put_prev_set_next() and hti the warning because we see A is is_blocked
> > when the sched_ext scheduler doesn't support it.
> > 
> > I'll work to prove this out a bit further next week.  I suspect we'll
> > need something somewhere between activate_blocked_waiters() ->
> > scx_do_enqueue_task() to skip enquing of is_blocked tasks when
> > SCX_OPS_ENQ_BLOCKED isn't set.
> 
> Your theory looks correct to me. I was also able to reproduce this. I think the
> problem is that scx_prepare_task_sched_change() calls sched_proxy_block_task(),
> but the latter has nothing to do when the donor is already off the runqueue
> behind a sleeping owner.
> 
> When the owner subsequently wakes, activate_blocked_waiters() unconditionally
> reactivates the donor while is_blocked is still set. Since this activation
> carries ENQUEUE_WAKEUP, sched_ext treats it as a normal wakeup rather than a
> blocked-donor admission. The donor can therefore enter an scx scheduler that
> doesn't set SCX_OPS_ENQ_BLOCKED (scx_pair in this case), eventually triggering
> the warning in put_prev_task_scx(): unexpected proxy donor without
> SCX_OPS_ENQ_BLOCKED set.
> 
> I think we can fix this by checking scx_allow_proxy_exec() in
> do_activate_blocked_waiter() before reactivating the donor. If the scx scheduler
> doesn't support proxy donors (SCX_OPS_ENQ_BLOCKED not set), the task should
> remain blocked and will be activated normally by the mutex wakeup.
> 
> While looking at this, I noticed another issue in the same path:
> activate_blocked_waiters() currently passes ENQUEUE_WAKEUP because the generic
> scheduling classes need wakeup-style enqueue accounting. However, this is not a
> real mutex wakeup, the donor remains blocked and is only being made runnable so
> that it can donate its scheduling context.
> 
> As a result, even a scheduler that sets SCX_OPS_ENQ_BLOCKED currently receives
> this sleeping-owner activation as SCX_ENQ_WAKEUP rather than SCX_ENQ_BLOCKED.
> That's because the existing sched_ext test:
> 
>   p->is_blocked && !(enq_flags & SCX_ENQ_WAKEUP)
> 
> doesn't distinguish a genuine mutex wakeup from this proxy activation.
> 
> Maybe we can address this by adding an internal ENQUEUE_PROXY flag to the
> sleeping-owner activation. The generic classes will continue to see
> ENQUEUE_WAKEUP, preserving their accounting behavior, while sched_ext will use
> ENQUEUE_PROXY to expose the event to BPF as SCX_ENQ_BLOCKED without
> SCX_ENQ_WAKEUP. And a genuine mutex wakeup will continue to be reported as
> SCX_ENQ_WAKEUP. What do you think?

FYI, I've applied your sleeping owner changes + the scx-proxy-exec patch series
with the changes mentioned above here:

 git://git.kernel.org/pub/scm/linux/kernel/git/arighi/linux.git scx-proxy-exec-next

Everything looks good on my side so far.

Thanks,
-Andrea