Re: [REPORT] sched_ext: sub-scheduler tasks are severely under-scheduled vs root-owned tasks

[Tao Cui <cui.tao@linux.dev>]

Hello， Tejun

在 2026/10/10 7:42, Tejun Heo 写道:
> Hello, Tao.
> 
> The following is a Claude-generated analysis.
> 
> On Fri, Oct 09, 2026 at 09:51:35AM +0800, Tao Cui wrote:
>> The scheduler is a single binary (scx_k2) with parent/root and
>> child/sub modes. The parent grants SCX_CAP_PERF | SCX_CAP_ENQ_IMMED
>> to the child over a cgroup, while the child only writes cpuperf
>> target 1 from ops.tick(). Both schedulers print their counters every
>> 2s.
> 
> Reproduced on for-7.4 with your sources in a 4 cpu VM. The starvation
> comes from the two schedulers, not the kernel.
> 
> The child's ops.enqueue() inserts every task into its own user DSQ. The
> only consumer of that DSQ is the child's ops.dispatch(), and a child's
> dispatch runs only when its parent calls scx_bpf_sub_dispatch() from its
> own ops.dispatch(). scx_k2's parent never does. See the comment above
> enum scx_cap_flags in kernel/sched/ext/internal.h and how scx_qmap
> dispatches its children. The SysRq-D dump during the stall shows the
> spinner in the child's DSQ 0 with a full slice while cpu0 runs root
> tasks.
> 
>> In an A/B test, having the parent kick the task's CPU from
>> ops.enqueue() with scx_bpf_kick_cid() and hand its dispatch turn to
>> the child with scx_bpf_sub_dispatch() prevents the starvation.
> 
> scx_bpf_sub_dispatch() is only callable from ops.dispatch(). With the
> parent calling it there, the child's dispatch does run, but every
> scx_bpf_dsq_move_to_local() is rejected because the grant lacks
> SCX_CAP_ENQ. SCX_CAP_ENQ_IMMED only covers IMMED inserts. The task goes
> to the reject DSQ and is reenqueued, SCX_EV_REENQ_REPEAT climbs on the
> child, and it still starves. With the sub dispatch call and SCX_CAP_ENQ
> granted as well, the spinner runs at full duty under the child.
> 
>> Actual: the child claims the task, and its ops.tick() initially
>> services it, but setcnt then stops increasing and the task receives
>> little or no CPU time - 0-91 ticks per 6s window across our runs,
>> most rounds 0.
> 
> The initial burst is the bypass DSQ drained during attach. In the
> move-after-attach order the task keeps running through dispatch's
> keep-last path until something else wakes on cpu0 and pushes it through
> the child's ops.enqueue(), which is why some windows look healthy. The
> stall watchdog catches it past your 6s window:
> 
>   sched_ext: BPF sub-scheduler "k2" disabled (runnable task stall)
>   sched_ext: k2: sh[1401] failed to run for 30.696s
> 
>> One possibility is that, on a NO_HZ idle CPU, no balance is
>> triggered for the child's DSQ, so the child's ops.dispatch() is
>> never called.
> 
> The parent's dispatch on cid 0 ran several times a second throughout
> the stall. It only consumed the parent's own DSQ.
> 

Thanks a lot for taking the time to reproduce this and for the detailed
analysis.

I completely missed that the child's ops.dispatch() only runs when the
parent explicitly delegates to it with scx_bpf_sub_dispatch(). I had
been assuming the core would eventually route balance to the child once
it owned the task, so I ended up chasing the wrong direction.

The SCX_CAP_ENQ vs SCX_CAP_ENQ_IMMED distinction also explains why my
A/B experiment wasn't actually proving what I thought it was. I hadn't
noticed that scx_bpf_dsq_move_to_local() requires SCX_CAP_ENQ, so the
reject/reenqueue path makes perfect sense now.

I'll update the scheduler to dispatch to the child as intended and
grant SCX_CAP_ENQ where appropriate, then rerun the tests.

Thanks again for the thorough explanation and for pointing me to the
internal.h comments and the scx_qmap implementation.

--
Tao

> Thanks.
>