Re: [PATCH sched_ext/for-7.1] sched_ext: Documentation: Add missing calls to quiescent(), runnable()

[Andrea Righi <arighi@nvidia.com>]

On Thu, Apr 09, 2026 at 01:30:55PM +0000, Kuba Piecuch wrote:
> On Thu Apr 9, 2026 at 9:46 AM UTC, Christian Loehle wrote:
> ...
> >>>
> >>>     ops.init_task();            /* A new task is created */
> >>>     ops.enable();               /* Enable BPF scheduling for the task */
> >>>
> >>>     while (task in SCHED_EXT) {
> >>>         if (task can migrate)
> >>>             ops.select_cpu();   /* Called on wakeup (optimization) */
> >>>
> >>>         ops.runnable();         /* Task becomes ready to run */
> >>>
> >>>         while (task_is_runnable(task)) {
> >>>             if (task is not in a DSQ || task->scx.slice == 0) {
> >>>                 ops.enqueue();  /* Task can be added to a DSQ */
> >>>
> >>>                 /* Task property change (i.e., affinity, nice, etc.)? */
> >>>                 if (sched_change(task)) {
> >>>                     ops.dequeue(); /* Exiting BPF scheduler custody */
> >>>                     ops.quiescent();
> >>>
> >>>                     /* Property change callback, e.g. ops.set_weight() */
> >>>
> >>>                     ops.runnable();
> >>>                     continue;
> >>>                 }
> >>>
> >>>                 /* Any usable CPU becomes available */
> >>>
> >>>                 ops.dispatch();     /* Task is moved to a local DSQ */
> >>>                 ops.dequeue();      /* Exiting BPF scheduler custody */
> > Is this true here? Any dispatch followed by a dequeue?
> 
> The comment next to ops.dispatch() says the task is moved to a local DSQ,
> so if we assume that, then I think it will always be followed by ops.dequeue().
> Same if we move the task to the global DSQ.

So, ops.dispatch() is not a "task callback", it's a "CPU callback", invoked when
a CPU becomes available. So having ops.dispatch() here can be a bit confusing.

The intent was to describe the workflow where, once the task is enqueued to a
non-terminal DSQ, then it can be consumed by an ops.dispatch() event and, in
that case, ops.dequeue() is also invoked when the task reaches a terminal DSQ.
Not sure if there's a better way to express this concept in the pseudocode.

> 
> Of course, you could do something weird like dispatch the task to a user DSQ,
> in which case there won't be a dequeue and the task won't start running, but
> that's weird enough that I don't think we need to consider it.

Right. For the records, scx_rustland_core does something similar: from
ops.dispatch() it consumes a task from a BPF user ringbuffer, inserts it into a
user DSQ and then consumes the first task from the user DSQ via
scx_bpf_dsq_move_to_local().

But that's a bit of a special use case, due to the unusual user-space scheduling
part. But in this case the pseudocode is still accurate, since the
scx_bpf_dsq_move_to_local() triggers an ops.dequeue().

> 
> You could also have a property change racing with the dispatch which would make
> the dispatch fail and not be followed by a dequeue, but again, we need to draw
> the line somewhere.

Yeah, the thing is that sched_change() introduces a lot of different edge cases
that are difficult to represent in the pseudocode. I guess the best we can do is
document in a descriptive manner the concept of "BPF scheduler's custody" and
the fact that a task can temporarily leave the custody when a sched_change()
event happens.

> 
> So, in other words, any _successful_ dispatch to a _terminal_ DSQ is always
> followed by a dequeue.
> 
> Another case that isn't handled here is direct dispatch to a terminal DSQ from
> ops.enqueue(), where we don't get ops.dispatch() or ops.dequeue() and go
> straight to ops.running(). If any of the above cases should be handled in the
> pseudocode, I'd say it's this one.

Right, in fact it should be: any _successful_ dispatch to a _terminal DSQ_, if
the task was in the BPF scheduler's custody. A direct dispatch to a terminal DSQ
either from ops.select_cpu() or ops.enqueue() doesn't trigger ops.dequeue(),
because the task doesn't enter the BPF scheduler's custody.

Thanks,
-Andrea