[PATCHSET sched_ext/for-7.3-fixes] sched_ext: Wait for SCX_OPSS_DISPATCHING on DSQ consume and move paths

[PATCHSET sched_ext/for-7.3-fixes] sched_ext: Wait for SCX_OPSS_DISPATCHING on DSQ consume and move paths

[Kuba Piecuch]

When a dispatch finishes into a non-local DSQ, scx_dispatch_enqueue()
links the task into the DSQ, updates its custody and drops the DSQ lock.
For the global and bypass DSQs, it then calls ops.dequeue() without
holding the task's rq lock. Only after that does it store SCX_OPSS_NONE:

  CPU A (dispatcher)                    CPU B (task_rq(p))
  ------------------                    ------------------
  finish_dispatch()
    QUEUED -> DISPATCHING
    scx_dispatch_enqueue(GLOBAL)
      lock dsq; link p; leave custody
      unlock dsq
                                        do_pick_task_scx()
                                          scx_consume_dispatch_q()
                                            lock dsq; unlink p; unlock dsq
                                            scx_move_local_task_to_local_dsq()
                                              /* modifies p->scx */
      ops.dequeue(p)  /* no rq lock */
      ops_state = NONE

ebf1ccff79c4 ("sched_ext: Fix ops.dequeue() semantics") moved the final
store after the DSQ unlock, so the task can be found on a DSQ while still
in SCX_OPSS_DISPATCHING. cb86607ada73 ("sched_ext: Don't run
ops.dequeue() with a DSQ lock held") also moved ops.dequeue() for
global and bypass DSQs out from under the DSQ lock, and the
scx_move_local_task_to_local_dsq() calls after the unlock as well. Per
the comment in scx_dispatch_enqueue(), running ops.dequeue() without the
rq lock is fine because waiting for SCX_OPSS_DISPATCHING to clear
serializes it against other users of the task.

dequeue_task_scx() and set_next_task_scx() wait in ops_dequeue(), and
7de9a6fb44ea ("sched_ext: Wait for SCX_OPSS_DISPATCHING before
reenqueueing a task") added the wait to the reenqueue paths. The DSQ
consume and move paths still don't wait:

- scx_consume_dispatch_q() -> scx_move_local_task_to_local_dsq() when
  the consuming CPU is task_rq(p), as above. This applies to the global,
  bypass and user DSQs (scx_bpf_dsq_move_to_local()). For a user DSQ, the
  consumer takes the task out of custody and calls ops.dequeue() itself
  while the dispatcher hasn't finished.

- move_task_between_dsqs() (scx_bpf_dsq_move[_vtime]()), which moves the
  task to another local or non-local DSQ.

In both, the consumer modifies p->scx (dsq, dsq_list, flags and, if
scx_resolve_local_dsq() diverts the task to the rescue or reject DSQ,
the reenqueue reason) and may call ops.dequeue() while the dispatcher
can still be running ops.dequeue() on the same task without the rq lock.

The remote consume path, unlink_dsq_and_switch_rq_lock() followed by
move_remote_task_to_local_dsq(), is currently covered by the wait in
deactivate_task() -> ops_dequeue(). The series adds the wait to
unlink_dsq_and_switch_rq_lock() as well, so that callers which act on
the task before or without deactivating it are covered too.

Reproduction
------------

The race is easy to hit by adding

  WARN_ON_ONCE(atomic_long_read(&p->scx.ops_state) != SCX_OPSS_NONE);

to scx_move_local_task_to_local_dsq() and move_task_between_dsqs() and
modifying the dequeue_remote selftest so that ops.dispatch() inserts
tasks whose CPU differs from the dispatching CPU into SCX_DSQ_GLOBAL and
ops.dequeue() kicks the task's CPU and spins for a bit to widen the
window. The warning fires within seconds in a vng guest:

  WARNING: kernel/sched/ext/ext.c:2416 at scx_move_local_task_to_local_dsq+0xf5/0x110, CPU#4: swapper/4/0
  RIP: 0010:scx_move_local_task_to_local_dsq+0xf5/0x110
  RAX: 0000000000000003 ...                    <- SCX_OPSS_DISPATCHING
  Call Trace:
   <TASK>
   scx_consume_dispatch_q+0x13c/0x290
   do_pick_task_scx+0x10a/0x980
   __schedule+0x586/0x1020
   schedule_idle+0x22/0x40
   cpu_startup_entry+0x29/0x30
   start_secondary+0xf3/0x100
   common_startup_64+0x13e/0x158
   </TASK>

With this series applied, the same setup doesn't trigger the warning,
and the dequeue, dequeue_iter and dequeue_remote selftests pass with a
clean dmesg.

Merging into for-7.4
--------------------

Based on sched-ext/for-7.3-fixes. Merging into for-7.4 conflicts in two
places:

- cea8b60e4aee ("sched_ext: Move reject DSQ draining into core") moved
  the reject DSQ drain from sub.c into scx_reenq_reject() in ext.c, which
  needs the rename too.

- 13ae184e4b82 ("sched_ext: Handle proxy-exec races in remote DSQ
  transfers") restructured unlink_dsq_and_switch_rq_lock(). The wait goes
  right after p->scx.dsq is cleared, before returning true. This covers
  the proxy-exec path in consume_remote_task(), which parks the task on
  the reject DSQ without going through ops_dequeue().

Kuba Piecuch (2):
  sched_ext: Rename scx_reenq_wait_dispatching() to
    scx_wait_dispatching()
  sched_ext: Wait for SCX_OPSS_DISPATCHING before moving tasks off DSQs

 kernel/sched/ext/ext.c      | 29 ++++++++++++++++++++---------
 kernel/sched/ext/internal.h |  2 +-
 kernel/sched/ext/sub.c      |  2 +-
 3 files changed, 22 insertions(+), 11 deletions(-)


base-commit: 069403dff337feb192bfc70fb28ddc8a171e20c1
-- 
2.56.0.385.gd3acb90ef8-goog

[PATCH 1/2] sched_ext: Rename scx_reenq_wait_dispatching() to scx_wait_dispatching()

[Kuba Piecuch]

Waiting for SCX_OPSS_DISPATCHING to clear isn't specific to the reenqueue
paths. Any path which takes a task off a DSQ can find it while the
dispatch is still in flight, and the next patch calls the helper from the
DSQ consume and move paths. Rename it to scx_wait_dispatching() and
update its comment accordingly.

No functional change.

Assisted-by: Claude:claude-opus-5.5
Signed-off-by: Kuba Piecuch <jpiecuch@google.com>
---
 kernel/sched/ext/ext.c      | 13 +++++++------
 kernel/sched/ext/internal.h |  2 +-
 kernel/sched/ext/sub.c      |  2 +-
 3 files changed, 9 insertions(+), 8 deletions(-)

diff --git a/kernel/sched/ext/ext.c b/kernel/sched/ext/ext.c
index a6a72ae7f815..acfeadf5503d 100644
--- a/kernel/sched/ext/ext.c
+++ b/kernel/sched/ext/ext.c
@@ -4483,11 +4483,12 @@ static bool local_task_should_reenq(struct rq *rq, struct task_struct *p,
 }
 
 /*
- * The dispatcher stores the final ops_state after dropping the DSQ lock, so @p
- * can be found on a DSQ while still %SCX_OPSS_DISPATCHING. Reenqueueing @p
- * before that store lands would have it clobber the new %SCX_OPSS_QUEUED.
+ * The dispatcher runs ops.dequeue() and stores the final ops_state after
+ * dropping the DSQ lock, so @p can be found on a DSQ while still
+ * %SCX_OPSS_DISPATCHING. Wait for the dispatch to finish before operating on
+ * @p.
  */
-void scx_reenq_wait_dispatching(struct task_struct *p)
+void scx_wait_dispatching(struct task_struct *p)
 {
 	if (unlikely(atomic_long_read_acquire(&p->scx.ops_state) == SCX_OPSS_DISPATCHING))
 		wait_ops_state(p, SCX_OPSS_DISPATCHING);
@@ -4536,7 +4537,7 @@ static u32 reenq_local(struct scx_sched *sch, struct rq *rq, u64 reenq_flags)
 		if (!local_task_should_reenq(rq, p, &reenq_flags, &reason))
 			continue;
 
-		scx_reenq_wait_dispatching(p);
+		scx_wait_dispatching(p);
 		scx_dispatch_dequeue(rq, p);
 
 		if (WARN_ON_ONCE(p->scx.flags & SCX_TASK_REENQ_REASON_MASK))
@@ -4660,7 +4661,7 @@ static void reenq_user(struct rq *rq, struct scx_dispatch_q *dsq, u64 reenq_flag
 		}
 
 		/* @p is on @dsq, its rq and @dsq are locked */
-		scx_reenq_wait_dispatching(p);
+		scx_wait_dispatching(p);
 		dispatch_dequeue_locked(p, dsq);
 		raw_spin_unlock(&dsq->lock);
 
diff --git a/kernel/sched/ext/internal.h b/kernel/sched/ext/internal.h
index 85579fcabf52..352547aedbb7 100644
--- a/kernel/sched/ext/internal.h
+++ b/kernel/sched/ext/internal.h
@@ -2123,7 +2123,7 @@ void scx_kick_cpu(struct scx_sched *sch, s32 cpu, u64 flags);
 u64 __scx_bpf_now(struct rq *rq);
 void schedule_dsq_reenq(struct scx_sched *sch, struct scx_dispatch_q *dsq,
 			u64 reenq_flags, struct rq *locked_rq);
-void scx_reenq_wait_dispatching(struct task_struct *p);
+void scx_wait_dispatching(struct task_struct *p);
 int __scx_init_task(struct scx_sched *sch, struct task_struct *p,
 		    struct cgroup *cgrp, bool fork);
 void scx_enable_task(struct scx_sched *sch, struct task_struct *p);
diff --git a/kernel/sched/ext/sub.c b/kernel/sched/ext/sub.c
index c9b44ab3bd0a..fee75fcdb59d 100644
--- a/kernel/sched/ext/sub.c
+++ b/kernel/sched/ext/sub.c
@@ -797,7 +797,7 @@ void scx_reenq_reject(struct rq *rq)
 		if (WARN_ON_ONCE(p->migration_pending))
 			continue;
 
-		scx_reenq_wait_dispatching(p);
+		scx_wait_dispatching(p);
 		scx_dispatch_dequeue(rq, p);
 
 		if (WARN_ON_ONCE(p->scx.flags & SCX_TASK_REENQ_REASON_MASK))
-- 
2.56.0.385.gd3acb90ef8-goog

[PATCH 2/2] sched_ext: Wait for SCX_OPSS_DISPATCHING before moving tasks off DSQs

[Kuba Piecuch]

A task can be found on a non-local DSQ while the CPU which dispatched it
is still running ops.dequeue() and hasn't cleared SCX_OPSS_DISPATCHING.
The DSQ consume and move paths don't wait for the dispatch to finish and
can operate on the task concurrently with the dispatcher.

Call scx_wait_dispatching() in scx_move_local_task_to_local_dsq(),
unlink_dsq_and_switch_rq_lock() and move_task_between_dsqs() before they
operate on the task. In scx_move_local_task_to_local_dsq(), wait before
scx_resolve_local_dsq() as it may update p->scx.flags.

Fixes: ebf1ccff79c4 ("sched_ext: Fix ops.dequeue() semantics")
Fixes: cb86607ada73 ("sched_ext: Don't run ops.dequeue() with a DSQ lock held")
Cc: stable@vger.kernel.org # v7.1+
Assisted-by: Claude:claude-opus-5.5
Signed-off-by: Kuba Piecuch <jpiecuch@google.com>
---
 kernel/sched/ext/ext.c | 16 +++++++++++++---
 1 file changed, 13 insertions(+), 3 deletions(-)

diff --git a/kernel/sched/ext/ext.c b/kernel/sched/ext/ext.c
index acfeadf5503d..d2a69dfda9cc 100644
--- a/kernel/sched/ext/ext.c
+++ b/kernel/sched/ext/ext.c
@@ -2407,11 +2407,14 @@ static void wakeup_preempt_scx(struct rq *rq, struct task_struct *p, int wake_fl
 void scx_move_local_task_to_local_dsq(struct scx_sched *sch, struct task_struct *p,
 				      u64 enq_flags, struct rq *dst_rq)
 {
-	struct scx_dispatch_q *dst_dsq = scx_resolve_local_dsq(sch, dst_rq, p, &enq_flags);
+	struct scx_dispatch_q *dst_dsq;
 
 	lockdep_assert_rq_held(dst_rq);
 
 	WARN_ON_ONCE(p->scx.holding_cpu >= 0);
+	scx_wait_dispatching(p);
+
+	dst_dsq = scx_resolve_local_dsq(sch, dst_rq, p, &enq_flags);
 
 	if (enq_flags & (SCX_ENQ_HEAD | SCX_ENQ_PREEMPT))
 		dsq_insert_head(dst_dsq, p);
@@ -2591,8 +2594,13 @@ static bool unlink_dsq_and_switch_rq_lock(struct task_struct *p,
 	switch_rq_lock(locked_rq, src_rq);
 
 	/* task_rq couldn't have changed if we're still the holding cpu */
-	return likely(p->scx.holding_cpu == cpu) &&
-		!WARN_ON_ONCE(src_rq != task_rq(p));
+	if (likely(p->scx.holding_cpu == cpu) &&
+	    !WARN_ON_ONCE(src_rq != task_rq(p))) {
+		scx_wait_dispatching(p);
+		return true;
+	}
+
+	return false;
 }
 
 static bool consume_remote_task(struct scx_sched *sch, struct rq *this_rq,
@@ -2635,6 +2643,8 @@ static struct rq *move_task_between_dsqs(struct scx_sched *sch,
 	lockdep_assert_held(&src_dsq->lock);
 	lockdep_assert_rq_held(src_rq);
 
+	scx_wait_dispatching(p);
+
 	if (dst_dsq->id == SCX_DSQ_LOCAL) {
 		dst_rq = container_of(dst_dsq, struct rq, scx.local_dsq);
 		if (src_rq != dst_rq &&
-- 
2.56.0.385.gd3acb90ef8-goog

Re: [PATCHSET sched_ext/for-7.3-fixes] sched_ext: Wait for SCX_OPSS_DISPATCHING on DSQ consume and move paths

[Andrea Righi]

On Mon, Oct 12, 2026 at 09:07:44PM +0000, Kuba Piecuch wrote:
> The DSQ consume and move paths still don't wait:

Thanks Kuba. The race is real, and the new waits cover the affected
consume and move paths. I don't see any regressions from this series.

For both patches:

Reviewed-by: Andrea Righi <arighi@nvidia.com>

Re: [PATCHSET sched_ext/for-7.3-fixes] sched_ext: Wait for SCX_OPSS_DISPATCHING on DSQ consume and move paths

[Tejun Heo]

Hello,

> Kuba Piecuch (2):
>   sched_ext: Rename scx_reenq_wait_dispatching() to
>     scx_wait_dispatching()
>   sched_ext: Wait for SCX_OPSS_DISPATCHING before moving tasks off DSQs

Applied 1-2 to sched_ext/for-7.3-fixes with Andrea's Reviewed-by added and
the holding_cpu condition in unlink_dsq_and_switch_rq_lock() joined on one
line.

Thanks.

--
tejun