[RFC][PATCH 0/6] clean up class switching

[RFC][PATCH 0/6] clean up class switching

[Peter Zijlstra]

Hi!

Recent changes (scx and especially sched_delayed) made bit of a mess of
sched_class changes.

The first patch fixes an scx bug (some day I'll remember scx), while the rest
cleans up the whole sched_class::switch*()-eroo stuff.

Patches are very lightly tested (as in, the test box boots).

[PATCH 1/6] sched/ext: Fix scx vs sched_delayed

[Peter Zijlstra]

Commit 98442f0ccd82 ("sched: Fix delayed_dequeue vs
switched_from_fair()") forgot about scx :/

Fixes: 98442f0ccd82 ("sched: Fix delayed_dequeue vs switched_from_fair()")
Reported-by: Tejun Heo <tj@kernel.org>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Link: https://lkml.kernel.org/r/20241030104934.GK14555@noisy.programming.kicks-ass.net
---
 kernel/sched/ext.c |   14 ++++++++++++--
 1 file changed, 12 insertions(+), 2 deletions(-)

--- a/kernel/sched/ext.c
+++ b/kernel/sched/ext.c
@@ -4489,11 +4489,16 @@ static void scx_ops_disable_workfn(struc
 	scx_task_iter_start(&sti);
 	while ((p = scx_task_iter_next_locked(&sti))) {
 		const struct sched_class *old_class = p->sched_class;
+		const struct sched_class *new_class =
+			__setscheduler_class(p->policy, p->prio);
 		struct sched_enq_and_set_ctx ctx;
 
+		if (old_class != new_class && p->se.sched_delayed)
+			dequeue_task(task_rq(p), p, DEQUEUE_SLEEP | DEQUEUE_DELAYED);
+
 		sched_deq_and_put_task(p, DEQUEUE_SAVE | DEQUEUE_MOVE, &ctx);
 
-		p->sched_class = __setscheduler_class(p->policy, p->prio);
+		p->sched_class = new_class;
 		check_class_changing(task_rq(p), p, old_class);
 
 		sched_enq_and_set_task(&ctx);
@@ -5199,12 +5204,17 @@ static int scx_ops_enable(struct sched_e
 	scx_task_iter_start(&sti);
 	while ((p = scx_task_iter_next_locked(&sti))) {
 		const struct sched_class *old_class = p->sched_class;
+		const struct sched_class *new_class =
+			__setscheduler_class(p->policy, p->prio);
 		struct sched_enq_and_set_ctx ctx;
 
+		if (old_class != new_class && p->se.sched_delayed)
+			dequeue_task(task_rq(p), p, DEQUEUE_SLEEP | DEQUEE_DELAYED);
+
 		sched_deq_and_put_task(p, DEQUEUE_SAVE | DEQUEUE_MOVE, &ctx);
 
 		p->scx.slice = SCX_SLICE_DFL;
-		p->sched_class = __setscheduler_class(p->policy, p->prio);
+		p->sched_class = new_class;
 		check_class_changing(task_rq(p), p, old_class);
 
 		sched_enq_and_set_task(&ctx);

Re: [PATCH 1/6] sched/ext: Fix scx vs sched_delayed

[Christian Loehle]

On 10/30/24 15:12, Peter Zijlstra wrote:
> Commit 98442f0ccd82 ("sched: Fix delayed_dequeue vs
> switched_from_fair()") forgot about scx :/
> 
> Fixes: 98442f0ccd82 ("sched: Fix delayed_dequeue vs switched_from_fair()")
> Reported-by: Tejun Heo <tj@kernel.org>
> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
> Link: https://lkml.kernel.org/r/20241030104934.GK14555@noisy.programming.kicks-ass.net
> ---
>  kernel/sched/ext.c |   14 ++++++++++++--
>  1 file changed, 12 insertions(+), 2 deletions(-)
> 
> --- a/kernel/sched/ext.c
> +++ b/kernel/sched/ext.c
> @@ -4489,11 +4489,16 @@ static void scx_ops_disable_workfn(struc
>  	scx_task_iter_start(&sti);
>  	while ((p = scx_task_iter_next_locked(&sti))) {
>  		const struct sched_class *old_class = p->sched_class;
> +		const struct sched_class *new_class =
> +			__setscheduler_class(p->policy, p->prio);
>  		struct sched_enq_and_set_ctx ctx;
>  
> +		if (old_class != new_class && p->se.sched_delayed)
> +			dequeue_task(task_rq(p), p, DEQUEUE_SLEEP | DEQUEUE_DELAYED);
> +
>  		sched_deq_and_put_task(p, DEQUEUE_SAVE | DEQUEUE_MOVE, &ctx);
>  
> -		p->sched_class = __setscheduler_class(p->policy, p->prio);
> +		p->sched_class = new_class;
>  		check_class_changing(task_rq(p), p, old_class);
>  
>  		sched_enq_and_set_task(&ctx);
> @@ -5199,12 +5204,17 @@ static int scx_ops_enable(struct sched_e
>  	scx_task_iter_start(&sti);
>  	while ((p = scx_task_iter_next_locked(&sti))) {
>  		const struct sched_class *old_class = p->sched_class;
> +		const struct sched_class *new_class =
> +			__setscheduler_class(p->policy, p->prio);
>  		struct sched_enq_and_set_ctx ctx;
>  
> +		if (old_class != new_class && p->se.sched_delayed)
> +			dequeue_task(task_rq(p), p, DEQUEUE_SLEEP | DEQUEE_DELAYED);
> +

s/DEQUEE_DELAYED/DEQUEUE_DELAYED
Anyway, no luck for me applying the series onto sched/core scx's for-next or rc5.
Any hint or do you mind supplying a branch?

Re: [PATCH 1/6] sched/ext: Fix scx vs sched_delayed

[Peter Zijlstra]

On Wed, Oct 30, 2024 at 03:50:05PM +0000, Christian Loehle wrote:
> On 10/30/24 15:12, Peter Zijlstra wrote:
> > Commit 98442f0ccd82 ("sched: Fix delayed_dequeue vs
> > switched_from_fair()") forgot about scx :/
> > 
> > Fixes: 98442f0ccd82 ("sched: Fix delayed_dequeue vs switched_from_fair()")
> > Reported-by: Tejun Heo <tj@kernel.org>
> > Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
> > Link: https://lkml.kernel.org/r/20241030104934.GK14555@noisy.programming.kicks-ass.net
> > ---
> >  kernel/sched/ext.c |   14 ++++++++++++--
> >  1 file changed, 12 insertions(+), 2 deletions(-)
> > 
> > --- a/kernel/sched/ext.c
> > +++ b/kernel/sched/ext.c
> > @@ -4489,11 +4489,16 @@ static void scx_ops_disable_workfn(struc
> >  	scx_task_iter_start(&sti);
> >  	while ((p = scx_task_iter_next_locked(&sti))) {
> >  		const struct sched_class *old_class = p->sched_class;
> > +		const struct sched_class *new_class =
> > +			__setscheduler_class(p->policy, p->prio);
> >  		struct sched_enq_and_set_ctx ctx;
> >  
> > +		if (old_class != new_class && p->se.sched_delayed)
> > +			dequeue_task(task_rq(p), p, DEQUEUE_SLEEP | DEQUEUE_DELAYED);
> > +
> >  		sched_deq_and_put_task(p, DEQUEUE_SAVE | DEQUEUE_MOVE, &ctx);
> >  
> > -		p->sched_class = __setscheduler_class(p->policy, p->prio);
> > +		p->sched_class = new_class;
> >  		check_class_changing(task_rq(p), p, old_class);
> >  
> >  		sched_enq_and_set_task(&ctx);
> > @@ -5199,12 +5204,17 @@ static int scx_ops_enable(struct sched_e
> >  	scx_task_iter_start(&sti);
> >  	while ((p = scx_task_iter_next_locked(&sti))) {
> >  		const struct sched_class *old_class = p->sched_class;
> > +		const struct sched_class *new_class =
> > +			__setscheduler_class(p->policy, p->prio);
> >  		struct sched_enq_and_set_ctx ctx;
> >  
> > +		if (old_class != new_class && p->se.sched_delayed)
> > +			dequeue_task(task_rq(p), p, DEQUEUE_SLEEP | DEQUEE_DELAYED);
> > +
> 
> s/DEQUEE_DELAYED/DEQUEUE_DELAYED

Bah, typing so hard..

> Anyway, no luck for me applying the series onto sched/core scx's for-next or rc5.
> Any hint or do you mind supplying a branch?

I think I did it on top of tip/master..

Re: [PATCH 1/6] sched/ext: Fix scx vs sched_delayed

[Christian Loehle]

On 10/30/24 15:54, Peter Zijlstra wrote:
> On Wed, Oct 30, 2024 at 03:50:05PM +0000, Christian Loehle wrote:
>> On 10/30/24 15:12, Peter Zijlstra wrote:
>>> Commit 98442f0ccd82 ("sched: Fix delayed_dequeue vs
>>> switched_from_fair()") forgot about scx :/
>>>
>>> Fixes: 98442f0ccd82 ("sched: Fix delayed_dequeue vs switched_from_fair()")
>>> Reported-by: Tejun Heo <tj@kernel.org>
>>> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
>>> Link: https://lkml.kernel.org/r/20241030104934.GK14555@noisy.programming.kicks-ass.net
>>> ---
>>>  kernel/sched/ext.c |   14 ++++++++++++--
>>>  1 file changed, 12 insertions(+), 2 deletions(-)
>>>
>>> --- a/kernel/sched/ext.c
>>> +++ b/kernel/sched/ext.c
>>> @@ -4489,11 +4489,16 @@ static void scx_ops_disable_workfn(struc
>>>  	scx_task_iter_start(&sti);
>>>  	while ((p = scx_task_iter_next_locked(&sti))) {
>>>  		const struct sched_class *old_class = p->sched_class;
>>> +		const struct sched_class *new_class =
>>> +			__setscheduler_class(p->policy, p->prio);
>>>  		struct sched_enq_and_set_ctx ctx;
>>>  
>>> +		if (old_class != new_class && p->se.sched_delayed)
>>> +			dequeue_task(task_rq(p), p, DEQUEUE_SLEEP | DEQUEUE_DELAYED);
>>> +
>>>  		sched_deq_and_put_task(p, DEQUEUE_SAVE | DEQUEUE_MOVE, &ctx);
>>>  
>>> -		p->sched_class = __setscheduler_class(p->policy, p->prio);
>>> +		p->sched_class = new_class;
>>>  		check_class_changing(task_rq(p), p, old_class);
>>>  
>>>  		sched_enq_and_set_task(&ctx);
>>> @@ -5199,12 +5204,17 @@ static int scx_ops_enable(struct sched_e
>>>  	scx_task_iter_start(&sti);
>>>  	while ((p = scx_task_iter_next_locked(&sti))) {
>>>  		const struct sched_class *old_class = p->sched_class;
>>> +		const struct sched_class *new_class =
>>> +			__setscheduler_class(p->policy, p->prio);
>>>  		struct sched_enq_and_set_ctx ctx;
>>>  
>>> +		if (old_class != new_class && p->se.sched_delayed)
>>> +			dequeue_task(task_rq(p), p, DEQUEUE_SLEEP | DEQUEE_DELAYED);
>>> +
>>
>> s/DEQUEE_DELAYED/DEQUEUE_DELAYED
> 
> Bah, typing so hard..
> 
>> Anyway, no luck for me applying the series onto sched/core scx's for-next or rc5.
>> Any hint or do you mind supplying a branch?
> 
> I think I did it on top of tip/master..

0b9c113ad1c2 (tip/master) Merge branch into tip/master: 'x86/sev'
results in the following diff for me (line number mismatch by 5), stops applying from 2/6:
-- 

diff --git a/kernel/sched/ext.c b/kernel/sched/ext.c
index 65334c13ffa5..2b5f1742299a 100644
--- a/kernel/sched/ext.c
+++ b/kernel/sched/ext.c
@@ -4494,11 +4494,16 @@ static void scx_ops_disable_workfn(struct kthread_work *work)
        scx_task_iter_start(&sti);
        while ((p = scx_task_iter_next_locked(&sti))) {
                const struct sched_class *old_class = p->sched_class;
+               const struct sched_class *new_class =
+                       __setscheduler_class(p->policy, p->prio);
                struct sched_enq_and_set_ctx ctx;
 
+               if (old_class != new_class && p->se.sched_delayed)
+                       dequeue_task(task_rq(p), p, DEQUEUE_SLEEP | DEQUEUE_DELAYED);
+
                sched_deq_and_put_task(p, DEQUEUE_SAVE | DEQUEUE_MOVE, &ctx);
 
-               p->sched_class = __setscheduler_class(p->policy, p->prio);
+               p->sched_class = new_class;
                check_class_changing(task_rq(p), p, old_class);
 
                sched_enq_and_set_task(&ctx);

Re: [PATCH 1/6] sched/ext: Fix scx vs sched_delayed

[Tejun Heo]

On Wed, Oct 30, 2024 at 04:12:56PM +0100, Peter Zijlstra wrote:
> Commit 98442f0ccd82 ("sched: Fix delayed_dequeue vs
> switched_from_fair()") forgot about scx :/
> 
> Fixes: 98442f0ccd82 ("sched: Fix delayed_dequeue vs switched_from_fair()")
> Reported-by: Tejun Heo <tj@kernel.org>
> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
> Link: https://lkml.kernel.org/r/20241030104934.GK14555@noisy.programming.kicks-ass.net

Sans the compile issue:

 Acked-by: Tejun Heo <tj@kernel.org>

Thanks.

-- 
tejun

[RFC][PATCH 2/6] sched: Employ sched_change guards

[Peter Zijlstra]

As proposed a long while ago -- and half done by scx -- wrap the
scheduler's 'change' pattern in a guard helper.

Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
---
 include/linux/cleanup.h |    5 +
 kernel/sched/core.c     |  158 ++++++++++++++++++------------------------------
 kernel/sched/ext.c      |   33 +++-------
 kernel/sched/sched.h    |   21 +++---
 kernel/sched/syscalls.c |   65 ++++++-------------
 5 files changed, 112 insertions(+), 170 deletions(-)

--- a/include/linux/cleanup.h
+++ b/include/linux/cleanup.h
@@ -297,6 +297,11 @@ static inline class_##_name##_t class_##
 #define __DEFINE_CLASS_IS_CONDITIONAL(_name, _is_cond)	\
 static __maybe_unused const bool class_##_name##_is_conditional = _is_cond
 
+#define DEFINE_CLASS_IS_UNCONDITIONAL(_name)		\
+	__DEFINE_CLASS_IS_CONDITIONAL(_name, false);	\
+	static inline void * class_##_name##_lock_ptr(class_##_name##_t *_T) \
+	{ return (void *)1; }
+
 #define DEFINE_GUARD(_name, _type, _lock, _unlock) \
 	__DEFINE_CLASS_IS_CONDITIONAL(_name, false); \
 	DEFINE_CLASS(_name, _type, if (_T) { _unlock; }, ({ _lock; _T; }), _type _T); \
--- a/kernel/sched/core.c
+++ b/kernel/sched/core.c
@@ -7099,7 +7099,7 @@ void rt_mutex_post_schedule(void)
  */
 void rt_mutex_setprio(struct task_struct *p, struct task_struct *pi_task)
 {
-	int prio, oldprio, queued, running, queue_flag =
+	int prio, oldprio, queue_flag =
 		DEQUEUE_SAVE | DEQUEUE_MOVE | DEQUEUE_NOCLOCK;
 	const struct sched_class *prev_class, *next_class;
 	struct rq_flags rf;
@@ -7164,52 +7164,42 @@ void rt_mutex_setprio(struct task_struct
 	if (prev_class != next_class && p->se.sched_delayed)
 		dequeue_task(rq, p, DEQUEUE_SLEEP | DEQUEUE_DELAYED | DEQUEUE_NOCLOCK);
 
-	queued = task_on_rq_queued(p);
-	running = task_current_donor(rq, p);
-	if (queued)
-		dequeue_task(rq, p, queue_flag);
-	if (running)
-		put_prev_task(rq, p);
-
-	/*
-	 * Boosting condition are:
-	 * 1. -rt task is running and holds mutex A
-	 *      --> -dl task blocks on mutex A
-	 *
-	 * 2. -dl task is running and holds mutex A
-	 *      --> -dl task blocks on mutex A and could preempt the
-	 *          running task
-	 */
-	if (dl_prio(prio)) {
-		if (!dl_prio(p->normal_prio) ||
-		    (pi_task && dl_prio(pi_task->prio) &&
-		     dl_entity_preempt(&pi_task->dl, &p->dl))) {
-			p->dl.pi_se = pi_task->dl.pi_se;
-			queue_flag |= ENQUEUE_REPLENISH;
+	scoped_guard (sched_change, p, queue_flag) {
+		/*
+		 * Boosting condition are:
+		 * 1. -rt task is running and holds mutex A
+		 *      --> -dl task blocks on mutex A
+		 *
+		 * 2. -dl task is running and holds mutex A
+		 *      --> -dl task blocks on mutex A and could preempt the
+		 *          running task
+		 */
+		if (dl_prio(prio)) {
+			if (!dl_prio(p->normal_prio) ||
+			    (pi_task && dl_prio(pi_task->prio) &&
+			     dl_entity_preempt(&pi_task->dl, &p->dl))) {
+				p->dl.pi_se = pi_task->dl.pi_se;
+				scope.flags |= ENQUEUE_REPLENISH;
+			} else {
+				p->dl.pi_se = &p->dl;
+			}
+		} else if (rt_prio(prio)) {
+			if (dl_prio(oldprio))
+				p->dl.pi_se = &p->dl;
+			if (oldprio < prio)
+				scope.flags |= ENQUEUE_HEAD;
 		} else {
-			p->dl.pi_se = &p->dl;
+			if (dl_prio(oldprio))
+				p->dl.pi_se = &p->dl;
+			if (rt_prio(oldprio))
+				p->rt.timeout = 0;
 		}
-	} else if (rt_prio(prio)) {
-		if (dl_prio(oldprio))
-			p->dl.pi_se = &p->dl;
-		if (oldprio < prio)
-			queue_flag |= ENQUEUE_HEAD;
-	} else {
-		if (dl_prio(oldprio))
-			p->dl.pi_se = &p->dl;
-		if (rt_prio(oldprio))
-			p->rt.timeout = 0;
-	}
-
-	p->sched_class = next_class;
-	p->prio = prio;
 
-	check_class_changing(rq, p, prev_class);
+		p->sched_class = next_class;
+		p->prio = prio;
 
-	if (queued)
-		enqueue_task(rq, p, queue_flag);
-	if (running)
-		set_next_task(rq, p);
+		check_class_changing(rq, p, prev_class);
+	}
 
 	check_class_changed(rq, p, prev_class, oldprio);
 out_unlock:
@@ -7819,26 +7809,9 @@ int migrate_task_to(struct task_struct *
  */
 void sched_setnuma(struct task_struct *p, int nid)
 {
-	bool queued, running;
-	struct rq_flags rf;
-	struct rq *rq;
-
-	rq = task_rq_lock(p, &rf);
-	queued = task_on_rq_queued(p);
-	running = task_current_donor(rq, p);
-
-	if (queued)
-		dequeue_task(rq, p, DEQUEUE_SAVE);
-	if (running)
-		put_prev_task(rq, p);
-
-	p->numa_preferred_nid = nid;
-
-	if (queued)
-		enqueue_task(rq, p, ENQUEUE_RESTORE | ENQUEUE_NOCLOCK);
-	if (running)
-		set_next_task(rq, p);
-	task_rq_unlock(rq, p, &rf);
+	guard(task_rq_lock)(p);
+	scoped_guard (sched_change, p, DEQUEUE_SAVE)
+		p->numa_preferred_nid = nid;
 }
 #endif /* CONFIG_NUMA_BALANCING */
 
@@ -8957,9 +8930,10 @@ static void sched_change_group(struct ta
  */
 void sched_move_task(struct task_struct *tsk)
 {
-	int queued, running, queue_flags =
+	unsigned int queue_flags =
 		DEQUEUE_SAVE | DEQUEUE_MOVE | DEQUEUE_NOCLOCK;
 	struct task_group *group;
+	bool resched = false;
 	struct rq *rq;
 
 	CLASS(task_rq_lock, rq_guard)(tsk);
@@ -8975,21 +8949,14 @@ void sched_move_task(struct task_struct
 
 	update_rq_clock(rq);
 
-	running = task_current_donor(rq, tsk);
-	queued = task_on_rq_queued(tsk);
+	scoped_guard (sched_change, tsk, queue_flags) {
+		sched_change_group(tsk, group);
+		scx_move_task(tsk);
+		if (scope.running)
+			resched = true;
+	}
 
-	if (queued)
-		dequeue_task(rq, tsk, queue_flags);
-	if (running)
-		put_prev_task(rq, tsk);
-
-	sched_change_group(tsk, group);
-	scx_move_task(tsk);
-
-	if (queued)
-		enqueue_task(rq, tsk, queue_flags);
-	if (running) {
-		set_next_task(rq, tsk);
+	if (resched) {
 		/*
 		 * After changing group, the running task may have joined a
 		 * throttled one but it's still the running task. Trigger a
@@ -10580,37 +10547,34 @@ void sched_mm_cid_fork(struct task_struc
 }
 #endif
 
-#ifdef CONFIG_SCHED_CLASS_EXT
-void sched_deq_and_put_task(struct task_struct *p, int queue_flags,
-			    struct sched_enq_and_set_ctx *ctx)
+struct sched_change_ctx sched_change_begin(struct task_struct *p, unsigned int flags)
 {
 	struct rq *rq = task_rq(p);
-
-	lockdep_assert_rq_held(rq);
-
-	*ctx = (struct sched_enq_and_set_ctx){
+	struct sched_change_ctx ctx = {
 		.p = p,
-		.queue_flags = queue_flags,
+		.flags = flags,
 		.queued = task_on_rq_queued(p),
 		.running = task_current(rq, p),
 	};
 
-	update_rq_clock(rq);
-	if (ctx->queued)
-		dequeue_task(rq, p, queue_flags | DEQUEUE_NOCLOCK);
-	if (ctx->running)
+	lockdep_assert_rq_held(rq);
+
+	if (ctx.queued)
+		dequeue_task(rq, p, flags);
+	if (ctx.running)
 		put_prev_task(rq, p);
+
+	return ctx;
 }
 
-void sched_enq_and_set_task(struct sched_enq_and_set_ctx *ctx)
+void sched_change_end(struct sched_change_ctx ctx)
 {
-	struct rq *rq = task_rq(ctx->p);
+	struct rq *rq = task_rq(ctx.p);
 
 	lockdep_assert_rq_held(rq);
 
-	if (ctx->queued)
-		enqueue_task(rq, ctx->p, ctx->queue_flags | ENQUEUE_NOCLOCK);
-	if (ctx->running)
-		set_next_task(rq, ctx->p);
+	if (ctx.queued)
+		enqueue_task(rq, ctx.p, ctx.flags | ENQUEUE_NOCLOCK);
+	if (ctx.running)
+		set_next_task(rq, ctx.p);
 }
-#endif	/* CONFIG_SCHED_CLASS_EXT */
--- a/kernel/sched/ext.c
+++ b/kernel/sched/ext.c
@@ -4355,11 +4355,10 @@ static void scx_ops_bypass(bool bypass)
 		 */
 		list_for_each_entry_safe_reverse(p, n, &rq->scx.runnable_list,
 						 scx.runnable_node) {
-			struct sched_enq_and_set_ctx ctx;
-
 			/* cycling deq/enq is enough, see the function comment */
-			sched_deq_and_put_task(p, DEQUEUE_SAVE | DEQUEUE_MOVE, &ctx);
-			sched_enq_and_set_task(&ctx);
+			scoped_guard (sched_change, p, DEQUEUE_SAVE | DEQUEUE_MOVE) {
+				/* nothing */ ;
+			}
 		}
 
 		rq_unlock_irqrestore(rq, &rf);
@@ -4491,17 +4490,14 @@ static void scx_ops_disable_workfn(struc
 		const struct sched_class *old_class = p->sched_class;
 		const struct sched_class *new_class =
 			__setscheduler_class(p->policy, p->prio);
-		struct sched_enq_and_set_ctx ctx;
 
 		if (old_class != new_class && p->se.sched_delayed)
 			dequeue_task(task_rq(p), p, DEQUEUE_SLEEP | DEQUEUE_DELAYED);
 
-		sched_deq_and_put_task(p, DEQUEUE_SAVE | DEQUEUE_MOVE, &ctx);
-
-		p->sched_class = new_class;
-		check_class_changing(task_rq(p), p, old_class);
-
-		sched_enq_and_set_task(&ctx);
+		scoped_guard (sched_change, p, DEQUEUE_SAVE | DEQUEUE_MOVE) {
+			p->sched_class = new_class;
+			check_class_changing(task_rq(p), p, old_class);
+		}
 
 		check_class_changed(task_rq(p), p, old_class, p->prio);
 		scx_ops_exit_task(p);
@@ -5206,18 +5202,15 @@ static int scx_ops_enable(struct sched_e
 		const struct sched_class *old_class = p->sched_class;
 		const struct sched_class *new_class =
 			__setscheduler_class(p->policy, p->prio);
-		struct sched_enq_and_set_ctx ctx;
 
 		if (old_class != new_class && p->se.sched_delayed)
-			dequeue_task(task_rq(p), p, DEQUEUE_SLEEP | DEQUEE_DELAYED);
-
-		sched_deq_and_put_task(p, DEQUEUE_SAVE | DEQUEUE_MOVE, &ctx);
-
-		p->scx.slice = SCX_SLICE_DFL;
-		p->sched_class = new_class;
-		check_class_changing(task_rq(p), p, old_class);
+			dequeue_task(task_rq(p), p, DEQUEUE_SLEEP | DEQUEUE_DELAYED);
 
-		sched_enq_and_set_task(&ctx);
+		scoped_guard (sched_change, p, DEQUEUE_SAVE | DEQUEUE_MOVE) {
+			p->scx.slice = SCX_SLICE_DFL;
+			p->sched_class = new_class;
+			check_class_changing(task_rq(p), p, old_class);
+		}
 
 		check_class_changed(task_rq(p), p, old_class, p->prio);
 	}
--- a/kernel/sched/sched.h
+++ b/kernel/sched/sched.h
@@ -3921,23 +3921,22 @@ static inline void balance_callbacks(str
 
 #endif
 
-#ifdef CONFIG_SCHED_CLASS_EXT
-/*
- * Used by SCX in the enable/disable paths to move tasks between sched_classes
- * and establish invariants.
- */
-struct sched_enq_and_set_ctx {
+struct sched_change_ctx {
 	struct task_struct	*p;
-	int			queue_flags;
+	unsigned int		flags;
 	bool			queued;
 	bool			running;
 };
 
-void sched_deq_and_put_task(struct task_struct *p, int queue_flags,
-			    struct sched_enq_and_set_ctx *ctx);
-void sched_enq_and_set_task(struct sched_enq_and_set_ctx *ctx);
+struct sched_change_ctx sched_change_begin(struct task_struct *p, unsigned int flags);
+void sched_change_end(struct sched_change_ctx ctx);
 
-#endif /* CONFIG_SCHED_CLASS_EXT */
+DEFINE_CLASS(sched_change, struct sched_change_ctx,
+	     sched_change_end(_T),
+	     sched_change_begin(p, flags),
+	     struct task_struct *p, unsigned int flags)
+
+DEFINE_CLASS_IS_UNCONDITIONAL(sched_change)
 
 #include "ext.h"
 
--- a/kernel/sched/syscalls.c
+++ b/kernel/sched/syscalls.c
@@ -64,7 +64,6 @@ static int effective_prio(struct task_st
 
 void set_user_nice(struct task_struct *p, long nice)
 {
-	bool queued, running;
 	struct rq *rq;
 	int old_prio;
 
@@ -90,22 +89,12 @@ void set_user_nice(struct task_struct *p
 		return;
 	}
 
-	queued = task_on_rq_queued(p);
-	running = task_current_donor(rq, p);
-	if (queued)
-		dequeue_task(rq, p, DEQUEUE_SAVE | DEQUEUE_NOCLOCK);
-	if (running)
-		put_prev_task(rq, p);
-
-	p->static_prio = NICE_TO_PRIO(nice);
-	set_load_weight(p, true);
-	old_prio = p->prio;
-	p->prio = effective_prio(p);
-
-	if (queued)
-		enqueue_task(rq, p, ENQUEUE_RESTORE | ENQUEUE_NOCLOCK);
-	if (running)
-		set_next_task(rq, p);
+	scoped_guard (sched_change, p, DEQUEUE_SAVE | DEQUEUE_NOCLOCK) {
+		p->static_prio = NICE_TO_PRIO(nice);
+		set_load_weight(p, true);
+		old_prio = p->prio;
+		p->prio = effective_prio(p);
+	}
 
 	/*
 	 * If the task increased its priority or is running and
@@ -528,7 +517,7 @@ int __sched_setscheduler(struct task_str
 			 bool user, bool pi)
 {
 	int oldpolicy = -1, policy = attr->sched_policy;
-	int retval, oldprio, newprio, queued, running;
+	int retval, oldprio, newprio;
 	const struct sched_class *prev_class, *next_class;
 	struct balance_callback *head;
 	struct rq_flags rf;
@@ -712,33 +701,25 @@ int __sched_setscheduler(struct task_str
 	if (prev_class != next_class && p->se.sched_delayed)
 		dequeue_task(rq, p, DEQUEUE_SLEEP | DEQUEUE_DELAYED | DEQUEUE_NOCLOCK);
 
-	queued = task_on_rq_queued(p);
-	running = task_current_donor(rq, p);
-	if (queued)
-		dequeue_task(rq, p, queue_flags);
-	if (running)
-		put_prev_task(rq, p);
-
-	if (!(attr->sched_flags & SCHED_FLAG_KEEP_PARAMS)) {
-		__setscheduler_params(p, attr);
-		p->sched_class = next_class;
-		p->prio = newprio;
-	}
-	__setscheduler_uclamp(p, attr);
-	check_class_changing(rq, p, prev_class);
+	scoped_guard (sched_change, p, queue_flags) {
 
-	if (queued) {
-		/*
-		 * We enqueue to tail when the priority of a task is
-		 * increased (user space view).
-		 */
-		if (oldprio < p->prio)
-			queue_flags |= ENQUEUE_HEAD;
+		if (!(attr->sched_flags & SCHED_FLAG_KEEP_PARAMS)) {
+			__setscheduler_params(p, attr);
+			p->sched_class = next_class;
+			p->prio = newprio;
+		}
+		__setscheduler_uclamp(p, attr);
+		check_class_changing(rq, p, prev_class);
 
-		enqueue_task(rq, p, queue_flags);
+		if (scope.queued) {
+			/*
+			 * We enqueue to tail when the priority of a task is
+			 * increased (user space view).
+			 */
+			if (oldprio < p->prio)
+				scope.flags |= ENQUEUE_HEAD;
+		}
 	}
-	if (running)
-		set_next_task(rq, p);
 
 	check_class_changed(rq, p, prev_class, oldprio);
 

Re: [RFC][PATCH 2/6] sched: Employ sched_change guards

[Tejun Heo]

On Wed, Oct 30, 2024 at 04:12:57PM +0100, Peter Zijlstra wrote:
> --- a/kernel/sched/ext.c
> +++ b/kernel/sched/ext.c
...
> @@ -5206,18 +5202,15 @@ static int scx_ops_enable(struct sched_e
>  		const struct sched_class *old_class = p->sched_class;
>  		const struct sched_class *new_class =
>  			__setscheduler_class(p->policy, p->prio);
> -		struct sched_enq_and_set_ctx ctx;
>  
>  		if (old_class != new_class && p->se.sched_delayed)
> -			dequeue_task(task_rq(p), p, DEQUEUE_SLEEP | DEQUEE_DELAYED);
> -
> -		sched_deq_and_put_task(p, DEQUEUE_SAVE | DEQUEUE_MOVE, &ctx);
> -
> -		p->scx.slice = SCX_SLICE_DFL;
> -		p->sched_class = new_class;
> -		check_class_changing(task_rq(p), p, old_class);
> +			dequeue_task(task_rq(p), p, DEQUEUE_SLEEP | DEQUEUE_DELAYED);
>  
> -		sched_enq_and_set_task(&ctx);
> +		scoped_guard (sched_change, p, DEQUEUE_SAVE | DEQUEUE_MOVE) {
> +			p->scx.slice = SCX_SLICE_DFL;
> +			p->sched_class = new_class;
> +			check_class_changing(task_rq(p), p, old_class);
> +		}
>  
>  		check_class_changed(task_rq(p), p, old_class, p->prio);
>  	}

I get the following from missing update_rq_lock():

  rq->clock_update_flags < RQCF_ACT_SKIP
  WARNING: CPU: 2 PID: 1692 at kernel/sched/sched.h:1647 update_load_avg+0x7c3/0x8c0
  Modules linked in:
  CPU: 2 UID: 0 PID: 1692 Comm: runner Not tainted 6.12.0-rc5-work-00336-g9bfae8f5ca65-dirty #515
  Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS unknown 2/2/2022
  Sched_ext: maximal (enabling+all)
  RIP: 0010:update_load_avg+0x7c3/0x8c0
  Code: 00 4c 2b bb c8 01 00 00 40 f6 c5 02 0f 84 e7 f8 ff ff e9 fa f8 ff ff c6 05 28 1f 81 02 01 48 c7 c7 f9 c5 dd 82 e8 1d 04 fb ff <0f> 0b e9 aa f8 ff ff 0f 0b 41 83 be f0 0c 00 00 01 0f 86 8d f8 ff
  RSP: 0018:ffffc900003c7c60 EFLAGS: 00010086
  RAX: 0000000000000026 RBX: ffff88810163d400 RCX: 0000000000000027
  RDX: 0000000000000002 RSI: 00000000ffffdfff RDI: ffff888237c9b448
  RBP: 0000000000000000 R08: 0000000000001fff R09: ffffffff8368dff0
  R10: 0000000000005ffd R11: 0000000000000004 R12: ffffffff82edb890
  R13: ffff888100398080 R14: ffff888237c30180 R15: ffff888100398000
  FS:  00007f850b4006c0(0000) GS:ffff888237c80000(0000) knlGS:0000000000000000
  CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
  CR2: 00007f84fc000020 CR3: 0000000103bfa000 CR4: 0000000000750eb0
  PKRU: 55555554
  Call Trace:
   <TASK>
   detach_task_cfs_rq+0x31/0xf0
   check_class_changed+0x29/0x70
   bpf_scx_reg+0xa72/0xc30
   bpf_struct_ops_link_create+0xf8/0x140
   __sys_bpf+0x348/0x510
   __x64_sys_bpf+0x18/0x20
   do_syscall_64+0x7b/0x140
   ? exc_page_fault+0x6b/0xb0
   entry_SYSCALL_64_after_hwframe+0x76/0x7e
  RIP: 0033:0x7f850c0551fd
  Code: ff c3 66 2e 0f 1f 84 00 00 00 00 00 90 f3 0f 1e fa 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d e3 fa 0c 00 f7 d8 64 89 01 48
  RSP: 002b:00007f850b3ffba8 EFLAGS: 00000202 ORIG_RAX: 0000000000000141
  RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f850c0551fd
  RDX: 0000000000000040 RSI: 00007f850b3ffdc0 RDI: 000000000000001c
  RBP: 00007f850b3ffbd0 R08: 0000000000000000 R09: 0000000000000000
  R10: 0000000000000001 R11: 0000000000000202 R12: 00007f850b4006c0
  R13: ffffffffffffff80 R14: 000000000000005f R15: 00007ffdf6c8de30
   </TASK>

The following patch fixes it. Thanks.

---
 kernel/sched/ext.c |    4 ++++
 1 file changed, 4 insertions(+)

--- a/kernel/sched/ext.c
+++ b/kernel/sched/ext.c
@@ -4496,6 +4496,8 @@ static void scx_ops_disable_workfn(struc
 		const struct sched_class *new_class =
 			__setscheduler_class(p->policy, p->prio);
 
+		update_rq_clock(task_rq(p));
+
 		if (old_class != new_class && p->se.sched_delayed)
 			dequeue_task(task_rq(p), p, DEQUEUE_SLEEP | DEQUEUE_DELAYED);
 
@@ -5208,6 +5210,8 @@ static int scx_ops_enable(struct sched_e
 		const struct sched_class *new_class =
 			__setscheduler_class(p->policy, p->prio);
 
+		update_rq_clock(task_rq(p));
+
 		if (old_class != new_class && p->se.sched_delayed)
 			dequeue_task(task_rq(p), p, DEQUEUE_SLEEP | DEQUEUE_DELAYED);

Re: [RFC][PATCH 2/6] sched: Employ sched_change guards

[Tejun Heo]

On Wed, Oct 30, 2024 at 10:58:40AM -1000, Tejun Heo wrote:
> On Wed, Oct 30, 2024 at 04:12:57PM +0100, Peter Zijlstra wrote:
> --- a/kernel/sched/ext.c
> +++ b/kernel/sched/ext.c
> @@ -4496,6 +4496,8 @@ static void scx_ops_disable_workfn(struc
>  		const struct sched_class *new_class =
>  			__setscheduler_class(p->policy, p->prio);
>  
> +		update_rq_clock(task_rq(p));
> +

Oh, this probably should be paired with DEQUEUE_NOCLOCK.

Thanks.

-- 
tejun

Re: [RFC][PATCH 2/6] sched: Employ sched_change guards

[Peter Zijlstra]

On Wed, Oct 30, 2024 at 11:09:56AM -1000, Tejun Heo wrote:
> On Wed, Oct 30, 2024 at 10:58:40AM -1000, Tejun Heo wrote:
> > On Wed, Oct 30, 2024 at 04:12:57PM +0100, Peter Zijlstra wrote:
> > --- a/kernel/sched/ext.c
> > +++ b/kernel/sched/ext.c
> > @@ -4496,6 +4496,8 @@ static void scx_ops_disable_workfn(struc
> >  		const struct sched_class *new_class =
> >  			__setscheduler_class(p->policy, p->prio);
> >  
> > +		update_rq_clock(task_rq(p));
> > +
> 
> Oh, this probably should be paired with DEQUEUE_NOCLOCK.

Yeah, it is like that at the end of the series. I'll shuffle things
around and make it so earlier.

Thanks!

[RFC][PATCH 3/6] sched: Re-arrange the {EN,DE}QUEUE flags

[Peter Zijlstra]

Ensure the matched flags are in the low byte while the unmatched flags
go into the second byte.

Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
---
 kernel/sched/sched.h |   27 +++++++++++++--------------
 1 file changed, 13 insertions(+), 14 deletions(-)

--- a/kernel/sched/sched.h
+++ b/kernel/sched/sched.h
@@ -2337,26 +2337,25 @@ extern const u32		sched_prio_to_wmult[40
 #define DEQUEUE_SAVE		0x02 /* Matches ENQUEUE_RESTORE */
 #define DEQUEUE_MOVE		0x04 /* Matches ENQUEUE_MOVE */
 #define DEQUEUE_NOCLOCK		0x08 /* Matches ENQUEUE_NOCLOCK */
-#define DEQUEUE_SPECIAL		0x10
-#define DEQUEUE_MIGRATING	0x100 /* Matches ENQUEUE_MIGRATING */
-#define DEQUEUE_DELAYED		0x200 /* Matches ENQUEUE_DELAYED */
+
+#define DEQUEUE_MIGRATING	0x10 /* Matches ENQUEUE_MIGRATING */
+#define DEQUEUE_DELAYED		0x20 /* Matches ENQUEUE_DELAYED */
+
+#define DEQUEUE_SPECIAL		0x0100
 
 #define ENQUEUE_WAKEUP		0x01
 #define ENQUEUE_RESTORE		0x02
 #define ENQUEUE_MOVE		0x04
 #define ENQUEUE_NOCLOCK		0x08
 
-#define ENQUEUE_HEAD		0x10
-#define ENQUEUE_REPLENISH	0x20
-#ifdef CONFIG_SMP
-#define ENQUEUE_MIGRATED	0x40
-#else
-#define ENQUEUE_MIGRATED	0x00
-#endif
-#define ENQUEUE_INITIAL		0x80
-#define ENQUEUE_MIGRATING	0x100
-#define ENQUEUE_DELAYED		0x200
-#define ENQUEUE_RQ_SELECTED	0x400
+#define ENQUEUE_MIGRATING	0x10
+#define ENQUEUE_DELAYED		0x20
+
+#define ENQUEUE_HEAD		0x0100
+#define ENQUEUE_REPLENISH	0x0200
+#define ENQUEUE_MIGRATED	(0x0400*IS_ENABLED(CONFIG_SMP))
+#define ENQUEUE_INITIAL		0x0800
+#define ENQUEUE_RQ_SELECTED	0x1000
 
 #define RETRY_TASK		((void *)-1UL)
 

[RFC][PATCH 4/6] sched: Fold sched_class::switch{ing,ed}_{to,from}() into the change pattern

[Peter Zijlstra]

Add {DE,EN}QUEUE_CLASS and fold the sched_class::switch* methods into
the change pattern. This completes and makes the pattern more
symmetric.

This changes the order of callbacks slightly:

				|
				|  switching_from()
  dequeue_task();		|  dequeue_task()
  put_prev_task();		|  put_prev_task()
				|  switched_from()
				|
  ... change task ...		|  ... change task ...
				|
  switching_to();		|  switching_to()
  enqueue_task();		|  enqueue_task()
  set_next_task();		|  set_next_task()
  prev_class->switched_from()	|
  switched_to()			|  switched_to()
				|

Notably, it moves the switched_from() callback right after the
dequeue/put. Existing implementations don't appear to be affected by
this change in location -- specifically the task isn't enqueued on the
class in question in either location.

Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
---
 kernel/sched/core.c      |   61 ++++++++++++++++++++---------------------------
 kernel/sched/ext.c       |   19 ++++++++++----
 kernel/sched/idle.c      |    4 +--
 kernel/sched/rt.c        |    4 +--
 kernel/sched/sched.h     |   21 ++++++----------
 kernel/sched/stop_task.c |    4 +--
 kernel/sched/syscalls.c  |    7 +++--
 7 files changed, 59 insertions(+), 61 deletions(-)

--- a/kernel/sched/core.c
+++ b/kernel/sched/core.c
@@ -2102,34 +2102,9 @@ inline int task_curr(const struct task_s
 	return cpu_curr(task_cpu(p)) == p;
 }
 
-/*
- * ->switching_to() is called with the pi_lock and rq_lock held and must not
- * mess with locking.
- */
-void check_class_changing(struct rq *rq, struct task_struct *p,
-			  const struct sched_class *prev_class)
+void check_prio_changed(struct rq *rq, struct task_struct *p, int oldprio)
 {
-	if (prev_class != p->sched_class && p->sched_class->switching_to)
-		p->sched_class->switching_to(rq, p);
-}
-
-/*
- * switched_from, switched_to and prio_changed must _NOT_ drop rq->lock,
- * use the balance_callback list if you want balancing.
- *
- * this means any call to check_class_changed() must be followed by a call to
- * balance_callback().
- */
-void check_class_changed(struct rq *rq, struct task_struct *p,
-			 const struct sched_class *prev_class,
-			 int oldprio)
-{
-	if (prev_class != p->sched_class) {
-		if (prev_class->switched_from)
-			prev_class->switched_from(rq, p);
-
-		p->sched_class->switched_to(rq, p);
-	} else if (oldprio != p->prio || dl_task(p))
+	if (oldprio != p->prio || dl_task(p))
 		p->sched_class->prio_changed(rq, p, oldprio);
 }
 
@@ -7161,6 +7136,9 @@ void rt_mutex_setprio(struct task_struct
 	prev_class = p->sched_class;
 	next_class = __setscheduler_class(p->policy, prio);
 
+	if (prev_class != next_class)
+		queue_flag |= DEQUEUE_CLASS;
+
 	if (prev_class != next_class && p->se.sched_delayed)
 		dequeue_task(rq, p, DEQUEUE_SLEEP | DEQUEUE_DELAYED | DEQUEUE_NOCLOCK);
 
@@ -7197,11 +7175,10 @@ void rt_mutex_setprio(struct task_struct
 
 		p->sched_class = next_class;
 		p->prio = prio;
-
-		check_class_changing(rq, p, prev_class);
 	}
 
-	check_class_changed(rq, p, prev_class, oldprio);
+	if (!(queue_flag & DEQUEUE_CLASS))
+		check_prio_changed(rq, p, oldprio);
 out_unlock:
 	/* Avoid rq from going away on us: */
 	preempt_disable();
@@ -10550,6 +10527,12 @@ void sched_mm_cid_fork(struct task_struc
 struct sched_change_ctx sched_change_begin(struct task_struct *p, unsigned int flags)
 {
 	struct rq *rq = task_rq(p);
+
+	lockdep_assert_rq_held(rq);
+
+	if ((flags & DEQUEUE_CLASS) && p->sched_class->switching_from)
+		p->sched_class->switching_from(rq, p);
+
 	struct sched_change_ctx ctx = {
 		.p = p,
 		.flags = flags,
@@ -10557,24 +10540,32 @@ struct sched_change_ctx sched_change_beg
 		.running = task_current(rq, p),
 	};
 
-	lockdep_assert_rq_held(rq);
-
 	if (ctx.queued)
 		dequeue_task(rq, p, flags);
 	if (ctx.running)
 		put_prev_task(rq, p);
 
+	if ((flags & DEQUEUE_CLASS) && p->sched_class->switched_from)
+		p->sched_class->switched_from(rq, p);
+
 	return ctx;
 }
 
 void sched_change_end(struct sched_change_ctx ctx)
 {
-	struct rq *rq = task_rq(ctx.p);
+	struct task_struct *p = ctx.p;
+	struct rq *rq = task_rq(p);
 
 	lockdep_assert_rq_held(rq);
 
+	if ((ctx.flags & ENQUEUE_CLASS) && p->sched_class->switching_to)
+		p->sched_class->switching_to(rq, p);
+
 	if (ctx.queued)
-		enqueue_task(rq, ctx.p, ctx.flags | ENQUEUE_NOCLOCK);
+		enqueue_task(rq, p, ctx.flags | ENQUEUE_NOCLOCK);
 	if (ctx.running)
-		set_next_task(rq, ctx.p);
+		set_next_task(rq, p);
+
+	if ((ctx.flags & ENQUEUE_CLASS) && p->sched_class->switched_to)
+		p->sched_class->switched_to(rq, p);
 }
--- a/kernel/sched/ext.c
+++ b/kernel/sched/ext.c
@@ -4487,19 +4487,24 @@ static void scx_ops_disable_workfn(struc
 
 	scx_task_iter_start(&sti);
 	while ((p = scx_task_iter_next_locked(&sti))) {
+		unsigned int queue_flags = DEQUEUE_SAVE | DEQUEUE_MOVE;
 		const struct sched_class *old_class = p->sched_class;
 		const struct sched_class *new_class =
 			__setscheduler_class(p->policy, p->prio);
 
+		if (old_class != new_class)
+			queue_flags |= DEQUEUE_CLASS;
+
 		if (old_class != new_class && p->se.sched_delayed)
 			dequeue_task(task_rq(p), p, DEQUEUE_SLEEP | DEQUEUE_DELAYED);
 
-		scoped_guard (sched_change, p, DEQUEUE_SAVE | DEQUEUE_MOVE) {
+		scoped_guard (sched_change, p, queue_flags) {
 			p->sched_class = new_class;
-			check_class_changing(task_rq(p), p, old_class);
 		}
 
-		check_class_changed(task_rq(p), p, old_class, p->prio);
+		if (!(queue_flags & DEQUEUE_CLASS))
+			check_prio_changed(task_rq(p), p, p->prio);
+
 		scx_ops_exit_task(p);
 	}
 	scx_task_iter_stop(&sti);
@@ -5199,20 +5204,24 @@ static int scx_ops_enable(struct sched_e
 	percpu_down_write(&scx_fork_rwsem);
 	scx_task_iter_start(&sti);
 	while ((p = scx_task_iter_next_locked(&sti))) {
+		unsigned int queue_flags = DEQUEUE_SAVE | DEQUEUE_MOVE;
 		const struct sched_class *old_class = p->sched_class;
 		const struct sched_class *new_class =
 			__setscheduler_class(p->policy, p->prio);
 
+		if (old_class != new_class)
+			queue_flags |= DEQUEUE_CLASS;
+
 		if (old_class != new_class && p->se.sched_delayed)
 			dequeue_task(task_rq(p), p, DEQUEUE_SLEEP | DEQUEUE_DELAYED);
 
 		scoped_guard (sched_change, p, DEQUEUE_SAVE | DEQUEUE_MOVE) {
 			p->scx.slice = SCX_SLICE_DFL;
 			p->sched_class = new_class;
-			check_class_changing(task_rq(p), p, old_class);
 		}
 
-		check_class_changed(task_rq(p), p, old_class, p->prio);
+		if (!(queue_flags & DEQUEUE_CLASS))
+			check_prio_changed(task_rq(p), p, p->prio);
 	}
 	scx_task_iter_stop(&sti);
 	percpu_up_write(&scx_fork_rwsem);
--- a/kernel/sched/idle.c
+++ b/kernel/sched/idle.c
@@ -494,7 +494,7 @@ static void task_tick_idle(struct rq *rq
 {
 }
 
-static void switched_to_idle(struct rq *rq, struct task_struct *p)
+static void switching_to_idle(struct rq *rq, struct task_struct *p)
 {
 	BUG();
 }
@@ -534,6 +534,6 @@ DEFINE_SCHED_CLASS(idle) = {
 	.task_tick		= task_tick_idle,
 
 	.prio_changed		= prio_changed_idle,
-	.switched_to		= switched_to_idle,
+	.switched_to		= switching_to_idle,
 	.update_curr		= update_curr_idle,
 };
--- a/kernel/sched/rt.c
+++ b/kernel/sched/rt.c
@@ -2633,7 +2633,6 @@ DEFINE_SCHED_CLASS(rt) = {
 	.rq_online              = rq_online_rt,
 	.rq_offline             = rq_offline_rt,
 	.task_woken		= task_woken_rt,
-	.switched_from		= switched_from_rt,
 	.find_lock_rq		= find_lock_lowest_rq,
 #endif
 
@@ -2641,8 +2640,9 @@ DEFINE_SCHED_CLASS(rt) = {
 
 	.get_rr_interval	= get_rr_interval_rt,
 
-	.prio_changed		= prio_changed_rt,
+	.switched_from		= switched_from_rt,
 	.switched_to		= switched_to_rt,
+	.prio_changed		= prio_changed_rt,
 
 	.update_curr		= update_curr_rt,
 
--- a/kernel/sched/sched.h
+++ b/kernel/sched/sched.h
@@ -2340,6 +2340,7 @@ extern const u32		sched_prio_to_wmult[40
 
 #define DEQUEUE_MIGRATING	0x10 /* Matches ENQUEUE_MIGRATING */
 #define DEQUEUE_DELAYED		0x20 /* Matches ENQUEUE_DELAYED */
+#define DEQUEUE_CLASS		0x40 /* Matches ENQUEUE_CLASS */
 
 #define DEQUEUE_SPECIAL		0x0100
 
@@ -2350,6 +2351,7 @@ extern const u32		sched_prio_to_wmult[40
 
 #define ENQUEUE_MIGRATING	0x10
 #define ENQUEUE_DELAYED		0x20
+#define ENQUEUE_CLASS		0x40
 
 #define ENQUEUE_HEAD		0x0100
 #define ENQUEUE_REPLENISH	0x0200
@@ -2415,14 +2417,11 @@ struct sched_class {
 	void (*task_fork)(struct task_struct *p);
 	void (*task_dead)(struct task_struct *p);
 
-	/*
-	 * The switched_from() call is allowed to drop rq->lock, therefore we
-	 * cannot assume the switched_from/switched_to pair is serialized by
-	 * rq->lock. They are however serialized by p->pi_lock.
-	 */
-	void (*switching_to) (struct rq *this_rq, struct task_struct *task);
-	void (*switched_from)(struct rq *this_rq, struct task_struct *task);
-	void (*switched_to)  (struct rq *this_rq, struct task_struct *task);
+	void (*switching_from)(struct rq *this_rq, struct task_struct *task);
+	void (*switched_from) (struct rq *this_rq, struct task_struct *task);
+	void (*switching_to)  (struct rq *this_rq, struct task_struct *task);
+	void (*switched_to)   (struct rq *this_rq, struct task_struct *task);
+
 	void (*reweight_task)(struct rq *this_rq, struct task_struct *task,
 			      const struct load_weight *lw);
 	void (*prio_changed) (struct rq *this_rq, struct task_struct *task,
@@ -3898,11 +3897,7 @@ extern void set_load_weight(struct task_
 extern void enqueue_task(struct rq *rq, struct task_struct *p, int flags);
 extern bool dequeue_task(struct rq *rq, struct task_struct *p, int flags);
 
-extern void check_class_changing(struct rq *rq, struct task_struct *p,
-				 const struct sched_class *prev_class);
-extern void check_class_changed(struct rq *rq, struct task_struct *p,
-				const struct sched_class *prev_class,
-				int oldprio);
+extern void check_prio_changed(struct rq *rq, struct task_struct *p, int oldprio);
 
 #ifdef CONFIG_SMP
 extern struct balance_callback *splice_balance_callbacks(struct rq *rq);
--- a/kernel/sched/stop_task.c
+++ b/kernel/sched/stop_task.c
@@ -76,7 +76,7 @@ static void task_tick_stop(struct rq *rq
 {
 }
 
-static void switched_to_stop(struct rq *rq, struct task_struct *p)
+static void switching_to_stop(struct rq *rq, struct task_struct *p)
 {
 	BUG(); /* its impossible to change to this class */
 }
@@ -115,6 +115,6 @@ DEFINE_SCHED_CLASS(stop) = {
 	.task_tick		= task_tick_stop,
 
 	.prio_changed		= prio_changed_stop,
-	.switched_to		= switched_to_stop,
+	.switching_to		= switching_to_stop,
 	.update_curr		= update_curr_stop,
 };
--- a/kernel/sched/syscalls.c
+++ b/kernel/sched/syscalls.c
@@ -698,6 +698,9 @@ int __sched_setscheduler(struct task_str
 	prev_class = p->sched_class;
 	next_class = __setscheduler_class(policy, newprio);
 
+	if (prev_class != next_class)
+		queue_flags |= DEQUEUE_CLASS;
+
 	if (prev_class != next_class && p->se.sched_delayed)
 		dequeue_task(rq, p, DEQUEUE_SLEEP | DEQUEUE_DELAYED | DEQUEUE_NOCLOCK);
 
@@ -709,7 +712,6 @@ int __sched_setscheduler(struct task_str
 			p->prio = newprio;
 		}
 		__setscheduler_uclamp(p, attr);
-		check_class_changing(rq, p, prev_class);
 
 		if (scope.queued) {
 			/*
@@ -721,7 +723,8 @@ int __sched_setscheduler(struct task_str
 		}
 	}
 
-	check_class_changed(rq, p, prev_class, oldprio);
+	if (!(queue_flags & DEQUEUE_CLASS))
+		check_prio_changed(rq, p, oldprio);
 
 	/* Avoid rq from going away on us: */
 	preempt_disable();

Re: [RFC][PATCH 4/6] sched: Fold sched_class::switch{ing,ed}_{to,from}() into the change pattern

[Tejun Heo]

On Wed, Oct 30, 2024 at 04:12:59PM +0100, Peter Zijlstra wrote:
> --- a/kernel/sched/ext.c
> +++ b/kernel/sched/ext.c
> @@ -5199,20 +5204,24 @@ static int scx_ops_enable(struct sched_e
>  	percpu_down_write(&scx_fork_rwsem);
>  	scx_task_iter_start(&sti);
>  	while ((p = scx_task_iter_next_locked(&sti))) {
> +		unsigned int queue_flags = DEQUEUE_SAVE | DEQUEUE_MOVE;
>  		const struct sched_class *old_class = p->sched_class;
>  		const struct sched_class *new_class =
>  			__setscheduler_class(p->policy, p->prio);
>  
> +		if (old_class != new_class)
> +			queue_flags |= DEQUEUE_CLASS;
> +
>  		if (old_class != new_class && p->se.sched_delayed)
>  			dequeue_task(task_rq(p), p, DEQUEUE_SLEEP | DEQUEUE_DELAYED);
>  
>  		scoped_guard (sched_change, p, DEQUEUE_SAVE | DEQUEUE_MOVE) {
                                               ^
					       queue_flags
					       
>  			p->scx.slice = SCX_SLICE_DFL;
>  			p->sched_class = new_class;
> -			check_class_changing(task_rq(p), p, old_class);
>  		}
>  
> -		check_class_changed(task_rq(p), p, old_class, p->prio);
> +		if (!(queue_flags & DEQUEUE_CLASS))
> +			check_prio_changed(task_rq(p), p, p->prio);

Maybe prio_changed can be moved into scoped_guard?

Thanks.

-- 
tejun

Re: [RFC][PATCH 4/6] sched: Fold sched_class::switch{ing,ed}_{to,from}() into the change pattern

[Peter Zijlstra]

On Wed, Oct 30, 2024 at 11:12:32AM -1000, Tejun Heo wrote:
> On Wed, Oct 30, 2024 at 04:12:59PM +0100, Peter Zijlstra wrote:
> > --- a/kernel/sched/ext.c
> > +++ b/kernel/sched/ext.c
> > @@ -5199,20 +5204,24 @@ static int scx_ops_enable(struct sched_e
> >  	percpu_down_write(&scx_fork_rwsem);
> >  	scx_task_iter_start(&sti);
> >  	while ((p = scx_task_iter_next_locked(&sti))) {
> > +		unsigned int queue_flags = DEQUEUE_SAVE | DEQUEUE_MOVE;
> >  		const struct sched_class *old_class = p->sched_class;
> >  		const struct sched_class *new_class =
> >  			__setscheduler_class(p->policy, p->prio);
> >  
> > +		if (old_class != new_class)
> > +			queue_flags |= DEQUEUE_CLASS;
> > +
> >  		if (old_class != new_class && p->se.sched_delayed)
> >  			dequeue_task(task_rq(p), p, DEQUEUE_SLEEP | DEQUEUE_DELAYED);
> >  
> >  		scoped_guard (sched_change, p, DEQUEUE_SAVE | DEQUEUE_MOVE) {
>                                                ^
> 					       queue_flags
> 					       
> >  			p->scx.slice = SCX_SLICE_DFL;
> >  			p->sched_class = new_class;
> > -			check_class_changing(task_rq(p), p, old_class);
> >  		}
> >  
> > -		check_class_changed(task_rq(p), p, old_class, p->prio);
> > +		if (!(queue_flags & DEQUEUE_CLASS))
> > +			check_prio_changed(task_rq(p), p, p->prio);
> 
> Maybe prio_changed can be moved into scoped_guard?

It wasn't before -- do you have need for it to be inside?

Re: [RFC][PATCH 4/6] sched: Fold sched_class::switch{ing,ed}_{to,from}() into the change pattern

[Tejun Heo]

On Wed, Oct 30, 2024 at 10:15:06PM +0100, Peter Zijlstra wrote:
...
> > > +		if (!(queue_flags & DEQUEUE_CLASS))
> > > +			check_prio_changed(task_rq(p), p, p->prio);
> > 
> > Maybe prio_changed can be moved into scoped_guard?
> 
> It wasn't before -- do you have need for it to be inside?

No, was just wondering whether that'd make things a bit more compact. Either
way is fine.

Thanks.

-- 
tejun

Re: [RFC][PATCH 4/6] sched: Fold sched_class::switch{ing,ed}_{to,from}() into the change pattern

[Peter Zijlstra]

On Wed, Oct 30, 2024 at 11:29:09AM -1000, Tejun Heo wrote:
> On Wed, Oct 30, 2024 at 10:15:06PM +0100, Peter Zijlstra wrote:
> ...
> > > > +		if (!(queue_flags & DEQUEUE_CLASS))
> > > > +			check_prio_changed(task_rq(p), p, p->prio);
> > > 
> > > Maybe prio_changed can be moved into scoped_guard?
> > 
> > It wasn't before -- do you have need for it to be inside?
> 
> No, was just wondering whether that'd make things a bit more compact. Either
> way is fine.

Oh, did you perhaps mean into sched_change_end() ? I suppose that's
possible indeed. Initially I thought that would require yet another
flags, but looking at it again, that doesn't seem to be the case. All
sched_change users lacking it never change the prio anyway.

I'll have a look at doing that tomorrow, with a slightly fresher brain.

I also think that adding flags to the switch*() methods isn't at all
needed, but perhaps it makes sense anyway.

Re: [RFC][PATCH 4/6] sched: Fold sched_class::switch{ing,ed}_{to,from}() into the change pattern

[Tejun Heo]

Hello,

On Wed, Oct 30, 2024 at 10:37:35PM +0100, Peter Zijlstra wrote:
> On Wed, Oct 30, 2024 at 11:29:09AM -1000, Tejun Heo wrote:
> > On Wed, Oct 30, 2024 at 10:15:06PM +0100, Peter Zijlstra wrote:
> > ...
> > > > > +		if (!(queue_flags & DEQUEUE_CLASS))
> > > > > +			check_prio_changed(task_rq(p), p, p->prio);
> > > > 
> > > > Maybe prio_changed can be moved into scoped_guard?
> > > 
> > > It wasn't before -- do you have need for it to be inside?
> > 
> > No, was just wondering whether that'd make things a bit more compact. Either
> > way is fine.
> 
> Oh, did you perhaps mean into sched_change_end() ? I suppose that's
> possible indeed. Initially I thought that would require yet another

Oh yeah, that's what I meant. Sorry about not being clearer.

> flags, but looking at it again, that doesn't seem to be the case. All
> sched_change users lacking it never change the prio anyway.
> 
> I'll have a look at doing that tomorrow, with a slightly fresher brain.
> 
> I also think that adding flags to the switch*() methods isn't at all
> needed, but perhaps it makes sense anyway.

Fantastic. Thanks.

-- 
tejun

[RFC][PATCH 5/6] sched: Add flags to the switch{ing,ed}_{to,from}() methods

[Peter Zijlstra]

... and remove switched_to_scx(), since all methods are now optional.

Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
---
 kernel/sched/core.c      |    8 ++++----
 kernel/sched/deadline.c  |    4 ++--
 kernel/sched/ext.c       |    6 ++----
 kernel/sched/fair.c      |    4 ++--
 kernel/sched/idle.c      |    2 +-
 kernel/sched/rt.c        |    4 ++--
 kernel/sched/sched.h     |    8 ++++----
 kernel/sched/stop_task.c |    2 +-
 8 files changed, 18 insertions(+), 20 deletions(-)

--- a/kernel/sched/core.c
+++ b/kernel/sched/core.c
@@ -10531,7 +10531,7 @@ struct sched_change_ctx sched_change_beg
 	lockdep_assert_rq_held(rq);
 
 	if ((flags & DEQUEUE_CLASS) && p->sched_class->switching_from)
-		p->sched_class->switching_from(rq, p);
+		p->sched_class->switching_from(rq, p, flags);
 
 	struct sched_change_ctx ctx = {
 		.p = p,
@@ -10546,7 +10546,7 @@ struct sched_change_ctx sched_change_beg
 		put_prev_task(rq, p);
 
 	if ((flags & DEQUEUE_CLASS) && p->sched_class->switched_from)
-		p->sched_class->switched_from(rq, p);
+		p->sched_class->switched_from(rq, p, flags);
 
 	return ctx;
 }
@@ -10559,7 +10559,7 @@ void sched_change_end(struct sched_chang
 	lockdep_assert_rq_held(rq);
 
 	if ((ctx.flags & ENQUEUE_CLASS) && p->sched_class->switching_to)
-		p->sched_class->switching_to(rq, p);
+		p->sched_class->switching_to(rq, p, ctx.flags);
 
 	if (ctx.queued)
 		enqueue_task(rq, p, ctx.flags | ENQUEUE_NOCLOCK);
@@ -10567,5 +10567,5 @@ void sched_change_end(struct sched_chang
 		set_next_task(rq, p);
 
 	if ((ctx.flags & ENQUEUE_CLASS) && p->sched_class->switched_to)
-		p->sched_class->switched_to(rq, p);
+		p->sched_class->switched_to(rq, p, ctx.flags);
 }
--- a/kernel/sched/deadline.c
+++ b/kernel/sched/deadline.c
@@ -2968,7 +2968,7 @@ void dl_clear_root_domain(struct root_do
 
 #endif /* CONFIG_SMP */
 
-static void switched_from_dl(struct rq *rq, struct task_struct *p)
+static void switched_from_dl(struct rq *rq, struct task_struct *p, int flags)
 {
 	/*
 	 * task_non_contending() can start the "inactive timer" (if the 0-lag
@@ -3022,7 +3022,7 @@ static void switched_from_dl(struct rq *
  * When switching to -deadline, we may overload the rq, then
  * we try to push someone off, if possible.
  */
-static void switched_to_dl(struct rq *rq, struct task_struct *p)
+static void switched_to_dl(struct rq *rq, struct task_struct *p, int flags)
 {
 	if (hrtimer_try_to_cancel(&p->dl.inactive_timer) == 1)
 		put_task_struct(p);
--- a/kernel/sched/ext.c
+++ b/kernel/sched/ext.c
@@ -3663,7 +3663,7 @@ static void prio_changed_scx(struct rq *
 {
 }
 
-static void switching_to_scx(struct rq *rq, struct task_struct *p)
+static void switching_to_scx(struct rq *rq, struct task_struct *p, int flags)
 {
 	scx_ops_enable_task(p);
 
@@ -3676,13 +3676,12 @@ static void switching_to_scx(struct rq *
 				 (struct cpumask *)p->cpus_ptr);
 }
 
-static void switched_from_scx(struct rq *rq, struct task_struct *p)
+static void switched_from_scx(struct rq *rq, struct task_struct *p, int flags)
 {
 	scx_ops_disable_task(p);
 }
 
 static void wakeup_preempt_scx(struct rq *rq, struct task_struct *p,int wake_flags) {}
-static void switched_to_scx(struct rq *rq, struct task_struct *p) {}
 
 int scx_check_setscheduler(struct task_struct *p, int policy)
 {
@@ -3971,7 +3970,6 @@ DEFINE_SCHED_CLASS(ext) = {
 
 	.switching_to		= switching_to_scx,
 	.switched_from		= switched_from_scx,
-	.switched_to		= switched_to_scx,
 	.reweight_task		= reweight_task_scx,
 	.prio_changed		= prio_changed_scx,
 
--- a/kernel/sched/fair.c
+++ b/kernel/sched/fair.c
@@ -13175,12 +13175,12 @@ static void attach_task_cfs_rq(struct ta
 	attach_entity_cfs_rq(se);
 }
 
-static void switched_from_fair(struct rq *rq, struct task_struct *p)
+static void switched_from_fair(struct rq *rq, struct task_struct *p, int flags)
 {
 	detach_task_cfs_rq(p);
 }
 
-static void switched_to_fair(struct rq *rq, struct task_struct *p)
+static void switched_to_fair(struct rq *rq, struct task_struct *p, int flags)
 {
 	SCHED_WARN_ON(p->se.sched_delayed);
 
--- a/kernel/sched/idle.c
+++ b/kernel/sched/idle.c
@@ -494,7 +494,7 @@ static void task_tick_idle(struct rq *rq
 {
 }
 
-static void switching_to_idle(struct rq *rq, struct task_struct *p)
+static void switching_to_idle(struct rq *rq, struct task_struct *p, int flags)
 {
 	BUG();
 }
--- a/kernel/sched/rt.c
+++ b/kernel/sched/rt.c
@@ -2420,7 +2420,7 @@ static void rq_offline_rt(struct rq *rq)
  * When switch from the rt queue, we bring ourselves to a position
  * that we might want to pull RT tasks from other runqueues.
  */
-static void switched_from_rt(struct rq *rq, struct task_struct *p)
+static void switched_from_rt(struct rq *rq, struct task_struct *p, int flags)
 {
 	/*
 	 * If there are other RT tasks then we will reschedule
@@ -2451,7 +2451,7 @@ void __init init_sched_rt_class(void)
  * with RT tasks. In this case we try to push them off to
  * other runqueues.
  */
-static void switched_to_rt(struct rq *rq, struct task_struct *p)
+static void switched_to_rt(struct rq *rq, struct task_struct *p, int flags)
 {
 	/*
 	 * If we are running, update the avg_rt tracking, as the running time
--- a/kernel/sched/sched.h
+++ b/kernel/sched/sched.h
@@ -2417,10 +2417,10 @@ struct sched_class {
 	void (*task_fork)(struct task_struct *p);
 	void (*task_dead)(struct task_struct *p);
 
-	void (*switching_from)(struct rq *this_rq, struct task_struct *task);
-	void (*switched_from) (struct rq *this_rq, struct task_struct *task);
-	void (*switching_to)  (struct rq *this_rq, struct task_struct *task);
-	void (*switched_to)   (struct rq *this_rq, struct task_struct *task);
+	void (*switching_from)(struct rq *this_rq, struct task_struct *task, int flags);
+	void (*switched_from) (struct rq *this_rq, struct task_struct *task, int flags);
+	void (*switching_to)  (struct rq *this_rq, struct task_struct *task, int flags);
+	void (*switched_to)   (struct rq *this_rq, struct task_struct *task, int flags);
 
 	void (*reweight_task)(struct rq *this_rq, struct task_struct *task,
 			      const struct load_weight *lw);
--- a/kernel/sched/stop_task.c
+++ b/kernel/sched/stop_task.c
@@ -76,7 +76,7 @@ static void task_tick_stop(struct rq *rq
 {
 }
 
-static void switching_to_stop(struct rq *rq, struct task_struct *p)
+static void switching_to_stop(struct rq *rq, struct task_struct *p, int flags)
 {
 	BUG(); /* its impossible to change to this class */
 }

[RFC][PATCH 6/6] sched: Cleanup sched_delayed handling for class switches

[Peter Zijlstra]

Use the new sched_class::switching_from() method to dequeue delayed
tasks before switching to another class.

Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
---
 kernel/sched/core.c     |   12 ++++++++----
 kernel/sched/ext.c      |   12 ++++--------
 kernel/sched/fair.c     |    7 +++++++
 kernel/sched/syscalls.c |    3 ---
 4 files changed, 19 insertions(+), 15 deletions(-)

--- a/kernel/sched/core.c
+++ b/kernel/sched/core.c
@@ -7139,9 +7139,6 @@ void rt_mutex_setprio(struct task_struct
 	if (prev_class != next_class)
 		queue_flag |= DEQUEUE_CLASS;
 
-	if (prev_class != next_class && p->se.sched_delayed)
-		dequeue_task(rq, p, DEQUEUE_SLEEP | DEQUEUE_DELAYED | DEQUEUE_NOCLOCK);
-
 	scoped_guard (sched_change, p, queue_flag) {
 		/*
 		 * Boosting condition are:
@@ -10530,8 +10527,15 @@ struct sched_change_ctx sched_change_beg
 
 	lockdep_assert_rq_held(rq);
 
-	if ((flags & DEQUEUE_CLASS) && p->sched_class->switching_from)
+	if ((flags & DEQUEUE_CLASS) && p->sched_class->switching_from) {
+		/*
+		 * switching_from_fair() assumes CLASS implies NOCLOCK; fixing
+		 * this assumption would mean switching_from() would need to be
+		 * able to change flags.
+		 */
+		SCHED_WARN_ON(!(flags & DEQUEUE_NOCLOCK));
 		p->sched_class->switching_from(rq, p, flags);
+	}
 
 	struct sched_change_ctx ctx = {
 		.p = p,
--- a/kernel/sched/ext.c
+++ b/kernel/sched/ext.c
@@ -4485,7 +4485,7 @@ static void scx_ops_disable_workfn(struc
 
 	scx_task_iter_start(&sti);
 	while ((p = scx_task_iter_next_locked(&sti))) {
-		unsigned int queue_flags = DEQUEUE_SAVE | DEQUEUE_MOVE;
+		unsigned int queue_flags = DEQUEUE_SAVE | DEQUEUE_MOVE | DEQUEUE_NOCLOCK;
 		const struct sched_class *old_class = p->sched_class;
 		const struct sched_class *new_class =
 			__setscheduler_class(p->policy, p->prio);
@@ -4493,9 +4493,7 @@ static void scx_ops_disable_workfn(struc
 		if (old_class != new_class)
 			queue_flags |= DEQUEUE_CLASS;
 
-		if (old_class != new_class && p->se.sched_delayed)
-			dequeue_task(task_rq(p), p, DEQUEUE_SLEEP | DEQUEUE_DELAYED);
-
+		update_rq_clock(task_rq(p));
 		scoped_guard (sched_change, p, queue_flags) {
 			p->sched_class = new_class;
 		}
@@ -5202,7 +5200,7 @@ static int scx_ops_enable(struct sched_e
 	percpu_down_write(&scx_fork_rwsem);
 	scx_task_iter_start(&sti);
 	while ((p = scx_task_iter_next_locked(&sti))) {
-		unsigned int queue_flags = DEQUEUE_SAVE | DEQUEUE_MOVE;
+		unsigned int queue_flags = DEQUEUE_SAVE | DEQUEUE_MOVE | DEQUEUE_NOCLOCK;
 		const struct sched_class *old_class = p->sched_class;
 		const struct sched_class *new_class =
 			__setscheduler_class(p->policy, p->prio);
@@ -5210,9 +5208,7 @@ static int scx_ops_enable(struct sched_e
 		if (old_class != new_class)
 			queue_flags |= DEQUEUE_CLASS;
 
-		if (old_class != new_class && p->se.sched_delayed)
-			dequeue_task(task_rq(p), p, DEQUEUE_SLEEP | DEQUEUE_DELAYED);
-
+		update_rq_clock(task_rq(p));
 		scoped_guard (sched_change, p, DEQUEUE_SAVE | DEQUEUE_MOVE) {
 			p->scx.slice = SCX_SLICE_DFL;
 			p->sched_class = new_class;
--- a/kernel/sched/fair.c
+++ b/kernel/sched/fair.c
@@ -13175,6 +13175,12 @@ static void attach_task_cfs_rq(struct ta
 	attach_entity_cfs_rq(se);
 }
 
+static void switching_from_fair(struct rq *rq, struct task_struct *p, int flags)
+{
+	if ((flags & DEQUEUE_CLASS) && p->se.sched_delayed)
+		dequeue_task(rq, p, DEQUEUE_SLEEP | DEQUEUE_DELAYED | DEQUEUE_NOCLOCK);
+}
+
 static void switched_from_fair(struct rq *rq, struct task_struct *p, int flags)
 {
 	detach_task_cfs_rq(p);
@@ -13592,6 +13598,7 @@ DEFINE_SCHED_CLASS(fair) = {
 
 	.reweight_task		= reweight_task_fair,
 	.prio_changed		= prio_changed_fair,
+	.switching_from		= switching_from_fair,
 	.switched_from		= switched_from_fair,
 	.switched_to		= switched_to_fair,
 
--- a/kernel/sched/syscalls.c
+++ b/kernel/sched/syscalls.c
@@ -701,9 +701,6 @@ int __sched_setscheduler(struct task_str
 	if (prev_class != next_class)
 		queue_flags |= DEQUEUE_CLASS;
 
-	if (prev_class != next_class && p->se.sched_delayed)
-		dequeue_task(rq, p, DEQUEUE_SLEEP | DEQUEUE_DELAYED | DEQUEUE_NOCLOCK);
-
 	scoped_guard (sched_change, p, queue_flags) {
 
 		if (!(attr->sched_flags & SCHED_FLAG_KEEP_PARAMS)) {

Re: [RFC][PATCH 6/6] sched: Cleanup sched_delayed handling for class switches

[Peter Zijlstra]

On Wed, Oct 30, 2024 at 04:13:01PM +0100, Peter Zijlstra wrote:

> --- a/kernel/sched/ext.c
> +++ b/kernel/sched/ext.c

> @@ -5202,7 +5200,7 @@ static int scx_ops_enable(struct sched_e
>  	percpu_down_write(&scx_fork_rwsem);
>  	scx_task_iter_start(&sti);
>  	while ((p = scx_task_iter_next_locked(&sti))) {
> -		unsigned int queue_flags = DEQUEUE_SAVE | DEQUEUE_MOVE;
> +		unsigned int queue_flags = DEQUEUE_SAVE | DEQUEUE_MOVE | DEQUEUE_NOCLOCK;
>  		const struct sched_class *old_class = p->sched_class;
>  		const struct sched_class *new_class =
>  			__setscheduler_class(p->policy, p->prio);
> @@ -5210,9 +5208,7 @@ static int scx_ops_enable(struct sched_e
>  		if (old_class != new_class)
>  			queue_flags |= DEQUEUE_CLASS;
>  
> -		if (old_class != new_class && p->se.sched_delayed)
> -			dequeue_task(task_rq(p), p, DEQUEUE_SLEEP | DEQUEUE_DELAYED);
> -
> +		update_rq_clock(task_rq(p));
>  		scoped_guard (sched_change, p, DEQUEUE_SAVE | DEQUEUE_MOVE) {
>  			p->scx.slice = SCX_SLICE_DFL;
>  			p->sched_class = new_class;

TJ, strictly speaking you should probably do a __balance_callbacks() in
__scx_task_irq_rq_unlock(). The various switched_from() methods like to
queue_balance_callback().

I think you're currently good by the fact that only RT/DL do this, and
you're not touching those in these loops.

Re: [RFC][PATCH 6/6] sched: Cleanup sched_delayed handling for class switches

[Tejun Heo]

On Wed, Oct 30, 2024 at 04:45:51PM +0100, Peter Zijlstra wrote:
> TJ, strictly speaking you should probably do a __balance_callbacks() in
> __scx_task_irq_rq_unlock(). The various switched_from() methods like to
> queue_balance_callback().
> 
> I think you're currently good by the fact that only RT/DL do this, and
> you're not touching those in these loops.

Ah, will send a patch soon.

Thanks.

-- 
tejun