From mboxrd@z Thu Jan 1 00:00:00 1970 Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.remote-ip=100.103.45.18 ARC-Seal: i=1; d=subspace.kernel.org; s=arc20261015; a=rsa-sha256; cv=none; t=1794587673; b=Xoi1TJQLjJfL1I7Pe/9z3OPWoS6KKr7G482kaLegEM2ej79GkyAbMEkMGjNAtM51sn6/ 55dz8SudG9PoiMGZxYVafRHJfxXwCMBjTyyPDqDA4VIb4aXAhHeRpHr1l5/ayt1MAeBPH M+S3QTeSPs8jl7RW++9soj/DwrsOwurE6lRdZXJdUm50oQOfk1LhFT0VD0XnXQxJCbDJS dd2G5suK3WvkLceMqyMbDNS/wTTtWspAEDOD5e1e4SUWoK+bieC5/PLHDEFAPS4C+g0Uy AfTOA88Z2N7K2cUz4smsD70NU2po49njkKnaCVMYIcQ2CjMB62CDkQLy58ggi3m1lxQ== ARC-Message-Signature: i=1; d=subspace.kernel.org; s=arc20261015; a=rsa-sha256; c=relaxed/relaxed; t=1794587673; h=Received:DKIM-Signature:From:To:Cc:Subject:Date:Message-ID:X-Mailer: In-Reply-To:References:MIME-Version:Content-Transfer-Encoding; bh=HuPqCBL72/mppOuVzQLZ7BYTQg1KEZjO6/BoIREVCFM=; b=eTTXt52JCzUTQEe4lrK2uqHReP37LrmHUoIsHgpAHIJBNWs460xMh1yNTeU92pZyRMGw 38t+ihoFLxeykbq8kHQInGGrFWA+wb4cuPJNvbHaqf3Uoz3dB5kHpX0qlXC34Jkz7v+QG YzRRnMQKS70KNjG8/6aAF8m5aKSC986+ahNxTvSVN0mIvZQU7WHpNPdYe2jWqi6U78l6i UV2n5tQyBPGuA0XmU9iQJNuyzvDBYgJOszq5qYhfN6U46uO0nRMZarLLq2pZ3tCt/RFnx +O+UmyTCAV04bo7h2uEYCZCrP68kpVLOW+aeVsZEKJ64hiyV65xllxQC6PR8pU4obVg== ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass header.d=kernel.org header.i=@kernel.org header.a=rsa-sha256 header.s=k20260515 header.b=mfhuiSpp; arc=none smtp.remote-ip=100.103.45.18 Received: from smtp.kernel.org (quasi.space.kernel.org [100.103.45.18]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (prime256v1)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 69A63606A330 for ; Fri, 13 Nov 2026 16:34:33 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.a=rsa-sha256 header.s=k20260515 header.b=mfhuiSpp Received: by smtp.kernel.org (Postfix) with ESMTPSA id 5D3311F000C7; Fri, 13 Nov 2026 16:34:32 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel.org; s=k20260515; t=1794587673; bh=HuPqCBL72/mppOuVzQLZ7BYTQg1KEZjO6/BoIREVCFM=; h=From:To:Cc:Subject:Date:In-Reply-To:References; b=mfhuiSppfdhiuLCsnDHkrNfgfWkmZ7Fon36/iKIQODkaONh/E27YR4j1g9SQosjIs nrOvdQi76UVfCSkS4eQD+PkNHL9l5WND2lYmNUTHFP+ZuHc/SF1jYZahG911sWzpra fCbZE+rSJE3rc5uHd+0Cis74+SPTie9YNtw8AuYFbTMx3TZbpSUzbtCt3qRbURTTsj JVM/Vt+Nk5I7+2Tm9Rx30P7J694zWp6Owb70n33C2V9UkiFRWl/1t2Gmy52XbwDn4J v67k9wW9C/lYiFEFeabxpDv4WI0KN+JxcLxRyssjWoinwD+aNDm8/O1/j6/On/o+FW bzbdkEMetcSUA== From: Sasha Levin To: stable@vger.kernel.org Cc: Peter Zijlstra , Juri Lelli , Tejun Heo , Vincent Guittot , Sasha Levin Subject: [PATCH 6.18.y 2/3] sched: Fold sched_class::switch{ing,ed}_{to,from}() into the change pattern Date: Fri, 13 Nov 2026 11:34:29 -0500 Message-ID: <20261113163420.2086653-2-sashal@kernel.org> X-Mailer: git-send-email 2.53.0 In-Reply-To: <20261113163420.2086653-1-sashal@kernel.org> References: <20261109200556-failed-6.18-c282051fc2ca@kernel.org> <20261113163420.2086653-1-sashal@kernel.org> X-Mailing-List: stable@vger.kernel.org List-Id: List-Post: List-Help: List-Subscribe: List-Unsubscribe: List-Owner: Precedence: list MIME-Version: 1.0 Content-Transfer-Encoding: 8bit From: Peter Zijlstra [ Upstream commit 637b0682821b144d5993211cf0a768b322138a69 ] Add {DE,EN}QUEUE_CLASS and fold the sched_class::switch* methods into the change pattern. This completes and makes the pattern more symmetric. This changes the order of callbacks slightly: OLD NEW | | switching_from() dequeue_task(); | dequeue_task() put_prev_task(); | put_prev_task() | switched_from() | ... change task ... | ... change task ... | switching_to(); | switching_to() enqueue_task(); | enqueue_task() set_next_task(); | set_next_task() prev_class->switched_from() | switched_to() | switched_to() | Notably, it moves the switched_from() callback right after the dequeue/put. Existing implementations don't appear to be affected by this change in location -- specifically the task isn't enqueued on the class in question in either location. Make (CLASS)^(SAVE|MOVE), because there is nothing to save-restore when changing scheduling classes. Signed-off-by: Peter Zijlstra (Intel) Reviewed-by: Juri Lelli Acked-by: Tejun Heo Acked-by: Vincent Guittot Stable-dep-of: c282051fc2ca ("sched/fair: Clear throttle state when leaving the fair class") Signed-off-by: Sasha Levin --- kernel/sched/core.c | 51 +++++++++++++++++----------------------- kernel/sched/ext.c | 22 ++++++++++++----- kernel/sched/idle.c | 4 ++-- kernel/sched/rt.c | 2 +- kernel/sched/sched.h | 22 +++++++---------- kernel/sched/stop_task.c | 4 ++-- kernel/sched/syscalls.c | 7 ++++-- 7 files changed, 55 insertions(+), 57 deletions(-) diff --git a/kernel/sched/core.c b/kernel/sched/core.c index dbd871c6235f4..57ba5fcff1339 100644 --- a/kernel/sched/core.c +++ b/kernel/sched/core.c @@ -2185,34 +2185,9 @@ inline int task_curr(const struct task_struct *p) return cpu_curr(task_cpu(p)) == p; } -/* - * ->switching_to() is called with the pi_lock and rq_lock held and must not - * mess with locking. - */ -void check_class_changing(struct rq *rq, struct task_struct *p, - const struct sched_class *prev_class) -{ - if (prev_class != p->sched_class && p->sched_class->switching_to) - p->sched_class->switching_to(rq, p); -} - -/* - * switched_from, switched_to and prio_changed must _NOT_ drop rq->lock, - * use the balance_callback list if you want balancing. - * - * this means any call to check_class_changed() must be followed by a call to - * balance_callback(). - */ -void check_class_changed(struct rq *rq, struct task_struct *p, - const struct sched_class *prev_class, - int oldprio) +void check_prio_changed(struct rq *rq, struct task_struct *p, int oldprio) { - if (prev_class != p->sched_class) { - if (prev_class->switched_from) - prev_class->switched_from(rq, p); - - p->sched_class->switched_to(rq, p); - } else if (oldprio != p->prio || dl_task(p)) + if (oldprio != p->prio || dl_task(p)) p->sched_class->prio_changed(rq, p, oldprio); } @@ -7413,6 +7388,9 @@ void rt_mutex_setprio(struct task_struct *p, struct task_struct *pi_task) prev_class = p->sched_class; next_class = __setscheduler_class(p->policy, prio); + if (prev_class != next_class) + queue_flag |= DEQUEUE_CLASS; + if (prev_class != next_class && p->se.sched_delayed) dequeue_task(rq, p, DEQUEUE_SLEEP | DEQUEUE_DELAYED | DEQUEUE_NOCLOCK); @@ -7449,11 +7427,10 @@ void rt_mutex_setprio(struct task_struct *p, struct task_struct *pi_task) p->sched_class = next_class; p->prio = prio; - - check_class_changing(rq, p, prev_class); } - check_class_changed(rq, p, prev_class, oldprio); + if (!(queue_flag & DEQUEUE_CLASS)) + check_prio_changed(rq, p, oldprio); out_unlock: /* Avoid rq from going away on us: */ preempt_disable(); @@ -10892,6 +10869,11 @@ struct sched_change_ctx *sched_change_begin(struct task_struct *p, unsigned int lockdep_assert_rq_held(rq); + if (flags & DEQUEUE_CLASS) { + if (p->sched_class->switching_from) + p->sched_class->switching_from(rq, p); + } + *ctx = (struct sched_change_ctx){ .p = p, .flags = flags, @@ -10904,6 +10886,9 @@ struct sched_change_ctx *sched_change_begin(struct task_struct *p, unsigned int if (ctx->running) put_prev_task(rq, p); + if ((flags & DEQUEUE_CLASS) && p->sched_class->switched_from) + p->sched_class->switched_from(rq, p); + return ctx; } @@ -10914,8 +10899,14 @@ void sched_change_end(struct sched_change_ctx *ctx) lockdep_assert_rq_held(rq); + if ((ctx->flags & ENQUEUE_CLASS) && p->sched_class->switching_to) + p->sched_class->switching_to(rq, p); + if (ctx->queued) enqueue_task(rq, p, ctx->flags | ENQUEUE_NOCLOCK); if (ctx->running) set_next_task(rq, p); + + if ((ctx->flags & ENQUEUE_CLASS) && p->sched_class->switched_to) + p->sched_class->switched_to(rq, p); } diff --git a/kernel/sched/ext.c b/kernel/sched/ext.c index 3117d1ef869c5..48b8fb873b852 100644 --- a/kernel/sched/ext.c +++ b/kernel/sched/ext.c @@ -4079,20 +4079,25 @@ static void scx_root_disable(struct scx_sched *sch) scx_task_iter_start(&sti); while ((p = scx_task_iter_next_locked(&sti))) { + unsigned int queue_flags = DEQUEUE_SAVE | DEQUEUE_MOVE | DEQUEUE_NOCLOCK; const struct sched_class *old_class = p->sched_class; const struct sched_class *new_class = scx_setscheduler_class(p); update_rq_clock(task_rq(p)); + if (old_class != new_class) + queue_flags |= DEQUEUE_CLASS; + if (old_class != new_class && p->se.sched_delayed) dequeue_task(task_rq(p), p, DEQUEUE_SLEEP | DEQUEUE_DELAYED | DEQUEUE_NOCLOCK); - scoped_guard (sched_change, p, DEQUEUE_SAVE | DEQUEUE_MOVE | DEQUEUE_NOCLOCK) { + scoped_guard (sched_change, p, queue_flags) { p->sched_class = new_class; - check_class_changing(task_rq(p), p, old_class); } - check_class_changed(task_rq(p), p, old_class, p->prio); + if (!(queue_flags & DEQUEUE_CLASS)) + check_prio_changed(task_rq(p), p, p->prio); + scx_exit_task(p); } scx_task_iter_stop(&sti); @@ -4932,6 +4937,7 @@ static void scx_root_enable_workfn(struct kthread_work *work) percpu_down_write(&scx_fork_rwsem); scx_task_iter_start(&sti); while ((p = scx_task_iter_next_locked(&sti))) { + unsigned int queue_flags = DEQUEUE_SAVE | DEQUEUE_MOVE | DEQUEUE_NOCLOCK; const struct sched_class *old_class = p->sched_class; const struct sched_class *new_class = scx_setscheduler_class(p); @@ -4940,16 +4946,20 @@ static void scx_root_enable_workfn(struct kthread_work *work) update_rq_clock(task_rq(p)); + if (old_class != new_class) + queue_flags |= DEQUEUE_CLASS; + if (old_class != new_class && p->se.sched_delayed) dequeue_task(task_rq(p), p, DEQUEUE_SLEEP | DEQUEUE_DELAYED | DEQUEUE_NOCLOCK); - scoped_guard (sched_change, p, DEQUEUE_SAVE | DEQUEUE_MOVE | DEQUEUE_NOCLOCK) { + scoped_guard (sched_change, p, queue_flags) { p->scx.slice = SCX_SLICE_DFL; p->sched_class = new_class; - check_class_changing(task_rq(p), p, old_class); } - check_class_changed(task_rq(p), p, old_class, p->prio); + if (!(queue_flags & DEQUEUE_CLASS)) + check_prio_changed(task_rq(p), p, p->prio); + put_task_struct(p); } scx_task_iter_stop(&sti); diff --git a/kernel/sched/idle.c b/kernel/sched/idle.c index 220fc9e7c22db..d5607b2150ba4 100644 --- a/kernel/sched/idle.c +++ b/kernel/sched/idle.c @@ -525,7 +525,7 @@ static void task_tick_idle(struct rq *rq, struct task_struct *curr, int queued) { } -static void switched_to_idle(struct rq *rq, struct task_struct *p) +static void switching_to_idle(struct rq *rq, struct task_struct *p) { BUG(); } @@ -563,6 +563,6 @@ DEFINE_SCHED_CLASS(idle) = { .task_tick = task_tick_idle, .prio_changed = prio_changed_idle, - .switched_to = switched_to_idle, + .switching_to = switching_to_idle, .update_curr = update_curr_idle, }; diff --git a/kernel/sched/rt.c b/kernel/sched/rt.c index fc9737decfb73..6274bda1d5706 100644 --- a/kernel/sched/rt.c +++ b/kernel/sched/rt.c @@ -2609,8 +2609,8 @@ DEFINE_SCHED_CLASS(rt) = { .get_rr_interval = get_rr_interval_rt, - .prio_changed = prio_changed_rt, .switched_to = switched_to_rt, + .prio_changed = prio_changed_rt, .update_curr = update_curr_rt, diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h index 2e0604b3a337f..81ccbfbb81521 100644 --- a/kernel/sched/sched.h +++ b/kernel/sched/sched.h @@ -21,7 +21,6 @@ #include #include #include - #include #include #include @@ -2384,6 +2383,7 @@ extern const u32 sched_prio_to_wmult[40]; #define DEQUEUE_MIGRATING 0x0010 /* Matches ENQUEUE_MIGRATING */ #define DEQUEUE_DELAYED 0x0020 /* Matches ENQUEUE_DELAYED */ +#define DEQUEUE_CLASS 0x0040 /* Matches ENQUEUE_CLASS */ #define DEQUEUE_SPECIAL 0x00010000 #define DEQUEUE_THROTTLE 0x00020000 @@ -2395,6 +2395,7 @@ extern const u32 sched_prio_to_wmult[40]; #define ENQUEUE_MIGRATING 0x0010 #define ENQUEUE_DELAYED 0x0020 +#define ENQUEUE_CLASS 0x0040 #define ENQUEUE_HEAD 0x00010000 #define ENQUEUE_REPLENISH 0x00020000 @@ -2458,14 +2459,11 @@ struct sched_class { void (*task_fork)(struct task_struct *p); void (*task_dead)(struct task_struct *p); - /* - * The switched_from() call is allowed to drop rq->lock, therefore we - * cannot assume the switched_from/switched_to pair is serialized by - * rq->lock. They are however serialized by p->pi_lock. - */ - void (*switching_to) (struct rq *this_rq, struct task_struct *task); - void (*switched_from)(struct rq *this_rq, struct task_struct *task); - void (*switched_to) (struct rq *this_rq, struct task_struct *task); + void (*switching_from)(struct rq *this_rq, struct task_struct *task); + void (*switched_from) (struct rq *this_rq, struct task_struct *task); + void (*switching_to) (struct rq *this_rq, struct task_struct *task); + void (*switched_to) (struct rq *this_rq, struct task_struct *task); + void (*reweight_task)(struct rq *this_rq, struct task_struct *task, const struct load_weight *lw); void (*prio_changed) (struct rq *this_rq, struct task_struct *task, @@ -3892,11 +3890,7 @@ extern void set_load_weight(struct task_struct *p, bool update_load); extern void enqueue_task(struct rq *rq, struct task_struct *p, int flags); extern bool dequeue_task(struct rq *rq, struct task_struct *p, int flags); -extern void check_class_changing(struct rq *rq, struct task_struct *p, - const struct sched_class *prev_class); -extern void check_class_changed(struct rq *rq, struct task_struct *p, - const struct sched_class *prev_class, - int oldprio); +extern void check_prio_changed(struct rq *rq, struct task_struct *p, int oldprio); extern struct balance_callback *splice_balance_callbacks(struct rq *rq); extern void balance_callbacks(struct rq *rq, struct balance_callback *head); diff --git a/kernel/sched/stop_task.c b/kernel/sched/stop_task.c index 73aeb0743aa2e..a781f4b44abf9 100644 --- a/kernel/sched/stop_task.c +++ b/kernel/sched/stop_task.c @@ -75,7 +75,7 @@ static void task_tick_stop(struct rq *rq, struct task_struct *curr, int queued) { } -static void switched_to_stop(struct rq *rq, struct task_struct *p) +static void switching_to_stop(struct rq *rq, struct task_struct *p) { BUG(); /* its impossible to change to this class */ } @@ -112,6 +112,6 @@ DEFINE_SCHED_CLASS(stop) = { .task_tick = task_tick_stop, .prio_changed = prio_changed_stop, - .switched_to = switched_to_stop, + .switching_to = switching_to_stop, .update_curr = update_curr_stop, }; diff --git a/kernel/sched/syscalls.c b/kernel/sched/syscalls.c index 77b663a5dfb2b..b044a3aa4a0ad 100644 --- a/kernel/sched/syscalls.c +++ b/kernel/sched/syscalls.c @@ -713,6 +713,9 @@ int __sched_setscheduler(struct task_struct *p, prev_class = p->sched_class; next_class = __setscheduler_class(policy, newprio); + if (prev_class != next_class) + queue_flags |= DEQUEUE_CLASS; + if (prev_class != next_class && p->se.sched_delayed) dequeue_task(rq, p, DEQUEUE_SLEEP | DEQUEUE_DELAYED | DEQUEUE_NOCLOCK); @@ -725,7 +728,6 @@ int __sched_setscheduler(struct task_struct *p, __setscheduler_dl_pi(newprio, policy, p, scope); } __setscheduler_uclamp(p, attr); - check_class_changing(rq, p, prev_class); if (scope->queued) { /* @@ -737,7 +739,8 @@ int __sched_setscheduler(struct task_struct *p, } } - check_class_changed(rq, p, prev_class, oldprio); + if (!(queue_flags & DEQUEUE_CLASS)) + check_prio_changed(rq, p, oldprio); /* Avoid rq from going away on us: */ preempt_disable(); -- 2.53.0