[PATCH] sched: Optimize core cookie matching check

[PATCH] sched: Optimize core cookie matching check

[Fernand Sieber]

Early return true if the core cookie matches. This avoids the SMT mask
loop to check for an idle core, which might be more expensive on wide
platforms.

Signed-off-by: Fernand Sieber <sieberf@amazon.com>
---
 kernel/sched/sched.h | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)

diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h
index adfb6e3409d7..381cd561e99b 100644
--- a/kernel/sched/sched.h
+++ b/kernel/sched/sched.h
@@ -1432,6 +1432,9 @@ static inline bool sched_core_cookie_match(struct rq *rq, struct task_struct *p)
 	if (!sched_core_enabled(rq))
 		return true;
 
+	if (rq->core->core_cookie == p->core_cookie)
+		return true;
+
 	for_each_cpu(cpu, cpu_smt_mask(cpu_of(rq))) {
 		if (!available_idle_cpu(cpu)) {
 			idle_core = false;
@@ -1443,7 +1446,7 @@ static inline bool sched_core_cookie_match(struct rq *rq, struct task_struct *p)
 	 * A CPU in an idle core is always the best choice for tasks with
 	 * cookies.
 	 */
-	return idle_core || rq->core->core_cookie == p->core_cookie;
+	return idle_core;
 }
 
 static inline bool sched_group_cookie_match(struct rq *rq,
-- 
2.43.0




Amazon Development Centre (South Africa) (Proprietary) Limited
29 Gogosoa Street, Observatory, Cape Town, Western Cape, 7925, South Africa
Registration Number: 2004 / 034463 / 07

Re: [PATCH] sched: Optimize core cookie matching check

[K Prateek Nayak]

Hello Fernand,

On 11/5/2025 8:55 PM, Fernand Sieber wrote:
> Early return true if the core cookie matches. This avoids the SMT mask
> loop to check for an idle core, which might be more expensive on wide
> platforms.
> 
> Signed-off-by: Fernand Sieber <sieberf@amazon.com>
> ---
>  kernel/sched/sched.h | 5 ++++-
>  1 file changed, 4 insertions(+), 1 deletion(-)
> 
> diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h
> index adfb6e3409d7..381cd561e99b 100644
> --- a/kernel/sched/sched.h
> +++ b/kernel/sched/sched.h
> @@ -1432,6 +1432,9 @@ static inline bool sched_core_cookie_match(struct rq *rq, struct task_struct *p)
>  	if (!sched_core_enabled(rq))
>  		return true;
>  
> +	if (rq->core->core_cookie == p->core_cookie)
> +		return true;

nit. We can use sched_cpu_cookie_match(rq, p) to check for the
above two conditions. but even this is good.

Apart from that, I think this optimization makes sense.

> +
>  	for_each_cpu(cpu, cpu_smt_mask(cpu_of(rq))) {
>  		if (!available_idle_cpu(cpu)) {
>  			idle_core = false;
> @@ -1443,7 +1446,7 @@ static inline bool sched_core_cookie_match(struct rq *rq, struct task_struct *p)
>  	 * A CPU in an idle core is always the best choice for tasks with
>  	 * cookies.
>  	 */
> -	return idle_core || rq->core->core_cookie == p->core_cookie;
> +	return idle_core;

Peter, do we care about checking the core_cookie again before
returning just in case the task cookie was selected between the
check above an here?

If not, then this looks good to me. Feel free to include:

Reviewed-by: K Prateek Nayak <kprateek.nayak@amd.com>

>  }
>  
>  static inline bool sched_group_cookie_match(struct rq *rq,

-- 
Thanks and Regards,
Prateek

Re: [PATCH] sched: Optimize core cookie matching check

[Peter Zijlstra]

On Wed, Nov 05, 2025 at 10:04:01PM +0530, K Prateek Nayak wrote:
> Hello Fernand,
> 
> On 11/5/2025 8:55 PM, Fernand Sieber wrote:
> > Early return true if the core cookie matches. This avoids the SMT mask
> > loop to check for an idle core, which might be more expensive on wide
> > platforms.
> > 
> > Signed-off-by: Fernand Sieber <sieberf@amazon.com>
> > ---
> >  kernel/sched/sched.h | 5 ++++-
> >  1 file changed, 4 insertions(+), 1 deletion(-)
> > 
> > diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h
> > index adfb6e3409d7..381cd561e99b 100644
> > --- a/kernel/sched/sched.h
> > +++ b/kernel/sched/sched.h
> > @@ -1432,6 +1432,9 @@ static inline bool sched_core_cookie_match(struct rq *rq, struct task_struct *p)
> >  	if (!sched_core_enabled(rq))
> >  		return true;
> >  
> > +	if (rq->core->core_cookie == p->core_cookie)
> > +		return true;
> 
> nit. We can use sched_cpu_cookie_match(rq, p) to check for the
> above two conditions. but even this is good.
> 
> Apart from that, I think this optimization makes sense.
> 
> > +
> >  	for_each_cpu(cpu, cpu_smt_mask(cpu_of(rq))) {
> >  		if (!available_idle_cpu(cpu)) {
> >  			idle_core = false;
> > @@ -1443,7 +1446,7 @@ static inline bool sched_core_cookie_match(struct rq *rq, struct task_struct *p)
> >  	 * A CPU in an idle core is always the best choice for tasks with
> >  	 * cookies.
> >  	 */
> > -	return idle_core || rq->core->core_cookie == p->core_cookie;
> > +	return idle_core;
> 
> Peter, do we care about checking the core_cookie again before
> returning just in case the task cookie was selected between the
> check above an here?

I don't think it really matters, but someone what runs this stuff would
probably know better than me ;-)

> If not, then this looks good to me. Feel free to include:
> 
> Reviewed-by: K Prateek Nayak <kprateek.nayak@amd.com>

Thanks all!

Re: [PATCH] sched: Optimize core cookie matching check

[Madadi Vineeth Reddy]

Hi Fernand,

On 05/11/25 20:55, Fernand Sieber wrote:
> Early return true if the core cookie matches. This avoids the SMT mask
> loop to check for an idle core, which might be more expensive on wide
> platforms.
> 
> Signed-off-by: Fernand Sieber <sieberf@amazon.com>
> ---
>  kernel/sched/sched.h | 5 ++++-
>  1 file changed, 4 insertions(+), 1 deletion(-)
> 
> diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h
> index adfb6e3409d7..381cd561e99b 100644
> --- a/kernel/sched/sched.h
> +++ b/kernel/sched/sched.h
> @@ -1432,6 +1432,9 @@ static inline bool sched_core_cookie_match(struct rq *rq, struct task_struct *p)
>  	if (!sched_core_enabled(rq))
>  		return true;
>  
> +	if (rq->core->core_cookie == p->core_cookie)
> +		return true;
> +
>  	for_each_cpu(cpu, cpu_smt_mask(cpu_of(rq))) {
>  		if (!available_idle_cpu(cpu)) {
>  			idle_core = false;
> @@ -1443,7 +1446,7 @@ static inline bool sched_core_cookie_match(struct rq *rq, struct task_struct *p)
>  	 * A CPU in an idle core is always the best choice for tasks with
>  	 * cookies.
>  	 */
> -	return idle_core || rq->core->core_cookie == p->core_cookie;
> +	return idle_core;
>  }

LGTM.
Reviewed-by: Madadi Vineeth Reddy <vineethr@linux.ibm.com>

Thanks,
Madadi Vineeth Reddy

>  
>  static inline bool sched_group_cookie_match(struct rq *rq,

[tip: sched/core] sched/core: Optimize core cookie matching check

[tip-bot2 for Fernand Sieber]

The following commit has been merged into the sched/core branch of tip:

Commit-ID:     7f829bde94b1c97b1804fa5860e066ea49dbfca3
Gitweb:        https://git.kernel.org/tip/7f829bde94b1c97b1804fa5860e066ea49dbfca3
Author:        Fernand Sieber <sieberf@amazon.com>
AuthorDate:    Wed, 05 Nov 2025 17:25:37 +02:00
Committer:     Peter Zijlstra <peterz@infradead.org>
CommitterDate: Tue, 11 Nov 2025 12:33:37 +01:00

sched/core: Optimize core cookie matching check

Early return true if the core cookie matches. This avoids the SMT mask
loop to check for an idle core, which might be more expensive on wide
platforms.

Signed-off-by: Fernand Sieber <sieberf@amazon.com>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Reviewed-by: K Prateek Nayak <kprateek.nayak@amd.com>
Reviewed-by: Madadi Vineeth Reddy <vineethr@linux.ibm.com>
Link: https://patch.msgid.link/20251105152538.470586-1-sieberf@amazon.com
---
 kernel/sched/sched.h | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)

diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h
index d04e007..82e74e8 100644
--- a/kernel/sched/sched.h
+++ b/kernel/sched/sched.h
@@ -1432,6 +1432,9 @@ static inline bool sched_core_cookie_match(struct rq *rq, struct task_struct *p)
 	if (!sched_core_enabled(rq))
 		return true;
 
+	if (rq->core->core_cookie == p->core_cookie)
+		return true;
+
 	for_each_cpu(cpu, cpu_smt_mask(cpu_of(rq))) {
 		if (!available_idle_cpu(cpu)) {
 			idle_core = false;
@@ -1443,7 +1446,7 @@ static inline bool sched_core_cookie_match(struct rq *rq, struct task_struct *p)
 	 * A CPU in an idle core is always the best choice for tasks with
 	 * cookies.
 	 */
-	return idle_core || rq->core->core_cookie == p->core_cookie;
+	return idle_core;
 }
 
 static inline bool sched_group_cookie_match(struct rq *rq,