[PATCH] sched/fair: Skip sched-domain traversal when wake affine is disabled

[PATCH] sched/fair: Skip sched-domain traversal when wake affine is disabled

[Srikar Dronamraju]

In select_task_rq_fair(), TTWU wakeups for which wake_wide() returns
true, or where the waking CPU is not in the task's allowed CPU mask,
want_affine is set to false.

For these wakeups, the scheduler does not perform wake-affine placement.
The existing code already treats this as a fast-path by exiting the
sched-domain walk before locating an SD_BALANCE_WAKE domain, ultimately
falling through to select_idle_sibling().

However, select_task_rq_fair() still enters the sched-domain traversal
only to break out almost immediately. Avoid this unnecessary work by
jumping directly to the existing fast path when want_affine is false.

As part of this cleanup:

  - Move want_affine into the WF_TTWU block where it is used.
  - Skip the sched-domain traversal entirely when want_affine is false.
  - Remove the now redundant want_affine test from the
    SD_WAKE_AFFINE check.
  - Remove the corresponding !want_affine loop termination logic.

This preserves existing wakeup placement behavior while avoiding
unnecessary sched-domain iteration for TTWU wakeups that cannot make
use of wake-affine placement.

Signed-off-by: Srikar Dronamraju <srikar@linux.ibm.com>
---
 kernel/sched/fair.c | 9 +++++----
 1 file changed, 5 insertions(+), 4 deletions(-)

diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
index 3ebec186f982..58ff55e093cb 100644
--- a/kernel/sched/fair.c
+++ b/kernel/sched/fair.c
@@ -8832,7 +8832,6 @@ select_task_rq_fair(struct task_struct *p, int prev_cpu, int wake_flags)
 	struct sched_domain *tmp, *sd = NULL;
 	int cpu = smp_processor_id();
 	int new_cpu = prev_cpu;
-	int want_affine = 0;
 	/* SD_flags and WF_flags share the first nibble */
 	int sd_flag = wake_flags & 0xF;
 
@@ -8841,6 +8840,8 @@ select_task_rq_fair(struct task_struct *p, int prev_cpu, int wake_flags)
 	 */
 	lockdep_assert_held(&p->pi_lock);
 	if (wake_flags & WF_TTWU) {
+		int want_affine;
+
 		record_wakee(p);
 
 		if ((wake_flags & WF_CURRENT_CPU) &&
@@ -8855,6 +8856,8 @@ select_task_rq_fair(struct task_struct *p, int prev_cpu, int wake_flags)
 		}
 
 		want_affine = !wake_wide(p) && cpumask_test_cpu(cpu, p->cpus_ptr);
+		if (!want_affine)
+			goto fast_path;
 	}
 
 	for_each_domain(cpu, tmp) {
@@ -8862,7 +8865,7 @@ select_task_rq_fair(struct task_struct *p, int prev_cpu, int wake_flags)
 		 * If both 'cpu' and 'prev_cpu' are part of this domain,
 		 * cpu is a valid SD_WAKE_AFFINE target.
 		 */
-		if (want_affine && (tmp->flags & SD_WAKE_AFFINE) &&
+		if ((tmp->flags & SD_WAKE_AFFINE) &&
 		    cpumask_test_cpu(prev_cpu, sched_domain_span(tmp))) {
 			if (cpu != prev_cpu)
 				new_cpu = wake_affine(tmp, p, cpu, prev_cpu, sync);
@@ -8878,8 +8881,6 @@ select_task_rq_fair(struct task_struct *p, int prev_cpu, int wake_flags)
 		 */
 		if (tmp->flags & sd_flag)
 			sd = tmp;
-		else if (!want_affine)
-			break;
 	}
 
 	/* Slow path */
@@ -8887,6 +8888,7 @@ select_task_rq_fair(struct task_struct *p, int prev_cpu, int wake_flags)
 		return sched_balance_find_dst_cpu(sd, p, cpu, prev_cpu, sd_flag);
 
 	/* Fast path */
+fast_path:
 	if (wake_flags & WF_TTWU)
 		return select_idle_sibling(p, prev_cpu, new_cpu);
 
-- 
2.43.0

Re: [PATCH] sched/fair: Skip sched-domain traversal when wake affine is disabled

[Kayra Cizmeci]

Hi!

On Mon, Nov 09, 2026 at 05:35:22PM +0530, Srikar Dronamraju wrote:
> In select_task_rq_fair(), TTWU wakeups for which wake_wide() returns
> true, or where the waking CPU is not in the task's allowed CPU mask,
> want_affine is set to false.
> 
> For these wakeups, the scheduler does not perform wake-affine placement.
> The existing code already treats this as a fast-path by exiting the
> sched-domain walk before locating an SD_BALANCE_WAKE domain, ultimately
> falling through to select_idle_sibling().

Not before locating. If it doesn't locates SD_BALANCE_WAKE in the first
check it just leaves.

AFAICT SD_BALANCE_WAKE is not set anywhere tho.

So if you wanna do this, maybe you can do something like:

diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
index d4cc0caee626..b40b2eb5e577 100644
--- a/kernel/sched/fair.c
+++ b/kernel/sched/fair.c
@@ -10016,6 +10016,9 @@ select_task_rq_fair(struct task_struct *p, int prev_cpu, int wake_flags)
 		}
 
 		want_affine = !wake_wide(p) && cpumask_test_cpu(cpu, p->cpus_ptr);
+
+		if (!want_affine)
+			return select_idle_sibling(p, prev_cpu, new_cpu);
 	}
 
 	for_each_domain(cpu, tmp) {

> 
> However, select_task_rq_fair() still enters the sched-domain traversal
> only to break out almost immediately. Avoid this unnecessary work by
> jumping directly to the existing fast path when want_affine is false.
> 
> As part of this cleanup:
> 
>   - Move want_affine into the WF_TTWU block where it is used.
>   - Skip the sched-domain traversal entirely when want_affine is false.
>   - Remove the now redundant want_affine test from the
>     SD_WAKE_AFFINE check.
>   - Remove the corresponding !want_affine loop termination logic.
> 
> This preserves existing wakeup placement behavior while avoiding
> unnecessary sched-domain iteration for TTWU wakeups that cannot make
> use of wake-affine placement.
> 
> Signed-off-by: Srikar Dronamraju <srikar@linux.ibm.com>
> ---
>  kernel/sched/fair.c | 9 +++++----
>  1 file changed, 5 insertions(+), 4 deletions(-)
> 
> diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
> index 3ebec186f982..58ff55e093cb 100644
> --- a/kernel/sched/fair.c
> +++ b/kernel/sched/fair.c
> @@ -8832,7 +8832,6 @@ select_task_rq_fair(struct task_struct *p, int prev_cpu, int wake_flags)
>  	struct sched_domain *tmp, *sd = NULL;
>  	int cpu = smp_processor_id();
>  	int new_cpu = prev_cpu;
> -	int want_affine = 0;
>  	/* SD_flags and WF_flags share the first nibble */
>  	int sd_flag = wake_flags & 0xF;
>  
> @@ -8841,6 +8840,8 @@ select_task_rq_fair(struct task_struct *p, int prev_cpu, int wake_flags)
>  	 */
>  	lockdep_assert_held(&p->pi_lock);
>  	if (wake_flags & WF_TTWU) {
> +		int want_affine;
> +
>  		record_wakee(p);
>  
>  		if ((wake_flags & WF_CURRENT_CPU) &&
> @@ -8855,6 +8856,8 @@ select_task_rq_fair(struct task_struct *p, int prev_cpu, int wake_flags)
>  		}
>  
>  		want_affine = !wake_wide(p) && cpumask_test_cpu(cpu, p->cpus_ptr);
> +		if (!want_affine)
> +			goto fast_path;
>  	}
>  
>  	for_each_domain(cpu, tmp) {
> @@ -8862,7 +8865,7 @@ select_task_rq_fair(struct task_struct *p, int prev_cpu, int wake_flags)
>  		 * If both 'cpu' and 'prev_cpu' are part of this domain,
>  		 * cpu is a valid SD_WAKE_AFFINE target.
>  		 */
> -		if (want_affine && (tmp->flags & SD_WAKE_AFFINE) &&
> +		if ((tmp->flags & SD_WAKE_AFFINE) &&
>  		    cpumask_test_cpu(prev_cpu, sched_domain_span(tmp))) {
>  			if (cpu != prev_cpu)
>  				new_cpu = wake_affine(tmp, p, cpu, prev_cpu, sync);
> @@ -8878,8 +8881,6 @@ select_task_rq_fair(struct task_struct *p, int prev_cpu, int wake_flags)
>  		 */
>  		if (tmp->flags & sd_flag)
>  			sd = tmp;
> -		else if (!want_affine)
> -			break;
>  	}


This just changes the behavior.

Old Code WF_FORK 1, WF_TTWU 0:

want_affine = 0, so we skip the first if block in the loop then leave
once we find a domain without our flag. And go with slow path.


New Code:

want_affine is moved away and early exit is removed. So we enter the if
block and try to find a domain with prev_cpu. Because WF_TTWU 
is 0, we just return new_cpu. Or if we can't fimd the domain with
prev_cpu, slow path.


Cheers,

Kayra

Re: [PATCH] sched/fair: Skip sched-domain traversal when wake affine is disabled

[Srikar Dronamraju]

* Kayra Cizmeci <kayracizmeci@gmail.com> [2026-11-09 18:09:56]:

Hi Kayra,

> 
> On Mon, Nov 09, 2026 at 05:35:22PM +0530, Srikar Dronamraju wrote:
> > In select_task_rq_fair(), TTWU wakeups for which wake_wide() returns
> > true, or where the waking CPU is not in the task's allowed CPU mask,
> > want_affine is set to false.
> > 
> > For these wakeups, the scheduler does not perform wake-affine placement.
> > The existing code already treats this as a fast-path by exiting the
> > sched-domain walk before locating an SD_BALANCE_WAKE domain, ultimately
> > falling through to select_idle_sibling().
> 
> Not before locating. If it doesn't locates SD_BALANCE_WAKE in the first
> check it just leaves.
> 
> AFAICT SD_BALANCE_WAKE is not set anywhere tho.

from kernel/sched/sched.h
#define WF_TTWU                 0x08 /* Wakeup;            maps to SD_BALANCE_WAKE *
The code searches for WF_TTWU in lieu of SD_BALANCE_WAKE.

> 
> So if you wanna do this, maybe you can do something like:
> 
> diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
> index d4cc0caee626..b40b2eb5e577 100644
> --- a/kernel/sched/fair.c
> +++ b/kernel/sched/fair.c
> @@ -10016,6 +10016,9 @@ select_task_rq_fair(struct task_struct *p, int prev_cpu, int wake_flags)
>  		}
>  
>  		want_affine = !wake_wide(p) && cpumask_test_cpu(cpu, p->cpus_ptr);
> +
> +		if (!want_affine)
> +			return select_idle_sibling(p, prev_cpu, new_cpu);
>  	}

Yes, this also has the same effect.

>  
>  	for_each_domain(cpu, tmp) {
> 
> > 
> > However, select_task_rq_fair() still enters the sched-domain traversal
> > only to break out almost immediately. Avoid this unnecessary work by
> > jumping directly to the existing fast path when want_affine is false.
> > 
> > As part of this cleanup:
> > 
> >   - Move want_affine into the WF_TTWU block where it is used.
> >   - Skip the sched-domain traversal entirely when want_affine is false.
> >   - Remove the now redundant want_affine test from the
> >     SD_WAKE_AFFINE check.
> >   - Remove the corresponding !want_affine loop termination logic.
> > 
> > This preserves existing wakeup placement behavior while avoiding
> > unnecessary sched-domain iteration for TTWU wakeups that cannot make
> > use of wake-affine placement.
> > 
> > Signed-off-by: Srikar Dronamraju <srikar@linux.ibm.com>
> > ---
> >  kernel/sched/fair.c | 9 +++++----
> >  1 file changed, 5 insertions(+), 4 deletions(-)
> > 
> > diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
> > index 3ebec186f982..58ff55e093cb 100644
> > --- a/kernel/sched/fair.c
> > +++ b/kernel/sched/fair.c
> > @@ -8832,7 +8832,6 @@ select_task_rq_fair(struct task_struct *p, int prev_cpu, int wake_flags)
> >  	struct sched_domain *tmp, *sd = NULL;
> >  	int cpu = smp_processor_id();
> >  	int new_cpu = prev_cpu;
> > -	int want_affine = 0;
> >  	/* SD_flags and WF_flags share the first nibble */
> >  	int sd_flag = wake_flags & 0xF;
> >  
> > @@ -8841,6 +8840,8 @@ select_task_rq_fair(struct task_struct *p, int prev_cpu, int wake_flags)
> >  	 */
> >  	lockdep_assert_held(&p->pi_lock);
> >  	if (wake_flags & WF_TTWU) {
> > +		int want_affine;
> > +
> >  		record_wakee(p);
> >  
> >  		if ((wake_flags & WF_CURRENT_CPU) &&
> > @@ -8855,6 +8856,8 @@ select_task_rq_fair(struct task_struct *p, int prev_cpu, int wake_flags)
> >  		}
> >  
> >  		want_affine = !wake_wide(p) && cpumask_test_cpu(cpu, p->cpus_ptr);
> > +		if (!want_affine)
> > +			goto fast_path;
> >  	}
> >  
> >  	for_each_domain(cpu, tmp) {
> > @@ -8862,7 +8865,7 @@ select_task_rq_fair(struct task_struct *p, int prev_cpu, int wake_flags)
> >  		 * If both 'cpu' and 'prev_cpu' are part of this domain,
> >  		 * cpu is a valid SD_WAKE_AFFINE target.
> >  		 */
> > -		if (want_affine && (tmp->flags & SD_WAKE_AFFINE) &&
> > +		if ((tmp->flags & SD_WAKE_AFFINE) &&
> >  		    cpumask_test_cpu(prev_cpu, sched_domain_span(tmp))) {
> >  			if (cpu != prev_cpu)
> >  				new_cpu = wake_affine(tmp, p, cpu, prev_cpu, sync);
> > @@ -8878,8 +8881,6 @@ select_task_rq_fair(struct task_struct *p, int prev_cpu, int wake_flags)
> >  		 */
> >  		if (tmp->flags & sd_flag)
> >  			sd = tmp;
> > -		else if (!want_affine)
> > -			break;
> >  	}
> 
> 
> This just changes the behavior.
> 
> Old Code WF_FORK 1, WF_TTWU 0:
> 
> want_affine = 0, so we skip the first if block in the loop then leave
> once we find a domain without our flag. And go with slow path.
> 
> 
> New Code:
> 
> want_affine is moved away and early exit is removed. So we enter the if
> block and try to find a domain with prev_cpu. Because WF_TTWU 
> is 0, we just return new_cpu. Or if we can't fimd the domain with
> prev_cpu, slow path.
> 

When WF_TTWU is set, we are always suppose take the fast path.
slowpath is for WF_FORK and WF_EXEC.

> 
> Cheers,
> 
> Kayra

-- 
Thanks and Regards
Srikar Dronamraju

Re: [PATCH] sched/fair: Skip sched-domain traversal when wake affine is disabled

[Kayra Cizmeci]

On Mon, Nov 09, 2026 at 10:01:20PM +0530, Srikar Dronamraju wrote:
> * Kayra Cizmeci <kayracizmeci@gmail.com> [2026-11-09 18:09:56]:
> 
> Hi Kayra,
> 
> > 
> > On Mon, Nov 09, 2026 at 05:35:22PM +0530, Srikar Dronamraju wrote:
> > > In select_task_rq_fair(), TTWU wakeups for which wake_wide() returns
> > > true, or where the waking CPU is not in the task's allowed CPU mask,
> > > want_affine is set to false.
> > > 
> > > For these wakeups, the scheduler does not perform wake-affine placement.
> > > The existing code already treats this as a fast-path by exiting the
> > > sched-domain walk before locating an SD_BALANCE_WAKE domain, ultimately
> > > falling through to select_idle_sibling().
> > 
> > Not before locating. If it doesn't locates SD_BALANCE_WAKE in the first
> > check it just leaves.
> > 
> > AFAICT SD_BALANCE_WAKE is not set anywhere tho.
> 
> from kernel/sched/sched.h
> #define WF_TTWU                 0x08 /* Wakeup;            maps to SD_BALANCE_WAKE *
> The code searches for WF_TTWU in lieu of SD_BALANCE_WAKE.

Yeah I know. But I couldn't find a place where SD_BALANCE_WAKE gets set
in the domain.

> 
> > 
> > So if you wanna do this, maybe you can do something like:
> > 
> > diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
> > index d4cc0caee626..b40b2eb5e577 100644
> > --- a/kernel/sched/fair.c
> > +++ b/kernel/sched/fair.c
> > @@ -10016,6 +10016,9 @@ select_task_rq_fair(struct task_struct *p, int prev_cpu, int wake_flags)
> >  		}
> >  
> >  		want_affine = !wake_wide(p) && cpumask_test_cpu(cpu, p->cpus_ptr);
> > +
> > +		if (!want_affine)
> > +			return select_idle_sibling(p, prev_cpu, new_cpu);
> >  	}
> 
> Yes, this also has the same effect.
> 
> >  
> >  	for_each_domain(cpu, tmp) {
> > 
> > > 
> > > However, select_task_rq_fair() still enters the sched-domain traversal
> > > only to break out almost immediately. Avoid this unnecessary work by
> > > jumping directly to the existing fast path when want_affine is false.
> > > 
> > > As part of this cleanup:
> > > 
> > >   - Move want_affine into the WF_TTWU block where it is used.
> > >   - Skip the sched-domain traversal entirely when want_affine is false.
> > >   - Remove the now redundant want_affine test from the
> > >     SD_WAKE_AFFINE check.
> > >   - Remove the corresponding !want_affine loop termination logic.
> > > 
> > > This preserves existing wakeup placement behavior while avoiding
> > > unnecessary sched-domain iteration for TTWU wakeups that cannot make
> > > use of wake-affine placement.
> > > 
> > > Signed-off-by: Srikar Dronamraju <srikar@linux.ibm.com>
> > > ---
> > >  kernel/sched/fair.c | 9 +++++----
> > >  1 file changed, 5 insertions(+), 4 deletions(-)
> > > 
> > > diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
> > > index 3ebec186f982..58ff55e093cb 100644
> > > --- a/kernel/sched/fair.c
> > > +++ b/kernel/sched/fair.c
> > > @@ -8832,7 +8832,6 @@ select_task_rq_fair(struct task_struct *p, int prev_cpu, int wake_flags)
> > >  	struct sched_domain *tmp, *sd = NULL;
> > >  	int cpu = smp_processor_id();
> > >  	int new_cpu = prev_cpu;
> > > -	int want_affine = 0;
> > >  	/* SD_flags and WF_flags share the first nibble */
> > >  	int sd_flag = wake_flags & 0xF;
> > >  
> > > @@ -8841,6 +8840,8 @@ select_task_rq_fair(struct task_struct *p, int prev_cpu, int wake_flags)
> > >  	 */
> > >  	lockdep_assert_held(&p->pi_lock);
> > >  	if (wake_flags & WF_TTWU) {
> > > +		int want_affine;
> > > +
> > >  		record_wakee(p);
> > >  
> > >  		if ((wake_flags & WF_CURRENT_CPU) &&
> > > @@ -8855,6 +8856,8 @@ select_task_rq_fair(struct task_struct *p, int prev_cpu, int wake_flags)
> > >  		}
> > >  
> > >  		want_affine = !wake_wide(p) && cpumask_test_cpu(cpu, p->cpus_ptr);
> > > +		if (!want_affine)
> > > +			goto fast_path;
> > >  	}
> > >  
> > >  	for_each_domain(cpu, tmp) {
> > > @@ -8862,7 +8865,7 @@ select_task_rq_fair(struct task_struct *p, int prev_cpu, int wake_flags)
> > >  		 * If both 'cpu' and 'prev_cpu' are part of this domain,
> > >  		 * cpu is a valid SD_WAKE_AFFINE target.
> > >  		 */
> > > -		if (want_affine && (tmp->flags & SD_WAKE_AFFINE) &&
> > > +		if ((tmp->flags & SD_WAKE_AFFINE) &&
> > >  		    cpumask_test_cpu(prev_cpu, sched_domain_span(tmp))) {
> > >  			if (cpu != prev_cpu)
> > >  				new_cpu = wake_affine(tmp, p, cpu, prev_cpu, sync);
> > > @@ -8878,8 +8881,6 @@ select_task_rq_fair(struct task_struct *p, int prev_cpu, int wake_flags)
> > >  		 */
> > >  		if (tmp->flags & sd_flag)
> > >  			sd = tmp;
> > > -		else if (!want_affine)
> > > -			break;
> > >  	}
> > 
> > 
> > This just changes the behavior.
> > 
> > Old Code WF_FORK 1, WF_TTWU 0:
> > 
> > want_affine = 0, so we skip the first if block in the loop then leave
> > once we find a domain without our flag. And go with slow path.
> > 
> > 
> > New Code:
> > 
> > want_affine is moved away and early exit is removed. So we enter the if
> > block and try to find a domain with prev_cpu. Because WF_TTWU 
> > is 0, we just return new_cpu. Or if we can't fimd the domain with
> > prev_cpu, slow path.
> > 
> 
> When WF_TTWU is set, we are always suppose take the fast path.
> slowpath is for WF_FORK and WF_EXEC.

I know. And that is the exact problem with the above diff :-).

Like let's say prev_cpu is in the first domain we are looking, the
domain has SD_WAKE_AFFINE and SD_BALANCE_FORK set and next domain has
only SD_WAKE_AFFINE set:

Old Code:

sd = first domain and we enter the slow path with the first domain.

With your patch:

sd = NULL and we return new_cpu


Or am I missing something?

Cheers,

Kayra

Re: [PATCH] sched/fair: Skip sched-domain traversal when wake affine is disabled

[Srikar Dronamraju]

* Kayra Cizmeci <kayracizmeci@gmail.com> [2026-11-09 21:10:46]:

> On Mon, Nov 09, 2026 at 10:01:20PM +0530, Srikar Dronamraju wrote:
> > * Kayra Cizmeci <kayracizmeci@gmail.com> [2026-11-09 18:09:56]:
> > 
> > Hi Kayra,
> > 
> > > 
> > > On Mon, Nov 09, 2026 at 05:35:22PM +0530, Srikar Dronamraju wrote:
> > > > In select_task_rq_fair(), TTWU wakeups for which wake_wide() returns
> > > > true, or where the waking CPU is not in the task's allowed CPU mask,
> > > > want_affine is set to false.
> > > > 
> > > 
> > > 
> > > This just changes the behavior.
> > > 
> > > Old Code WF_FORK 1, WF_TTWU 0:
> > > 
> > > want_affine = 0, so we skip the first if block in the loop then leave
> > > once we find a domain without our flag. And go with slow path.
> > > 
> > > 
> > > New Code:
> > > 
> > > want_affine is moved away and early exit is removed. So we enter the if
> > > block and try to find a domain with prev_cpu. Because WF_TTWU 
> > > is 0, we just return new_cpu. Or if we can't fimd the domain with
> > > prev_cpu, slow path.
> > > 
> > 
> > When WF_TTWU is set, we are always suppose take the fast path.
> > slowpath is for WF_FORK and WF_EXEC.
> 
> I know. And that is the exact problem with the above diff :-).
> 
> Like let's say prev_cpu is in the first domain we are looking, the
> domain has SD_WAKE_AFFINE and SD_BALANCE_FORK set and next domain has
> only SD_WAKE_AFFINE set:
> 
> Old Code:
> 
> sd = first domain and we enter the slow path with the first domain.
> 
> With your patch:
> 
> sd = NULL and we return new_cpu
> 

Oh thats a fair point, thanks for the reviewing it.

> 
> Or am I missing something?
> 
> Cheers,
> 
> Kayra

-- 
Thanks and Regards
Srikar Dronamraju