[PATCH] sched/fair: Let sync wakeups target the waker's core

[PATCH] sched/fair: Let sync wakeups target the waker's core

[Madadi Vineeth Reddy]

WF_SYNC tells the scheduler the waker is about to block, and
wake_affine_idle() already acts on it: when the waker's runqueue holds a
single runnable task it returns the waker's CPU. select_idle_sibling()
then discards that decision, because available_idle_cpu() is false for a
CPU that is still running the waker, and the scan continues elsewhere in
the LLC.

When the wakee's previous CPU is idle and shares cache with the target,
SIS returns it early and nothing is lost. Once prev_cpu is busy, though,
the scan has no way to distinguish "this core is busy" from "this core is
running only the task that is about to sleep", so the wakee is placed on
a cold CPU even though the waker's core is about to have capacity and
already holds the data in cache.

Pass the waker's CPU down to select_idle_core() in that case and let it
count as idle. The waker's core then remains an idle-core candidate and
the wakee is placed on one of its sibling threads.

The change is a no-op when the waker's core has no idle sibling, when the
waker's runqueue holds more than one runnable task, when the wakee's
previous CPU is already a valid target, and on non-SMT systems.

Tested on POWER11, SMT8, 160 CPUs / 20 cores. 5 runs per case, all
figures normalised to baseline.

POWER11:
[producer_consumer] (time/access, median, lower is better)
==================================================
case             load       base   base+this patch
time/access      -l 5       1.00   0.89 (+11.11%)
time/access      -l 10      1.00   0.92 ( +7.69%)
time/access      -l 20      1.00   1.00 ( +0.00%)
time/access      -l 100     1.00   1.00 ( +0.00%)

A strict two-task handoff. As the per-iteration work grows, the
wakeup path becomes a smaller fraction of each iteration and
placement matters less: 11% at -l 5, nothing from -l 20 upwards.

[hackbench] (mean completion time, lower is better; 150000 loops)
=================================================================
case             load       baseline   base+this patch   sd%
process-pipe     1-group    1.00       0.94 ( +6.39%)     6.2
thread-pipe      1-group    1.00       0.92 ( +7.83%)     8.4
process-pipe     2-group    1.00       0.94 ( +6.18%)     7.1
thread-pipe      2-group    1.00       0.97 ( +3.12%)     7.9
process-pipe     4-group    1.00       0.98 ( +1.69%)     7.2
thread-pipe      4-group    1.00       1.09 ( -8.80%)    11.9
process-pipe     8-group    1.00       1.02 ( -2.32%)     3.8
thread-pipe      8-group    1.00       0.99 ( +0.56%)     2.6
process-socket   2-group    1.00       0.95 ( +5.20%)     6.1
thread-socket    2-group    1.00       1.02 ( -2.13%)     5.8

Each group is 40 tasks, so 1 group is 25% of the 160 CPUs and 8 groups is
200%. The patch needs an idle core in the LLC, so its impact shrinks as
the utilization increase. Mostly numbers are positive and within
run to run variation.

[schbench] (mean p99 wakeup latency, lower is better)
=====================================================
case             load       baseline   base+this patch   sd%
p99-latency        8-wkr    1.00       1.06 ( -6.06%)     8.3
p99-latency       40-wkr    1.00       0.95 ( +5.26%)     7.8
p99-latency       80-wkr    1.00       1.08 ( -7.58%)     9.9
p99-latency      240-wkr    1.00       1.00 ( -0.05%)     0.6

[schbench] (mean current rps, higher is better)
===============================================
case             load       baseline   base+this patch   sd%
rps                8-wkr    1.00       1.00 ( -0.17%)     0.4
rps               40-wkr    1.00       1.01 ( +0.87%)     4.9
rps               80-wkr    1.00       0.94 ( -5.55%)     7.9
rps              240-wkr    1.00       1.00 ( -0.28%)     0.3

schbench numbers are within run to run variation and hence not
much impacted with this patch.

Signed-off-by: Madadi Vineeth Reddy <vineethr@linux.ibm.com>

---

Shubhang Kaushik returns the waker CPU directly from the wake-affine
branch in select_task_rq_fair(); v3 restricts that to
!sched_smt_active(), since on SMT it stacks the pair onto one hardware
thread while siblings sit idle:
https://lore.kernel.org/lkml/20260727-b4-sched-sync-wakeup-v3-1-90cf481dbd85@gentwo.org/

Prateek proposed moving the decision into select_idle_sibling()
under if (!has_idle_core), preferring an idle sibling of prev and then
stacking on the waker:
https://lore.kernel.org/lkml/f3d5530f-3811-42af-8c34-c40cf314deed@amd.com/

This patch covers the disjoint case: on a sync wakeup the waker's core
already holds the data, so where an idle core exists in the LLC this
redirects the wakee onto a sibling of the waker's core rather than a cold
one, and where none exists it is a no-op. I mentioned this approach on
the v2 thread of Shubhang's patch:
https://lore.kernel.org/all/60a584c5-25ac-4077-a725-a2f9ee74318d@linux.ibm.com/
---
 kernel/sched/fair.c | 49 +++++++++++++++++++++++++++++++++++----------
 1 file changed, 38 insertions(+), 11 deletions(-)

diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
index df8c9c2c7918..bb75b228c817 100644
--- a/kernel/sched/fair.c
+++ b/kernel/sched/fair.c
@@ -1301,7 +1301,7 @@ static bool update_deadline(struct cfs_rq *cfs_rq, struct sched_entity *se)
 
 #include "pelt.h"
 
-static int select_idle_sibling(struct task_struct *p, int prev_cpu, int cpu);
+static int select_idle_sibling(struct task_struct *p, int prev_cpu, int cpu, int sync_cpu);
 static unsigned long task_h_load(struct task_struct *p);
 static unsigned long capacity_of(int cpu);
 
@@ -8604,13 +8604,24 @@ void __update_idle_core(struct rq *rq)
  * sd_balance_shared->has_idle_cores and enabled through update_idle_core()
  * above.
  */
-static int select_idle_core(struct task_struct *p, int core, struct cpumask *cpus, int *idle_cpu)
+static int select_idle_core(struct task_struct *p, int core, struct cpumask *cpus,
+				int *idle_cpu, int sync_cpu)
 {
 	bool idle = true;
 	int cpu;
 
 	for_each_cpu(cpu, cpu_smt_mask(core)) {
-		if (!available_idle_cpu(cpu)) {
+		bool sync_waker = (cpu == sync_cpu);
+
+		/*
+		 * @sync_cpu, if set, is running a waker that is about to
+		 * block with nothing else runnable behind it. Treat it as
+		 * idle so this core stays an idle-core candidate: placing
+		 * the wakee on a sibling keeps the cache sharing that
+		 * stacking on the waker's rq would get, without serialising
+		 * the wakee behind the waker's remaining work.
+		 */
+		if (!available_idle_cpu(cpu) && !sync_waker) {
 			idle = false;
 			if (*idle_cpu == -1) {
 				if (choose_sched_idle_rq(cpu_rq(cpu), p) &&
@@ -8622,7 +8633,12 @@ static int select_idle_core(struct task_struct *p, int core, struct cpumask *cpu
 			}
 			break;
 		}
-		if (*idle_cpu == -1 && cpumask_test_cpu(cpu, cpus))
+
+		/*
+		 * The waker is not idle yet, so it must not be offered as
+		 * the fallback target if this core turns out to be busy.
+		 */
+		if (!sync_waker && *idle_cpu == -1 && cpumask_test_cpu(cpu, cpus))
 			*idle_cpu = cpu;
 	}
 
@@ -8661,7 +8677,8 @@ static int select_idle_smt(struct task_struct *p, struct sched_domain *sd, int t
  * comparing the average scan cost (tracked in sd->avg_scan_cost) against the
  * average idle time for this rq (as found in rq->avg_idle).
  */
-static int select_idle_cpu(struct task_struct *p, struct sched_domain *sd, bool has_idle_core, int target)
+static int select_idle_cpu(struct task_struct *p, struct sched_domain *sd, bool has_idle_core,
+				int target, int sync_cpu)
 {
 	struct cpumask *cpus = this_cpu_cpumask_var_ptr(select_rq_mask);
 	int i, cpu, idle_cpu = -1, nr = INT_MAX;
@@ -8694,7 +8711,7 @@ static int select_idle_cpu(struct task_struct *p, struct sched_domain *sd, bool
 					continue;
 
 				if (has_idle_core) {
-					i = select_idle_core(p, cpu, cpus, &idle_cpu);
+					i = select_idle_core(p, cpu, cpus, &idle_cpu, sync_cpu);
 					if ((unsigned int)i < nr_cpumask_bits)
 						return i;
 				} else {
@@ -8711,7 +8728,7 @@ static int select_idle_cpu(struct task_struct *p, struct sched_domain *sd, bool
 
 	for_each_cpu_wrap(cpu, cpus, target + 1) {
 		if (has_idle_core) {
-			i = select_idle_core(p, cpu, cpus, &idle_cpu);
+			i = select_idle_core(p, cpu, cpus, &idle_cpu, sync_cpu);
 			if ((unsigned int)i < nr_cpumask_bits)
 				return i;
 
@@ -8928,7 +8945,7 @@ static inline bool asym_fits_cpu(unsigned long util,
 /*
  * Try and locate an idle core/thread in the LLC cache domain.
  */
-static int select_idle_sibling(struct task_struct *p, int prev, int target)
+static int select_idle_sibling(struct task_struct *p, int prev, int target, int sync_cpu)
 {
 	bool has_idle_core = false;
 	struct sched_domain *sd;
@@ -9037,7 +9054,7 @@ static int select_idle_sibling(struct task_struct *p, int prev, int target)
 		}
 	}
 
-	i = select_idle_cpu(p, sd, has_idle_core, target);
+	i = select_idle_cpu(p, sd, has_idle_core, target, sync_cpu);
 	if ((unsigned)i < nr_cpumask_bits)
 		return i;
 
@@ -9733,8 +9750,18 @@ select_task_rq_fair(struct task_struct *p, int prev_cpu, int wake_flags)
 		return sched_balance_find_dst_cpu(sd, p, cpu, prev_cpu, sd_flag);
 
 	/* Fast path */
-	if (wake_flags & WF_TTWU)
-		return select_idle_sibling(p, prev_cpu, new_cpu);
+	if (wake_flags & WF_TTWU) {
+		int sync_cpu = -1;
+
+		if (want_affine && sync && new_cpu == cpu) {
+			struct rq *rq = cpu_rq(cpu);
+
+			if ((rq->nr_running - cfs_h_nr_delayed(rq)) == 1)
+				sync_cpu = cpu;
+		}
+
+		return select_idle_sibling(p, prev_cpu, new_cpu, sync_cpu);
+	}
 
 	return new_cpu;
 }
-- 
2.43.0

Re: [PATCH] sched/fair: Let sync wakeups target the waker's core

[Zhan Xusheng]

From: Zhan Xusheng <zhanxusheng1024@gmail.com>

On Sat, Aug 01, 2026 at 09:25:32AM +0530, Madadi Vineeth Reddy wrote:
> Pass the waker's CPU down to select_idle_core() in that case and let it
> count as idle. The waker's core then remains an idle-core candidate and
> the wakee is placed on one of its sibling threads.

Nice, and thanks for laying out the other approaches in the changelog.

If I read select_idle_cpu() right, which sibling the wakee actually lands
on depends on the wrap scan order: it iterates
for_each_cpu_wrap(cpu, cpus, target + 1) and select_idle_core() returns the
first fully-idle core it hits.

On POWER SMT8 the SMT siblings are numbered contiguously, so the waker's
sibling sits right at target + 1 and is reached first - exactly the intent.

On layouts where the siblings are not adjacent to the waker - e.g. the
common x86 enumeration where the sibling is at cpu + nr_cores rather than
cpu + 1 - the scan visits other cores first, and if any of them is fully
idle it returns that (cold) core before ever reaching the waker's sibling.
There the cache-sharing win would not materialise; the wakee lands where
plain SIS would have put it. It should not regress, but the benefit looks
like it could be largely specific to contiguously numbered SMT.

Would you be able to share x86 / arm64 SMT2 numbers? That would help show
whether the waker's sibling actually gets picked on those layouts, or
whether the effect is mostly seen on contiguously numbered SMT.

Thanks,
Zhan Xusheng

Re: [PATCH] sched/fair: Let sync wakeups target the waker's core

[K Prateek Nayak]

Hello Vineeth,

On 8/1/2026 9:25 AM, Madadi Vineeth Reddy wrote:
> -static int select_idle_core(struct task_struct *p, int core, struct cpumask *cpus, int *idle_cpu)
> +static int select_idle_core(struct task_struct *p, int core, struct cpumask *cpus,
> +				int *idle_cpu, int sync_cpu)
>  {
>  	bool idle = true;
>  	int cpu;
>  
>  	for_each_cpu(cpu, cpu_smt_mask(core)) {
> -		if (!available_idle_cpu(cpu)) {
> +		bool sync_waker = (cpu == sync_cpu);
> +
> +		/*
> +		 * @sync_cpu, if set, is running a waker that is about to
> +		 * block with nothing else runnable behind it. Treat it as
> +		 * idle so this core stays an idle-core candidate: placing
> +		 * the wakee on a sibling keeps the cache sharing that
> +		 * stacking on the waker's rq would get, without serialising
> +		 * the wakee behind the waker's remaining work.
> +		 */
> +		if (!available_idle_cpu(cpu) && !sync_waker) {

If I'm not wrong, all you want to make is the sync_waker appear idle and
then see if you can then consider that core as idle core or not right?

Why can't this be done in select_idle_sibling() extending that early
check for (!has_idle_core && cpus_share_cache(prev, target)) condition
and then initializing "idle_cpu" in select_idle_cpu() accordingly?

Something along the lines of:

  (Only build tested)

diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
index df8c9c2c7918..dd62bceb3838 100644
--- a/kernel/sched/fair.c
+++ b/kernel/sched/fair.c
@@ -1301,7 +1301,6 @@ static bool update_deadline(struct cfs_rq *cfs_rq, struct sched_entity *se)
 
 #include "pelt.h"
 
-static int select_idle_sibling(struct task_struct *p, int prev_cpu, int cpu);
 static unsigned long task_h_load(struct task_struct *p);
 static unsigned long capacity_of(int cpu);
 
@@ -8661,10 +8660,11 @@ static int select_idle_smt(struct task_struct *p, struct sched_domain *sd, int t
  * comparing the average scan cost (tracked in sd->avg_scan_cost) against the
  * average idle time for this rq (as found in rq->avg_idle).
  */
-static int select_idle_cpu(struct task_struct *p, struct sched_domain *sd, bool has_idle_core, int target)
+static int select_idle_cpu(struct task_struct *p, struct sched_domain *sd, bool has_idle_core,
+			   int target, int idle_cpu)
 {
 	struct cpumask *cpus = this_cpu_cpumask_var_ptr(select_rq_mask);
-	int i, cpu, idle_cpu = -1, nr = INT_MAX;
+	int i, cpu, nr = INT_MAX;
 
 	if (sched_feat(SIS_UTIL) && sd->shared) {
 		/*
@@ -8928,7 +8928,7 @@ static inline bool asym_fits_cpu(unsigned long util,
 /*
  * Try and locate an idle core/thread in the LLC cache domain.
  */
-static int select_idle_sibling(struct task_struct *p, int prev, int target)
+static int select_idle_sibling(struct task_struct *p, int prev, int target, int sync_cpu)
 {
 	bool has_idle_core = false;
 	struct sched_domain *sd;
@@ -9028,16 +9028,19 @@ static int select_idle_sibling(struct task_struct *p, int prev, int target)
 		return target;
 
 	if (sched_smt_active()) {
+		int cpu = ((unsigned)sync_cpu < nr_cpumask_bits) ? sync_cpu : prev;
+
 		has_idle_core = test_idle_cores(target);
 
-		if (!has_idle_core && cpus_share_cache(prev, target)) {
-			i = select_idle_smt(p, sd, prev);
-			if ((unsigned int)i < nr_cpumask_bits)
+		if (sync_cpu == target || (!has_idle_core && cpus_share_cache(prev, target))) {
+			i = select_idle_smt(p, sd, cpu);
+
+			if (!has_idle_core && ((unsigned int)i < nr_cpumask_bits))
 				return i;
 		}
 	}
 
-	i = select_idle_cpu(p, sd, has_idle_core, target);
+	i = select_idle_cpu(p, sd, has_idle_core, target, i);
 	if ((unsigned)i < nr_cpumask_bits)
 		return i;
 
@@ -9733,8 +9736,18 @@ select_task_rq_fair(struct task_struct *p, int prev_cpu, int wake_flags)
 		return sched_balance_find_dst_cpu(sd, p, cpu, prev_cpu, sd_flag);
 
 	/* Fast path */
-	if (wake_flags & WF_TTWU)
-		return select_idle_sibling(p, prev_cpu, new_cpu);
+	if (wake_flags & WF_TTWU) {
+		int sync_cpu = -1;
+
+		if (want_affine && sync && new_cpu == cpu) {
+			struct rq *rq = cpu_rq(cpu);
+
+			if ((rq->nr_running - cfs_h_nr_delayed(rq)) == 1)
+				sync_cpu = cpu;
+		}
+
+		return select_idle_sibling(p, prev_cpu, new_cpu, sync_cpu);
+	}
 
 	return new_cpu;
 }
---

You can probably infer sync hint by checking
"target == smp_preocessor_id()" too in select_idle_sibling() instead of
passing it on.

>  			idle = false;
>  			if (*idle_cpu == -1) {
>  				if (choose_sched_idle_rq(cpu_rq(cpu), p) &&
-- 
Thanks and Regards,
Prateek

Re: [PATCH] sched/fair: Let sync wakeups target the waker's core

[Madadi Vineeth Reddy]

Hello Prateek,

On 04/08/26 10:19, K Prateek Nayak wrote:
> Hello Vineeth,
> 
> On 8/1/2026 9:25 AM, Madadi Vineeth Reddy wrote:
>> -static int select_idle_core(struct task_struct *p, int core, struct cpumask *cpus, int *idle_cpu)
>> +static int select_idle_core(struct task_struct *p, int core, struct cpumask *cpus,
>> +				int *idle_cpu, int sync_cpu)
>>  {
>>  	bool idle = true;
>>  	int cpu;
>>  
>>  	for_each_cpu(cpu, cpu_smt_mask(core)) {
>> -		if (!available_idle_cpu(cpu)) {
>> +		bool sync_waker = (cpu == sync_cpu);
>> +
>> +		/*
>> +		 * @sync_cpu, if set, is running a waker that is about to
>> +		 * block with nothing else runnable behind it. Treat it as
>> +		 * idle so this core stays an idle-core candidate: placing
>> +		 * the wakee on a sibling keeps the cache sharing that
>> +		 * stacking on the waker's rq would get, without serialising
>> +		 * the wakee behind the waker's remaining work.
>> +		 */
>> +		if (!available_idle_cpu(cpu) && !sync_waker) {
> 
> If I'm not wrong, all you want to make is the sync_waker appear idle and
> then see if you can then consider that core as idle core or not right?
> 

Correct.

> Why can't this be done in select_idle_sibling() extending that early
> check for (!has_idle_core && cpus_share_cache(prev, target)) condition
> and then initializing "idle_cpu" in select_idle_cpu() accordingly?
> 
> Something along the lines of:
> 
>   (Only build tested)
> 
> diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
> index df8c9c2c7918..dd62bceb3838 100644
> --- a/kernel/sched/fair.c
> +++ b/kernel/sched/fair.c
> @@ -1301,7 +1301,6 @@ static bool update_deadline(struct cfs_rq *cfs_rq, struct sched_entity *se)
>  
>  #include "pelt.h"
>  
> -static int select_idle_sibling(struct task_struct *p, int prev_cpu, int cpu);
>  static unsigned long task_h_load(struct task_struct *p);
>  static unsigned long capacity_of(int cpu);
>  
> @@ -8661,10 +8660,11 @@ static int select_idle_smt(struct task_struct *p, struct sched_domain *sd, int t
>   * comparing the average scan cost (tracked in sd->avg_scan_cost) against the
>   * average idle time for this rq (as found in rq->avg_idle).
>   */
> -static int select_idle_cpu(struct task_struct *p, struct sched_domain *sd, bool has_idle_core, int target)
> +static int select_idle_cpu(struct task_struct *p, struct sched_domain *sd, bool has_idle_core,
> +			   int target, int idle_cpu)
>  {
>  	struct cpumask *cpus = this_cpu_cpumask_var_ptr(select_rq_mask);
> -	int i, cpu, idle_cpu = -1, nr = INT_MAX;
> +	int i, cpu, nr = INT_MAX;
>  
>  	if (sched_feat(SIS_UTIL) && sd->shared) {
>  		/*
> @@ -8928,7 +8928,7 @@ static inline bool asym_fits_cpu(unsigned long util,
>  /*
>   * Try and locate an idle core/thread in the LLC cache domain.
>   */
> -static int select_idle_sibling(struct task_struct *p, int prev, int target)
> +static int select_idle_sibling(struct task_struct *p, int prev, int target, int sync_cpu)
>  {
>  	bool has_idle_core = false;
>  	struct sched_domain *sd;
> @@ -9028,16 +9028,19 @@ static int select_idle_sibling(struct task_struct *p, int prev, int target)
>  		return target;
>  
>  	if (sched_smt_active()) {
> +		int cpu = ((unsigned)sync_cpu < nr_cpumask_bits) ? sync_cpu : prev;
> +
>  		has_idle_core = test_idle_cores(target);
>  
> -		if (!has_idle_core && cpus_share_cache(prev, target)) {
> -			i = select_idle_smt(p, sd, prev);
> -			if ((unsigned int)i < nr_cpumask_bits)
> +		if (sync_cpu == target || (!has_idle_core && cpus_share_cache(prev, target))) {
> +			i = select_idle_smt(p, sd, cpu);
> +
> +			if (!has_idle_core && ((unsigned int)i < nr_cpumask_bits))
>  				return i;
>  		}
>  	}
>  
> -	i = select_idle_cpu(p, sd, has_idle_core, target);
> +	i = select_idle_cpu(p, sd, has_idle_core, target, i);

`i` which is passed could be garbage value if we don't enter sched_smt_active block.

>  	if ((unsigned)i < nr_cpumask_bits)
>  		return i;
>  

This is different from what I wanted to achieve in a couple of ways.

- Calling `select_idle_smt()` in sync case, would only give an idle CPU in that core but doesn't
test if that core is idle. That would be lost.
- You return `i` only when `!has_idle_core`, but I wanted to return waker core given that rest of the siblings
in that waker core are idle even though there are other idle cores present in the LLC.

I agree that this could be done in `select_idle_sibling` but with a helper function.
Something like below (build tested)

diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
index df8c9c2c7918..448af3c4b183 100644
--- a/kernel/sched/fair.c
+++ b/kernel/sched/fair.c
@@ -1301,7 +1301,7 @@ static bool update_deadline(struct cfs_rq *cfs_rq, struct sched_entity *se)
 
 #include "pelt.h"
 
-static int select_idle_sibling(struct task_struct *p, int prev_cpu, int cpu);
+static int select_idle_sibling(struct task_struct *p, int prev_cpu, int cpu, bool sync_core);
 static unsigned long task_h_load(struct task_struct *p);
 static unsigned long capacity_of(int cpu);
 
@@ -8656,6 +8656,27 @@ static int select_idle_smt(struct task_struct *p, struct sched_domain *sd, int t
 	return -1;
 }
 
+static int select_idle_sync_core(struct task_struct *p, struct sched_domain *sd,
+				 int target)
+{
+	int cpu, idle_sibling = -1;
+
+	for_each_cpu(cpu, cpu_smt_mask(target)) {
+		if (cpu == target)
+			continue;
+
+		if (!available_idle_cpu(cpu))
+			return -1;
+
+		if (idle_sibling == -1 &&
+		    cpumask_test_cpu(cpu, sched_domain_span(sd)) &&
+		    cpumask_test_cpu(cpu, p->cpus_ptr))
+			idle_sibling = cpu;
+	}
+
+	return idle_sibling;
+}
+
 /*
  * Scan the LLC domain for idle CPUs; this is dynamically regulated by
  * comparing the average scan cost (tracked in sd->avg_scan_cost) against the
@@ -8928,7 +8949,7 @@ static inline bool asym_fits_cpu(unsigned long util,
 /*
  * Try and locate an idle core/thread in the LLC cache domain.
  */
-static int select_idle_sibling(struct task_struct *p, int prev, int target)
+static int select_idle_sibling(struct task_struct *p, int prev, int target, bool sync_core)
 {
 	bool has_idle_core = false;
 	struct sched_domain *sd;
@@ -9035,6 +9056,12 @@ static int select_idle_sibling(struct task_struct *p, int prev, int target)
 			if ((unsigned int)i < nr_cpumask_bits)
 				return i;
 		}
+
+		if (sync_core) {
+			i = select_idle_sync_core(p, sd, target);
+			if ((unsigned int)i < nr_cpumask_bits)
+				return i;
+		}
 	}
 
 	i = select_idle_cpu(p, sd, has_idle_core, target);
@@ -9733,8 +9760,16 @@ select_task_rq_fair(struct task_struct *p, int prev_cpu, int wake_flags)
 		return sched_balance_find_dst_cpu(sd, p, cpu, prev_cpu, sd_flag);
 
 	/* Fast path */
-	if (wake_flags & WF_TTWU)
-		return select_idle_sibling(p, prev_cpu, new_cpu);
+	if (wake_flags & WF_TTWU) {
+		bool sync_core = false;
+		if (want_affine && sync && new_cpu == cpu) {
+			struct rq *rq = cpu_rq(cpu);
+
+			sync_core = (rq->nr_running - cfs_h_nr_delayed(rq)) == 1;
+		}
+
+		return select_idle_sibling(p, prev_cpu, new_cpu, sync_core);
+	}
 
 	return new_cpu;
 }

This should also solve the issue raised by Zhan Xusheng and first target waker core given
that it is idle by giving exception to waker cpu.

Thoughts?

Thanks,
Vineeth

> @@ -9733,8 +9736,18 @@ select_task_rq_fair(struct task_struct *p, int prev_cpu, int wake_flags)
>  		return sched_balance_find_dst_cpu(sd, p, cpu, prev_cpu, sd_flag);
>  
>  	/* Fast path */
> -	if (wake_flags & WF_TTWU)
> -		return select_idle_sibling(p, prev_cpu, new_cpu);
> +	if (wake_flags & WF_TTWU) {
> +		int sync_cpu = -1;
> +
> +		if (want_affine && sync && new_cpu == cpu) {
> +			struct rq *rq = cpu_rq(cpu);
> +
> +			if ((rq->nr_running - cfs_h_nr_delayed(rq)) == 1)
> +				sync_cpu = cpu;
> +		}
> +
> +		return select_idle_sibling(p, prev_cpu, new_cpu, sync_cpu);
> +	}
>  
>  	return new_cpu;
>  }
> ---
> 
> You can probably infer sync hint by checking
> "target == smp_preocessor_id()" too in select_idle_sibling() instead of
> passing it on.
> 
>>  			idle = false;
>>  			if (*idle_cpu == -1) {
>>  				if (choose_sched_idle_rq(cpu_rq(cpu), p) &&

Re: [PATCH] sched/fair: Let sync wakeups target the waker's core

[K Prateek Nayak]

Hello Vineeth,

On 8/4/2026 5:43 PM, Madadi Vineeth Reddy wrote:
> Hello Prateek,
> 
> On 04/08/26 10:19, K Prateek Nayak wrote:
>> Hello Vineeth,
>>
>> On 8/1/2026 9:25 AM, Madadi Vineeth Reddy wrote:
>>> -static int select_idle_core(struct task_struct *p, int core, struct cpumask *cpus, int *idle_cpu)
>>> +static int select_idle_core(struct task_struct *p, int core, struct cpumask *cpus,
>>> +				int *idle_cpu, int sync_cpu)
>>>  {
>>>  	bool idle = true;
>>>  	int cpu;
>>>  
>>>  	for_each_cpu(cpu, cpu_smt_mask(core)) {
>>> -		if (!available_idle_cpu(cpu)) {
>>> +		bool sync_waker = (cpu == sync_cpu);
>>> +
>>> +		/*
>>> +		 * @sync_cpu, if set, is running a waker that is about to
>>> +		 * block with nothing else runnable behind it. Treat it as
>>> +		 * idle so this core stays an idle-core candidate: placing
>>> +		 * the wakee on a sibling keeps the cache sharing that
>>> +		 * stacking on the waker's rq would get, without serialising
>>> +		 * the wakee behind the waker's remaining work.
>>> +		 */
>>> +		if (!available_idle_cpu(cpu) && !sync_waker) {
>>
>> If I'm not wrong, all you want to make is the sync_waker appear idle and
>> then see if you can then consider that core as idle core or not right?
>>
> 
> Correct.
> 
>> Why can't this be done in select_idle_sibling() extending that early
>> check for (!has_idle_core && cpus_share_cache(prev, target)) condition
>> and then initializing "idle_cpu" in select_idle_cpu() accordingly?
>>
>> Something along the lines of:
>>
>>   (Only build tested)
>>
>> diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
>> index df8c9c2c7918..dd62bceb3838 100644
>> --- a/kernel/sched/fair.c
>> +++ b/kernel/sched/fair.c
>> @@ -1301,7 +1301,6 @@ static bool update_deadline(struct cfs_rq *cfs_rq, struct sched_entity *se)
>>  
>>  #include "pelt.h"
>>  
>> -static int select_idle_sibling(struct task_struct *p, int prev_cpu, int cpu);
>>  static unsigned long task_h_load(struct task_struct *p);
>>  static unsigned long capacity_of(int cpu);
>>  
>> @@ -8661,10 +8660,11 @@ static int select_idle_smt(struct task_struct *p, struct sched_domain *sd, int t
>>   * comparing the average scan cost (tracked in sd->avg_scan_cost) against the
>>   * average idle time for this rq (as found in rq->avg_idle).
>>   */
>> -static int select_idle_cpu(struct task_struct *p, struct sched_domain *sd, bool has_idle_core, int target)
>> +static int select_idle_cpu(struct task_struct *p, struct sched_domain *sd, bool has_idle_core,
>> +			   int target, int idle_cpu)
>>  {
>>  	struct cpumask *cpus = this_cpu_cpumask_var_ptr(select_rq_mask);
>> -	int i, cpu, idle_cpu = -1, nr = INT_MAX;
>> +	int i, cpu, nr = INT_MAX;
>>  
>>  	if (sched_feat(SIS_UTIL) && sd->shared) {
>>  		/*
>> @@ -8928,7 +8928,7 @@ static inline bool asym_fits_cpu(unsigned long util,
>>  /*
>>   * Try and locate an idle core/thread in the LLC cache domain.
>>   */
>> -static int select_idle_sibling(struct task_struct *p, int prev, int target)
>> +static int select_idle_sibling(struct task_struct *p, int prev, int target, int sync_cpu)
>>  {
>>  	bool has_idle_core = false;
>>  	struct sched_domain *sd;
>> @@ -9028,16 +9028,19 @@ static int select_idle_sibling(struct task_struct *p, int prev, int target)
>>  		return target;
>>  
>>  	if (sched_smt_active()) {
>> +		int cpu = ((unsigned)sync_cpu < nr_cpumask_bits) ? sync_cpu : prev;
>> +
>>  		has_idle_core = test_idle_cores(target);
>>  
>> -		if (!has_idle_core && cpus_share_cache(prev, target)) {
>> -			i = select_idle_smt(p, sd, prev);
>> -			if ((unsigned int)i < nr_cpumask_bits)
>> +		if (sync_cpu == target || (!has_idle_core && cpus_share_cache(prev, target))) {
>> +			i = select_idle_smt(p, sd, cpu);
>> +
>> +			if (!has_idle_core && ((unsigned int)i < nr_cpumask_bits))
>>  				return i;
>>  		}
>>  	}
>>  
>> -	i = select_idle_cpu(p, sd, has_idle_core, target);
>> +	i = select_idle_cpu(p, sd, has_idle_core, target, i);
> 
> `i` which is passed could be garbage value if we don't enter sched_smt_active block.
> 
>>  	if ((unsigned)i < nr_cpumask_bits)
>>  		return i;
>>  
> 
> This is different from what I wanted to achieve in a couple of ways.
> 
> - Calling `select_idle_smt()` in sync case, would only give an idle CPU in that core but doesn't
> test if that core is idle. That would be lost.
> - You return `i` only when `!has_idle_core`, but I wanted to return waker core given that rest of the siblings
> in that waker core are idle even though there are other idle cores present in the LLC.
> 
> I agree that this could be done in `select_idle_sibling` but with a helper function.
> Something like below (build tested)
> 
> diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
> index df8c9c2c7918..448af3c4b183 100644
> --- a/kernel/sched/fair.c
> +++ b/kernel/sched/fair.c
> @@ -1301,7 +1301,7 @@ static bool update_deadline(struct cfs_rq *cfs_rq, struct sched_entity *se)
>  
>  #include "pelt.h"
>  
> -static int select_idle_sibling(struct task_struct *p, int prev_cpu, int cpu);
> +static int select_idle_sibling(struct task_struct *p, int prev_cpu, int cpu, bool sync_core);

Is this forward declaration necessary? I think you can remove it
entirely.

>  static unsigned long task_h_load(struct task_struct *p);
>  static unsigned long capacity_of(int cpu);
>  
> @@ -8656,6 +8656,27 @@ static int select_idle_smt(struct task_struct *p, struct sched_domain *sd, int t
>  	return -1;
>  }
>  
> +static int select_idle_sync_core(struct task_struct *p, struct sched_domain *sd,
> +				 int target)
> +{
> +	int cpu, idle_sibling = -1;
> +
> +	for_each_cpu(cpu, cpu_smt_mask(target)) {
> +		if (cpu == target)
> +			continue;
> +
> +		if (!available_idle_cpu(cpu))
> +			return -1;

Reads a lot like select_idle_smt(). Perhaps you can pass "has_idle_core"
hint to the same and return idle CPU early if !has_idle_core or otherwise
return the first idle sibling if core is idle.

That way even sync_core case is handled by the same function. You can
gate the call with (sync_core || !has_idle_core)

> +
> +		if (idle_sibling == -1 &&
> +		    cpumask_test_cpu(cpu, sched_domain_span(sd)) &&
> +		    cpumask_test_cpu(cpu, p->cpus_ptr))
> +			idle_sibling = cpu;
> +	}
> +
> +	return idle_sibling;
> +}
> +
>  /*
>   * Scan the LLC domain for idle CPUs; this is dynamically regulated by
>   * comparing the average scan cost (tracked in sd->avg_scan_cost) against the
> @@ -8928,7 +8949,7 @@ static inline bool asym_fits_cpu(unsigned long util,
>  /*
>   * Try and locate an idle core/thread in the LLC cache domain.
>   */
> -static int select_idle_sibling(struct task_struct *p, int prev, int target)
> +static int select_idle_sibling(struct task_struct *p, int prev, int target, bool sync_core)
>  {
>  	bool has_idle_core = false;
>  	struct sched_domain *sd;
> @@ -9035,6 +9056,12 @@ static int select_idle_sibling(struct task_struct *p, int prev, int target)
>  			if ((unsigned int)i < nr_cpumask_bits)
>  				return i;
>  		}
> +
> +		if (sync_core) {
> +			i = select_idle_sync_core(p, sd, target);
> +			if ((unsigned int)i < nr_cpumask_bits)
> +				return i;
> +		}
>  	}
>  
>  	i = select_idle_cpu(p, sd, has_idle_core, target);
> @@ -9733,8 +9760,16 @@ select_task_rq_fair(struct task_struct *p, int prev_cpu, int wake_flags)
>  		return sched_balance_find_dst_cpu(sd, p, cpu, prev_cpu, sd_flag);
>  
>  	/* Fast path */
> -	if (wake_flags & WF_TTWU)
> -		return select_idle_sibling(p, prev_cpu, new_cpu);
> +	if (wake_flags & WF_TTWU) {
> +		bool sync_core = false;
> +		if (want_affine && sync && new_cpu == cpu) {
> +			struct rq *rq = cpu_rq(cpu);
> +
> +			sync_core = (rq->nr_running - cfs_h_nr_delayed(rq)) == 1;

Instead of computing this twice on wake_affine path, you can have a task
flag like sched_task_hot that you just set in wake_affine_idle() before
it returns early from the sync branch.

You can clear it up top in select_task_rq_fair() and check it
select_idle_sibling() to conditionally call select_idle_sync_core().

> +		}
> +
> +		return select_idle_sibling(p, prev_cpu, new_cpu, sync_core);
> +	}
>  
>  	return new_cpu;
>  }
> 
> This should also solve the issue raised by Zhan Xusheng and first target waker core given
> that it is idle by giving exception to waker cpu.
> 
> Thoughts?

Let me give it a spin. I'll report back if I see anything unexpected.

-- 
Thanks and Regards,
Prateek

Re: [PATCH] sched/fair: Let sync wakeups target the waker's core

[Madadi Vineeth Reddy]

On 05/08/26 09:00, K Prateek Nayak wrote:
> Hello Vineeth,
> 
> On 8/4/2026 5:43 PM, Madadi Vineeth Reddy wrote:
>> Hello Prateek,
>>
>> On 04/08/26 10:19, K Prateek Nayak wrote:
>>> Hello Vineeth,
>>>
>>> On 8/1/2026 9:25 AM, Madadi Vineeth Reddy wrote:
>>>> -static int select_idle_core(struct task_struct *p, int core, struct cpumask *cpus, int *idle_cpu)
>>>> +static int select_idle_core(struct task_struct *p, int core, struct cpumask *cpus,
>>>> +				int *idle_cpu, int sync_cpu)
>>>>  {
>>>>  	bool idle = true;
>>>>  	int cpu;
>>>>  
>>>>  	for_each_cpu(cpu, cpu_smt_mask(core)) {
>>>> -		if (!available_idle_cpu(cpu)) {
>>>> +		bool sync_waker = (cpu == sync_cpu);
>>>> +
>>>> +		/*
>>>> +		 * @sync_cpu, if set, is running a waker that is about to
>>>> +		 * block with nothing else runnable behind it. Treat it as
>>>> +		 * idle so this core stays an idle-core candidate: placing
>>>> +		 * the wakee on a sibling keeps the cache sharing that
>>>> +		 * stacking on the waker's rq would get, without serialising
>>>> +		 * the wakee behind the waker's remaining work.
>>>> +		 */
>>>> +		if (!available_idle_cpu(cpu) && !sync_waker) {
>>>
>>> If I'm not wrong, all you want to make is the sync_waker appear idle and
>>> then see if you can then consider that core as idle core or not right?
>>>
>>
>> Correct.
>>
>>> Why can't this be done in select_idle_sibling() extending that early
>>> check for (!has_idle_core && cpus_share_cache(prev, target)) condition
>>> and then initializing "idle_cpu" in select_idle_cpu() accordingly?
>>>
>>> Something along the lines of:
>>>
>>>   (Only build tested)
>>>
>>> diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
>>> index df8c9c2c7918..dd62bceb3838 100644
>>> --- a/kernel/sched/fair.c
>>> +++ b/kernel/sched/fair.c
>>> @@ -1301,7 +1301,6 @@ static bool update_deadline(struct cfs_rq *cfs_rq, struct sched_entity *se)
>>>  
>>>  #include "pelt.h"
>>>  
>>> -static int select_idle_sibling(struct task_struct *p, int prev_cpu, int cpu);
>>>  static unsigned long task_h_load(struct task_struct *p);
>>>  static unsigned long capacity_of(int cpu);
>>>  
>>> @@ -8661,10 +8660,11 @@ static int select_idle_smt(struct task_struct *p, struct sched_domain *sd, int t
>>>   * comparing the average scan cost (tracked in sd->avg_scan_cost) against the
>>>   * average idle time for this rq (as found in rq->avg_idle).
>>>   */
>>> -static int select_idle_cpu(struct task_struct *p, struct sched_domain *sd, bool has_idle_core, int target)
>>> +static int select_idle_cpu(struct task_struct *p, struct sched_domain *sd, bool has_idle_core,
>>> +			   int target, int idle_cpu)
>>>  {
>>>  	struct cpumask *cpus = this_cpu_cpumask_var_ptr(select_rq_mask);
>>> -	int i, cpu, idle_cpu = -1, nr = INT_MAX;
>>> +	int i, cpu, nr = INT_MAX;
>>>  
>>>  	if (sched_feat(SIS_UTIL) && sd->shared) {
>>>  		/*
>>> @@ -8928,7 +8928,7 @@ static inline bool asym_fits_cpu(unsigned long util,
>>>  /*
>>>   * Try and locate an idle core/thread in the LLC cache domain.
>>>   */
>>> -static int select_idle_sibling(struct task_struct *p, int prev, int target)
>>> +static int select_idle_sibling(struct task_struct *p, int prev, int target, int sync_cpu)
>>>  {
>>>  	bool has_idle_core = false;
>>>  	struct sched_domain *sd;
>>> @@ -9028,16 +9028,19 @@ static int select_idle_sibling(struct task_struct *p, int prev, int target)
>>>  		return target;
>>>  
>>>  	if (sched_smt_active()) {
>>> +		int cpu = ((unsigned)sync_cpu < nr_cpumask_bits) ? sync_cpu : prev;
>>> +
>>>  		has_idle_core = test_idle_cores(target);
>>>  
>>> -		if (!has_idle_core && cpus_share_cache(prev, target)) {
>>> -			i = select_idle_smt(p, sd, prev);
>>> -			if ((unsigned int)i < nr_cpumask_bits)
>>> +		if (sync_cpu == target || (!has_idle_core && cpus_share_cache(prev, target))) {
>>> +			i = select_idle_smt(p, sd, cpu);
>>> +
>>> +			if (!has_idle_core && ((unsigned int)i < nr_cpumask_bits))
>>>  				return i;
>>>  		}
>>>  	}
>>>  
>>> -	i = select_idle_cpu(p, sd, has_idle_core, target);
>>> +	i = select_idle_cpu(p, sd, has_idle_core, target, i);
>>
>> `i` which is passed could be garbage value if we don't enter sched_smt_active block.
>>
>>>  	if ((unsigned)i < nr_cpumask_bits)
>>>  		return i;
>>>  
>>
>> This is different from what I wanted to achieve in a couple of ways.
>>
>> - Calling `select_idle_smt()` in sync case, would only give an idle CPU in that core but doesn't
>> test if that core is idle. That would be lost.
>> - You return `i` only when `!has_idle_core`, but I wanted to return waker core given that rest of the siblings
>> in that waker core are idle even though there are other idle cores present in the LLC.
>>
>> I agree that this could be done in `select_idle_sibling` but with a helper function.
>> Something like below (build tested)
>>
>> diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
>> index df8c9c2c7918..448af3c4b183 100644
>> --- a/kernel/sched/fair.c
>> +++ b/kernel/sched/fair.c
>> @@ -1301,7 +1301,7 @@ static bool update_deadline(struct cfs_rq *cfs_rq, struct sched_entity *se)
>>  
>>  #include "pelt.h"
>>  
>> -static int select_idle_sibling(struct task_struct *p, int prev_cpu, int cpu);
>> +static int select_idle_sibling(struct task_struct *p, int prev_cpu, int cpu, bool sync_core);
> 
> Is this forward declaration necessary? I think you can remove it
> entirely.

Right. This can be removed as definition is above the only caller.

> 
>>  static unsigned long task_h_load(struct task_struct *p);
>>  static unsigned long capacity_of(int cpu);
>>  
>> @@ -8656,6 +8656,27 @@ static int select_idle_smt(struct task_struct *p, struct sched_domain *sd, int t
>>  	return -1;
>>  }
>>  
>> +static int select_idle_sync_core(struct task_struct *p, struct sched_domain *sd,
>> +				 int target)
>> +{
>> +	int cpu, idle_sibling = -1;
>> +
>> +	for_each_cpu(cpu, cpu_smt_mask(target)) {
>> +		if (cpu == target)
>> +			continue;
>> +
>> +		if (!available_idle_cpu(cpu))
>> +			return -1;
> 
> Reads a lot like select_idle_smt(). Perhaps you can pass "has_idle_core"
> hint to the same and return idle CPU early if !has_idle_core or otherwise
> return the first idle sibling if core is idle.
> 
> That way even sync_core case is handled by the same function. You can
> gate the call with (sync_core || !has_idle_core)

Agreed. When has_idle_core is clear the LLC scan is not going to find a
fully idle core anyway, so an idle sibling on the waker's core is at least
as good.

> 
>> +
>> +		if (idle_sibling == -1 &&
>> +		    cpumask_test_cpu(cpu, sched_domain_span(sd)) &&
>> +		    cpumask_test_cpu(cpu, p->cpus_ptr))
>> +			idle_sibling = cpu;
>> +	}
>> +
>> +	return idle_sibling;
>> +}
>> +
>>  /*
>>   * Scan the LLC domain for idle CPUs; this is dynamically regulated by
>>   * comparing the average scan cost (tracked in sd->avg_scan_cost) against the
>> @@ -8928,7 +8949,7 @@ static inline bool asym_fits_cpu(unsigned long util,
>>  /*
>>   * Try and locate an idle core/thread in the LLC cache domain.
>>   */
>> -static int select_idle_sibling(struct task_struct *p, int prev, int target)
>> +static int select_idle_sibling(struct task_struct *p, int prev, int target, bool sync_core)
>>  {
>>  	bool has_idle_core = false;
>>  	struct sched_domain *sd;
>> @@ -9035,6 +9056,12 @@ static int select_idle_sibling(struct task_struct *p, int prev, int target)
>>  			if ((unsigned int)i < nr_cpumask_bits)
>>  				return i;
>>  		}
>> +
>> +		if (sync_core) {
>> +			i = select_idle_sync_core(p, sd, target);
>> +			if ((unsigned int)i < nr_cpumask_bits)
>> +				return i;
>> +		}
>>  	}
>>  
>>  	i = select_idle_cpu(p, sd, has_idle_core, target);
>> @@ -9733,8 +9760,16 @@ select_task_rq_fair(struct task_struct *p, int prev_cpu, int wake_flags)
>>  		return sched_balance_find_dst_cpu(sd, p, cpu, prev_cpu, sd_flag);
>>  
>>  	/* Fast path */
>> -	if (wake_flags & WF_TTWU)
>> -		return select_idle_sibling(p, prev_cpu, new_cpu);
>> +	if (wake_flags & WF_TTWU) {
>> +		bool sync_core = false;
>> +		if (want_affine && sync && new_cpu == cpu) {
>> +			struct rq *rq = cpu_rq(cpu);
>> +
>> +			sync_core = (rq->nr_running - cfs_h_nr_delayed(rq)) == 1;
> 
> Instead of computing this twice on wake_affine path, you can have a task
> flag like sched_task_hot that you just set in wake_affine_idle() before
> it returns early from the sync branch.
> 
> You can clear it up top in select_task_rq_fair() and check it
> select_idle_sibling() to conditionally call select_idle_sync_core().
> 

Fair point, will include this in the next version.

>> +		}
>> +
>> +		return select_idle_sibling(p, prev_cpu, new_cpu, sync_core);
>> +	}
>>  
>>  	return new_cpu;
>>  }
>>
>> This should also solve the issue raised by Zhan Xusheng and first target waker core given
>> that it is idle by giving exception to waker cpu.
>>
>> Thoughts?
> 
> Let me give it a spin. I'll report back if I see anything unexpected.

Thank you. Will post a new version with these set of changes and include numbers from Power10
box.

Thanks,
Vineeth

> 

Re: [PATCH] sched/fair: Let sync wakeups target the waker's core

[Chen Yu]

Hi Madadi,

On Thu, Aug 06, 2026 at 10:20:38AM +0530, Madadi Vineeth Reddy wrote:

[ ... ]

> >> -static int select_idle_sibling(struct task_struct *p, int prev, int target)
> >> +static int select_idle_sibling(struct task_struct *p, int prev, int target, bool sync_core)
> >>  {
> >>  	bool has_idle_core = false;
> >>  	struct sched_domain *sd;
> >> @@ -9035,6 +9056,12 @@ static int select_idle_sibling(struct task_struct *p, int prev, int target)
> >>  			if ((unsigned int)i < nr_cpumask_bits)
> >>  				return i;
> >>  		}
> >> +
> >> +		if (sync_core) {
> >> +			i = select_idle_sync_core(p, sd, target);
> >> +			if ((unsigned int)i < nr_cpumask_bits)
> >> +				return i;
> >> +		}
> >>  	}
> >>  
> >>  	i = select_idle_cpu(p, sd, has_idle_core, target);
> >> @@ -9733,8 +9760,16 @@ select_task_rq_fair(struct task_struct *p, int prev_cpu, int wake_flags)
> >>  		return sched_balance_find_dst_cpu(sd, p, cpu, prev_cpu, sd_flag);
> >>  
> >>  	/* Fast path */
> >> -	if (wake_flags & WF_TTWU)
> >> -		return select_idle_sibling(p, prev_cpu, new_cpu);
> >> +	if (wake_flags & WF_TTWU) {
> >> +		bool sync_core = false;
> >> +		if (want_affine && sync && new_cpu == cpu) {
> >> +			struct rq *rq = cpu_rq(cpu);
> >> +
> >> +			sync_core = (rq->nr_running - cfs_h_nr_delayed(rq)) == 1;

If I understand correctly, the goal is to choose an idle SMT sibling as the waker
CPU, if:

1. the wakeup has WF_SYNC, and
2. the waker's SMT sibling CPUs are all idle, and
3. the waker is about to release the CPU.
In this way, we can "stack" the wakee on a core that is about to become idle to
get better cache locality.

Condition 3 above might not always hold true, because WF_SYNC is not restricted to
task context. softirq may also call wake_up_interruptible_sync_poll() with WF_SYNC,
and in that case, current is whatever task the softirq happened to interrupt.

Given that, would it be reasonable to add in_task() check to gate the softirq case?


=======================================================================================
BTW, in your git log:
"WF_SYNC tells the scheduler the waker is about to block ... when the waker's runqueue
holds a single runnable task it returns the waker's CPU, select_idle_sibling() then
discards that decision, because available_idle_cpu() is false for a CPU that is still
running the waker"

Thanks for this description. I realized that WF_SYNC is not what I previously thought:
stacking the wakee on the same CPU as the waker - that's not exactly right.
Now my understanding is that, WF_SYNC is actually asking the wakee to find an idle CPU
in the waker's LLC domain within select_idle_sibling(), humm, not sure if I missed anything:

sd = rcu_dereference_all(per_cpu(sd_llc, target));
======================================================================================

thanks,
Chenyu

Re: [PATCH] sched/fair: Let sync wakeups target the waker's core

[Kayra Cizmeci]

Hi Vineeth,

Since all the numbers on this thread are from POWER, I ran this on x86 (Zen 3). I ran the test 3 times on both
baseline and baseline + patch. 

I first got the baseline from 0d83957076 that was like that: 

=num===cycles===cache-misses= 
[ 1: 11266587459: 19064897  ]
[ 2: 11329575367: 18860444  ]
[ 3: 11283159019: 19121436  ]

Then, I got the 0d83957076 + this patch:

=num===cycles===cache-misses=
[ 1: 11128721747: 19886131  ]
[ 2: 11163014797: 20019508  ]
[ 3: 11111832330: 19764849  ]

I tested them with AMD Ryzen 7 5700X 8 Cores, 16 Threads or by other means SMT2. Busybox, minimal initramfs and 
perf or to be more specific, this command: perf stat -r 10 -e cycles,cache-misses perf bench sched pipe

There is a 1.40% decrease of cycles baseline to this patch while there is a 4.60% increase in cache-misses. 
Both of these values are average values of the decrease and the increase. The values also don't overlap.

I don't really have explanations for why this values came back like that, 
I wasn't expecting them to be like this. Especially cache-misses.

Tested-by: Kayra Cizmeci <kayracizmeci@gmail.com>

Thanks,
Kayra