[PATCH] sched/fair: Add more core cookie check in wake up fast path

[Fernand Sieber <sieberf@amazon.com>]

The fast path in select_idle_sibling() can place tasks on CPUs without
considering core scheduling constraints, potentially causing immediate
force idle when the sibling runs an incompatible task.

Add cookie compatibility checks before selecting a CPU in the fast path.
This prevents placing waking tasks on CPUs where the sibling is running
an incompatible task, reducing force idle occurrences.

Testing
=======

Perf testing using 10 threads on 8 CPUs on a platform with each core
sporting two hyperthreads.
Each thread is running 1200 iterations of a cycle consisting of a random
period of work between 0-10ms followed by a random period of sleep 0-30ms.

The goal of this configuration is to apply a light load on the system
(33%), with randomization causing opportunities for scheduling placement
including all cases of idle cores, partially idle cores and fully busy
cores.

Each test configuration is run 3 times and time to work completion is
measured and averaged. First configuration doesn't use cookies, second
configuration groups the 10 threads in 5 pairs of cookies, third
configuration also groups the 10 threads in 5 pairs of cookies, but with
this patch applied.

Test Results (seconds)

Configuration          Run 1    Run 2    Run 3    Average  Overhead
No cookie              24.653   24.454   24.586   24.564   0.000
Cookie                 25.896   26.256   25.979   26.044   1.480
Cookie + patch         25.346   25.007   25.042   25.132   0.568

The patch reduces cookie overhead by 61.7% (0.568/1.480 = 0.383)

Signed-off-by: Fernand Sieber <sieberf@amazon.com>
---
 kernel/sched/fair.c | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)

diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
index 5b752324270b..90ceb3da2251 100644
--- a/kernel/sched/fair.c
+++ b/kernel/sched/fair.c
@@ -7647,7 +7647,7 @@ static int select_idle_smt(struct task_struct *p, struct sched_domain *sd, int t
 		 */
 		if (!cpumask_test_cpu(cpu, sched_domain_span(sd)))
 			continue;
-		if (available_idle_cpu(cpu) || sched_idle_cpu(cpu))
+		if (__select_idle_cpu(cpu, p) != -1)
 			return cpu;
 	}

@@ -7841,6 +7841,7 @@ static int select_idle_sibling(struct task_struct *p, int prev, int target)
 	lockdep_assert_irqs_disabled();

 	if ((available_idle_cpu(target) || sched_idle_cpu(target)) &&
+	    sched_core_cookie_match(cpu_rq(target), p) &&
 	    asym_fits_cpu(task_util, util_min, util_max, target))
 		return target;

@@ -7849,6 +7850,7 @@ static int select_idle_sibling(struct task_struct *p, int prev, int target)
 	 */
 	if (prev != target && cpus_share_cache(prev, target) &&
 	    (available_idle_cpu(prev) || sched_idle_cpu(prev)) &&
+	    sched_core_cookie_match(cpu_rq(prev), p) &&
 	    asym_fits_cpu(task_util, util_min, util_max, prev)) {

 		if (!static_branch_unlikely(&sched_cluster_active) ||
@@ -7881,6 +7883,7 @@ static int select_idle_sibling(struct task_struct *p, int prev, int target)
 	    recent_used_cpu != target &&
 	    cpus_share_cache(recent_used_cpu, target) &&
 	    (available_idle_cpu(recent_used_cpu) || sched_idle_cpu(recent_used_cpu)) &&
+	    sched_core_cookie_match(cpu_rq(recent_used_cpu), p) &&
 	    cpumask_test_cpu(recent_used_cpu, p->cpus_ptr) &&
 	    asym_fits_cpu(task_util, util_min, util_max, recent_used_cpu)) {

--
2.34.1




Amazon Development Centre (South Africa) (Proprietary) Limited
29 Gogosoa Street, Observatory, Cape Town, Western Cape, 7925, South Africa
Registration Number: 2004 / 034463 / 07