The Linux Kernel Mailing List
 help / color / mirror / Atom feed
From: Chen Yu <chen.yu@linux.dev>
To: Andrea Righi <arighi@nvidia.com>
Cc: Vincent Guittot <vincent.guittot@linaro.org>,
	Rik van Riel <riel@surriel.com>,
	Usama Arif <usama.arif@linux.dev>,
	peterz@infradead.org, mingo@redhat.com, kprateek.nayak@amd.com,
	juri.lelli@redhat.com, dietmar.eggemann@arm.com,
	rostedt@goodmis.org, bsegall@google.com, mgorman@suse.de,
	vschneid@redhat.com, samuelcrossley@gmail.com,
	linux-kernel@vger.kernel.org, stable@vger.kernel.org,
	kas@kernel.org, Arnav Aghav <aghav@meta.com>
Subject: Re: [PATCH] Revert "sched/fair: Add SIS_UTIL support to select_idle_capacity()"
Date: Mon, 16 Nov 2026 21:00:14 +0800	[thread overview]
Message-ID: <avr-XumgYsrnA6vZ@three-body> (raw)
In-Reply-To: <avglp8nnc4u9OCb5@gpd4>

On Sat, Nov 14, 2026 at 10:11:03AM +0100, Andrea Righi wrote:
> Date: Sat, 14 Nov 2026 10:11:03 +0100
> From: Andrea Righi <arighi@nvidia.com>
> To: Vincent Guittot <vincent.guittot@linaro.org>
> Cc: Rik van Riel <riel@surriel.com>, Usama Arif <usama.arif@linux.dev>,
>  peterz@infradead.org, mingo@redhat.com, kprateek.nayak@amd.com,
>  juri.lelli@redhat.com, dietmar.eggemann@arm.com, rostedt@goodmis.org,
>  bsegall@google.com, mgorman@suse.de, vschneid@redhat.com,
>  samuelcrossley@gmail.com, linux-kernel@vger.kernel.org,
>  stable@vger.kernel.org, kas@kernel.org, Arnav Aghav <aghav@meta.com>
> Subject: Re: [PATCH] Revert "sched/fair: Add SIS_UTIL support to
>  select_idle_capacity()"
> 
> On Sat, Nov 14, 2026 at 09:18:30AM +0100, Vincent Guittot wrote:
> > On Fri, 13 Nov 2026 at 20:32, Rik van Riel <riel@surriel.com> wrote:
> > >
> > > On Fri, 2026-11-13 at 07:31 -0800, Usama Arif wrote:
> > > >
> > > > On a 72-CPU NVIDIA grace arm64 server, small reported capacity
> > > > differences
> > > > (1013-1024) within the single LLC activate asymmetric selection. Both
> > > > the reference and affected kernels were observed with a zero scan
> > > > hint. The reference kernel ignored it and still searched the domain.
> > > > A latency-sensitive, wakeup-heavy production workload saw its
> > > > reported
> > > > minute p50 latencies average 129 ms on the reference kernel and 223
> > > > ms
> > > > on the affected kernel.
> > > >
> > > > Disabling SIS_UTIL reduced the aggregate involuntary context switch
> > > > rate of worker main threads from 71.6k/s to 13.3k/s; re-enabling it
> > > > raised the rate to 73.7k/s.
> > >
> > > Afaik, the asymmetrical selection is supposed
> > > to solve a very specific problem: when a task
> > > requires X amount of CPU capacity to run, we
> > > try to avoid scheduling it on a CPU that has
> > > only a fraction of X available.
> > >
> > > This creates the issue that a task that appears
> > > to "need" exactly X, because it's been running
> > > on a CPU with full available capacity, ends up
> > > not being scheduled at all when the available
> > > idle core has X-3% available capacity, because
> > > that CPU happens to spend 3% of its time handling
> > > interrupts, or running something else.
> > 
> > The task will be scheduled on this idle cpu/core of (X-3% capacity) if
> > no other idle CPU with X capacity is available.
> > The selection in select_idle_capacity is:
> > - Idle core with enough capacity
> > - Idle core with enough capacity but doesn't fullfill uclamp_min
> > - Idle core that doesn't fit
> > - Idle CPU with enough capacity
> > - Idle CPU with enough capacity but doesn't fullfill uclamp_min
> > - Idle CPU that doesn't fit
> 
> Right, and when the scan budget is exhausted select_idle_capacity()
> returns the best idle candidate found so far, even if it doesn't fit.
> 
> However, in the zero-budget case, we return before examining any
> candidates in the scan domain. Relaxing the capacity-fit rule inside
> that scan wouldn't change the outcome, because we never reach that
> check.
> 
> Maybe Rik's suggestion could instead help with the earlier checks in
> select_idle_sibling()? Accepting an idle target, previous or recently
> used CPU whose capacity is "close enough" could avoid reaching the
> zero-budget bailout.
> 
> The tradeoff is that we'd accept a potentially weaker CPU immediately,
> without searching for a better one.
>

Do you something similar to cluster wakeup, give the wakeup a second chance to find an idle
CPU in the LLC. I suppose what you suggested is:

 
diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
index 470ba0dc6719..d1f3b179013e 100644
--- a/kernel/sched/fair.c
+++ b/kernel/sched/fair.c
@@ -9141,6 +9141,24 @@ static inline bool asym_fits_cpu(unsigned long util,
 	return true;
 }
 
+static int asym_idle_retry(struct task_struct *p, int prev, int target,
+			   int recent)
+{
+	if (choose_idle_cpu(target, p))
+		return target;
+
+	if (prev != target && cpus_share_cache(prev, target) &&
+	    choose_idle_cpu(prev, p))
+		return prev;
+
+	if (recent != prev && recent != target &&
+	    cpus_share_cache(recent, target) &&
+	    choose_idle_cpu(recent, p))
+		return recent;
+
+	return target;
+}
+
 /*
  * Try and locate an idle core/thread in the LLC cache domain.
  */
@@ -9149,7 +9167,7 @@ static int select_idle_sibling(struct task_struct *p, int prev, int target)
 	bool has_idle_core = false;
 	struct sched_domain *sd;
 	unsigned long task_util, util_min, util_max;
-	int i, recent_used_cpu, prev_aff = -1;
+	int i, recent_used_cpu, recent, prev_aff = -1;
 
 	/*
 	 * On asymmetric system, update task utilization because we will check
@@ -9206,6 +9224,7 @@ static int select_idle_sibling(struct task_struct *p, int prev, int target)
 
 	/* Check a recently used CPU as a potential idle candidate: */
 	recent_used_cpu = p->recent_used_cpu;
+	recent = recent_used_cpu;
 	p->recent_used_cpu = prev;
 	if (recent_used_cpu != prev &&
 	    recent_used_cpu != target &&
@@ -9244,7 +9263,7 @@ static int select_idle_sibling(struct task_struct *p, int prev, int target)
 				target = i;
 				goto select_smt_priority;
 			}
-			return target;
+			return asym_idle_retry(p, prev, target, recent);
 		}
 	}
 
-- 
2.43.0

 
> Thanks,
> -Andrea
> 
> > 
> > >
> > > In that situation, it will be better to run the
> > > task on the CPU with 97% capacity right now,
> > > than to not run the task until a CPU frees up
> > > later.
> > >
> > > Things like SMT and power-driven hardware
> > > scaling complicate the matter further, because
> > > we simply cannot see the full amount of capacity
> > > that is available on each CPU.
> > >
> > > That makes me wonder if we should simply accept
> > > any CPU that's "close enough to X" when dealing
> > > with asymmetrical placement.
> > >
> > > Say, anything that's above 3/4 of the capacity
> > > of X would be considered good enough to run the task.
> > >
> > > That still prevents us from placing a CPU intensive
> > > task on a little core in a big.LITTLE system, or
> > > from placing a task on a CPU that spends 80% of its
> > > time handling interrupts, but it will allow a task
> > > to run right now, when the capacity imbalance is small.
> > >
> > > Does that make sense, or am I forgetting something
> > > important?
> > >
> > > --
> > > All Rights Reversed.

      reply	other threads:[~2026-11-16 12:59 UTC|newest]

Thread overview: 10+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2026-11-13 15:31 [PATCH] Revert "sched/fair: Add SIS_UTIL support to select_idle_capacity()" Usama Arif
2026-11-13 17:28 ` Andrea Righi
2026-11-13 18:26   ` Usama Arif
2026-11-13 21:32     ` Andrea Righi
2026-11-14  9:06       ` Vincent Guittot
2026-11-16 15:37         ` Chen Yu
2026-11-13 19:31 ` Rik van Riel
2026-11-14  8:18   ` Vincent Guittot
2026-11-14  9:11     ` Andrea Righi
2026-11-16 13:00       ` Chen Yu [this message]

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=avr-XumgYsrnA6vZ@three-body \
    --to=chen.yu@linux.dev \
    --cc=aghav@meta.com \
    --cc=arighi@nvidia.com \
    --cc=bsegall@google.com \
    --cc=dietmar.eggemann@arm.com \
    --cc=juri.lelli@redhat.com \
    --cc=kas@kernel.org \
    --cc=kprateek.nayak@amd.com \
    --cc=linux-kernel@vger.kernel.org \
    --cc=mgorman@suse.de \
    --cc=mingo@redhat.com \
    --cc=peterz@infradead.org \
    --cc=riel@surriel.com \
    --cc=rostedt@goodmis.org \
    --cc=samuelcrossley@gmail.com \
    --cc=stable@vger.kernel.org \
    --cc=usama.arif@linux.dev \
    --cc=vincent.guittot@linaro.org \
    --cc=vschneid@redhat.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox