[PATCH v3] sched/fair: Prefer fully idle cores for NOHZ balancing

[PATCH v3] sched/fair: Prefer fully idle cores for NOHZ balancing

[Andrea Righi]

find_new_ilb() selects the first idle housekeeping CPU without
considering whether another thread is running on the same physical core.
On an SMT system, the idle load balancer can therefore activate both
siblings even when another housekeeping CPU has an entirely idle core.

On most SMT systems, this is not problematic because the idle load
balancer is a short-lived activity and the transient wakeup of a sibling
has negligible performance impact.

However, this can be particularly costly on NVIDIA Olympus cores used in
Vera. Briefly activating an otherwise idle sibling can reduce the
performance available to the other sibling and this effect does not
necessarily end once the activated sibling becomes idle: after the ILB
finishes and its CPU enters WFI, full single-thread performance is
restored only after the sibling has remained idle for a qualification
interval (10 Ki cycles on the tested Vera system). Repeated short
sibling wakeups can therefore sustain the interference even with little
actual overlap.

Prevent this by preferring an idle housekeeping CPU whose entire SMT
core is idle. Retain the first idle CPU as a fallback when no fully idle
core is available, so NOHZ balancing continues to make forward progress.
Once a partially busy core has been examined, skip its remaining SMT
siblings to avoid repeating the core-idle check on wide SMT systems.

Tests performed using an ad hoc GEMM benchmark running one CPU-intensive
task per SMT core within its CPU affinity mask improved from
approximately 6.2 TFLOP/s to 9.4 TFLOP/s.

Note that this preference may wake a fully idle physical core instead of
using an idle sibling of an active core, potentially increasing ILB
wakeup latency or energy consumption on some architectures. It may also
scan additional CPUs before selecting the one to run the ILB. The
selection falls back to the first idle CPU when no fully idle SMT core
is available. Non-SMT systems continue to select the first idle
housekeeping CPU.

Cc: Mete Durlu <meted@linux.ibm.com>
Cc: K Prateek Nayak <kprateek.nayak@amd.com>
Cc: Shrikanth Hegde <sshegde@linux.ibm.com>
Signed-off-by: Andrea Righi <arighi@nvidia.com>
---
Changes in v3:
 - After finding an idle fallback, skip all siblings when a busy CPU is
   encountered, avoiding per-CPU traversal of known-busy cores (Mete Durlu)
 - Link to v2: https://lore.kernel.org/all/20260729163225.1987068-1-arighi@nvidia.com/

Changes in v2:
 - Avoid repeated is_core_idle() checks on wide SMT systems by pruning
   the remaining siblings of a partially busy core (Prateek Nayak)
 - Link to v1: https://lore.kernel.org/r/20260728214442.1648483-1-arighi@nvidia.com/

 kernel/sched/fair.c | 56 +++++++++++++++++++++++++++++++++++++--------
 1 file changed, 47 insertions(+), 9 deletions(-)

diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
index 37001c63452e5..574b6b3ee922a 100644
--- a/kernel/sched/fair.c
+++ b/kernel/sched/fair.c
@@ -13965,28 +13965,66 @@ static inline int on_null_domain(struct rq *rq)
 static inline int find_new_ilb(void)
 {
 	int this_cpu = smp_processor_id();
-	const struct cpumask *hk_mask;
-	int ilb_cpu;
+	struct cpumask *ilb_cpus;
+	int ilb_cpu, fallback = -1;
+
+	lockdep_assert_irqs_disabled();
 
-	hk_mask = housekeeping_cpumask(HK_TYPE_KERNEL_NOISE);
+	/*
+	 * Reuse the per-CPU select_rq_mask, which is protected from concurrent
+	 * use on this CPU by having interrupts disabled.
+	 */
+	ilb_cpus = this_cpu_cpumask_var_ptr(select_rq_mask);
+	cpumask_and(ilb_cpus, nohz.idle_cpus_mask,
+		    housekeeping_cpumask(HK_TYPE_KERNEL_NOISE));
 
-	for_each_cpu_and(ilb_cpu, nohz.idle_cpus_mask, hk_mask) {
+	for_each_cpu(ilb_cpu, ilb_cpus) {
 		if (ilb_cpu == this_cpu)
 			continue;
 
-		if (idle_cpu(ilb_cpu))
-			return ilb_cpu;
+		if (!idle_cpu(ilb_cpu)) {
+			/*
+			 * Once an idle fallback exists, a busy CPU proves that
+			 * this core cannot be fully idle. Skip its siblings.
+			 */
+			if (sched_smt_active() && fallback >= 0)
+				cpumask_andnot(ilb_cpus, ilb_cpus,
+					       cpu_smt_mask(ilb_cpu));
+			continue;
+		}
+
+		/*
+		 * Running the idle load balancer on an idle sibling of a busy
+		 * SMT core can reduce the capacity available to its sibling. Prefer
+		 * a CPU whose entire core is idle, but retain the first idle CPU as
+		 * a fallback so idle balancing can still make progress when no fully
+		 * idle core exists.
+		 */
+		if (sched_smt_active() && !is_core_idle(ilb_cpu)) {
+			if (fallback < 0)
+				fallback = ilb_cpu;
+
+			/*
+			 * The core is not idle, so there is no need to check
+			 * any of its other SMT siblings.
+			 */
+			cpumask_andnot(ilb_cpus, ilb_cpus,
+				       cpu_smt_mask(ilb_cpu));
+			continue;
+		}
+
+		return ilb_cpu;
 	}
 
-	return -1;
+	return fallback;
 }
 
 /*
  * Kick a CPU to do the NOHZ balancing, if it is time for it, via a cross-CPU
  * SMP function call (IPI).
  *
- * We pick the first idle CPU in the HK_TYPE_KERNEL_NOISE housekeeping set
- * (if there is one).
+ * Prefer a CPU on a fully idle core in the HK_TYPE_KERNEL_NOISE housekeeping
+ * set. Fall back to the first idle CPU when no fully idle core exists.
  */
 static void kick_ilb(unsigned int flags)
 {
-- 
2.55.0

Re: [PATCH v3] sched/fair: Prefer fully idle cores for NOHZ balancing

[Vincent Guittot]

On Fri, 31 Jul 2026 at 21:20, Andrea Righi <arighi@nvidia.com> wrote:
>
> find_new_ilb() selects the first idle housekeeping CPU without
> considering whether another thread is running on the same physical core.
> On an SMT system, the idle load balancer can therefore activate both
> siblings even when another housekeeping CPU has an entirely idle core.
>
> On most SMT systems, this is not problematic because the idle load
> balancer is a short-lived activity and the transient wakeup of a sibling
> has negligible performance impact.
>
> However, this can be particularly costly on NVIDIA Olympus cores used in
> Vera. Briefly activating an otherwise idle sibling can reduce the
> performance available to the other sibling and this effect does not
> necessarily end once the activated sibling becomes idle: after the ILB
> finishes and its CPU enters WFI, full single-thread performance is
> restored only after the sibling has remained idle for a qualification
> interval (10 Ki cycles on the tested Vera system). Repeated short
> sibling wakeups can therefore sustain the interference even with little
> actual overlap.
>
> Prevent this by preferring an idle housekeeping CPU whose entire SMT
> core is idle. Retain the first idle CPU as a fallback when no fully idle
> core is available, so NOHZ balancing continues to make forward progress.
> Once a partially busy core has been examined, skip its remaining SMT
> siblings to avoid repeating the core-idle check on wide SMT systems.
>
> Tests performed using an ad hoc GEMM benchmark running one CPU-intensive
> task per SMT core within its CPU affinity mask improved from
> approximately 6.2 TFLOP/s to 9.4 TFLOP/s.
>
> Note that this preference may wake a fully idle physical core instead of
> using an idle sibling of an active core, potentially increasing ILB
> wakeup latency or energy consumption on some architectures. It may also
> scan additional CPUs before selecting the one to run the ILB. The
> selection falls back to the first idle CPU when no fully idle SMT core
> is available. Non-SMT systems continue to select the first idle
> housekeeping CPU.
>
> Cc: Mete Durlu <meted@linux.ibm.com>
> Cc: K Prateek Nayak <kprateek.nayak@amd.com>
> Cc: Shrikanth Hegde <sshegde@linux.ibm.com>
> Signed-off-by: Andrea Righi <arighi@nvidia.com>
> ---
> Changes in v3:
>  - After finding an idle fallback, skip all siblings when a busy CPU is
>    encountered, avoiding per-CPU traversal of known-busy cores (Mete Durlu)
>  - Link to v2: https://lore.kernel.org/all/20260729163225.1987068-1-arighi@nvidia.com/
>
> Changes in v2:
>  - Avoid repeated is_core_idle() checks on wide SMT systems by pruning
>    the remaining siblings of a partially busy core (Prateek Nayak)
>  - Link to v1: https://lore.kernel.org/r/20260728214442.1648483-1-arighi@nvidia.com/
>
>  kernel/sched/fair.c | 56 +++++++++++++++++++++++++++++++++++++--------
>  1 file changed, 47 insertions(+), 9 deletions(-)
>
> diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
> index 37001c63452e5..574b6b3ee922a 100644
> --- a/kernel/sched/fair.c
> +++ b/kernel/sched/fair.c
> @@ -13965,28 +13965,66 @@ static inline int on_null_domain(struct rq *rq)
>  static inline int find_new_ilb(void)
>  {
>         int this_cpu = smp_processor_id();
> -       const struct cpumask *hk_mask;
> -       int ilb_cpu;
> +       struct cpumask *ilb_cpus;
> +       int ilb_cpu, fallback = -1;
> +
> +       lockdep_assert_irqs_disabled();
>
> -       hk_mask = housekeeping_cpumask(HK_TYPE_KERNEL_NOISE);
> +       /*
> +        * Reuse the per-CPU select_rq_mask, which is protected from concurrent
> +        * use on this CPU by having interrupts disabled.
> +        */
> +       ilb_cpus = this_cpu_cpumask_var_ptr(select_rq_mask);
> +       cpumask_and(ilb_cpus, nohz.idle_cpus_mask,
> +                   housekeeping_cpumask(HK_TYPE_KERNEL_NOISE));
>
> -       for_each_cpu_and(ilb_cpu, nohz.idle_cpus_mask, hk_mask) {
> +       for_each_cpu(ilb_cpu, ilb_cpus) {
>                 if (ilb_cpu == this_cpu)

this_cpu is not idle so you can apply the same as below

Other than that looks good to me

>                         continue;
>
> -               if (idle_cpu(ilb_cpu))
> -                       return ilb_cpu;
> +               if (!idle_cpu(ilb_cpu)) {

+               if (ilb_cpu == this_cpu || !idle_cpu(ilb_cpu)) {

> +                       /*
> +                        * Once an idle fallback exists, a busy CPU proves that
> +                        * this core cannot be fully idle. Skip its siblings.
> +                        */
> +                       if (sched_smt_active() && fallback >= 0)
> +                               cpumask_andnot(ilb_cpus, ilb_cpus,
> +                                              cpu_smt_mask(ilb_cpu));
> +                       continue;
> +               }
> +
> +               /*
> +                * Running the idle load balancer on an idle sibling of a busy
> +                * SMT core can reduce the capacity available to its sibling. Prefer
> +                * a CPU whose entire core is idle, but retain the first idle CPU as
> +                * a fallback so idle balancing can still make progress when no fully
> +                * idle core exists.
> +                */
> +               if (sched_smt_active() && !is_core_idle(ilb_cpu)) {
> +                       if (fallback < 0)
> +                               fallback = ilb_cpu;
> +
> +                       /*
> +                        * The core is not idle, so there is no need to check
> +                        * any of its other SMT siblings.
> +                        */
> +                       cpumask_andnot(ilb_cpus, ilb_cpus,
> +                                      cpu_smt_mask(ilb_cpu));
> +                       continue;
> +               }
> +
> +               return ilb_cpu;
>         }
>
> -       return -1;
> +       return fallback;
>  }
>
>  /*
>   * Kick a CPU to do the NOHZ balancing, if it is time for it, via a cross-CPU
>   * SMP function call (IPI).
>   *
> - * We pick the first idle CPU in the HK_TYPE_KERNEL_NOISE housekeeping set
> - * (if there is one).
> + * Prefer a CPU on a fully idle core in the HK_TYPE_KERNEL_NOISE housekeeping
> + * set. Fall back to the first idle CPU when no fully idle core exists.
>   */
>  static void kick_ilb(unsigned int flags)
>  {
> --
> 2.55.0

Re: [PATCH v3] sched/fair: Prefer fully idle cores for NOHZ balancing

[K Prateek Nayak]

Hello Vincent,

On 8/4/2026 2:12 PM, Vincent Guittot wrote:
>> diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
>> index 37001c63452e5..574b6b3ee922a 100644
>> --- a/kernel/sched/fair.c
>> +++ b/kernel/sched/fair.c
>> @@ -13965,28 +13965,66 @@ static inline int on_null_domain(struct rq *rq)
>>  static inline int find_new_ilb(void)
>>  {
>>         int this_cpu = smp_processor_id();
>> -       const struct cpumask *hk_mask;
>> -       int ilb_cpu;
>> +       struct cpumask *ilb_cpus;
>> +       int ilb_cpu, fallback = -1;
>> +
>> +       lockdep_assert_irqs_disabled();
>>
>> -       hk_mask = housekeeping_cpumask(HK_TYPE_KERNEL_NOISE);
>> +       /*
>> +        * Reuse the per-CPU select_rq_mask, which is protected from concurrent
>> +        * use on this CPU by having interrupts disabled.
>> +        */
>> +       ilb_cpus = this_cpu_cpumask_var_ptr(select_rq_mask);
>> +       cpumask_and(ilb_cpus, nohz.idle_cpus_mask,
>> +                   housekeeping_cpumask(HK_TYPE_KERNEL_NOISE));
>>
>> -       for_each_cpu_and(ilb_cpu, nohz.idle_cpus_mask, hk_mask) {
>> +       for_each_cpu(ilb_cpu, ilb_cpus) {
>>                 if (ilb_cpu == this_cpu)
> 
> this_cpu is not idle so you can apply the same as below

Dumb question: nohz_balancer_kick() already does a
nohz_balance_exit_idle(rq) before trying to find the ilb_cpu via
find_new_ilb() so is it even possible for this_cpu to be set on the
nohz.idle_cpus_mask here?

Even on weakly ordered systems, reads and writes to the same location
(nohz.idle_cpus_mask) cannot get reordered right?

> 
> Other than that looks good to me
> 
>>                         continue;
>>
>> -               if (idle_cpu(ilb_cpu))
>> -                       return ilb_cpu;
>> +               if (!idle_cpu(ilb_cpu)) {
> 
> +               if (ilb_cpu == this_cpu || !idle_cpu(ilb_cpu)) {

Otherwise, this makes sense.

-- 
Thanks and Regards,
Prateek

Re: [PATCH v3] sched/fair: Prefer fully idle cores for NOHZ balancing

[Vincent Guittot]

On Tue, 4 Aug 2026 at 11:49, K Prateek Nayak <kprateek.nayak@amd.com> wrote:
>
> Hello Vincent,
>
> On 8/4/2026 2:12 PM, Vincent Guittot wrote:
> >> diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
> >> index 37001c63452e5..574b6b3ee922a 100644
> >> --- a/kernel/sched/fair.c
> >> +++ b/kernel/sched/fair.c
> >> @@ -13965,28 +13965,66 @@ static inline int on_null_domain(struct rq *rq)
> >>  static inline int find_new_ilb(void)
> >>  {
> >>         int this_cpu = smp_processor_id();
> >> -       const struct cpumask *hk_mask;
> >> -       int ilb_cpu;
> >> +       struct cpumask *ilb_cpus;
> >> +       int ilb_cpu, fallback = -1;
> >> +
> >> +       lockdep_assert_irqs_disabled();
> >>
> >> -       hk_mask = housekeeping_cpumask(HK_TYPE_KERNEL_NOISE);
> >> +       /*
> >> +        * Reuse the per-CPU select_rq_mask, which is protected from concurrent
> >> +        * use on this CPU by having interrupts disabled.
> >> +        */
> >> +       ilb_cpus = this_cpu_cpumask_var_ptr(select_rq_mask);
> >> +       cpumask_and(ilb_cpus, nohz.idle_cpus_mask,
> >> +                   housekeeping_cpumask(HK_TYPE_KERNEL_NOISE));
> >>
> >> -       for_each_cpu_and(ilb_cpu, nohz.idle_cpus_mask, hk_mask) {
> >> +       for_each_cpu(ilb_cpu, ilb_cpus) {
> >>                 if (ilb_cpu == this_cpu)
> >
> > this_cpu is not idle so you can apply the same as below
>
> Dumb question: nohz_balancer_kick() already does a
> nohz_balance_exit_idle(rq) before trying to find the ilb_cpu via
> find_new_ilb() so is it even possible for this_cpu to be set on the
> nohz.idle_cpus_mask here?

Yes that's a good point, the test is useless

>
> Even on weakly ordered systems, reads and writes to the same location
> (nohz.idle_cpus_mask) cannot get reordered right?

Yes

>
> >
> > Other than that looks good to me
> >
> >>                         continue;
> >>
> >> -               if (idle_cpu(ilb_cpu))
> >> -                       return ilb_cpu;
> >> +               if (!idle_cpu(ilb_cpu)) {
> >
> > +               if (ilb_cpu == this_cpu || !idle_cpu(ilb_cpu)) {
>
> Otherwise, this makes sense.
>
> --
> Thanks and Regards,
> Prateek
>

Re: [PATCH v3] sched/fair: Prefer fully idle cores for NOHZ balancing

[Andrea Righi]

Hi Vincent and Prateek,

On Tue, Aug 04, 2026 at 12:30:22PM +0200, Vincent Guittot wrote:
> On Tue, 4 Aug 2026 at 11:49, K Prateek Nayak <kprateek.nayak@amd.com> wrote:
> >
> > Hello Vincent,
> >
> > On 8/4/2026 2:12 PM, Vincent Guittot wrote:
> > >> diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
> > >> index 37001c63452e5..574b6b3ee922a 100644
> > >> --- a/kernel/sched/fair.c
> > >> +++ b/kernel/sched/fair.c
> > >> @@ -13965,28 +13965,66 @@ static inline int on_null_domain(struct rq *rq)
> > >>  static inline int find_new_ilb(void)
> > >>  {
> > >>         int this_cpu = smp_processor_id();
> > >> -       const struct cpumask *hk_mask;
> > >> -       int ilb_cpu;
> > >> +       struct cpumask *ilb_cpus;
> > >> +       int ilb_cpu, fallback = -1;
> > >> +
> > >> +       lockdep_assert_irqs_disabled();
> > >>
> > >> -       hk_mask = housekeeping_cpumask(HK_TYPE_KERNEL_NOISE);
> > >> +       /*
> > >> +        * Reuse the per-CPU select_rq_mask, which is protected from concurrent
> > >> +        * use on this CPU by having interrupts disabled.
> > >> +        */
> > >> +       ilb_cpus = this_cpu_cpumask_var_ptr(select_rq_mask);
> > >> +       cpumask_and(ilb_cpus, nohz.idle_cpus_mask,
> > >> +                   housekeeping_cpumask(HK_TYPE_KERNEL_NOISE));
> > >>
> > >> -       for_each_cpu_and(ilb_cpu, nohz.idle_cpus_mask, hk_mask) {
> > >> +       for_each_cpu(ilb_cpu, ilb_cpus) {
> > >>                 if (ilb_cpu == this_cpu)
> > >
> > > this_cpu is not idle so you can apply the same as below
> >
> > Dumb question: nohz_balancer_kick() already does a
> > nohz_balance_exit_idle(rq) before trying to find the ilb_cpu via
> > find_new_ilb() so is it even possible for this_cpu to be set on the
> > nohz.idle_cpus_mask here?
> 
> Yes that's a good point, the test is useless
> 
> >
> > Even on weakly ordered systems, reads and writes to the same location
> > (nohz.idle_cpus_mask) cannot get reordered right?
> 
> Yes

Right, checking this_cpu is redundant, I'll remove it and send a v4.

Thanks!
-Andrea

> 
> >
> > >
> > > Other than that looks good to me
> > >
> > >>                         continue;
> > >>
> > >> -               if (idle_cpu(ilb_cpu))
> > >> -                       return ilb_cpu;
> > >> +               if (!idle_cpu(ilb_cpu)) {
> > >
> > > +               if (ilb_cpu == this_cpu || !idle_cpu(ilb_cpu)) {
> >
> > Otherwise, this makes sense.
> >
> > --
> > Thanks and Regards,
> > Prateek
> >

Re: [PATCH v3] sched/fair: Prefer fully idle cores for NOHZ balancing

[Mete Durlu]

Hi,

> find_new_ilb() selects the first idle housekeeping CPU without
> considering whether another thread is running on the same physical core.
> On an SMT system, the idle load balancer can therefore activate both
> siblings even when another housekeeping CPU has an entirely idle core.
> 
> On most SMT systems, this is not problematic because the idle load
> balancer is a short-lived activity and the transient wakeup of a sibling
> has negligible performance impact.
> 
> However, this can be particularly costly on NVIDIA Olympus cores used in
> Vera. Briefly activating an otherwise idle sibling can reduce the
> performance available to the other sibling and this effect does not
> necessarily end once the activated sibling becomes idle: after the ILB
> finishes and its CPU enters WFI, full single-thread performance is
> restored only after the sibling has remained idle for a qualification
> interval (10 Ki cycles on the tested Vera system). Repeated short
> sibling wakeups can therefore sustain the interference even with little
> actual overlap.
> 
> Prevent this by preferring an idle housekeeping CPU whose entire SMT
> core is idle. Retain the first idle CPU as a fallback when no fully idle
> core is available, so NOHZ balancing continues to make forward progress.
> Once a partially busy core has been examined, skip its remaining SMT
> siblings to avoid repeating the core-idle check on wide SMT systems.
> 
> Tests performed using an ad hoc GEMM benchmark running one CPU-intensive
> task per SMT core within its CPU affinity mask improved from
> approximately 6.2 TFLOP/s to 9.4 TFLOP/s.

Although what you describe above with siblings suffering interference
does not really fit to s390, I'd like to hear more about what sort
of GEMM (general matrix multiplication) tests you did.

I tested this patch with a couple of different tools
- perf bench sched pipe
- hackbench
- uperf
- cyclictest
- stress-ng (3d-matrix and cyclic)

Didn't come across any meaningful difference in any of them on multiple
runs each. So I was curious about the exact sort of benchmark you
mention here.

One minor nit for the diff below;

> 
> diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
> index 37001c63452e5..574b6b3ee922a 100644
> --- a/kernel/sched/fair.c
> +++ b/kernel/sched/fair.c
> @@ -13965,28 +13965,66 @@ static inline int on_null_domain(struct rq *rq)
>   static inline int find_new_ilb(void)
>   {
>   	int this_cpu = smp_processor_id();
> -	const struct cpumask *hk_mask;
> -	int ilb_cpu;
> +	struct cpumask *ilb_cpus;
> +	int ilb_cpu, fallback = -1;
> +
> +	lockdep_assert_irqs_disabled();
>   
> -	hk_mask = housekeeping_cpumask(HK_TYPE_KERNEL_NOISE);
> +	/*
> +	 * Reuse the per-CPU select_rq_mask, which is protected from concurrent
> +	 * use on this CPU by having interrupts disabled.
> +	 */
> +	ilb_cpus = this_cpu_cpumask_var_ptr(select_rq_mask);
> +	cpumask_and(ilb_cpus, nohz.idle_cpus_mask,
> +		    housekeeping_cpumask(HK_TYPE_KERNEL_NOISE));
>   
> -	for_each_cpu_and(ilb_cpu, nohz.idle_cpus_mask, hk_mask) {
> +	for_each_cpu(ilb_cpu, ilb_cpus) {
>   		if (ilb_cpu == this_cpu)
>   			continue;
>   
> -		if (idle_cpu(ilb_cpu))
> -			return ilb_cpu;
> +		if (!idle_cpu(ilb_cpu)) {
> +			/*
> +			 * Once an idle fallback exists, a busy CPU proves that
> +			 * this core cannot be fully idle. Skip its siblings.
> +			 */
> +			if (sched_smt_active() && fallback >= 0)
> +				cpumask_andnot(ilb_cpus, ilb_cpus,
> +					       cpu_smt_mask(ilb_cpu));

nit;
With line break this if block is now taking multiple lines and deserves
its own curly braces.

With or without the nit, feel free to add my r-b to v4, I doubt removal
of the "this_cpu" check will change anything as it is a dud.

Reviewed By: Mete Durlu <meted@linux.ibm.com>

Re: [PATCH v3] sched/fair: Prefer fully idle cores for NOHZ balancing

[Andrea Righi]

Hi Mete,

On Tue, Aug 04, 2026 at 02:36:48PM +0200, Mete Durlu wrote:
> Hi,
> 
> > find_new_ilb() selects the first idle housekeeping CPU without
> > considering whether another thread is running on the same physical core.
> > On an SMT system, the idle load balancer can therefore activate both
> > siblings even when another housekeeping CPU has an entirely idle core.
> > 
> > On most SMT systems, this is not problematic because the idle load
> > balancer is a short-lived activity and the transient wakeup of a sibling
> > has negligible performance impact.
> > 
> > However, this can be particularly costly on NVIDIA Olympus cores used in
> > Vera. Briefly activating an otherwise idle sibling can reduce the
> > performance available to the other sibling and this effect does not
> > necessarily end once the activated sibling becomes idle: after the ILB
> > finishes and its CPU enters WFI, full single-thread performance is
> > restored only after the sibling has remained idle for a qualification
> > interval (10 Ki cycles on the tested Vera system). Repeated short
> > sibling wakeups can therefore sustain the interference even with little
> > actual overlap.
> > 
> > Prevent this by preferring an idle housekeeping CPU whose entire SMT
> > core is idle. Retain the first idle CPU as a fallback when no fully idle
> > core is available, so NOHZ balancing continues to make forward progress.
> > Once a partially busy core has been examined, skip its remaining SMT
> > siblings to avoid repeating the core-idle check on wide SMT systems.
> > 
> > Tests performed using an ad hoc GEMM benchmark running one CPU-intensive
> > task per SMT core within its CPU affinity mask improved from
> > approximately 6.2 TFLOP/s to 9.4 TFLOP/s.
> 
> Although what you describe above with siblings suffering interference
> does not really fit to s390, I'd like to hear more about what sort
> of GEMM (general matrix multiplication) tests you did.
> 
> I tested this patch with a couple of different tools
> - perf bench sched pipe
> - hackbench
> - uperf
> - cyclictest
> - stress-ng (3d-matrix and cyclic)
> 
> Didn't come across any meaningful difference in any of them on multiple
> runs each. So I was curious about the exact sort of benchmark you
> mention here.

Thanks for testing on s390!

The original benchmark I used is based on an internal NVPL container that I
can't share publicly. However, I tried with the public OpenBLAS SGEMM benchmark
and I can see exactly the same behavior and effects:

  https://github.com/OpenMathLib/OpenBLAS

My test machine has the following topology (arm64):

    CPUs:                   352
    Sockets:                2
    Cores per socket:       88
    Threads per core:       2
    NUMA node 0 CPUs:       0-87,176-263
    NUMA node 1 CPUs:       88-175,264-351

The SMT sibling pairs on NUMA node 0 are (0,176), (1,177), ..., (87,263). I only
used NUMA node 0, to prevent adding potential NUMA side effects.

I used OpenBLAS v0.3.33, built using GCC 13.3.0 with the ARMv8 SVE kernels and
OpenMP threading:

 $ make -j176 \
        TARGET=ARMV8SVE \
        USE_OPENMP=1 \
        NUM_THREADS=176 \
        NOFORTRAN=1

 $ make -C benchmark sgemm.goto \
        TARGET=ARMV8SVE \
        USE_OPENMP=1 \
        NUM_THREADS=176 \
        NOFORTRAN=1

The unpatched kernel was tip/master, while the patched kernel used the same base
with only this change applied.

A 10-loop runs produced:

  unpatched:  5.088 TFLOP/s
  patched:    7.143 TFLOP/s

Two additional 5-loop runs produced:

  unpatched:  5.338, 5.246 TFLOP/s
  patched:    7.082, 7.144 TFLOP/s

The median across all the measurements increased from 5.246 TFLOP/s to 7.143
TFLOP/s, an improvement of approximately 36.2%.

The absolute throughput is lower than my initial NVPL benchmark, as expected
from the different GEMM implementations, but the relative behavior seems to be
consistent.

> 
> One minor nit for the diff below;
> 
> > 
> > diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
> > index 37001c63452e5..574b6b3ee922a 100644
> > --- a/kernel/sched/fair.c
> > +++ b/kernel/sched/fair.c
> > @@ -13965,28 +13965,66 @@ static inline int on_null_domain(struct rq *rq)
> >   static inline int find_new_ilb(void)
> >   {
> >   	int this_cpu = smp_processor_id();
> > -	const struct cpumask *hk_mask;
> > -	int ilb_cpu;
> > +	struct cpumask *ilb_cpus;
> > +	int ilb_cpu, fallback = -1;
> > +
> > +	lockdep_assert_irqs_disabled();
> > -	hk_mask = housekeeping_cpumask(HK_TYPE_KERNEL_NOISE);
> > +	/*
> > +	 * Reuse the per-CPU select_rq_mask, which is protected from concurrent
> > +	 * use on this CPU by having interrupts disabled.
> > +	 */
> > +	ilb_cpus = this_cpu_cpumask_var_ptr(select_rq_mask);
> > +	cpumask_and(ilb_cpus, nohz.idle_cpus_mask,
> > +		    housekeeping_cpumask(HK_TYPE_KERNEL_NOISE));
> > -	for_each_cpu_and(ilb_cpu, nohz.idle_cpus_mask, hk_mask) {
> > +	for_each_cpu(ilb_cpu, ilb_cpus) {
> >   		if (ilb_cpu == this_cpu)
> >   			continue;
> > -		if (idle_cpu(ilb_cpu))
> > -			return ilb_cpu;
> > +		if (!idle_cpu(ilb_cpu)) {
> > +			/*
> > +			 * Once an idle fallback exists, a busy CPU proves that
> > +			 * this core cannot be fully idle. Skip its siblings.
> > +			 */
> > +			if (sched_smt_active() && fallback >= 0)
> > +				cpumask_andnot(ilb_cpus, ilb_cpus,
> > +					       cpu_smt_mask(ilb_cpu));
> 
> nit;
> With line break this if block is now taking multiple lines and deserves
> its own curly braces.

The cpumask_andnot() invocation fits within the line-length limit, so I'll move
it on the same line.

> 
> With or without the nit, feel free to add my r-b to v4, I doubt removal
> of the "this_cpu" check will change anything as it is a dud.
> 
> Reviewed By: Mete Durlu <meted@linux.ibm.com>
> 

Thanks,
-Andrea

Re: [PATCH v3] sched/fair: Prefer fully idle cores for NOHZ balancing

[K Prateek Nayak]

Hello Andrea,

On 8/1/2026 12:49 AM, Andrea Righi wrote:
> find_new_ilb() selects the first idle housekeeping CPU without
> considering whether another thread is running on the same physical core.
> On an SMT system, the idle load balancer can therefore activate both
> siblings even when another housekeeping CPU has an entirely idle core.
> 
> On most SMT systems, this is not problematic because the idle load
> balancer is a short-lived activity and the transient wakeup of a sibling
> has negligible performance impact.
> 
> However, this can be particularly costly on NVIDIA Olympus cores used in
> Vera. Briefly activating an otherwise idle sibling can reduce the
> performance available to the other sibling and this effect does not
> necessarily end once the activated sibling becomes idle: after the ILB
> finishes and its CPU enters WFI, full single-thread performance is
> restored only after the sibling has remained idle for a qualification
> interval (10 Ki cycles on the tested Vera system). Repeated short
> sibling wakeups can therefore sustain the interference even with little
> actual overlap.
> 
> Prevent this by preferring an idle housekeeping CPU whose entire SMT
> core is idle. Retain the first idle CPU as a fallback when no fully idle
> core is available, so NOHZ balancing continues to make forward progress.
> Once a partially busy core has been examined, skip its remaining SMT
> siblings to avoid repeating the core-idle check on wide SMT systems.
> 
> Tests performed using an ad hoc GEMM benchmark running one CPU-intensive
> task per SMT core within its CPU affinity mask improved from
> approximately 6.2 TFLOP/s to 9.4 TFLOP/s.
> 
> Note that this preference may wake a fully idle physical core instead of
> using an idle sibling of an active core, potentially increasing ILB
> wakeup latency or energy consumption on some architectures. It may also
> scan additional CPUs before selecting the one to run the ILB. The
> selection falls back to the first idle CPU when no fully idle SMT core
> is available. Non-SMT systems continue to select the first idle
> housekeeping CPU.

I tested this on my end and couldn't really find a workload that shows a
significant difference but I like the general idea of using an idle core
for newidle balance so FWIW, feel free to include:

Reviewed-by: K Prateek Nayak <kprateek.nayak@amd.com>
Tested-by: K Prateek Nayak <kprateek.nayak@amd.com>

[..snip..]

> diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
> index 37001c63452e5..574b6b3ee922a 100644
> --- a/kernel/sched/fair.c
> +++ b/kernel/sched/fair.c
> @@ -13965,28 +13965,66 @@ static inline int on_null_domain(struct rq *rq)
>  static inline int find_new_ilb(void)
>  {
>  	int this_cpu = smp_processor_id();
> -	const struct cpumask *hk_mask;
> -	int ilb_cpu;
> +	struct cpumask *ilb_cpus;
> +	int ilb_cpu, fallback = -1;

nit. In case you are re-spinning this, could you rearrange these
two line to follow reverse x-mas tree.

-- 
Thanks and Regards,
Prateek