[PATCH v4] sched/fair: Prefer fully idle cores for NOHZ balancing

[PATCH v4] sched/fair: Prefer fully idle cores for NOHZ balancing

[Andrea Righi]

find_new_ilb() selects the first idle housekeeping CPU without
considering whether another thread is running on the same physical core.
On an SMT system, the idle load balancer can therefore activate both
siblings even when another housekeeping CPU has an entirely idle core.

On most SMT systems, this is not problematic because the idle load
balancer is a short-lived activity and the transient wakeup of a sibling
has negligible performance impact.

However, this can be particularly costly on NVIDIA Olympus cores used in
Vera. Briefly activating an otherwise idle sibling can reduce the
performance available to the other sibling and this effect does not
necessarily end once the activated sibling becomes idle: after the ILB
finishes and its CPU enters WFI, full single-thread performance is
restored only after the sibling has remained idle for a qualification
interval (10 Ki cycles on the tested Vera system). Repeated short
sibling wakeups can therefore sustain the interference even with little
actual overlap.

Prevent this by preferring an idle housekeeping CPU whose entire SMT
core is idle. Retain the first idle CPU as a fallback when no fully idle
core is available, so NOHZ balancing continues to make forward progress.
Once a partially busy core has been examined, skip its remaining SMT
siblings to avoid repeating the core-idle check on wide SMT systems.

Tests performed using an ad hoc GEMM benchmark running one CPU-intensive
task per SMT core within its CPU affinity mask improved from
approximately 6.2 TFLOP/s to 9.4 TFLOP/s.

Note that this preference may wake a fully idle physical core instead of
using an idle sibling of an active core, potentially increasing ILB
wakeup latency or energy consumption on some architectures. It may also
scan additional CPUs before selecting the one to run the ILB. The
selection falls back to the first idle CPU when no fully idle SMT core
is available. Non-SMT systems continue to select the first idle
housekeeping CPU.

Cc: Vincent Guittot <vincent.guittot@linaro.org>
Cc: K Prateek Nayak <kprateek.nayak@amd.com>
Cc: Shrikanth Hegde <sshegde@linux.ibm.com>
Signed-off-by: Andrea Righi <arighi@nvidia.com>
Reviewed-by: Mete Durlu <meted@linux.ibm.com>
---
Changes in v4:
 - Remove redundant this_cpu check (Prateek Nayak, Vincent Guittot)
 - Link to v3: https://lore.kernel.org/all/20260731191957.3199642-1-arighi@nvidia.com/

Changes in v3:
 - After finding an idle fallback, skip all siblings when a busy CPU is
   encountered, avoiding per-CPU traversal of known-busy cores (Mete Durlu)
 - Link to v2: https://lore.kernel.org/all/20260729163225.1987068-1-arighi@nvidia.com/

Changes in v2:
 - Avoid repeated is_core_idle() checks on wide SMT systems by pruning
   the remaining siblings of a partially busy core (Prateek Nayak)
 - Link to v1: https://lore.kernel.org/r/20260728214442.1648483-1-arighi@nvidia.com/

 kernel/sched/fair.c | 55 ++++++++++++++++++++++++++++++++++++---------
 1 file changed, 44 insertions(+), 11 deletions(-)

diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
index 37001c63452e5..9a975a684b487 100644
--- a/kernel/sched/fair.c
+++ b/kernel/sched/fair.c
@@ -13964,29 +13964,62 @@ static inline int on_null_domain(struct rq *rq)
  */
 static inline int find_new_ilb(void)
 {
-	int this_cpu = smp_processor_id();
-	const struct cpumask *hk_mask;
-	int ilb_cpu;
+	struct cpumask *ilb_cpus;
+	int ilb_cpu, fallback = -1;
+
+	lockdep_assert_irqs_disabled();
+
+	/*
+	 * Reuse the per-CPU select_rq_mask, which is protected from concurrent
+	 * use on this CPU by having interrupts disabled.
+	 */
+	ilb_cpus = this_cpu_cpumask_var_ptr(select_rq_mask);
+	cpumask_and(ilb_cpus, nohz.idle_cpus_mask,
+		    housekeeping_cpumask(HK_TYPE_KERNEL_NOISE));
+
+	for_each_cpu(ilb_cpu, ilb_cpus) {
+		if (!idle_cpu(ilb_cpu)) {
+			/*
+			 * Once an idle fallback exists, a busy CPU proves that
+			 * this core cannot be fully idle. Skip its siblings.
+			 */
+			if (sched_smt_active() && fallback >= 0)
+				cpumask_andnot(ilb_cpus, ilb_cpus, cpu_smt_mask(ilb_cpu));
+			continue;
+		}
 
-	hk_mask = housekeeping_cpumask(HK_TYPE_KERNEL_NOISE);
+		/*
+		 * Running the idle load balancer on an idle sibling of a busy
+		 * SMT core can reduce the capacity available to its sibling. Prefer
+		 * a CPU whose entire core is idle, but retain the first idle CPU as
+		 * a fallback so idle balancing can still make progress when no fully
+		 * idle core exists.
+		 */
+		if (sched_smt_active() && !is_core_idle(ilb_cpu)) {
+			if (fallback < 0)
+				fallback = ilb_cpu;
 
-	for_each_cpu_and(ilb_cpu, nohz.idle_cpus_mask, hk_mask) {
-		if (ilb_cpu == this_cpu)
+			/*
+			 * The core is not idle, so there is no need to check
+			 * any of its other SMT siblings.
+			 */
+			cpumask_andnot(ilb_cpus, ilb_cpus,
+				       cpu_smt_mask(ilb_cpu));
 			continue;
+		}
 
-		if (idle_cpu(ilb_cpu))
-			return ilb_cpu;
+		return ilb_cpu;
 	}
 
-	return -1;
+	return fallback;
 }
 
 /*
  * Kick a CPU to do the NOHZ balancing, if it is time for it, via a cross-CPU
  * SMP function call (IPI).
  *
- * We pick the first idle CPU in the HK_TYPE_KERNEL_NOISE housekeeping set
- * (if there is one).
+ * Prefer a CPU on a fully idle core in the HK_TYPE_KERNEL_NOISE housekeeping
+ * set. Fall back to the first idle CPU when no fully idle core exists.
  */
 static void kick_ilb(unsigned int flags)
 {
-- 
2.55.0

Re: [PATCH v4] sched/fair: Prefer fully idle cores for NOHZ balancing

[Shrikanth Hegde]

Hi Andrea,

On 8/4/26 8:43 PM, Andrea Righi wrote:
> find_new_ilb() selects the first idle housekeeping CPU without
> considering whether another thread is running on the same physical core.
> On an SMT system, the idle load balancer can therefore activate both
> siblings even when another housekeeping CPU has an entirely idle core.
> 
> On most SMT systems, this is not problematic because the idle load
> balancer is a short-lived activity and the transient wakeup of a sibling
> has negligible performance impact.
> 
> However, this can be particularly costly on NVIDIA Olympus cores used in
> Vera. Briefly activating an otherwise idle sibling can reduce the
> performance available to the other sibling and this effect does not
> necessarily end once the activated sibling becomes idle: after the ILB
> finishes and its CPU enters WFI, full single-thread performance is
> restored only after the sibling has remained idle for a qualification
> interval (10 Ki cycles on the tested Vera system). Repeated short
> sibling wakeups can therefore sustain the interference even with little
> actual overlap.

Is this hardware cost has been accounted in things like cpufreq or cpuidle?
or have you set it already to max performance.

> 
> Prevent this by preferring an idle housekeeping CPU whose entire SMT
> core is idle. Retain the first idle CPU as a fallback when no fully idle
> core is available, so NOHZ balancing continues to make forward progress.
> Once a partially busy core has been examined, skip its remaining SMT
> siblings to avoid repeating the core-idle check on wide SMT systems.
> 

Just Curious, making nohz_full=<except first core> yeilds similar numbers?

> Tests performed using an ad hoc GEMM benchmark running one CPU-intensive
> task per SMT core within its CPU affinity mask improved from
> approximately 6.2 TFLOP/s to 9.4 TFLOP/s.
> 
> Note that this preference may wake a fully idle physical core instead of
> using an idle sibling of an active core, potentially increasing ILB
> wakeup latency or energy consumption on some architectures. It may also
> scan additional CPUs before selecting the one to run the ILB. The
> selection falls back to the first idle CPU when no fully idle SMT core
> is available. Non-SMT systems continue to select the first idle
> housekeeping CPU.
> 
> Cc: Vincent Guittot <vincent.guittot@linaro.org>
> Cc: K Prateek Nayak <kprateek.nayak@amd.com>
> Cc: Shrikanth Hegde <sshegde@linux.ibm.com>
> Signed-off-by: Andrea Righi <arighi@nvidia.com>
> Reviewed-by: Mete Durlu <meted@linux.ibm.com>
> ---
> Changes in v4:
>   - Remove redundant this_cpu check (Prateek Nayak, Vincent Guittot)
>   - Link to v3: https://lore.kernel.org/all/20260731191957.3199642-1-arighi@nvidia.com/
> 
> Changes in v3:
>   - After finding an idle fallback, skip all siblings when a busy CPU is
>     encountered, avoiding per-CPU traversal of known-busy cores (Mete Durlu)
>   - Link to v2: https://lore.kernel.org/all/20260729163225.1987068-1-arighi@nvidia.com/
> 
> Changes in v2:
>   - Avoid repeated is_core_idle() checks on wide SMT systems by pruning
>     the remaining siblings of a partially busy core (Prateek Nayak)
>   - Link to v1: https://lore.kernel.org/r/20260728214442.1648483-1-arighi@nvidia.com/
> 
>   kernel/sched/fair.c | 55 ++++++++++++++++++++++++++++++++++++---------
>   1 file changed, 44 insertions(+), 11 deletions(-)
> 
> diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
> index 37001c63452e5..9a975a684b487 100644
> --- a/kernel/sched/fair.c
> +++ b/kernel/sched/fair.c
> @@ -13964,29 +13964,62 @@ static inline int on_null_domain(struct rq *rq)
>    */
>   static inline int find_new_ilb(void)
>   {
> -	int this_cpu = smp_processor_id();
> -	const struct cpumask *hk_mask;
> -	int ilb_cpu;
> +	struct cpumask *ilb_cpus;
> +	int ilb_cpu, fallback = -1;
> +
> +	lockdep_assert_irqs_disabled();
> +
> +	/*
> +	 * Reuse the per-CPU select_rq_mask, which is protected from concurrent
> +	 * use on this CPU by having interrupts disabled.
> +	 */
> +	ilb_cpus = this_cpu_cpumask_var_ptr(select_rq_mask);
> +	cpumask_and(ilb_cpus, nohz.idle_cpus_mask,
> +		    housekeeping_cpumask(HK_TYPE_KERNEL_NOISE));
> +
> +	for_each_cpu(ilb_cpu, ilb_cpus) {
> +		if (!idle_cpu(ilb_cpu)) {
> +			/*
> +			 * Once an idle fallback exists, a busy CPU proves that
> +			 * this core cannot be fully idle. Skip its siblings.
> +			 */
> +			if (sched_smt_active() && fallback >= 0)
> +				cpumask_andnot(ilb_cpus, ilb_cpus, cpu_smt_mask(ilb_cpu));
> +			continue;
> +		}
>   
> -	hk_mask = housekeeping_cpumask(HK_TYPE_KERNEL_NOISE);
> +		/*
> +		 * Running the idle load balancer on an idle sibling of a busy
> +		 * SMT core can reduce the capacity available to its sibling. Prefer
> +		 * a CPU whose entire core is idle, but retain the first idle CPU as
> +		 * a fallback so idle balancing can still make progress when no fully
> +		 * idle core exists.
> +		 */
> +		if (sched_smt_active() && !is_core_idle(ilb_cpu)) {
> +			if (fallback < 0)
> +				fallback = ilb_cpu;
>   
> -	for_each_cpu_and(ilb_cpu, nohz.idle_cpus_mask, hk_mask) {
> -		if (ilb_cpu == this_cpu)
> +			/*
> +			 * The core is not idle, so there is no need to check
> +			 * any of its other SMT siblings.
> +			 */
> +			cpumask_andnot(ilb_cpus, ilb_cpus,
> +				       cpu_smt_mask(ilb_cpu));
>   			continue;
> +		}
>   
> -		if (idle_cpu(ilb_cpu))
> -			return ilb_cpu;
> +		return ilb_cpu;
>   	}
>   
> -	return -1;
> +	return fallback;
>   }
>   
>   /*
>    * Kick a CPU to do the NOHZ balancing, if it is time for it, via a cross-CPU
>    * SMP function call (IPI).
>    *
> - * We pick the first idle CPU in the HK_TYPE_KERNEL_NOISE housekeeping set
> - * (if there is one).
> + * Prefer a CPU on a fully idle core in the HK_TYPE_KERNEL_NOISE housekeeping
> + * set. Fall back to the first idle CPU when no fully idle core exists.
>    */
>   static void kick_ilb(unsigned int flags)
>   {

Re: [PATCH v4] sched/fair: Prefer fully idle cores for NOHZ balancing

[Andrea Righi]

Hi Shrikanth,

On Tue, Aug 04, 2026 at 08:49:01PM +0530, Shrikanth Hegde wrote:
> Hi Andrea,
> 
> On 8/4/26 8:43 PM, Andrea Righi wrote:
> > find_new_ilb() selects the first idle housekeeping CPU without
> > considering whether another thread is running on the same physical core.
> > On an SMT system, the idle load balancer can therefore activate both
> > siblings even when another housekeeping CPU has an entirely idle core.
> > 
> > On most SMT systems, this is not problematic because the idle load
> > balancer is a short-lived activity and the transient wakeup of a sibling
> > has negligible performance impact.
> > 
> > However, this can be particularly costly on NVIDIA Olympus cores used in
> > Vera. Briefly activating an otherwise idle sibling can reduce the
> > performance available to the other sibling and this effect does not
> > necessarily end once the activated sibling becomes idle: after the ILB
> > finishes and its CPU enters WFI, full single-thread performance is
> > restored only after the sibling has remained idle for a qualification
> > interval (10 Ki cycles on the tested Vera system). Repeated short
> > sibling wakeups can therefore sustain the interference even with little
> > actual overlap.
> 
> Is this hardware cost has been accounted in things like cpufreq or cpuidle?
> or have you set it already to max performance.

All the CPUs were using the cppc_cpufreq performance governor during the tests.
The scaling minimum frequency was between 95.5% and 97% of the maximum
frequency, depending on the CPU. The CPUs were not strictly pinned to their
maximum frequency, but the performance governor was in use.

The acpi_idle driver was enabled, with both LPI-0 and LPI-1 available. I didn't
disable the idle states.

> 
> > 
> > Prevent this by preferring an idle housekeeping CPU whose entire SMT
> > core is idle. Retain the first idle CPU as a fallback when no fully idle
> > core is available, so NOHZ balancing continues to make forward progress.
> > Once a partially busy core has been examined, skip its remaining SMT
> > siblings to avoid repeating the core-idle check on wide SMT systems.
> > 
> 
> Just Curious, making nohz_full=<except first core> yeilds similar numbers?

Yes, provided that the housekeeping core is also excluded from the workload.

I tested this booting with:

  nohz_full=1-175,177-351

This leaves CPUs 0 and 176 for housekeeping. I excluded that core from the
workload and ran 87 OpenMP tasks on the remaining node-0 cores (1 task per core,
excluding the housekeeping one):

 $ env OMP_NUM_THREADS=87 \
       OMP_DYNAMIC=false \
       OPENBLAS_LOOPS=10 \
       OPENBLAS_PARAM_M=16384 \
       OPENBLAS_PARAM_N=16384 \
       OPENBLAS_PARAM_K=16384 \
       numactl -C 1-87,177-263 --membind=0 \
       ./benchmark/sgemm.goto 1 1 1

Results:

    unpatched           : 5.246 TFLOP/s
    unpatched+nohz_full : 6.953 TFLOP/s
    patched             : 6.861 TFLOP/s
    patched+nohz_full   : 6.984 TFLOP/s

So, nohz_full seems to prevent the problematic ILB wakeups and can produce
similar results for this CPU-bound workload. However, it shouldn't be considered
a sobstiute for the ILB fix, since it requires explicit partitioning and
reserved housekeeping CPUs. Full dyntick also enables context tracking on the
isolated CPUs, adding kernel entry/exit overhead.

Thanks,
-Andrea

> 
> > Tests performed using an ad hoc GEMM benchmark running one CPU-intensive
> > task per SMT core within its CPU affinity mask improved from
> > approximately 6.2 TFLOP/s to 9.4 TFLOP/s.
> > 
> > Note that this preference may wake a fully idle physical core instead of
> > using an idle sibling of an active core, potentially increasing ILB
> > wakeup latency or energy consumption on some architectures. It may also
> > scan additional CPUs before selecting the one to run the ILB. The
> > selection falls back to the first idle CPU when no fully idle SMT core
> > is available. Non-SMT systems continue to select the first idle
> > housekeeping CPU.
> > 
> > Cc: Vincent Guittot <vincent.guittot@linaro.org>
> > Cc: K Prateek Nayak <kprateek.nayak@amd.com>
> > Cc: Shrikanth Hegde <sshegde@linux.ibm.com>
> > Signed-off-by: Andrea Righi <arighi@nvidia.com>
> > Reviewed-by: Mete Durlu <meted@linux.ibm.com>
> > ---
> > Changes in v4:
> >   - Remove redundant this_cpu check (Prateek Nayak, Vincent Guittot)
> >   - Link to v3: https://lore.kernel.org/all/20260731191957.3199642-1-arighi@nvidia.com/
> > 
> > Changes in v3:
> >   - After finding an idle fallback, skip all siblings when a busy CPU is
> >     encountered, avoiding per-CPU traversal of known-busy cores (Mete Durlu)
> >   - Link to v2: https://lore.kernel.org/all/20260729163225.1987068-1-arighi@nvidia.com/
> > 
> > Changes in v2:
> >   - Avoid repeated is_core_idle() checks on wide SMT systems by pruning
> >     the remaining siblings of a partially busy core (Prateek Nayak)
> >   - Link to v1: https://lore.kernel.org/r/20260728214442.1648483-1-arighi@nvidia.com/
> > 
> >   kernel/sched/fair.c | 55 ++++++++++++++++++++++++++++++++++++---------
> >   1 file changed, 44 insertions(+), 11 deletions(-)
> > 
> > diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
> > index 37001c63452e5..9a975a684b487 100644
> > --- a/kernel/sched/fair.c
> > +++ b/kernel/sched/fair.c
> > @@ -13964,29 +13964,62 @@ static inline int on_null_domain(struct rq *rq)
> >    */
> >   static inline int find_new_ilb(void)
> >   {
> > -	int this_cpu = smp_processor_id();
> > -	const struct cpumask *hk_mask;
> > -	int ilb_cpu;
> > +	struct cpumask *ilb_cpus;
> > +	int ilb_cpu, fallback = -1;
> > +
> > +	lockdep_assert_irqs_disabled();
> > +
> > +	/*
> > +	 * Reuse the per-CPU select_rq_mask, which is protected from concurrent
> > +	 * use on this CPU by having interrupts disabled.
> > +	 */
> > +	ilb_cpus = this_cpu_cpumask_var_ptr(select_rq_mask);
> > +	cpumask_and(ilb_cpus, nohz.idle_cpus_mask,
> > +		    housekeeping_cpumask(HK_TYPE_KERNEL_NOISE));
> > +
> > +	for_each_cpu(ilb_cpu, ilb_cpus) {
> > +		if (!idle_cpu(ilb_cpu)) {
> > +			/*
> > +			 * Once an idle fallback exists, a busy CPU proves that
> > +			 * this core cannot be fully idle. Skip its siblings.
> > +			 */
> > +			if (sched_smt_active() && fallback >= 0)
> > +				cpumask_andnot(ilb_cpus, ilb_cpus, cpu_smt_mask(ilb_cpu));
> > +			continue;
> > +		}
> > -	hk_mask = housekeeping_cpumask(HK_TYPE_KERNEL_NOISE);
> > +		/*
> > +		 * Running the idle load balancer on an idle sibling of a busy
> > +		 * SMT core can reduce the capacity available to its sibling. Prefer
> > +		 * a CPU whose entire core is idle, but retain the first idle CPU as
> > +		 * a fallback so idle balancing can still make progress when no fully
> > +		 * idle core exists.
> > +		 */
> > +		if (sched_smt_active() && !is_core_idle(ilb_cpu)) {
> > +			if (fallback < 0)
> > +				fallback = ilb_cpu;
> > -	for_each_cpu_and(ilb_cpu, nohz.idle_cpus_mask, hk_mask) {
> > -		if (ilb_cpu == this_cpu)
> > +			/*
> > +			 * The core is not idle, so there is no need to check
> > +			 * any of its other SMT siblings.
> > +			 */
> > +			cpumask_andnot(ilb_cpus, ilb_cpus,
> > +				       cpu_smt_mask(ilb_cpu));
> >   			continue;
> > +		}
> > -		if (idle_cpu(ilb_cpu))
> > -			return ilb_cpu;
> > +		return ilb_cpu;
> >   	}
> > -	return -1;
> > +	return fallback;
> >   }
> >   /*
> >    * Kick a CPU to do the NOHZ balancing, if it is time for it, via a cross-CPU
> >    * SMP function call (IPI).
> >    *
> > - * We pick the first idle CPU in the HK_TYPE_KERNEL_NOISE housekeeping set
> > - * (if there is one).
> > + * Prefer a CPU on a fully idle core in the HK_TYPE_KERNEL_NOISE housekeeping
> > + * set. Fall back to the first idle CPU when no fully idle core exists.
> >    */
> >   static void kick_ilb(unsigned int flags)
> >   {
> 

Re: [PATCH v4] sched/fair: Prefer fully idle cores for NOHZ balancing

[Shrikanth Hegde]

On 8/4/26 11:39 PM, Andrea Righi wrote:
> Hi Shrikanth,
> 

>> Just Curious, making nohz_full=<except first core> yeilds similar numbers?
> 
> Yes, provided that the housekeeping core is also excluded from the workload.
> 
> I tested this booting with:
> 
>    nohz_full=1-175,177-351
> 
> This leaves CPUs 0 and 176 for housekeeping. I excluded that core from the
> workload and ran 87 OpenMP tasks on the remaining node-0 cores (1 task per core,
> excluding the housekeeping one):
> 
>   $ env OMP_NUM_THREADS=87 \
>         OMP_DYNAMIC=false \
>         OPENBLAS_LOOPS=10 \
>         OPENBLAS_PARAM_M=16384 \
>         OPENBLAS_PARAM_N=16384 \
>         OPENBLAS_PARAM_K=16384 \
>         numactl -C 1-87,177-263 --membind=0 \
>         ./benchmark/sgemm.goto 1 1 1
> 
> Results:
> 
>      unpatched           : 5.246 TFLOP/s
>      unpatched+nohz_full : 6.953 TFLOP/s
>      patched             : 6.861 TFLOP/s
>      patched+nohz_full   : 6.984 TFLOP/s
> 
> So, nohz_full seems to prevent the problematic ILB wakeups and can produce
> similar results for this CPU-bound workload. However, it shouldn't be considered
> a sobstiute for the ILB fix, since it requires explicit partitioning and
> reserved housekeeping CPUs. Full dyntick also enables context tracking on the
> isolated CPUs, adding kernel entry/exit overhead.
> 

That's good to know. Thanks for running them.
I put "just curious", knowing nohz_full is not a solution.

Re: [PATCH v4] sched/fair: Prefer fully idle cores for NOHZ balancing

[Vincent Guittot]

On Tue, 4 Aug 2026 at 17:13, Andrea Righi <arighi@nvidia.com> wrote:
>
> find_new_ilb() selects the first idle housekeeping CPU without
> considering whether another thread is running on the same physical core.
> On an SMT system, the idle load balancer can therefore activate both
> siblings even when another housekeeping CPU has an entirely idle core.
>
> On most SMT systems, this is not problematic because the idle load
> balancer is a short-lived activity and the transient wakeup of a sibling
> has negligible performance impact.
>
> However, this can be particularly costly on NVIDIA Olympus cores used in
> Vera. Briefly activating an otherwise idle sibling can reduce the
> performance available to the other sibling and this effect does not
> necessarily end once the activated sibling becomes idle: after the ILB
> finishes and its CPU enters WFI, full single-thread performance is
> restored only after the sibling has remained idle for a qualification
> interval (10 Ki cycles on the tested Vera system). Repeated short
> sibling wakeups can therefore sustain the interference even with little
> actual overlap.
>
> Prevent this by preferring an idle housekeeping CPU whose entire SMT
> core is idle. Retain the first idle CPU as a fallback when no fully idle
> core is available, so NOHZ balancing continues to make forward progress.
> Once a partially busy core has been examined, skip its remaining SMT
> siblings to avoid repeating the core-idle check on wide SMT systems.
>
> Tests performed using an ad hoc GEMM benchmark running one CPU-intensive
> task per SMT core within its CPU affinity mask improved from
> approximately 6.2 TFLOP/s to 9.4 TFLOP/s.
>
> Note that this preference may wake a fully idle physical core instead of
> using an idle sibling of an active core, potentially increasing ILB
> wakeup latency or energy consumption on some architectures. It may also
> scan additional CPUs before selecting the one to run the ILB. The
> selection falls back to the first idle CPU when no fully idle SMT core
> is available. Non-SMT systems continue to select the first idle
> housekeeping CPU.
>
> Cc: Vincent Guittot <vincent.guittot@linaro.org>
> Cc: K Prateek Nayak <kprateek.nayak@amd.com>
> Cc: Shrikanth Hegde <sshegde@linux.ibm.com>
> Signed-off-by: Andrea Righi <arighi@nvidia.com>
> Reviewed-by: Mete Durlu <meted@linux.ibm.com>

Reviewed-by: Vincent Guittot <vincent.guittot@linaro.org>

> ---
> Changes in v4:
>  - Remove redundant this_cpu check (Prateek Nayak, Vincent Guittot)
>  - Link to v3: https://lore.kernel.org/all/20260731191957.3199642-1-arighi@nvidia.com/
>
> Changes in v3:
>  - After finding an idle fallback, skip all siblings when a busy CPU is
>    encountered, avoiding per-CPU traversal of known-busy cores (Mete Durlu)
>  - Link to v2: https://lore.kernel.org/all/20260729163225.1987068-1-arighi@nvidia.com/
>
> Changes in v2:
>  - Avoid repeated is_core_idle() checks on wide SMT systems by pruning
>    the remaining siblings of a partially busy core (Prateek Nayak)
>  - Link to v1: https://lore.kernel.org/r/20260728214442.1648483-1-arighi@nvidia.com/
>
>  kernel/sched/fair.c | 55 ++++++++++++++++++++++++++++++++++++---------
>  1 file changed, 44 insertions(+), 11 deletions(-)
>
> diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
> index 37001c63452e5..9a975a684b487 100644
> --- a/kernel/sched/fair.c
> +++ b/kernel/sched/fair.c
> @@ -13964,29 +13964,62 @@ static inline int on_null_domain(struct rq *rq)
>   */
>  static inline int find_new_ilb(void)
>  {
> -       int this_cpu = smp_processor_id();
> -       const struct cpumask *hk_mask;
> -       int ilb_cpu;
> +       struct cpumask *ilb_cpus;
> +       int ilb_cpu, fallback = -1;
> +
> +       lockdep_assert_irqs_disabled();
> +
> +       /*
> +        * Reuse the per-CPU select_rq_mask, which is protected from concurrent
> +        * use on this CPU by having interrupts disabled.
> +        */
> +       ilb_cpus = this_cpu_cpumask_var_ptr(select_rq_mask);
> +       cpumask_and(ilb_cpus, nohz.idle_cpus_mask,
> +                   housekeeping_cpumask(HK_TYPE_KERNEL_NOISE));
> +
> +       for_each_cpu(ilb_cpu, ilb_cpus) {
> +               if (!idle_cpu(ilb_cpu)) {
> +                       /*
> +                        * Once an idle fallback exists, a busy CPU proves that
> +                        * this core cannot be fully idle. Skip its siblings.
> +                        */
> +                       if (sched_smt_active() && fallback >= 0)
> +                               cpumask_andnot(ilb_cpus, ilb_cpus, cpu_smt_mask(ilb_cpu));
> +                       continue;
> +               }
>
> -       hk_mask = housekeeping_cpumask(HK_TYPE_KERNEL_NOISE);
> +               /*
> +                * Running the idle load balancer on an idle sibling of a busy
> +                * SMT core can reduce the capacity available to its sibling. Prefer
> +                * a CPU whose entire core is idle, but retain the first idle CPU as
> +                * a fallback so idle balancing can still make progress when no fully
> +                * idle core exists.
> +                */
> +               if (sched_smt_active() && !is_core_idle(ilb_cpu)) {
> +                       if (fallback < 0)
> +                               fallback = ilb_cpu;
>
> -       for_each_cpu_and(ilb_cpu, nohz.idle_cpus_mask, hk_mask) {
> -               if (ilb_cpu == this_cpu)
> +                       /*
> +                        * The core is not idle, so there is no need to check
> +                        * any of its other SMT siblings.
> +                        */
> +                       cpumask_andnot(ilb_cpus, ilb_cpus,
> +                                      cpu_smt_mask(ilb_cpu));
>                         continue;
> +               }
>
> -               if (idle_cpu(ilb_cpu))
> -                       return ilb_cpu;
> +               return ilb_cpu;
>         }
>
> -       return -1;
> +       return fallback;
>  }
>
>  /*
>   * Kick a CPU to do the NOHZ balancing, if it is time for it, via a cross-CPU
>   * SMP function call (IPI).
>   *
> - * We pick the first idle CPU in the HK_TYPE_KERNEL_NOISE housekeeping set
> - * (if there is one).
> + * Prefer a CPU on a fully idle core in the HK_TYPE_KERNEL_NOISE housekeeping
> + * set. Fall back to the first idle CPU when no fully idle core exists.
>   */
>  static void kick_ilb(unsigned int flags)
>  {
> --
> 2.55.0

Re: [PATCH v4] sched/fair: Prefer fully idle cores for NOHZ balancing

[Shrikanth Hegde]

Hi Andrea,

On 8/4/26 8:43 PM, Andrea Righi wrote:
> find_new_ilb() selects the first idle housekeeping CPU without
> considering whether another thread is running on the same physical core.
> On an SMT system, the idle load balancer can therefore activate both
> siblings even when another housekeeping CPU has an entirely idle core.
> 
> On most SMT systems, this is not problematic because the idle load
> balancer is a short-lived activity and the transient wakeup of a sibling
> has negligible performance impact.
> 
> However, this can be particularly costly on NVIDIA Olympus cores used in
> Vera. Briefly activating an otherwise idle sibling can reduce the
> performance available to the other sibling and this effect does not
> necessarily end once the activated sibling becomes idle: after the ILB
> finishes and its CPU enters WFI, full single-thread performance is
> restored only after the sibling has remained idle for a qualification
> interval (10 Ki cycles on the tested Vera system). Repeated short
> sibling wakeups can therefore sustain the interference even with little
> actual overlap.
> 
> Prevent this by preferring an idle housekeeping CPU whose entire SMT
> core is idle. Retain the first idle CPU as a fallback when no fully idle
> core is available, so NOHZ balancing continues to make forward progress.
> Once a partially busy core has been examined, skip its remaining SMT
> siblings to avoid repeating the core-idle check on wide SMT systems.
> 
> Tests performed using an ad hoc GEMM benchmark running one CPU-intensive
> task per SMT core within its CPU affinity mask improved from
> approximately 6.2 TFLOP/s to 9.4 TFLOP/s.
> 
> Note that this preference may wake a fully idle physical core instead of
> using an idle sibling of an active core, potentially increasing ILB
> wakeup latency or energy consumption on some architectures. It may also
> scan additional CPUs before selecting the one to run the ILB. The
> selection falls back to the first idle CPU when no fully idle SMT core
> is available. Non-SMT systems continue to select the first idle
> housekeeping CPU.
> 
> Cc: Vincent Guittot <vincent.guittot@linaro.org>
> Cc: K Prateek Nayak <kprateek.nayak@amd.com>
> Cc: Shrikanth Hegde <sshegde@linux.ibm.com>
> Signed-off-by: Andrea Righi <arighi@nvidia.com>
> Reviewed-by: Mete Durlu <meted@linux.ibm.com>
> ---
> Changes in v4:
>   - Remove redundant this_cpu check (Prateek Nayak, Vincent Guittot)
>   - Link to v3: https://lore.kernel.org/all/20260731191957.3199642-1-arighi@nvidia.com/
> 
> Changes in v3:
>   - After finding an idle fallback, skip all siblings when a busy CPU is
>     encountered, avoiding per-CPU traversal of known-busy cores (Mete Durlu)
>   - Link to v2: https://lore.kernel.org/all/20260729163225.1987068-1-arighi@nvidia.com/
> 
> Changes in v2:
>   - Avoid repeated is_core_idle() checks on wide SMT systems by pruning
>     the remaining siblings of a partially busy core (Prateek Nayak)
>   - Link to v1: https://lore.kernel.org/r/20260728214442.1648483-1-arighi@nvidia.com/
> 
>   kernel/sched/fair.c | 55 ++++++++++++++++++++++++++++++++++++---------
>   1 file changed, 44 insertions(+), 11 deletions(-)
> 
> diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
> index 37001c63452e5..9a975a684b487 100644
> --- a/kernel/sched/fair.c
> +++ b/kernel/sched/fair.c
> @@ -13964,29 +13964,62 @@ static inline int on_null_domain(struct rq *rq)
>    */
>   static inline int find_new_ilb(void)
>   {
> -	int this_cpu = smp_processor_id();
> -	const struct cpumask *hk_mask;
> -	int ilb_cpu;
> +	struct cpumask *ilb_cpus;
> +	int ilb_cpu, fallback = -1;
> +
> +	lockdep_assert_irqs_disabled();
> +
> +	/*
> +	 * Reuse the per-CPU select_rq_mask, which is protected from concurrent
> +	 * use on this CPU by having interrupts disabled.
> +	 */
> +	ilb_cpus = this_cpu_cpumask_var_ptr(select_rq_mask);
> +	cpumask_and(ilb_cpus, nohz.idle_cpus_mask,
> +		    housekeeping_cpumask(HK_TYPE_KERNEL_NOISE));
> +
> +	for_each_cpu(ilb_cpu, ilb_cpus) {
> +		if (!idle_cpu(ilb_cpu)) {
> +			/*
> +			 * Once an idle fallback exists, a busy CPU proves that
> +			 * this core cannot be fully idle. Skip its siblings.
> +			 */
> +			if (sched_smt_active() && fallback >= 0)
> +				cpumask_andnot(ilb_cpus, ilb_cpus, cpu_smt_mask(ilb_cpu));
> +			continue;
> +		}
>   
> -	hk_mask = housekeeping_cpumask(HK_TYPE_KERNEL_NOISE);
> +		/*
> +		 * Running the idle load balancer on an idle sibling of a busy
> +		 * SMT core can reduce the capacity available to its sibling. Prefer
> +		 * a CPU whose entire core is idle, but retain the first idle CPU as
> +		 * a fallback so idle balancing can still make progress when no fully
> +		 * idle core exists.
> +		 */
> +		if (sched_smt_active() && !is_core_idle(ilb_cpu)) {
> +			if (fallback < 0)
> +				fallback = ilb_cpu;
>   
> -	for_each_cpu_and(ilb_cpu, nohz.idle_cpus_mask, hk_mask) {
> -		if (ilb_cpu == this_cpu)
> +			/*
> +			 * The core is not idle, so there is no need to check
> +			 * any of its other SMT siblings.
> +			 */
> +			cpumask_andnot(ilb_cpus, ilb_cpus,
> +				       cpu_smt_mask(ilb_cpu));
>   			continue;
> +		}
>   
> -		if (idle_cpu(ilb_cpu))
> -			return ilb_cpu;
> +		return ilb_cpu;
>   	}
>   
> -	return -1;
> +	return fallback;
>   }
>   
>   /*
>    * Kick a CPU to do the NOHZ balancing, if it is time for it, via a cross-CPU
>    * SMP function call (IPI).
>    *
> - * We pick the first idle CPU in the HK_TYPE_KERNEL_NOISE housekeeping set
> - * (if there is one).
> + * Prefer a CPU on a fully idle core in the HK_TYPE_KERNEL_NOISE housekeeping
> + * set. Fall back to the first idle CPU when no fully idle core exists.
>    */
>   static void kick_ilb(unsigned int flags)
>   {


I have only a small system at this moment.
I don't see any noticeable regression/improvement in SMT4.

FWIW, patch looks okay to me.
Reviewed-by: Shrikanth Hegde <sshegde@linux.ibm.com>


BTW, Andrea, are you planning to do that refactoring of select_rq_mask
or you want me to pick it up?

Re: [PATCH v4] sched/fair: Prefer fully idle cores for NOHZ balancing

[Andrea Righi]

Hi Shrikanth,

On Thu, Aug 06, 2026 at 03:56:20PM +0530, Shrikanth Hegde wrote:
...
> I have only a small system at this moment.
> I don't see any noticeable regression/improvement in SMT4.
> 
> FWIW, patch looks okay to me.
> Reviewed-by: Shrikanth Hegde <sshegde@linux.ibm.com>

Thanks for testing!

> 
> 
> BTW, Andrea, are you planning to do that refactoring of select_rq_mask
> or you want me to pick it up?

I was waiting for this patch to land before looking at the select_rq_mask
refactoring. If you have time and would like to pick it up, please go ahead,
I'll be happy to review it.

-Andrea

Re: [PATCH v4] sched/fair: Prefer fully idle cores for NOHZ balancing

[Shrikanth Hegde]

Hi Andrea, Peter, Prateek,

>> BTW, Andrea, are you planning to do that refactoring of select_rq_mask
>> or you want me to pick it up?
> 
> I was waiting for this patch to land before looking at the select_rq_mask
> refactoring. If you have time and would like to pick it up, please go ahead,
> I'll be happy to review it.
> 
> -Andrea

"Copying peter's reply from v1"
>> Peter, do you have a preference on whether the mask renaming/accessor work
>> should be addressed separately, or included as a preparatory patch for this?

>Separate would be fine. Perhaps another way to do it is using the fancy
>new clang context analysis.

I am not sure if below is what peter meant.
Below diff uses scope based stuff we have in kernel.

This is only a draft, this may need to cleaned up and split into 5 patches.
If the patch below makes sense, please let me know I can work on sending the
series.

Further Notes:
- I checked all the current usecases in sched, all have irq disabled. So
   have lockdep_assert_irqs_disabled in the constructor.
- Its usage is limited to scheduler as for now, there may be few such callsites
   outside. But i don't think its worth making it as generic api usecase.
- Variables names are not well thought off. I just scribbled something. I am sure
   it can be better. Please suggest.
- May need PROVE_LOCKING/debug build check for in_use and those WARN_ON's so that
   it remains minimal. or let it be as is?


I have run hackbench on small 40 CPU system, didn;t see any difference in performance
numbers, not any warnings. Please try and see if it works.


Subject: [PATCH] sched: introduce generic scratch cpumask

---
  kernel/sched/core.c     | 18 +++++++++--
  kernel/sched/deadline.c | 18 +++--------
  kernel/sched/ext/idle.c | 28 +++++------------
  kernel/sched/fair.c     | 42 +++++++++-----------------
  kernel/sched/rt.c       | 15 ++-------
  kernel/sched/sched.h    | 67 +++++++++++++++++++++++++++++++++++++++--
  6 files changed, 109 insertions(+), 79 deletions(-)

diff --git a/kernel/sched/core.c b/kernel/sched/core.c
index 2e7cde033a31..eb44b562abfb 100644
--- a/kernel/sched/core.c
+++ b/kernel/sched/core.c
@@ -130,6 +130,7 @@ EXPORT_TRACEPOINT_SYMBOL_GPL(sched_dl_server_stop_tp);
  
  DEFINE_PER_CPU_SHARED_ALIGNED(struct rq, runqueues);
  DEFINE_PER_CPU(struct rnd_state, sched_rnd_state);
+DEFINE_PER_CPU(struct sched_scratchmask_pool, sched_scratchmask_irq);
  
  #ifdef CONFIG_SCHED_PROXY_EXEC
  DEFINE_STATIC_KEY_TRUE(__sched_proxy_exec);
@@ -8875,9 +8876,6 @@ void __init sched_init_smp(void)
  	current->flags &= ~PF_NO_SETAFFINITY;
  	sched_init_granularity();
  
-	init_sched_rt_class();
-	init_sched_dl_class();
-
  	sched_init_dl_servers();
  
  	sched_smp_initialized = true;
@@ -8897,6 +8895,18 @@ int in_sched_functions(unsigned long addr)
  		&& addr < (unsigned long)__sched_text_end);
  }
  
+void __init init_scratchmasks(void)
+{
+	struct sched_scratchmask_pool *pool;
+	int cpu, slot;
+
+	for_each_possible_cpu(cpu) {
+		pool = per_cpu_ptr(&sched_scratchmask_irq, cpu);
+		for (slot = 0; slot < SCHED_SCRATCHMASK_IRQ_SLOTS; slot++)
+			zalloc_cpumask_var_node(&pool->mask[slot], GFP_KERNEL, cpu_to_node(cpu));
+	}
+}
+
  #ifdef CONFIG_CGROUP_SCHED
  /*
   * Default task group.
@@ -8946,6 +8956,8 @@ void __init sched_init(void)
  
  #endif /* CONFIG_RT_GROUP_SCHED */
  
+	init_scratchmasks();
+
  	init_defrootdomain();
  
  #ifdef CONFIG_RT_GROUP_SCHED
diff --git a/kernel/sched/deadline.c b/kernel/sched/deadline.c
index 857dbe3519a8..5d64536acfef 100644
--- a/kernel/sched/deadline.c
+++ b/kernel/sched/deadline.c
@@ -2927,13 +2927,11 @@ static struct task_struct *pick_earliest_pushable_dl_task(struct rq *rq, int cpu
  	return NULL;
  }
  
-/* Access rule: must be called on local CPU with preemption disabled */
-static DEFINE_PER_CPU(cpumask_var_t, local_cpu_mask_dl);
-
  static int find_later_rq(struct task_struct *task)
  {
  	struct sched_domain *sd;
-	struct cpumask *later_mask = this_cpu_cpumask_var_ptr(local_cpu_mask_dl);
+	CLASS(sched_scratchmask_irq, scratch)(0);
+	struct cpumask *later_mask = scratch.mask;
  	int this_cpu = smp_processor_id();
  	int cpu = task_cpu(task);
  
@@ -3382,15 +3380,6 @@ static void rq_offline_dl(struct rq *rq)
  	cpudl_clear(&rq->rd->cpudl, rq->cpu, false);
  }
  
-void __init init_sched_dl_class(void)
-{
-	unsigned int i;
-
-	for_each_possible_cpu(i)
-		zalloc_cpumask_var_node(&per_cpu(local_cpu_mask_dl, i),
-					GFP_KERNEL, cpu_to_node(i));
-}
-
  /*
   * This function always returns a non-empty bitmap in @cpus. This is because
   * if a root domain has reserved bandwidth for DL tasks, the DL bandwidth
@@ -3435,7 +3424,8 @@ void dl_add_task_root_domain(struct task_struct *p)
  		return;
  	}
  
-	msk = this_cpu_cpumask_var_ptr(local_cpu_mask_dl);
+	CLASS(sched_scratchmask_irq, scratch)(0);
+	msk = scratch.mask;
  	dl_get_task_effective_cpus(p, msk);
  	cpu = cpumask_first_and(cpu_active_mask, msk);
  	BUG_ON(cpu >= nr_cpu_ids);
diff --git a/kernel/sched/ext/idle.c b/kernel/sched/ext/idle.c
index 6f93cc32b650..056cb9016129 100644
--- a/kernel/sched/ext/idle.c
+++ b/kernel/sched/ext/idle.c
@@ -47,13 +47,6 @@ static struct scx_idle_cpus scx_idle_global_masks;
   */
  static struct scx_idle_cpus **scx_idle_node_masks;
  
-/*
- * Local per-CPU cpumasks (used to generate temporary idle cpumasks).
- */
-static DEFINE_PER_CPU(cpumask_var_t, local_idle_cpumask);
-static DEFINE_PER_CPU(cpumask_var_t, local_llc_idle_cpumask);
-static DEFINE_PER_CPU(cpumask_var_t, local_numa_idle_cpumask);
-
  /*
   * Return the idle masks associated to a target @node.
   *
@@ -468,8 +461,13 @@ s32 scx_select_cpu_dfl(struct task_struct *p, s32 prev_cpu, u64 wake_flags,
  	/*
  	 * Determine the subset of CPUs usable by @p within @cpus_allowed.
  	 */
+
+	CLASS(sched_scratchmask_irq, scratch)(0);
+	CLASS(sched_scratchmask_irq, scratch1)(1);
+	CLASS(sched_scratchmask_irq, scratch2)(2);
+
  	if (allowed != p->cpus_ptr) {
-		struct cpumask *local_cpus = this_cpu_cpumask_var_ptr(local_idle_cpumask);
+		struct cpumask *local_cpus = scratch.mask;
  
  		if (task_affinity_all(p)) {
  			allowed = cpus_allowed;
@@ -500,7 +498,7 @@ s32 scx_select_cpu_dfl(struct task_struct *p, s32 prev_cpu, u64 wake_flags,
  	 * directly.
  	 */
  	if (static_branch_maybe(CONFIG_NUMA, &scx_selcpu_topo_numa)) {
-		struct cpumask *local_cpus = this_cpu_cpumask_var_ptr(local_numa_idle_cpumask);
+		struct cpumask *local_cpus = scratch1.mask;
  		const struct cpumask *cpus = numa_span(prev_cpu);
  
  		if (allowed == p->cpus_ptr && task_affinity_all(p))
@@ -510,7 +508,7 @@ s32 scx_select_cpu_dfl(struct task_struct *p, s32 prev_cpu, u64 wake_flags,
  	}
  
  	if (static_branch_maybe(CONFIG_SCHED_MC, &scx_selcpu_topo_llc)) {
-		struct cpumask *local_cpus = this_cpu_cpumask_var_ptr(local_llc_idle_cpumask);
+		struct cpumask *local_cpus = scratch2.mask;
  		const struct cpumask *cpus = llc_span(prev_cpu);
  
  		if (allowed == p->cpus_ptr && task_affinity_all(p))
@@ -695,16 +693,6 @@ void scx_idle_init_masks(void)
  		BUG_ON(!alloc_cpumask_var_node(&scx_idle_node_masks[i]->cpu, GFP_KERNEL, i));
  		BUG_ON(!alloc_cpumask_var_node(&scx_idle_node_masks[i]->smt, GFP_KERNEL, i));
  	}
-
-	/* Allocate local per-cpu idle cpumasks */
-	for_each_possible_cpu(i) {
-		BUG_ON(!alloc_cpumask_var_node(&per_cpu(local_idle_cpumask, i),
-					       GFP_KERNEL, cpu_to_node(i)));
-		BUG_ON(!alloc_cpumask_var_node(&per_cpu(local_llc_idle_cpumask, i),
-					       GFP_KERNEL, cpu_to_node(i)));
-		BUG_ON(!alloc_cpumask_var_node(&per_cpu(local_numa_idle_cpumask, i),
-					       GFP_KERNEL, cpu_to_node(i)));
-	}
  }
  
  static void update_builtin_idle(int cpu, bool idle)
diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
index 9a975a684b48..770363a59b0c 100644
--- a/kernel/sched/fair.c
+++ b/kernel/sched/fair.c
@@ -8190,11 +8190,6 @@ static inline unsigned int cfs_h_nr_delayed(struct rq *rq)
  	return (rq->cfs.h_nr_queued - rq->cfs.h_nr_runnable);
  }
  
-/* Working cpumask for: sched_balance_rq(), sched_balance_newidle(). */
-static DEFINE_PER_CPU(cpumask_var_t, load_balance_mask);
-static DEFINE_PER_CPU(cpumask_var_t, select_rq_mask);
-static DEFINE_PER_CPU(cpumask_var_t, should_we_balance_tmpmask);
-
  #ifdef CONFIG_NO_HZ_COMMON
  
  static struct {
@@ -8665,7 +8660,8 @@ static int select_idle_smt(struct task_struct *p, struct sched_domain *sd, int t
   */
  static int select_idle_cpu(struct task_struct *p, struct sched_domain *sd, bool has_idle_core, int target)
  {
-	struct cpumask *cpus = this_cpu_cpumask_var_ptr(select_rq_mask);
+	CLASS(sched_scratchmask_irq, scratch)(0);
+	struct cpumask *cpus = scratch.mask;
  	int i, cpu, idle_cpu = -1, nr = INT_MAX;
  
  	if (sched_feat(SIS_UTIL) && sd->shared) {
@@ -8800,7 +8796,8 @@ select_idle_capacity(struct task_struct *p, struct sched_domain *sd, int target)
  	struct cpumask *cpus;
  	int nr = INT_MAX;
  
-	cpus = this_cpu_cpumask_var_ptr(select_rq_mask);
+	CLASS(sched_scratchmask_irq, scratch)(0);
+	cpus = scratch.mask;
  	cpumask_and(cpus, sched_domain_span(sd), p->cpus_ptr);
  
  	task_util = task_util_est(p);
@@ -8948,9 +8945,6 @@ static int select_idle_sibling(struct task_struct *p, int prev, int target)
  		util_max = uclamp_eff_value(p, UCLAMP_MAX);
  	}
  
-	/*
-	 * per-cpu select_rq_mask usage
-	 */
  	lockdep_assert_irqs_disabled();
  
  	if (choose_idle_cpu(target, p) &&
@@ -9487,7 +9481,8 @@ compute_energy(struct energy_env *eenv, struct perf_domain *pd,
   */
  static int find_energy_efficient_cpu(struct task_struct *p, int prev_cpu)
  {
-	struct cpumask *cpus = this_cpu_cpumask_var_ptr(select_rq_mask);
+	CLASS(sched_scratchmask_irq, scratch)(0);
+	struct cpumask *cpus = scratch.mask;
  	unsigned long prev_delta = ULONG_MAX, best_delta = ULONG_MAX;
  	unsigned long p_util_min = uclamp_is_used() ? uclamp_eff_value(p, UCLAMP_MIN) : 0;
  	unsigned long p_util_max = uclamp_is_used() ? uclamp_eff_value(p, UCLAMP_MAX) : 1024;
@@ -13271,7 +13266,9 @@ static int active_load_balance_cpu_stop(void *data);
  
  static int should_we_balance(struct lb_env *env)
  {
-	struct cpumask *swb_cpus = this_cpu_cpumask_var_ptr(should_we_balance_tmpmask);
+	/* 0 is used already in sched_balance_rq */
+	CLASS(sched_scratchmask_irq, scratch1)(1);
+	struct cpumask *swb_cpus = scratch1.mask;
  	struct sched_group *sg = env->sd->groups;
  	int cpu, idle_smt = -1;
  
@@ -13387,7 +13384,8 @@ static int sched_balance_rq(int this_cpu, struct rq *this_rq,
  	struct sched_group *group;
  	struct rq *busiest;
  	struct rq_flags rf;
-	struct cpumask *cpus = this_cpu_cpumask_var_ptr(load_balance_mask);
+	CLASS(sched_scratchmask_irq, scratch)(0);
+	struct cpumask *cpus = scratch.mask;
  	struct lb_env env = {
  		.sd		= sd,
  		.dst_cpu	= this_cpu,
@@ -13967,13 +13965,9 @@ static inline int find_new_ilb(void)
  	struct cpumask *ilb_cpus;
  	int ilb_cpu, fallback = -1;
  
-	lockdep_assert_irqs_disabled();
+	CLASS(sched_scratchmask_irq, scratch)(0);
+	ilb_cpus = scratch.mask;
  
-	/*
-	 * Reuse the per-CPU select_rq_mask, which is protected from concurrent
-	 * use on this CPU by having interrupts disabled.
-	 */
-	ilb_cpus = this_cpu_cpumask_var_ptr(select_rq_mask);
  	cpumask_and(ilb_cpus, nohz.idle_cpus_mask,
  		    housekeeping_cpumask(HK_TYPE_KERNEL_NOISE));
  
@@ -15574,18 +15568,12 @@ __init void init_sched_fair_class(void)
  {
  	int i;
  
-	for_each_possible_cpu(i) {
-		zalloc_cpumask_var_node(&per_cpu(load_balance_mask, i), GFP_KERNEL, cpu_to_node(i));
-		zalloc_cpumask_var_node(&per_cpu(select_rq_mask,    i), GFP_KERNEL, cpu_to_node(i));
-		zalloc_cpumask_var_node(&per_cpu(should_we_balance_tmpmask, i),
-					GFP_KERNEL, cpu_to_node(i));
-
  #ifdef CONFIG_CFS_BANDWIDTH
+	for_each_possible_cpu(i) {
  		INIT_CSD(&cpu_rq(i)->cfsb_csd, __cfsb_csd_unthrottle, cpu_rq(i));
  		INIT_LIST_HEAD(&cpu_rq(i)->cfsb_csd_list);
-#endif
  	}
-
+#endif
  	open_softirq(SCHED_SOFTIRQ, sched_balance_softirq);
  
  #ifdef CONFIG_NO_HZ_COMMON
diff --git a/kernel/sched/rt.c b/kernel/sched/rt.c
index e6e5f8a2caaf..95b2b20a661d 100644
--- a/kernel/sched/rt.c
+++ b/kernel/sched/rt.c
@@ -1770,12 +1770,11 @@ static struct task_struct *pick_highest_pushable_task(struct rq *rq, int cpu)
  	return NULL;
  }
  
-static DEFINE_PER_CPU(cpumask_var_t, local_cpu_mask);
-
  static int find_lowest_rq(struct task_struct *task)
  {
  	struct sched_domain *sd;
-	struct cpumask *lowest_mask = this_cpu_cpumask_var_ptr(local_cpu_mask);
+	CLASS(sched_scratchmask_irq, scratch)(0);
+	struct cpumask *lowest_mask = scratch.mask;
  	int this_cpu = smp_processor_id();
  	int cpu      = task_cpu(task);
  	int ret;
@@ -2425,16 +2424,6 @@ static void switched_from_rt(struct rq *rq, struct task_struct *p)
  	rt_queue_pull_task(rq);
  }
  
-void __init init_sched_rt_class(void)
-{
-	unsigned int i;
-
-	for_each_possible_cpu(i) {
-		zalloc_cpumask_var_node(&per_cpu(local_cpu_mask, i),
-					GFP_KERNEL, cpu_to_node(i));
-	}
-}
-
  /*
   * When switching a task to RT, we may overload the runqueue
   * with RT tasks. In this case we try to push them off to
diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h
index 26ae13c86b69..afa44303b0da 100644
--- a/kernel/sched/sched.h
+++ b/kernel/sched/sched.h
@@ -1064,6 +1064,7 @@ struct root_domain {
  };
  
  extern void init_defrootdomain(void);
+extern void init_scratchmasks(void);
  extern int sched_init_domains(const struct cpumask *cpu_map);
  extern void rq_attach_root(struct rq *rq, struct root_domain *rd);
  extern void sched_get_rd(struct root_domain *rd);
@@ -2968,8 +2969,6 @@ extern void sysrq_sched_debug_show(void);
  extern void sched_init_granularity(void);
  extern void update_max_interval(void);
  
-extern void init_sched_dl_class(void);
-extern void init_sched_rt_class(void);
  extern void init_sched_fair_class(void);
  
  extern void resched_curr(struct rq *rq);
@@ -4230,4 +4229,68 @@ DEFINE_CLASS_IS_UNCONDITIONAL(sched_change)
  
  #include "ext/ext.h"
  
+#define SCHED_SCRATCHMASK_IRQ_SLOTS	3
+
+struct sched_scratchmask_pool {
+	cpumask_var_t mask[SCHED_SCRATCHMASK_IRQ_SLOTS];
+	unsigned long in_use;	/* bitwise slot usage */
+};
+
+DECLARE_PER_CPU(struct sched_scratchmask_pool, sched_scratchmask_irq);
+
+static __always_inline struct cpumask *
+sched_get_scratchmask_irq(unsigned int slot)
+{
+	struct sched_scratchmask_pool *pool = this_cpu_ptr(&sched_scratchmask_irq);
+
+	lockdep_assert_irqs_disabled();
+
+	if (WARN_ON_ONCE(slot >= SCHED_SCRATCHMASK_IRQ_SLOTS))
+		return NULL;
+
+	if (WARN_ON_ONCE(test_and_set_bit(slot, &pool->in_use)))
+		return NULL;
+
+	return pool->mask[slot];
+}
+
+static __always_inline void
+sched_put_scratchmask_irq(unsigned int slot)
+{
+	struct sched_scratchmask_pool *pool = this_cpu_ptr(&sched_scratchmask_irq);
+
+	lockdep_assert_irqs_disabled();
+
+	if (WARN_ON_ONCE(slot >= SCHED_SCRATCHMASK_IRQ_SLOTS))
+		return;
+
+	WARN_ON_ONCE(!test_and_clear_bit(slot, &pool->in_use));
+}
+
+struct sched_scratchmask {
+	struct cpumask	*mask;
+	unsigned int	slot;
+};
+
+static __always_inline struct sched_scratchmask
+sched_scratchmask_acquire_irq(unsigned int slot)
+{
+	return (struct sched_scratchmask) {
+		.mask = sched_get_scratchmask_irq(slot),
+		.slot = slot,
+	};
+}
+
+static __always_inline void
+sched_scratchmask_release_irq(struct sched_scratchmask *scratch)
+{
+	if (scratch->mask)
+		sched_put_scratchmask_irq(scratch->slot);
+}
+
+DEFINE_CLASS(sched_scratchmask_irq, struct sched_scratchmask,
+	     sched_scratchmask_release_irq(&_T),
+	     sched_scratchmask_acquire_irq(_slot),
+	     unsigned int _slot);
+
  #endif /* _KERNEL_SCHED_SCHED_H */
-- 
2.47.3

[tip: sched/core] sched/fair: Prefer fully idle cores for NOHZ balancing

[tip-bot2 for Andrea Righi]

The following commit has been merged into the sched/core branch of tip:

Commit-ID:     293f9611ae73564febc553935830074f0f300694
Gitweb:        https://git.kernel.org/tip/293f9611ae73564febc553935830074f0f300694
Author:        Andrea Righi <arighi@nvidia.com>
AuthorDate:    Tue, 04 Aug 2026 17:13:24 +02:00
Committer:     Peter Zijlstra <peterz@infradead.org>
CommitterDate: Fri, 07 Aug 2026 18:27:09 +02:00

sched/fair: Prefer fully idle cores for NOHZ balancing

find_new_ilb() selects the first idle housekeeping CPU without
considering whether another thread is running on the same physical core.
On an SMT system, the idle load balancer can therefore activate both
siblings even when another housekeeping CPU has an entirely idle core.

On most SMT systems, this is not problematic because the idle load
balancer is a short-lived activity and the transient wakeup of a sibling
has negligible performance impact.

However, this can be particularly costly on NVIDIA Olympus cores used in
Vera. Briefly activating an otherwise idle sibling can reduce the
performance available to the other sibling and this effect does not
necessarily end once the activated sibling becomes idle: after the ILB
finishes and its CPU enters WFI, full single-thread performance is
restored only after the sibling has remained idle for a qualification
interval (10 Ki cycles on the tested Vera system). Repeated short
sibling wakeups can therefore sustain the interference even with little
actual overlap.

Prevent this by preferring an idle housekeeping CPU whose entire SMT
core is idle. Retain the first idle CPU as a fallback when no fully idle
core is available, so NOHZ balancing continues to make forward progress.
Once a partially busy core has been examined, skip its remaining SMT
siblings to avoid repeating the core-idle check on wide SMT systems.

Tests performed using an ad hoc GEMM benchmark running one CPU-intensive
task per SMT core within its CPU affinity mask improved from
approximately 6.2 TFLOP/s to 9.4 TFLOP/s.

Note that this preference may wake a fully idle physical core instead of
using an idle sibling of an active core, potentially increasing ILB
wakeup latency or energy consumption on some architectures. It may also
scan additional CPUs before selecting the one to run the ILB. The
selection falls back to the first idle CPU when no fully idle SMT core
is available. Non-SMT systems continue to select the first idle
housekeeping CPU.

Signed-off-by: Andrea Righi <arighi@nvidia.com>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Reviewed-by: Mete Durlu <meted@linux.ibm.com>
Reviewed-by: Vincent Guittot <vincent.guittot@linaro.org>
Link: https://patch.msgid.link/20260804151324.918020-1-arighi@nvidia.com
---
 kernel/sched/fair.c | 55 +++++++++++++++++++++++++++++++++++---------
 1 file changed, 44 insertions(+), 11 deletions(-)

diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
index df8c9c2..a24dd20 100644
--- a/kernel/sched/fair.c
+++ b/kernel/sched/fair.c
@@ -13962,29 +13962,62 @@ static inline int on_null_domain(struct rq *rq)
  */
 static inline int find_new_ilb(void)
 {
-	int this_cpu = smp_processor_id();
-	const struct cpumask *hk_mask;
-	int ilb_cpu;
+	struct cpumask *ilb_cpus;
+	int ilb_cpu, fallback = -1;
+
+	lockdep_assert_irqs_disabled();
+
+	/*
+	 * Reuse the per-CPU select_rq_mask, which is protected from concurrent
+	 * use on this CPU by having interrupts disabled.
+	 */
+	ilb_cpus = this_cpu_cpumask_var_ptr(select_rq_mask);
+	cpumask_and(ilb_cpus, nohz.idle_cpus_mask,
+		    housekeeping_cpumask(HK_TYPE_KERNEL_NOISE));
+
+	for_each_cpu(ilb_cpu, ilb_cpus) {
+		if (!idle_cpu(ilb_cpu)) {
+			/*
+			 * Once an idle fallback exists, a busy CPU proves that
+			 * this core cannot be fully idle. Skip its siblings.
+			 */
+			if (sched_smt_active() && fallback >= 0)
+				cpumask_andnot(ilb_cpus, ilb_cpus, cpu_smt_mask(ilb_cpu));
+			continue;
+		}
 
-	hk_mask = housekeeping_cpumask(HK_TYPE_KERNEL_NOISE);
+		/*
+		 * Running the idle load balancer on an idle sibling of a busy
+		 * SMT core can reduce the capacity available to its sibling. Prefer
+		 * a CPU whose entire core is idle, but retain the first idle CPU as
+		 * a fallback so idle balancing can still make progress when no fully
+		 * idle core exists.
+		 */
+		if (sched_smt_active() && !is_core_idle(ilb_cpu)) {
+			if (fallback < 0)
+				fallback = ilb_cpu;
 
-	for_each_cpu_and(ilb_cpu, nohz.idle_cpus_mask, hk_mask) {
-		if (ilb_cpu == this_cpu)
+			/*
+			 * The core is not idle, so there is no need to check
+			 * any of its other SMT siblings.
+			 */
+			cpumask_andnot(ilb_cpus, ilb_cpus,
+				       cpu_smt_mask(ilb_cpu));
 			continue;
+		}
 
-		if (idle_cpu(ilb_cpu))
-			return ilb_cpu;
+		return ilb_cpu;
 	}
 
-	return -1;
+	return fallback;
 }
 
 /*
  * Kick a CPU to do the NOHZ balancing, if it is time for it, via a cross-CPU
  * SMP function call (IPI).
  *
- * We pick the first idle CPU in the HK_TYPE_KERNEL_NOISE housekeeping set
- * (if there is one).
+ * Prefer a CPU on a fully idle core in the HK_TYPE_KERNEL_NOISE housekeeping
+ * set. Fall back to the first idle CPU when no fully idle core exists.
  */
 static void kick_ilb(unsigned int flags)
 {