[PATCH] sched/nohz: Fix NOHZ imbalance by adding options for ILB CPU

[PATCH] sched/nohz: Fix NOHZ imbalance by adding options for ILB CPU

[Adam Li]

A qualified CPU to run NOHZ idle load balancing (ILB) has to be:
1) housekeeping CPU in housekeeping_cpumask(HK_TYPE_KERNEL_NOISE)
2) and in nohz.idle_cpus_mask
3) and idle
4) and not current CPU

If most CPUs are in nohz_full CPU list there is few housekeeping CPU left.
In the worst case if all CPUs are in nohz_full only the boot CPU is used
for housekeeping. And the housekeeping CPU is usually busier so it will
be unlikely added to nohz.idle_cpus_mask.

Therefore if there is few housekeeping CPUs, find_new_ilb() may likely
failed to find any CPU to do NOHZ idle load balancing. Some NOHZ CPUs may
stay idle while other CPUs are busy.

This patch adds fallback options when looking for ILB CPU if there is
no CPU meeting above requirements. Then it searches in bellow order:
1) Try looking for the first idle housekeeping CPU
2) Try looking for the first idle CPU in nohz.idle_cpus_mask if no SMT.
3) Select the first housekeeping CPU even if it is busy.

With this patch the NOHZ idle balancing happens more frequently.

Signed-off-by: Adam Li <adamli@os.amperecomputing.com>
---
 kernel/sched/fair.c | 32 +++++++++++++++++++++++++++++---
 1 file changed, 29 insertions(+), 3 deletions(-)

diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
index b173a059315c..12bcc3f81f9b 100644
--- a/kernel/sched/fair.c
+++ b/kernel/sched/fair.c
@@ -12194,19 +12194,45 @@ static inline int on_null_domain(struct rq *rq)
 static inline int find_new_ilb(void)
 {
 	const struct cpumask *hk_mask;
-	int ilb_cpu;
+	struct cpumask ilb_mask;
+	int ilb_cpu, this_cpu = smp_processor_id();
 
 	hk_mask = housekeeping_cpumask(HK_TYPE_KERNEL_NOISE);
 
-	for_each_cpu_and(ilb_cpu, nohz.idle_cpus_mask, hk_mask) {
+	/*
+	 * Look for an idle cpu who is both NOHZ_idle and housekeeping.
+	 * If no such cpu, look for an idle housekeeping cpu.
+	 */
+	if (!cpumask_and(&ilb_mask, nohz.idle_cpus_mask, hk_mask))
+		cpumask_copy(&ilb_mask, hk_mask);
 
-		if (ilb_cpu == smp_processor_id())
+	for_each_cpu(ilb_cpu, &ilb_mask) {
+		if (ilb_cpu == this_cpu)
 			continue;
 
 		if (idle_cpu(ilb_cpu))
 			return ilb_cpu;
 	}
 
+	/*
+	 * If CPU has no SMT, look for an idle NOHZ_idle cpu.
+	 * Run NOHZ ILB may cause jitter on SMT sibling CPU.
+	 */
+	if (!sched_smt_active()) {
+		for_each_cpu(ilb_cpu, nohz.idle_cpus_mask) {
+			if (ilb_cpu == this_cpu)
+				continue;
+
+			if (idle_cpu(ilb_cpu))
+				return ilb_cpu;
+		}
+	}
+
+	/* Select the first housekeeping cpu anyway. */
+	ilb_cpu = cpumask_first(hk_mask);
+	if (ilb_cpu < nr_cpu_ids)
+		return ilb_cpu;
+
 	return -1;
 }
 
-- 
2.34.1

Re: [PATCH] sched/nohz: Fix NOHZ imbalance by adding options for ILB CPU

[Valentin Schneider]

On 19/08/25 02:57, Adam Li wrote:
> A qualified CPU to run NOHZ idle load balancing (ILB) has to be:
> 1) housekeeping CPU in housekeeping_cpumask(HK_TYPE_KERNEL_NOISE)
> 2) and in nohz.idle_cpus_mask
> 3) and idle
> 4) and not current CPU
>
> If most CPUs are in nohz_full CPU list there is few housekeeping CPU left.
> In the worst case if all CPUs are in nohz_full only the boot CPU is used
> for housekeeping. And the housekeeping CPU is usually busier so it will
> be unlikely added to nohz.idle_cpus_mask.
>
> Therefore if there is few housekeeping CPUs, find_new_ilb() may likely
> failed to find any CPU to do NOHZ idle load balancing. Some NOHZ CPUs may
> stay idle while other CPUs are busy.
>
> This patch adds fallback options when looking for ILB CPU if there is
> no CPU meeting above requirements. Then it searches in bellow order:
> 1) Try looking for the first idle housekeeping CPU
> 2) Try looking for the first idle CPU in nohz.idle_cpus_mask if no SMT.
> 3) Select the first housekeeping CPU even if it is busy.
>
> With this patch the NOHZ idle balancing happens more frequently.
>

I'm not understanding why, in the scenarios outlined above, more NOHZ idle
balancing is a good thing.

Considering only housekeeping CPUs, they're all covered by wakeup, periodic
and idle balancing (on top of NOHZ idle balancing when relevant). So if
find_new_ilb() never finds a NOHZ-idle CPU, then that means your HK CPUs
are either always busy or never stopping the tick when going idle, IOW they
always have some work to do within a jiffy boundary.

Am I missing something?

Re: [PATCH] sched/nohz: Fix NOHZ imbalance by adding options for ILB CPU

[Adam Li]

Hi Valentin,

Thanks for your comments.
On 8/19/2025 10:00 PM, Valentin Schneider wrote:
> On 19/08/25 02:57, Adam Li wrote:
>> A qualified CPU to run NOHZ idle load balancing (ILB) has to be:
>> 1) housekeeping CPU in housekeeping_cpumask(HK_TYPE_KERNEL_NOISE)
>> 2) and in nohz.idle_cpus_mask
>> 3) and idle
>> 4) and not current CPU
>>
>> If most CPUs are in nohz_full CPU list there is few housekeeping CPU left.
>> In the worst case if all CPUs are in nohz_full only the boot CPU is used
>> for housekeeping. And the housekeeping CPU is usually busier so it will
>> be unlikely added to nohz.idle_cpus_mask.
>>
>> Therefore if there is few housekeeping CPUs, find_new_ilb() may likely
>> failed to find any CPU to do NOHZ idle load balancing. Some NOHZ CPUs may
>> stay idle while other CPUs are busy.
>>
>> This patch adds fallback options when looking for ILB CPU if there is
>> no CPU meeting above requirements. Then it searches in bellow order:
>> 1) Try looking for the first idle housekeeping CPU
>> 2) Try looking for the first idle CPU in nohz.idle_cpus_mask if no SMT.
>> 3) Select the first housekeeping CPU even if it is busy.
>>
>> With this patch the NOHZ idle balancing happens more frequently.
>>
> 
> I'm not understanding why, in the scenarios outlined above, more NOHZ idle
> balancing is a good thing.
> 
> Considering only housekeeping CPUs, they're all covered by wakeup, periodic
> and idle balancing (on top of NOHZ idle balancing when relevant). So if
> find_new_ilb() never finds a NOHZ-idle CPU, then that means your HK CPUs
> are either always busy or never stopping the tick when going idle, IOW they
> always have some work to do within a jiffy boundary.
> > Am I missing something?
>

I agree with your description about the housekeeping CPUs. In the worst case,
the system only has one housekeeping CPU and this housekeeping CPU is so busy
that:
1) This housekeeping CPU is unlikely idle;
2) and this housekeeping CPU is unlikely in 'nohz.idle_cpus_mask' because tick
is not stopped.
Therefore find_new_ilb() may very likely return -1. *No* CPU can be selected
to do NOHZ idle load balancing.

This patch tries to fix the imbalance of NOHZ idle CPUs (CPUs in nohz.idle_cpus_mask).
Here is more background:

When running llama on arm64 server, some CPUs *keep* idle while others
are 100% busy. All CPUs are in 'nohz_full=' cpu list, and CONFIG_NO_HZ_FULL
is set.

The problem is caused by two issues:
1) Some idle CPUs cannot be added to 'nohz.idle_cpus_mask',
this bug is fixed by another patch:
https://lore.kernel.org/all/20250815065115.289337-2-adamli@os.amperecomputing.com/

2) Even if the idle CPUs are in 'nohz.idle_cpus_mask', *no* CPU can be selected to
do NOHZ idle load balancing because conditions in find_new_ilb() is too strict.
This patch tries to solve this issue.

Hope this information helps.

Thanks,
-adam

 

Re: [PATCH] sched/nohz: Fix NOHZ imbalance by adding options for ILB CPU

[Valentin Schneider]

On 20/08/25 11:35, Adam Li wrote:
> On 8/19/2025 10:00 PM, Valentin Schneider wrote:
>>
>> I'm not understanding why, in the scenarios outlined above, more NOHZ idle
>> balancing is a good thing.
>>
>> Considering only housekeeping CPUs, they're all covered by wakeup, periodic
>> and idle balancing (on top of NOHZ idle balancing when relevant). So if
>> find_new_ilb() never finds a NOHZ-idle CPU, then that means your HK CPUs
>> are either always busy or never stopping the tick when going idle, IOW they
>> always have some work to do within a jiffy boundary.
>> > Am I missing something?
>>
>
> I agree with your description about the housekeeping CPUs. In the worst case,
> the system only has one housekeeping CPU and this housekeeping CPU is so busy
> that:
> 1) This housekeeping CPU is unlikely idle;
> 2) and this housekeeping CPU is unlikely in 'nohz.idle_cpus_mask' because tick
> is not stopped.
> Therefore find_new_ilb() may very likely return -1. *No* CPU can be selected
> to do NOHZ idle load balancing.
>
> This patch tries to fix the imbalance of NOHZ idle CPUs (CPUs in nohz.idle_cpus_mask).
> Here is more background:
>
> When running llama on arm64 server, some CPUs *keep* idle while others
> are 100% busy. All CPUs are in 'nohz_full=' cpu list, and CONFIG_NO_HZ_FULL
> is set.
>

I assume you mean all but one CPU is in 'nohz_full=' since you need at
least one housekeeping CPU. But in that case this becomes a slightly
different problem, since no CPU in 'nohz_full' will be in

  housekeeping_cpumask(HK_TYPE_KERNEL_NOISE)

> The problem is caused by two issues:
> 1) Some idle CPUs cannot be added to 'nohz.idle_cpus_mask',
> this bug is fixed by another patch:
> https://lore.kernel.org/all/20250815065115.289337-2-adamli@os.amperecomputing.com/
>
> 2) Even if the idle CPUs are in 'nohz.idle_cpus_mask', *no* CPU can be selected to
> do NOHZ idle load balancing because conditions in find_new_ilb() is too strict.
> This patch tries to solve this issue.
>
> Hope this information helps.
>

I hadn't seen that patch; that cclist is quite small, you'll want to add
the scheduler people to our next submission.

So IIUC:
- Pretty much all your CPUs are NOHZ_FULL
- When they go idle they remain so for a while despite work being available

My first question would be: is NOHZ_FULL really right for your workload?
It's mainly designed to be used with always-running userspace tasks,
generally affined to a CPU by the system administrator.
Here AIUI you're relying on the scheduler load balancing to distribute work
to the NOHZ_FULL CPUs, so you're going to be penalized a lot by the
NOHZ_FULL context switch overheads. What's the point? Wouldn't you have
less overhead with just NOHZ_IDLE?

As for the actual balancing, yeah if you have idle NOHZ_FULL CPUs they
won't do the periodic balance; the residual 1Hz remote tick doesn't do that
either. But they should still do the newidle balance to pull work before
going tickless idle, and wakeup balance should help as well, albeit that
also depends on your topology.

Could you share your system topology and your actual nohz_full cmdline?

> Thanks,
> -adam
>
>

Re: [PATCH] sched/nohz: Fix NOHZ imbalance by adding options for ILB CPU

[Adam Li]

On 8/20/2025 4:43 PM, Valentin Schneider wrote:
> On 20/08/25 11:35, Adam Li wrote:
>> On 8/19/2025 10:00 PM, Valentin Schneider wrote:
>>>
>>> I'm not understanding why, in the scenarios outlined above, more NOHZ idle
>>> balancing is a good thing.
>>>
>>> Considering only housekeeping CPUs, they're all covered by wakeup, periodic
>>> and idle balancing (on top of NOHZ idle balancing when relevant). So if
>>> find_new_ilb() never finds a NOHZ-idle CPU, then that means your HK CPUs
>>> are either always busy or never stopping the tick when going idle, IOW they
>>> always have some work to do within a jiffy boundary.
>>>> Am I missing something?
>>>
>>
>> I agree with your description about the housekeeping CPUs. In the worst case,
>> the system only has one housekeeping CPU and this housekeeping CPU is so busy
>> that:
>> 1) This housekeeping CPU is unlikely idle;
>> 2) and this housekeeping CPU is unlikely in 'nohz.idle_cpus_mask' because tick
>> is not stopped.
>> Therefore find_new_ilb() may very likely return -1. *No* CPU can be selected
>> to do NOHZ idle load balancing.
>>
>> This patch tries to fix the imbalance of NOHZ idle CPUs (CPUs in nohz.idle_cpus_mask).
>> Here is more background:
>>
>> When running llama on arm64 server, some CPUs *keep* idle while others
>> are 100% busy. All CPUs are in 'nohz_full=' cpu list, and CONFIG_NO_HZ_FULL
>> is set.
>>
> 
> I assume you mean all but one CPU is in 'nohz_full=' since you need at
> least one housekeeping CPU. But in that case this becomes a slightly
> different problem, since no CPU in 'nohz_full' will be in
> 
>   housekeeping_cpumask(HK_TYPE_KERNEL_NOISE)
> 

I ran llama workload on a system with 192 CPUs. I set "nohz_full=0-191" so all CPUs
are in 'nohz_full' list. In this case, kernel uses the boot CPU for housekeeping:

Kernel message: "Housekeeping: must include one present CPU, using boot CPU:0"

find_new_ilb() looks for qualified CPU from housekeeping CPUs. The searching
is likely to fail if there is only one housekeeping CPU.

>> The problem is caused by two issues:
>> 1) Some idle CPUs cannot be added to 'nohz.idle_cpus_mask',
>> this bug is fixed by another patch:
>> https://lore.kernel.org/all/20250815065115.289337-2-adamli@os.amperecomputing.com/
>>
>> 2) Even if the idle CPUs are in 'nohz.idle_cpus_mask', *no* CPU can be selected to
>> do NOHZ idle load balancing because conditions in find_new_ilb() is too strict.
>> This patch tries to solve this issue.
>>
>> Hope this information helps.
>>
> 
> I hadn't seen that patch; that cclist is quite small, you'll want to add
> the scheduler people to our next submission.
> 

Sure. The first patch involves both 'tick' and 'scheduler' subsystem. I can resend
the first patch to broader reviewers if you don't mind.

> So IIUC:
> - Pretty much all your CPUs are NOHZ_FULL
> - When they go idle they remain so for a while despite work being available
> 

Exactly.

> My first question would be: is NOHZ_FULL really right for your workload?
> It's mainly designed to be used with always-running userspace tasks,
> generally affined to a CPU by the system administrator.
> Here AIUI you're relying on the scheduler load balancing to distribute work
> to the NOHZ_FULL CPUs, so you're going to be penalized a lot by the
> NOHZ_FULL context switch overheads. What's the point? Wouldn't you have
> less overhead with just NOHZ_IDLE?
> 

I ran the llama workload to do 'Large Language Model' reference.
The workload creates 'always-running userspace' threads doing math computing.
There is *few* sleep, wakeup and context switch. IIUC NOHZ_IDLE cannot help
always-running task? 

'nohz_full' option is supposed to benefit performance by reducing kernel
noise I think. Could you please give more detail on
'NOHZ_FULL context switch overhead'?
  
> As for the actual balancing, yeah if you have idle NOHZ_FULL CPUs they
> won't do the periodic balance; the residual 1Hz remote tick doesn't do that
> either. But they should still do the newidle balance to pull work before
> going tickless idle, and wakeup balance should help as well, albeit that
> also depends on your topology.
>

I think the newidle balance and wakeup balance do not help in this case
because the workload has few sleep and wakeup.
 
> Could you share your system topology and your actual nohz_full cmdline?
>

The system has 192 CPUs. I set "nohz_full=0-191".

Thanks,
-adam

Re: [PATCH] sched/nohz: Fix NOHZ imbalance by adding options for ILB CPU

[Valentin Schneider]

On 20/08/25 19:05, Adam Li wrote:
> On 8/20/2025 4:43 PM, Valentin Schneider wrote:
>> On 20/08/25 11:35, Adam Li wrote:
>>> I agree with your description about the housekeeping CPUs. In the worst case,
>>> the system only has one housekeeping CPU and this housekeeping CPU is so busy
>>> that:
>>> 1) This housekeeping CPU is unlikely idle;
>>> 2) and this housekeeping CPU is unlikely in 'nohz.idle_cpus_mask' because tick
>>> is not stopped.
>>> Therefore find_new_ilb() may very likely return -1. *No* CPU can be selected
>>> to do NOHZ idle load balancing.
>>>
>>> This patch tries to fix the imbalance of NOHZ idle CPUs (CPUs in nohz.idle_cpus_mask).
>>> Here is more background:
>>>
>>> When running llama on arm64 server, some CPUs *keep* idle while others
>>> are 100% busy. All CPUs are in 'nohz_full=' cpu list, and CONFIG_NO_HZ_FULL
>>> is set.
>>>
>>
>> I assume you mean all but one CPU is in 'nohz_full=' since you need at
>> least one housekeeping CPU. But in that case this becomes a slightly
>> different problem, since no CPU in 'nohz_full' will be in
>>
>>   housekeeping_cpumask(HK_TYPE_KERNEL_NOISE)
>>
>
> I ran llama workload on a system with 192 CPUs. I set "nohz_full=0-191" so all CPUs
> are in 'nohz_full' list. In this case, kernel uses the boot CPU for housekeeping:
>
> Kernel message: "Housekeeping: must include one present CPU, using boot CPU:0"
>
> find_new_ilb() looks for qualified CPU from housekeeping CPUs. The searching
> is likely to fail if there is only one housekeeping CPU.
>

Right

>>> The problem is caused by two issues:
>>> 1) Some idle CPUs cannot be added to 'nohz.idle_cpus_mask',
>>> this bug is fixed by another patch:
>>> https://lore.kernel.org/all/20250815065115.289337-2-adamli@os.amperecomputing.com/
>>>
>>> 2) Even if the idle CPUs are in 'nohz.idle_cpus_mask', *no* CPU can be selected to
>>> do NOHZ idle load balancing because conditions in find_new_ilb() is too strict.
>>> This patch tries to solve this issue.
>>>
>>> Hope this information helps.
>>>
>>
>> I hadn't seen that patch; that cclist is quite small, you'll want to add
>> the scheduler people to our next submission.
>>
>
> Sure. The first patch involves both 'tick' and 'scheduler' subsystem. I can resend
> the first patch to broader reviewers if you don't mind.
>

I'd say resend the whole series with the right folks cc'd.

>> So IIUC:
>> - Pretty much all your CPUs are NOHZ_FULL
>> - When they go idle they remain so for a while despite work being available
>>
>
> Exactly.
>
>> My first question would be: is NOHZ_FULL really right for your workload?
>> It's mainly designed to be used with always-running userspace tasks,
>> generally affined to a CPU by the system administrator.
>> Here AIUI you're relying on the scheduler load balancing to distribute work
>> to the NOHZ_FULL CPUs, so you're going to be penalized a lot by the
>> NOHZ_FULL context switch overheads. What's the point? Wouldn't you have
>> less overhead with just NOHZ_IDLE?
>>
>
> I ran the llama workload to do 'Large Language Model' reference.
> The workload creates 'always-running userspace' threads doing math computing.
> There is *few* sleep, wakeup and context switch. IIUC NOHZ_IDLE cannot help
> always-running task?
>

Right, NOHZ_IDLE is really about power savings while a CPU is idle (and
IIRC it helps some virtualization cases).

> 'nohz_full' option is supposed to benefit performance by reducing kernel
> noise I think. Could you please give more detail on
> 'NOHZ_FULL context switch overhead'?
>

The doc briefly touches on that:

  https://docs.kernel.org/timers/no_hz.html#omit-scheduling-clock-ticks-for-cpus-with-only-one-runnable-task

The longer story is have a look at kernel/context_tracking.c; every
transition into and out of the kernel to and from user or idle requires
additional atomic operations and synchronization.

It would be worth for you to quantify how much these processes
sleep/context switch, it could be that keep the tick enabled incurs a lower
throughput penalty than the NO_HZ_FULL overheads.

>> As for the actual balancing, yeah if you have idle NOHZ_FULL CPUs they
>> won't do the periodic balance; the residual 1Hz remote tick doesn't do that
>> either. But they should still do the newidle balance to pull work before
>> going tickless idle, and wakeup balance should help as well, albeit that
>> also depends on your topology.
>>
>
> I think the newidle balance and wakeup balance do not help in this case
> because the workload has few sleep and wakeup.
>

Right. So other than the NO_HZ_FULL vs NO_HZ_IDLE considerations above, you
could manually affine the threads of the workload. Depending on how much
control you have over how many threads it spawn, you could either pin on
thread per CPU, or just spawn the workload into a cpuset covering the
NO_HZ_FULL CPUs.

Having the scheduler do the balancing is bit of a precarious
situation. Your single housekeeping CPU is pretty much going to be always
running things, does it make sense to have it run the NOHZ idle balance
when there are available idle NOHZ_FULL CPUs? And in the same sense, does
it make sense to disturb an idle NOHZ_FULL CPU to get it to spread load on
other NOHZ_FULL CPUs? Admins that manually affine their threads will
probably say no.

9b019acb72e4 ("sched/nohz: Run NOHZ idle load balancer on HK_FLAG_MISC CPUs")
also mentions SMT being an issue.

>> Could you share your system topology and your actual nohz_full cmdline?
>>
>
> The system has 192 CPUs. I set "nohz_full=0-191".
>
> Thanks,
> -adam

Re: [PATCH] sched/nohz: Fix NOHZ imbalance by adding options for ILB CPU

[Adam Li]

On 8/20/2025 7:46 PM, Valentin Schneider wrote:
> 
> I'd say resend the whole series with the right folks cc'd.
>
OK. I resent the patch series.
Please refer to: https://lore.kernel.org/all/20250821042707.62993-1-adamli@os.amperecomputing.com/

>> 'nohz_full' option is supposed to benefit performance by reducing kernel
>> noise I think. Could you please give more detail on
>> 'NOHZ_FULL context switch overhead'?
>>
> 
> The doc briefly touches on that:
> 
>   https://docs.kernel.org/timers/no_hz.html#omit-scheduling-clock-ticks-for-cpus-with-only-one-runnable-task
> 
> The longer story is have a look at kernel/context_tracking.c; every
> transition into and out of the kernel to and from user or idle requires
> additional atomic operations and synchronization.
> 
> It would be worth for you to quantify how much these processes
> sleep/context switch, it could be that keep the tick enabled incurs a lower
> throughput penalty than the NO_HZ_FULL overheads.
> 

Thanks for the information.

>>> As for the actual balancing, yeah if you have idle NOHZ_FULL CPUs they
>>> won't do the periodic balance; the residual 1Hz remote tick doesn't do that
>>> either. But they should still do the newidle balance to pull work before
>>> going tickless idle, and wakeup balance should help as well, albeit that
>>> also depends on your topology.
>>>
>>
>> I think the newidle balance and wakeup balance do not help in this case
>> because the workload has few sleep and wakeup.
>>
> 
> Right. So other than the NO_HZ_FULL vs NO_HZ_IDLE considerations above, you
> could manually affine the threads of the workload. Depending on how much
> control you have over how many threads it spawn, you could either pin on
> thread per CPU, or just spawn the workload into a cpuset covering the
> NO_HZ_FULL CPUs.
> 

Yes, binding the threads to CPU can work around the performance
issue caused by load imbalance. Should we document that 'nohz_full' may cause
the scheduler load balancing not working well and CPU affinity is preferred?

> Having the scheduler do the balancing is bit of a precarious
> situation. Your single housekeeping CPU is pretty much going to be always
> running things, does it make sense to have it run the NOHZ idle balance
> when there are available idle NOHZ_FULL CPUs? And in the same sense, does
> it make sense to disturb an idle NOHZ_FULL CPU to get it to spread load on
> other NOHZ_FULL CPUs? Admins that manually affine their threads will
> probably say no.
> 

I think when the NOHZ_FULL CPU is added to nohz.idle_cpus_mask and
its tick is stopped, the CPU is 'very' idle. We can safely assign some work to it.

> 9b019acb72e4 ("sched/nohz: Run NOHZ idle load balancer on HK_FLAG_MISC CPUs")
> also mentions SMT being an issue.
> 

From the commit message of 9b019acb72e4:
"The problem was observed with increased jitter on an application
running on CPU0, caused by NOHZ idle load balancing being run on
CPU1 (an SMT sibling)."

Can we say if *no* SMT, it is safe to run NOHZ idle load balancing
on CPU in nohz.idle_cpus_mask? My patch checks '!sched_smt_active()' when
searching from nohz.idle_cpus_mask.

Thanks,
-adam

Re: [PATCH] sched/nohz: Fix NOHZ imbalance by adding options for ILB CPU

[Valentin Schneider]

On 21/08/25 19:18, Adam Li wrote:
> On 8/20/2025 7:46 PM, Valentin Schneider wrote:
>> Right. So other than the NO_HZ_FULL vs NO_HZ_IDLE considerations above, you
>> could manually affine the threads of the workload. Depending on how much
>> control you have over how many threads it spawn, you could either pin on
>> thread per CPU, or just spawn the workload into a cpuset covering the
>> NO_HZ_FULL CPUs.
>>
>
> Yes, binding the threads to CPU can work around the performance
> issue caused by load imbalance. Should we document that 'nohz_full' may cause
> the scheduler load balancing not working well and CPU affinity is preferred?
>

Yeah I guess we could highlight that.

I think it's kind of a gray area; technically we could change load
balancing to make NO_HZ_FULL CPUs better at pulling tasks, but that only
works up to the point where, if you have N NO_HZ_FULL CPUs, you have pulled
N tasks. So there is an underlying assumption that the workload threading
matches your NO_HZ_FULL topology; and if that's the case, you might as well
affine the tasks by hand and avoid any surprises.

Put in another way: yes we can probably make load balancing better
for NO_HZ_FULL CPUs, but that only really works if we have one task to pull
per NO_HZ_FULL CPU, in which case manual affinity binding works just as
well, and I prefer that approach since it means we don't have to add a
NO_HZ_FULL load balancing logic which may end up interfering with
NO_HZ_FULL itself. At least, that is my opinion.

>> Having the scheduler do the balancing is bit of a precarious
>> situation. Your single housekeeping CPU is pretty much going to be always
>> running things, does it make sense to have it run the NOHZ idle balance
>> when there are available idle NOHZ_FULL CPUs? And in the same sense, does
>> it make sense to disturb an idle NOHZ_FULL CPU to get it to spread load on
>> other NOHZ_FULL CPUs? Admins that manually affine their threads will
>> probably say no.
>>
>
> I think when the NOHZ_FULL CPU is added to nohz.idle_cpus_mask and
> its tick is stopped, the CPU is 'very' idle. We can safely assign some work to it.
>
>> 9b019acb72e4 ("sched/nohz: Run NOHZ idle load balancer on HK_FLAG_MISC CPUs")
>> also mentions SMT being an issue.
>>
>
> From the commit message of 9b019acb72e4:
> "The problem was observed with increased jitter on an application
> running on CPU0, caused by NOHZ idle load balancing being run on
> CPU1 (an SMT sibling)."
>
> Can we say if *no* SMT, it is safe to run NOHZ idle load balancing
> on CPU in nohz.idle_cpus_mask? My patch checks '!sched_smt_active()' when
> searching from nohz.idle_cpus_mask.
>

I suppose we could still make this work for SMT with e.g. is_core_idle(),
but see my point above.

> Thanks,
> -adam

Re: [PATCH] sched/nohz: Fix NOHZ imbalance by adding options for ILB CPU

[Christoph Lameter (Ampere)]

On Thu, 28 Aug 2025, Valentin Schneider wrote:

> > Yes, binding the threads to CPU can work around the performance
> > issue caused by load imbalance. Should we document that 'nohz_full' may cause
> > the scheduler load balancing not working well and CPU affinity is preferred?
> >
>
> Yeah I guess we could highlight that.

We need to make sure that the idle cpus are used when available and
needed. Otherwise the scheduler is buggy.

Such a load balancing action means that there is a cpu that is running
multiple processes. Therefore the timer interrrupt and the scheduler
processing is active on at least one cpu. We can therefore do something
about the situation.

The scheduler needs to move one of the processes onto the idle cpu.

Re: [PATCH] sched/nohz: Fix NOHZ imbalance by adding options for ILB CPU

[Valentin Schneider]

On 28/08/25 08:44, Christoph Lameter (Ampere) wrote:
> On Thu, 28 Aug 2025, Valentin Schneider wrote:
>
>> > Yes, binding the threads to CPU can work around the performance
>> > issue caused by load imbalance. Should we document that 'nohz_full' may cause
>> > the scheduler load balancing not working well and CPU affinity is preferred?
>> >
>>
>> Yeah I guess we could highlight that.
>
> We need to make sure that the idle cpus are used when available and
> needed. Otherwise the scheduler is buggy.
>
> Such a load balancing action means that there is a cpu that is running
> multiple processes. Therefore the timer interrrupt and the scheduler
> processing is active on at least one cpu. We can therefore do something
> about the situation.
>
> The scheduler needs to move one of the processes onto the idle cpu.

AFAICT we have (at least) two options:
1) Trigger NOHZ balancing on a busy housekeeping CPU (what this patch does)

   This is somewhat against idle load balancing rules (only spend CPU time
   on that if there is no "genuine" work to run), but I guess from a CPU
   isolation PoV this can be tallied as just another housekeeping activity

2) Trigger NOHZ balancing on an idle NOHZ_FULL CPU

   That doesn't steal useful CPU time, but that also potentially causes
   interference, albeit only if racing with the NOHZ_FULL workload spawning
   (which shouldn't be the steady state).

The more I think about it the more I'm leaning towards 1), but I'd like
other folks' opinion.

Re: [PATCH] sched/nohz: Fix NOHZ imbalance by adding options for ILB CPU

[Vincent Guittot]

On Wed, 3 Sept 2025 at 14:35, Valentin Schneider <vschneid@redhat.com> wrote:
>
> On 28/08/25 08:44, Christoph Lameter (Ampere) wrote:
> > On Thu, 28 Aug 2025, Valentin Schneider wrote:
> >
> >> > Yes, binding the threads to CPU can work around the performance
> >> > issue caused by load imbalance. Should we document that 'nohz_full' may cause
> >> > the scheduler load balancing not working well and CPU affinity is preferred?
> >> >
> >>
> >> Yeah I guess we could highlight that.
> >
> > We need to make sure that the idle cpus are used when available and
> > needed. Otherwise the scheduler is buggy.
> >
> > Such a load balancing action means that there is a cpu that is running
> > multiple processes. Therefore the timer interrrupt and the scheduler
> > processing is active on at least one cpu. We can therefore do something
> > about the situation.
> >
> > The scheduler needs to move one of the processes onto the idle cpu.
>
> AFAICT we have (at least) two options:
> 1) Trigger NOHZ balancing on a busy housekeeping CPU (what this patch does)
>
>    This is somewhat against idle load balancing rules (only spend CPU time
>    on that if there is no "genuine" work to run), but I guess from a CPU
>    isolation PoV this can be tallied as just another housekeeping activity

In this case, this should only be done for full nohz case and not for
other cases because the ILB overhead is not negligible on a busy cpu
and I don't see anything that enable 1) only for full no hz

>
> 2) Trigger NOHZ balancing on an idle NOHZ_FULL CPU

this patch also does 2) for no smt case

I wonder why this happens only for no smt case ?   If the sibling is
used by another thread with full nohz, it already interferes with this
one

But we might want to do is_core_idle() instead

>
>    That doesn't steal useful CPU time, but that also potentially causes
>    interference, albeit only if racing with the NOHZ_FULL workload spawning
>    (which shouldn't be the steady state).
>
> The more I think about it the more I'm leaning towards 1), but I'd like
> other folks' opinion.
>

Re: [PATCH] sched/nohz: Fix NOHZ imbalance by adding options for ILB CPU

[Christoph Lameter (Ampere)]

On Wed, 3 Sep 2025, Vincent Guittot wrote:

> > AFAICT we have (at least) two options:
> > 1) Trigger NOHZ balancing on a busy housekeeping CPU (what this patch does)


Isnt there a third option?

3) Trigger load balancing if on a NOHZ_FULL cpu and multiple processes are
running on it.

Re: [PATCH] sched/nohz: Fix NOHZ imbalance by adding options for ILB CPU

[Vincent Guittot]

On Wed, 3 Sept 2025 at 22:44, Christoph Lameter (Ampere) <cl@gentwo.org> wrote:
>
> On Wed, 3 Sep 2025, Vincent Guittot wrote:
>
> > > AFAICT we have (at least) two options:
> > > 1) Trigger NOHZ balancing on a busy housekeeping CPU (what this patch does)
>
>
> Isnt there a third option?
>
> 3) Trigger load balancing if on a NOHZ_FULL cpu and multiple processes are
> running on it.

If there are several threads on the NOHZ_FULL CPu then the CPU has its
tick firing which will trigger the busy load balance for this CPU

>
>

Re: [PATCH] sched/nohz: Fix NOHZ imbalance by adding options for ILB CPU

[Christoph Lameter (Ampere)]

On Wed, 20 Aug 2025, Valentin Schneider wrote:

> My first question would be: is NOHZ_FULL really right for your workload?

Yes performance is improved. AI workloads are like HPC workloads in that
they need to do compute and then rendezvous for data exchange. Variations
in the runtime due to timer ticks cause idle periods where the rendezvous
cannot be completed because some cpus are delayed.

The more frequent rendezvous can be performed the better the performance
numbers will be.

> It's mainly designed to be used with always-running userspace
tasks, > generally affined to a CPU by the system administrator.

hohz full has been reworked somewhat since the early days and works in a
more general way today.

> Here AIUI you're relying on the scheduler load balancing to distribute work
> to the NOHZ_FULL CPUs, so you're going to be penalized a lot by the
> NOHZ_FULL context switch overheads. What's the point? Wouldn't you have
> less overhead with just NOHZ_IDLE?

The benchmarks show a regression of 10-20% if the tick is operational.
The context switch overhead is negligible since the cpus are doing compute
and not system calls.

> As for the actual balancing, yeah if you have idle NOHZ_FULL CPUs they
> won't do the periodic balance; the residual 1Hz remote tick doesn't do that
> either. But they should still do the newidle balance to pull work before
> going tickless idle, and wakeup balance should help as well, albeit that
> also depends on your topology.

That should work in general and not depend on any hardware topology. In
this case we have a linear sched domain including all processors.

Re: [PATCH] sched/nohz: Fix NOHZ imbalance by adding options for ILB CPU

[Valentin Schneider]

On 20/08/25 10:31, Christoph Lameter (Ampere) wrote:
> On Wed, 20 Aug 2025, Valentin Schneider wrote:
>
>> My first question would be: is NOHZ_FULL really right for your workload?
>
> Yes performance is improved. AI workloads are like HPC workloads in that
> they need to do compute and then rendezvous for data exchange. Variations
> in the runtime due to timer ticks cause idle periods where the rendezvous
> cannot be completed because some cpus are delayed.
>
> The more frequent rendezvous can be performed the better the performance
> numbers will be.

[...]

> The benchmarks show a regression of 10-20% if the tick is operational.
> The context switch overhead is negligible since the cpus are doing compute
> and not system calls.
>

Ah good, that's useful information, thanks!

>> As for the actual balancing, yeah if you have idle NOHZ_FULL CPUs they
>> won't do the periodic balance; the residual 1Hz remote tick doesn't do that
>> either. But they should still do the newidle balance to pull work before
>> going tickless idle, and wakeup balance should help as well, albeit that
>> also depends on your topology.
>
> That should work in general and not depend on any hardware topology. In
> this case we have a linear sched domain including all processors.

Wakeup balance not so much, select_idle_sibling() won't move tasks outside
of the waker's LLC - but AIUI in your case you have just a single node and
I'm assuming one big LLC.

Re: [PATCH] sched/nohz: Fix NOHZ imbalance by adding options for ILB CPU

[Frederic Weisbecker]

Le Wed, Aug 20, 2025 at 10:31:24AM -0700, Christoph Lameter (Ampere) a écrit :
> On Wed, 20 Aug 2025, Valentin Schneider wrote:
> 
> > My first question would be: is NOHZ_FULL really right for your workload?
> 
> Yes performance is improved. AI workloads are like HPC workloads in that
> they need to do compute and then rendezvous for data exchange.

Ok, I was about to say that this is the first known (for me) usecase of
nohz_full that is about performance and doesn't strictly require low-latency
guarantee. But...

> Variations
> in the runtime due to timer ticks cause idle periods where the rendezvous
> cannot be completed because some cpus are delayed.
> 
> The more frequent rendezvous can be performed the better the performance
> numbers will be.

...that is low-latency requirement...for performance :-)

That's an argument _not_ in favour of dynamic balancing such as ILB, even for
this usecase in nohz_full (all the other usecases of nohz_full I know really
want static affinity and no balancing at all).

So I have to ask, what would be wrong with static affinities to these tasks?

> 
> > It's mainly designed to be used with always-running userspace
> tasks, > generally affined to a CPU by the system administrator.
> 
> hohz full has been reworked somewhat since the early days and works in a
> more general way today.

Not sure about that. Although it was not initially intended to, it has
been very single purpose since the early days: ie: run a single task in
userspace without being disturbed.

> > Here AIUI you're relying on the scheduler load balancing to distribute work
> > to the NOHZ_FULL CPUs, so you're going to be penalized a lot by the
> > NOHZ_FULL context switch overheads. What's the point? Wouldn't you have
> > less overhead with just NOHZ_IDLE?
> 
> The benchmarks show a regression of 10-20% if the tick is operational.

Impressive!

> The context switch overhead is negligible since the cpus are doing compute
> and not system calls.

And not many syscalls, right?

Thanks.

-- 
Frederic Weisbecker
SUSE Labs

Re: [PATCH] sched/nohz: Fix NOHZ imbalance by adding options for ILB CPU

[Christoph Lameter (Ampere)]

[-- Attachment #1: Type: text/plain, Size: 2474 bytes --]

On Thu, 4 Sep 2025, Frederic Weisbecker wrote:

> Le Wed, Aug 20, 2025 at 10:31:24AM -0700, Christoph Lameter (Ampere) a écrit :
> > On Wed, 20 Aug 2025, Valentin Schneider wrote:
> >
> > > My first question would be: is NOHZ_FULL really right for your workload?
> >
> > Yes performance is improved. AI workloads are like HPC workloads in that
> > they need to do compute and then rendezvous for data exchange.
>
> Ok, I was about to say that this is the first known (for me) usecase of
> nohz_full that is about performance and doesn't strictly require low-latency
> guarantee. But...

For me it was always about both. Low latency is required for a high number
of compute cycles in HPC apps. It is a requiremen for high performance
parallelized compute.

> > The more frequent rendezvous can be performed the better the performance
> > numbers will be.
>
> ...that is low-latency requirement...for performance :-)

Yea thats why we want this in HPC/HFT and AI applications.

> That's an argument _not_ in favour of dynamic balancing such as ILB, even for
> this usecase in nohz_full (all the other usecases of nohz_full I know really
> want static affinity and no balancing at all).
>
> So I have to ask, what would be wrong with static affinities to these tasks?

Static affinities are great but they keep the tick active and thus the
rendevous can be off off one or the other compute thread.

> > hohz full has been reworked somewhat since the early days and works in a
> > more general way today.
>
> Not sure about that. Although it was not initially intended to, it has
> been very single purpose since the early days: ie: run a single task in
> userspace without being disturbed.

The restrictions have been reduced from what I see in the code and
syscalls are possible without incurring a 2 second penalty of ticks.


> > > Here AIUI you're relying on the scheduler load balancing to distribute work
> > > to the NOHZ_FULL CPUs, so you're going to be penalized a lot by the
> > > NOHZ_FULL context switch overheads. What's the point? Wouldn't you have
> > > less overhead with just NOHZ_IDLE?
> >
> > The benchmarks show a regression of 10-20% if the tick is operational.
>
> Impressive!
>
> > The context switch overhead is negligible since the cpus are doing compute
> > and not system calls.
>
> And not many syscalls, right?

Periodically the data needs to be saved but that can be done from special
threads or after a large number of compute cycles is complete.

Re: [PATCH] sched/nohz: Fix NOHZ imbalance by adding options for ILB CPU

[Frederic Weisbecker]

Le Thu, Sep 04, 2025 at 08:34:34AM -0700, Christoph Lameter (Ampere) a écrit :
> On Thu, 4 Sep 2025, Frederic Weisbecker wrote:
> 
> > Le Wed, Aug 20, 2025 at 10:31:24AM -0700, Christoph Lameter (Ampere) a écrit :
> > > On Wed, 20 Aug 2025, Valentin Schneider wrote:
> > >
> > > > My first question would be: is NOHZ_FULL really right for your workload?
> > >
> > > Yes performance is improved. AI workloads are like HPC workloads in that
> > > they need to do compute and then rendezvous for data exchange.
> >
> > Ok, I was about to say that this is the first known (for me) usecase of
> > nohz_full that is about performance and doesn't strictly require low-latency
> > guarantee. But...
> 
> For me it was always about both. Low latency is required for a high number
> of compute cycles in HPC apps. It is a requiremen for high performance
> parallelized compute.

Right, it's just that until now I was used to workloads that would even
be broken if the occasional jitter reached some threshold, which doesn't
appear to be your case.

> 
> > > The more frequent rendezvous can be performed the better the performance
> > > numbers will be.
> >
> > ...that is low-latency requirement...for performance :-)
> 
> Yea thats why we want this in HPC/HFT and AI applications.

Ok.

> > That's an argument _not_ in favour of dynamic balancing such as ILB, even for
> > this usecase in nohz_full (all the other usecases of nohz_full I know really
> > want static affinity and no balancing at all).
> >
> > So I have to ask, what would be wrong with static affinities to these tasks?
> 
> Static affinities are great but they keep the tick active and thus the
> rendevous can be off off one or the other compute thread.

How do static affinities keep the tick active?

> 
> > > hohz full has been reworked somewhat since the early days and works in a
> > > more general way today.
> >
> > Not sure about that. Although it was not initially intended to, it has
> > been very single purpose since the early days: ie: run a single task in
> > userspace without being disturbed.
> 
> The restrictions have been reduced from what I see in the code and
> syscalls are possible without incurring a 2 second penalty of ticks.

Yes the isolation has been improved overall but the basic constraints remain.

> > > > Here AIUI you're relying on the scheduler load balancing to distribute work
> > > > to the NOHZ_FULL CPUs, so you're going to be penalized a lot by the
> > > > NOHZ_FULL context switch overheads. What's the point? Wouldn't you have
> > > > less overhead with just NOHZ_IDLE?
> > >
> > > The benchmarks show a regression of 10-20% if the tick is operational.
> >
> > Impressive!
> >
> > > The context switch overhead is negligible since the cpus are doing compute
> > > and not system calls.
> >
> > And not many syscalls, right?
> 
> Periodically the data needs to be saved but that can be done from special
> threads or after a large number of compute cycles is complete.

Got it!

Thanks.

-- 
Frederic Weisbecker
SUSE Labs

Re: [PATCH] sched/nohz: Fix NOHZ imbalance by adding options for ILB CPU

[Christoph Lameter (Ampere)]

On Thu, 4 Sep 2025, Frederic Weisbecker wrote:

> > > That's an argument _not_ in favour of dynamic balancing such as ILB, even for
> > > this usecase in nohz_full (all the other usecases of nohz_full I know really
> > > want static affinity and no balancing at all).
> > >
> > > So I have to ask, what would be wrong with static affinities to these tasks?
> >
> > Static affinities are great but they keep the tick active and thus the
> > rendevous can be off off one or the other compute thread.
>
> How do static affinities keep the tick active?

If you dont use hohz_full and only do static affinities then the tick is
active.

My wish for the future would be that nohz_full would be the default and
that the scheduler does correct load balancing regardless of the cpu being
in tick mode or not. It should automatically switch the tick off if a cpu
goes into 100% compute and ideally there would be no performance
regression if the tick is off.

Re: [PATCH] sched/nohz: Fix NOHZ imbalance by adding options for ILB CPU

[Frederic Weisbecker]

Le Thu, Sep 04, 2025 at 09:18:42AM -0700, Christoph Lameter (Ampere) a écrit :
> On Thu, 4 Sep 2025, Frederic Weisbecker wrote:
> 
> > > > That's an argument _not_ in favour of dynamic balancing such as ILB, even for
> > > > this usecase in nohz_full (all the other usecases of nohz_full I know really
> > > > want static affinity and no balancing at all).
> > > >
> > > > So I have to ask, what would be wrong with static affinities to these tasks?
> > >
> > > Static affinities are great but they keep the tick active and thus the
> > > rendevous can be off off one or the other compute thread.
> >
> > How do static affinities keep the tick active?
> 
> If you dont use hohz_full and only do static affinities then the tick is
> active.

If you don't use nohz_full then the tick is active, yes :-)

> My wish for the future would be that nohz_full would be the default and
> that the scheduler does correct load balancing regardless of the cpu being
> in tick mode or not.

Provided we ever manage to run nohz_full without tradeoffs (offloaded 1Hz tick,
RCU user/kernel overhead, cputime accounting overhead, use hrtick when
preemption must be maintained) then yes it would make sense but we are very
far from that. I'm not even sure this will ever be possible.

> It should automatically switch the tick off if a cpu
> goes into 100% compute and ideally there would be no performance
> regression if the tick is off.

Automatically means the kernel has to guess and make conclusion after some
execution time. That doesn't sound right, especially when nohz_full is used
with strong jitter-free guarantee, which has been the case for every usecases
so far.

Thanks.

-- 
Frederic Weisbecker
SUSE Labs

Re: [PATCH] sched/nohz: Fix NOHZ imbalance by adding options for ILB CPU

[Christoph Lameter (Ampere)]

On Fri, 5 Sep 2025, Frederic Weisbecker wrote:

> > My wish for the future would be that nohz_full would be the default and
> > that the scheduler does correct load balancing regardless of the cpu being
> > in tick mode or not.
>
> Provided we ever manage to run nohz_full without tradeoffs (offloaded 1Hz tick,
> RCU user/kernel overhead, cputime accounting overhead, use hrtick when
> preemption must be maintained) then yes it would make sense but we are very
> far from that. I'm not even sure this will ever be possible.

We need a genius to have a bright idea on how to handle these tradeoffs to
avoid manual configurations.