Re: [PATCH v3 3/3] sched/fair: Remove nohz.nr_cpus and use weight of cpumask instead

[Shrikanth Hegde <sshegde@linux.ibm.com>]

Hi Valentin. Thanks for going through.

On 1/9/26 8:14 PM, Valentin Schneider wrote:
> On 07/01/26 12:21, Shrikanth Hegde wrote:
>> nohz.nr_cpus was observed as contended cacheline when running
>> enterprise workload on large systems.
>>
>> Fundamental scalability challenge with nohz.idle_cpus_mask
>> and nohz.nr_cpus is the following:
>>
>>   (1) nohz_balancer_kick() observes (reads) nohz.nr_cpus
>>       (or nohz.idle_cpu_mask) and nohz.has_blocked to  see whether there's
>>       any nohz balancing work to do, in every scheduler tick.
>>
>>   (2) nohz_balance_enter_idle() and nohz_balance_exit_idle()
>>       (through nohz_balancer_kick() via sched_tick()) modify (write)
>>       nohz.nr_cpus (and/or nohz.idle_cpu_mask) and nohz.has_blocked.
>>
> 
> My first reaction on reading the whole changelog was: "but .nr_cpus and
> .idle_cpus_mask are in the same cacheline?!", which as Ingo pointed out
> somewhere down [1] isn't true for CPUMASK_OFFSTACK, so this change
> effectively gets rid of the dirtying of one extra cacheline during idle
> entry/exit.
> 
> [1]: http://lore.kernel.org/r/aS3za7X9BLS5rg65@gmail.com
> 
> I'd suggest adding something like so in this part of the changelog:
> 
> """
> Note that nohz.idle_cpus_mask and nohz.nr_cpus reside in the same
> cacheline, however under CONFIG_CPUMASK_OFFSTACK the backing storage for
> nohz.idle_cpus_mask will be elsewhere. This implies two separate cachelines
> being dirtied upon idle entry / exit.
> """
> 

ok. Will do that. Thanks.

Even for CONFIG_CPUMASK_OFFSTACK=n, usual configuration is like 512/1024/
2048 or higher.

For 64 byte cacheline, 1 cacheline can hold 512 CPUs.
So idle_cpus_mask and rest of nohz fields including nr_cpus will be in different
cacheline.

Even for powerpc(128 byte cacheline), where CONFIG_CPUMASK_OFFSTACK=n,
default is NR_CPUS=2048. that means idle_cpus_mask will take 2 cachelines and rest
of nohz fields will be in third cacheline.

So in most of the cases, this implies dirtying one less cacheline.

data points with CONFIG_CPUMASK_OFFSTACK=y/n
[1]: https://lore.kernel.org/all/fdb378e7-7797-4aeb-a79f-12af4cb1b81a@linux.ibm.com/