sched/fair: Only update stats for allowed CPUs when looking for dst group

sched/fair: Only update stats for allowed CPUs when looking for dst group

[Erwan Velu]

Please backport commit 82d6e01a0699800efd8b048eb584c907ccb47b7a, 
introduced in 6.19 as all stable kernels are affected by a significant 
performance issue.


On systems with large NUMA domains like recent AMD processors, all 
current longterm kernels are affected by a scheduling issue that 
degrades CPU computing performance.

I've been reproducing the issue on 3 different generation of processors, 
and study the impact by applying this commit on top of the 6.18.38, 
latest longterm kernel at date.

With a very quick and automated test, I can see up to a 20% performance 
difference on the Zen5c machines but if you look at the numbers you'll 
see that the

coefficient of variation (CoV) is also greatly impacted.

The CoV is useful to study of how evenly the workers performed and 
you'll see that all systems are affected demonstrating how unfair the 
scheduling was between the cores.

You'll see that this patch can divide the CoV up to a 10x factor.

Please note this issue is not tied to AMD systems but I used them to 
show up the impact of this bug on NUMA-aware configuration.

Kudos to Adam contributing this patch.

Kudos to Colin adding the individual workers metrics and obviously for 
Stress-ng.


Please find below the details of my experiments:

The following results were generated with
- stress-ng 21.03 with commit 6aeb52c7421897e6156a225f2d00d268bf14721c 
that introduce per-worker metrics
- hwbench : https://github.com/criteo/hwbench with the following job 
configuration:
     [global]
     runtime=30
     monitor=all
     engine=stressng
     selected_cpus=numa-simple
     selected_cpus_scaling=iterate
     stressor_range=auto
     skip_method=wait

     [cpu]
     engine_module=cpu
     engine_module_parameter=int128

Columns (all values taken at this scaling step = 384 workers):
   Trace     = trace logical name; '*' marks the reference (first trace).
               'baseline' results are computed with the 6.18.38 Linux Kernel
               'patched'  results are computed with the 6.18.38 Linux 
Kernel + commit 82d6e01a0699800efd8b048eb584c907ccb47b7a
   Wrk       = worker count for this section.
   Perf      = benchmark performance (unit shown per section); Δperf = 
ratio to the reference.
   Perf/core = performance per physical CPU core (physical cores, not 
SMT threads or workers);
               Δperf/core = ratio to the reference.
   Power     = mean CPU.package power in Watts; Δpower = ratio to the 
reference.
   Perf/W    = performance per watt, as a ratio to the reference (Δperf 
/ Δpower).
   Clock     = mean CPU core frequency in MHz; Δclock = ratio to the 
reference.
   IPC       = mean instructions per cycle across cores; ΔIPC = ratio to 
the reference.
   CoV       = Coefficient of Variation (std-dev / mean, %) of the 
per-worker performance:
               how evenly the workers performed (lower = more 
homogeneous, 0% = identical);
               from the stress-ng per-worker detail.

All systems are running with the LastLevelCache (LLC) enabled in the 
BIOS to get the most precise NUMA table and map each L3 as a domain.


##################################################### Zen 5c 
########################################################
   (ref) Zen5c-baseline         2x AMD EPYC 9845 160-Core Processor - 
320 cores / 20 NUMA   scaling 32->640 workers
         Zen5c-patched          2x AMD EPYC 9845 160-Core Processor - 
320 cores / 20 NUMA   scaling 32->640 workers

### cpu/int128  [Bogo op/s]
Trace            Wrk     Perf  Δperf  Perf/core  Δperf/core Power  
Δpower  Perf/W  Clock  Δclock   IPC   ΔIPC    CoV
---------------  ---  -------  -----  ---------  ---------- -----  
------  ------  -----  ------  ----  -----  -----
Zen5c-baseline*  384  395.66K  1.00x      1.24K       1.00x  568W  
  1.00x   1.00x  3274M   1.00x  1.64  1.00x  63.4%
Zen5c-patched    384  470.31K  1.19x      1.47K       1.19x  568W  
  1.00x   1.19x  3141M   0.96x  1.31  0.80x   6.8%
#####################################################################################################################

##################################################### Zen 4c 
########################################################
   (ref) Zen4c-baseline         1x AMD EPYC 8534P 64-Core Processor -  
64 cores /  8 NUMA   scaling 16->128 workers
         Zen4c-patched          1x AMD EPYC 8534P 64-Core Processor -  
64 cores /  8 NUMA   scaling 16->128 workers

### cpu/int128  [Bogo op/s]
Trace            Wrk     Perf  Δperf  Perf/core  Δperf/core Power  
Δpower  Perf/W  Clock  Δclock   IPC   ΔIPC    CoV
---------------  ---  -------  -----  ---------  ---------- -----  
------  ------  -----  ------  ----  -----  -----
Zen4c-baseline*   96  107.60K  1.00x      1.68K       1.00x  160W  
  1.00x   1.00x  3094M   1.00x  1.40  1.00x  13.0%
Zen4c-patched     96  111.74K  1.04x      1.75K       1.04x  164W  
  1.02x   1.01x  3095M   1.00x  1.39  0.99x   1.4%
#####################################################################################################################


##################################################### Zen 2 
########################################################
   (ref) Zen2-baseline          1x AMD EPYC 7502P 32-Core Processor -  
32 cores /  8 NUMA   scaling 8->64 workers
         Zen2-patched           1x AMD EPYC 7502P 32-Core Processor -  
32 cores /  8 NUMA   scaling 8->64 workers

### cpu/int128  [Bogo op/s]
Trace           Wrk    Perf  Δperf  Perf/core  Δperf/core  Power Δpower  
Perf/W  Clock  Δclock   IPC   ΔIPC   CoV
--------------  ---  ------  -----  ---------  ----------  ----- ------  
------  -----  ------  ----  -----  ----
Zen2-baseline*   40  40.95K  1.00x      1.28K       1.00x   150W  1.00x  
  1.00x  2841M   1.00x  1.10  1.00x  7.0%
Zen2-patched     40  41.99K  1.03x      1.31K       1.03x   149W  1.00x  
  1.03x  2855M   1.00x  1.07  0.97x  0.1%
#####################################################################################################################

Re: sched/fair: Only update stats for allowed CPUs when looking for dst group

[Greg KH]

On Wed, Jul 15, 2026 at 04:29:09PM +0200, Erwan Velu wrote:
> Please backport commit 82d6e01a0699800efd8b048eb584c907ccb47b7a, introduced
> in 6.19 as all stable kernels are affected by a significant performance
> issue.

Now applied to the 6.18.y tree, thanks.

greg k-h

Re: sched/fair: Only update stats for allowed CPUs when looking for dst group

[Erwan Velu]

On 7/16/26 15:12, Greg KH wrote:
> On Wed, Jul 15, 2026 at 04:29:09PM +0200, Erwan Velu wrote:
>> Please backport commit 82d6e01a0699800efd8b048eb584c907ccb47b7a, introduced
>> in 6.19 as all stable kernels are affected by a significant performance
>> issue.
> Now applied to the 6.18.y tree, thanks.
>
Hey,

Any idea why it is not cherry-picked for 6.12, 6.6, 6.1, 5.15, 5.10  
while the patch applies perfectly ?

Will it be cherry-picked for the -rc2 ?

Thanks,

Erwan,

Re: sched/fair: Only update stats for allowed CPUs when looking for dst group

[Sasha Levin]

On Thu, Jul 16, 2026 at 03:12:00PM +0200, Greg KH wrote:
> On Wed, Jul 15, 2026 at 04:29:09PM +0200, Erwan Velu wrote:
> > Please backport commit 82d6e01a0699800efd8b048eb584c907ccb47b7a, introduced
> > in 6.19 as all stable kernels are affected by a significant performance
> > issue.
>
> Now applied to the 6.18.y tree, thanks.

Queued for 6.12, 6.6, 6.1, 5.15, and 5.10 as well, thanks.

-- 
Thanks,
Sasha

Re: sched/fair: Only update stats for allowed CPUs when looking for dst group

[Erwan Velu]

On 7/17/26 03:37, Sasha Levin wrote:
> [Vous ne recevez pas souvent de courriers de sashal@kernel.org. Découvrez pourquoi ceci est important à https://aka.ms/LearnAboutSenderIdentification ]
>
> On Thu, Jul 16, 2026 at 03:12:00PM +0200, Greg KH wrote:
>> On Wed, Jul 15, 2026 at 04:29:09PM +0200, Erwan Velu wrote:
>>> Please backport commit 82d6e01a0699800efd8b048eb584c907ccb47b7a, introduced
>>> in 6.19 as all stable kernels are affected by a significant performance
>>> issue.
>> Now applied to the 6.18.y tree, thanks.
> Queued for 6.12, 6.6, 6.1, 5.15, and 5.10 as well, thanks.
>
Oh thanks for confirming,

Erwan,