Cache-aware scheduling does not work well with amd big/little cores

Cache-aware scheduling does not work well with amd big/little cores

[Klaus Kusche]

Hello,

I'm running linux on an AMD Ryzen HX 370.
I use Gentoo and compile a lot of things (including the kernel)
with Clang using full LTO, which results in a single LTO process
running for minutes on an otherwise idle processor.

Before Cache-aware scheduling was introduced,
AMD's pstate driver informed the scheduler about big and little cores,
and the scheduler always moved those LTO processes to a big/fast core.

Now, the cache-aware scheduling takes precedence over big/little
scheduling: All big cores form one L3 cache domain,
and all little cores form the other L3 cache domain.

When a process is started on a little core (and LTO is in most cases),
it will no longer be moved to a big core,
even if it runs for minutes and all big cores are idle.

In case of LTO this hurts twice and badly
(slows down LTO not just noticeably, but dramatically:
In total, any compilation with LTO-linking takes much longer
compared to earlier kernels without cache-aware scheduling):

* The little cores run slower (3.3 GHz instead of 5.1 GHz).

* LTO linking is very cache intensive,
and the L3 cache of the little cores is only half the size (8 MB)
of the L3 cache of the big cores (16 MB).


So if a big core is idling, and a process on a little core
is long-running, big/little scheduling should overrule
cache-aware scheduling and move that process to a big core,
even if this moves the process to another cache domain.

Greetings

-- 
Prof. Dr. Klaus Kusche
Privat: Söllmnitz 32 d, D-07554 Gera/Söllmnitz
036695/859909 klaus.kusche@computerix.info https://www.computerix.info
Dienstlich: DHGE Gera, Weg der Freundschaft 4, D-07546 Gera
klaus.kusche@dhge.de https://www.dhge.de

Re: Cache-aware scheduling does not work well with amd big/little cores

[Mario Limonciello]

Add a few others who have worked on CAS.

On 8/29/26 10:42, Klaus Kusche wrote:
> 
> Hello,
> 
> I'm running linux on an AMD Ryzen HX 370.
> I use Gentoo and compile a lot of things (including the kernel)
> with Clang using full LTO, which results in a single LTO process
> running for minutes on an otherwise idle processor.
> 
> Before Cache-aware scheduling was introduced,
> AMD's pstate driver informed the scheduler about big and little cores,
> and the scheduler always moved those LTO processes to a big/fast core.
> 
> Now, the cache-aware scheduling takes precedence over big/little
> scheduling: All big cores form one L3 cache domain,
> and all little cores form the other L3 cache domain.
> 
> When a process is started on a little core (and LTO is in most cases),
> it will no longer be moved to a big core,
> even if it runs for minutes and all big cores are idle.
> 
> In case of LTO this hurts twice and badly
> (slows down LTO not just noticeably, but dramatically:
> In total, any compilation with LTO-linking takes much longer
> compared to earlier kernels without cache-aware scheduling):
> 
> * The little cores run slower (3.3 GHz instead of 5.1 GHz).
> 
> * LTO linking is very cache intensive,
> and the L3 cache of the little cores is only half the size (8 MB)
> of the L3 cache of the big cores (16 MB).
> 
> 
> So if a big core is idling, and a process on a little core
> is long-running, big/little scheduling should overrule
> cache-aware scheduling and move that process to a big core,
> even if this moves the process to another cache domain.
> 
> Greetings
> 

Not so sure if it would help this case but I will note there is a 
debugfs knob you might be able to use called aggr_tolerance (it was 
introduced in c1e7fe5e75ed11fa85368e5a186472afd3858f3a if you want to 
read more about it's background).

Re: Cache-aware scheduling does not work well with amd big/little cores

[Chen, Yu C]

Hi,

On 8/31/2026 9:53 AM, Mario Limonciello wrote:
> Add a few others who have worked on CAS.
> 
> On 8/29/26 10:42, Klaus Kusche wrote:
>>
>> Hello,
>>
>> I'm running linux on an AMD Ryzen HX 370.
>> I use Gentoo and compile a lot of things (including the kernel)
>> with Clang using full LTO, which results in a single LTO process
>> running for minutes on an otherwise idle processor.
>>
>> Before Cache-aware scheduling was introduced,
>> AMD's pstate driver informed the scheduler about big and little cores,
>> and the scheduler always moved those LTO processes to a big/fast core.
>>
>> Now, the cache-aware scheduling takes precedence over big/little
>> scheduling: All big cores form one L3 cache domain,
>> and all little cores form the other L3 cache domain.
>>
>> When a process is started on a little core (and LTO is in most cases),
>> it will no longer be moved to a big core,
>> even if it runs for minutes and all big cores are idle.
>>
>> In case of LTO this hurts twice and badly
>> (slows down LTO not just noticeably, but dramatically:
>> In total, any compilation with LTO-linking takes much longer
>> compared to earlier kernels without cache-aware scheduling):
>>
>> * The little cores run slower (3.3 GHz instead of 5.1 GHz).
>>
>> * LTO linking is very cache intensive,
>> and the L3 cache of the little cores is only half the size (8 MB)
>> of the L3 cache of the big cores (16 MB).
>>
>>
>> So if a big core is idling, and a process on a little core
>> is long-running, big/little scheduling should overrule
>> cache-aware scheduling and move that process to a big core,
>> even if this moves the process to another cache domain.
>>
>> Greetings
>>
> 
> Not so sure if it would help this case but I will note there is a 
> debugfs knob you might be able to use called aggr_tolerance (it was 
> introduced in c1e7fe5e75ed11fa85368e5a186472afd3858f3a if you want to 
> read more about it's background).

Thanks Mario and Kusche for bringing this issue up. Yes, we can tune Cache
Aware scheduling via debugfs or even turn it off. The issue in current code
is that the cache aware scheduling overwrites the strategy of asymmetric 
scheduling,
and Ricardo also found this issue previous and there is a patch to address
that, by honoring CPU capacity during cache aware scheduling, for the 
**misfit**
task. It would be helpful to see if this patch makes things better:
https://lore.kernel.org/lkml/20260825174112.2580942-1-tim.c.chen@linux.intel.com/

thanks,
Chenyu

Re: Cache-aware scheduling does not work well with amd big/little cores

[Klaus Kusche]

Hello,

both patches in combination seem to have the desired effect.

But I just look at a bar graph showing the current load
of each core. 
The graph suggests that long-running CPU-intensive processes 
migrate to fast cores when fast cores become available.
And I have the impression that LTO compilations
finish significantly faster now.

I don't have exact numbers or benchmarks.

P.S.: I'm currently on holiday (for three weeks).
So my responses will be much slower than usual.

Greetings

Prof. Dr. Klaus Kusche
Privat: Söllmnitz 32 d, D-07554 Gera/Söllmnitz
036695/859909 klaus.kusche@computerix.info https://www.computerix.info
Dienstlich: DHGE Gera, Weg der Freundschaft 4, D-07546 Gera
klaus.kusche@dhge.de https://www.dhge.de

On 31/08/2026 04:08, Chen, Yu C wrote:
> Hi,
> 
> On 8/31/2026 9:53 AM, Mario Limonciello wrote:
>> Add a few others who have worked on CAS.
>>
>> On 8/29/26 10:42, Klaus Kusche wrote:
>>>
>>> Hello,
>>>
>>> I'm running linux on an AMD Ryzen HX 370.
>>> I use Gentoo and compile a lot of things (including the kernel)
>>> with Clang using full LTO, which results in a single LTO process
>>> running for minutes on an otherwise idle processor.
>>>
>>> Before Cache-aware scheduling was introduced,
>>> AMD's pstate driver informed the scheduler about big and little cores,
>>> and the scheduler always moved those LTO processes to a big/fast core.
>>>
>>> Now, the cache-aware scheduling takes precedence over big/little
>>> scheduling: All big cores form one L3 cache domain,
>>> and all little cores form the other L3 cache domain.
>>>
>>> When a process is started on a little core (and LTO is in most cases),
>>> it will no longer be moved to a big core,
>>> even if it runs for minutes and all big cores are idle.
>>>
>>> In case of LTO this hurts twice and badly
>>> (slows down LTO not just noticeably, but dramatically:
>>> In total, any compilation with LTO-linking takes much longer
>>> compared to earlier kernels without cache-aware scheduling):
>>>
>>> * The little cores run slower (3.3 GHz instead of 5.1 GHz).
>>>
>>> * LTO linking is very cache intensive,
>>> and the L3 cache of the little cores is only half the size (8 MB)
>>> of the L3 cache of the big cores (16 MB).
>>>
>>>
>>> So if a big core is idling, and a process on a little core
>>> is long-running, big/little scheduling should overrule
>>> cache-aware scheduling and move that process to a big core,
>>> even if this moves the process to another cache domain.
>>>
>>> Greetings
>>>
>>
>> Not so sure if it would help this case but I will note there is a 
>> debugfs knob you might be able to use called aggr_tolerance (it was 
>> introduced in c1e7fe5e75ed11fa85368e5a186472afd3858f3a if you want to 
>> read more about it's background).
> 
> Thanks Mario and Kusche for bringing this issue up. Yes, we can tune Cache
> Aware scheduling via debugfs or even turn it off. The issue in current code
> is that the cache aware scheduling overwrites the strategy of asymmetric 
> scheduling,
> and Ricardo also found this issue previous and there is a patch to address
> that, by honoring CPU capacity during cache aware scheduling, for the 
> **misfit**
> task. It would be helpful to see if this patch makes things better:
> https://lore.kernel.org/lkml/20260825174112.2580942-1-tim.c.chen@linux.intel.com/
> 
> thanks,
> Chenyu

Re: Cache-aware scheduling does not work well with amd big/little cores

[Tim Chen]

On Mon, 2026-08-31 at 13:24 +0200, Klaus Kusche wrote:
> Hello,
> 
> both patches in combination seem to have the desired effect.
> 
> But I just look at a bar graph showing the current load
> of each core. 
> The graph suggests that long-running CPU-intensive processes 
> migrate to fast cores when fast cores become available.
> And I have the impression that LTO compilations
> finish significantly faster now.
> 
> I don't have exact numbers or benchmarks.

Thanks for testing the fix.

If you just apply https://lore.kernel.org/lkml/20260825174112.2580942-1-tim.c.chen@linux.intel.com/,
with default aggr_tolerance, what numbers do you see?

That will be helpful for further tuning.  Thanks.

Tim 

Re: Cache-aware scheduling does not work well with amd big/little cores

[Klaus Kusche]

This would need some more testing efforts,
because all my kernels are built without debugfs
(using an earlier version of Mario's patch),
and without Mario's patch and without debugfs,
I don't get any big/little scheduling at all.
So I'd need to build and test with debugfs
(tomorrow I'm on a mountain tour, perhaps later this week).

Prof. Dr. Klaus Kusche
Privat: Söllmnitz 32 d, D-07554 Gera/Söllmnitz
036695/859909 klaus.kusche@computerix.info https://www.computerix.info
Dienstlich: DHGE Gera, Weg der Freundschaft 4, D-07546 Gera
klaus.kusche@dhge.de https://www.dhge.de

On 31/08/2026 19:29, Tim Chen wrote:
> On Mon, 2026-08-31 at 13:24 +0200, Klaus Kusche wrote:
>> Hello,
>>
>> both patches in combination seem to have the desired effect.
>>
>> But I just look at a bar graph showing the current load
>> of each core. 
>> The graph suggests that long-running CPU-intensive processes 
>> migrate to fast cores when fast cores become available.
>> And I have the impression that LTO compilations
>> finish significantly faster now.
>>
>> I don't have exact numbers or benchmarks.
> 
> Thanks for testing the fix.
> 
> If you just apply https://lore.kernel.org/lkml/20260825174112.2580942-1-tim.c.chen@linux.intel.com/,
> with default aggr_tolerance, what numbers do you see?
> 
> That will be helpful for further tuning.  Thanks.
> 
> Tim 

Re: Cache-aware scheduling does not work well with amd big/little cores

[Mario Limonciello]

On 8/31/26 13:49, Klaus Kusche wrote:
> 
> This would need some more testing efforts,
> because all my kernels are built without debugfs
> (using an earlier version of Mario's patch),
> and without Mario's patch and without debugfs,
> I don't get any big/little scheduling at all.
> So I'd need to build and test with debugfs
> (tomorrow I'm on a mountain tour, perhaps later this week).
> 
The debugfs patch doesn't really add anything tangible.  I'd say tests 
with that and debugfs turned off are just as valid as tests with no 
patch and debugfs turned on.