Re: Linux 6.10-rc2 - massive performance regression

Re: Linux 6.10-rc2 - massive performance regression

[Linux regression tracking (Thorsten Leemhuis)]

On 09.06.24 00:00, Linus Torvalds wrote:
> On Sat, 8 Jun 2024 at 14:36, David Laight <David.Laight@aculab.com> wrote:
> [...]
>> I've done some tests.
>> I'm seeing a three-fold slow down on:
>> $ i=0; while [ $i -lt 1000000 ]; do i=$((i + 1)); done
>> which goes from 1 second to 3.
>>
>> I can run that with ftrace monitoring scheduler events (and a few
>> other things) and can't spot anywhere the process isn't running
>> for a significant time.
> 
> Sounds like cpu frequency. Almost certainly hw-specific. I went
> through that on my Threadripper in the 6.9 timeframe, but I'm not
> seeing any issues in this current release.

David, what kind of hardware do you use? Johan Hovold as
man-in-the-middle just reported "CPU frequency of the big cores on the
Lenovo ThinkPad X13s sometimes appears to get stuck at a low frequency
with 6.10-rc2" and confirmed "that once the cores are fully throttled
(using the stepwise thermal governor) due to the skin temperature
reaching the
first trip point, scaling_max_freq gets stuck at the next OPP".

For more details, see:
https://lore.kernel.org/all/ZmVfcEOxmjUHZTSX@hovoldconsulting.com/

> If you bisect it, we have somebody to blame and point fingers at...

Sadly that one is not bisected yet either. :-/

Ciao, Thorsten

RE: Linux 6.10-rc2 - massive performance regression

[David Laight]

From: Linux regression tracking (Thorsten Leemhuis)
> Sent: 09 June 2024 09:11
> 
> On 09.06.24 00:00, Linus Torvalds wrote:
> > On Sat, 8 Jun 2024 at 14:36, David Laight <David.Laight@aculab.com> wrote:
> > [...]
> >> I've done some tests.
> >> I'm seeing a three-fold slow down on:
> >> $ i=0; while [ $i -lt 1000000 ]; do i=$((i + 1)); done
> >> which goes from 1 second to 3.
> >>
> >> I can run that with ftrace monitoring scheduler events (and a few
> >> other things) and can't spot anywhere the process isn't running
> >> for a significant time.
> >
> > Sounds like cpu frequency. Almost certainly hw-specific. I went
> > through that on my Threadripper in the 6.9 timeframe, but I'm not
> > seeing any issues in this current release.
> 
> David, what kind of hardware do you use?

This is on an 17-7700 (4 cores + hyperthreading enabled = 8 cpu).

> Johan Hovold as
> man-in-the-middle just reported "CPU frequency of the big cores on the
> Lenovo ThinkPad X13s sometimes appears to get stuck at a low frequency
> with 6.10-rc2" and confirmed "that once the cores are fully throttled
> (using the stepwise thermal governor) due to the skin temperature
> reaching the
> first trip point, scaling_max_freq gets stuck at the next OPP".

That's not what I'm seeing.
I can turn the speed up and down by stopping/starting a daemon we use
for processing audio.
(I can give anyone a copy; it is freely downloadable from the company
web site - if you know exactly where to look!)
Basically that ends up running a bit of code on every cpu every 10ms.

There is a big difference in the number of sched_migrate_task traces
between 6.9 and 6.10 (15 v 83).

I suspect that the underlying problem is that the cpu governor
doesn't allow for a 'busy' process being moved to an idle cpu?
So if you bounce a process about it always runs an 800MHz.

My dmesg (6.9 and 6.10) has:
cpuidle: using governor idle
cpuidle: using governor ladder

But I had a feeling that some 'hardware magic' changes the cpu
speed on these systems?

	David

-
Registered Address Lakeside, Bramley Road, Mount Farm, Milton Keynes, MK1 1PT, UK
Registration No: 1397386 (Wales)