linux-6.6.y regression on amd-pstate

linux-6.6.y regression on amd-pstate

[Mario Limonciello]

+ stable
+ regressions
New subject

Great news.

Greg, Sasha,

Can you please pull in these 3 commits specifically to 6.6.y to fix a 
regression that was reported by Morgan in 6.6.y:

commit 12753d71e8c5 ("ACPI: CPPC: Add helper to get the highest 
performance value")
commit ed429c686b79 ("cpufreq: amd-pstate: Enable amd-pstate preferred 
core support")
commit 3d291fe47fe1 ("cpufreq: amd-pstate: fix the highest frequency 
issue which limits performance")

Further details are below.

Thanks!

On 9/5/2024 16:09, Jones, Morgan wrote:
> Mario,
> 
> Confirmed. Thank you for the help! Slightly different refs on my end:
> 
> Remotes:
> 
> next    https://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git (fetch)
> next    https://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git (push)
> origin  git://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git (fetch)
> origin  git://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git (push)
> superm1 https://git.kernel.org/pub/scm/linux/kernel/git/superm1/linux.git/ (fetch)
> superm1 https://git.kernel.org/pub/scm/linux/kernel/git/superm1/linux.git/ (push)
> torvalds        git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git (fetch)
> torvalds        git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git (push)
> 
> Patches:
> 
> git format-patch 12753d71e8c5^..12753d71e8c5
> git format-patch f3a052391822b772b4e27f2594526cf1eb103cab^..f3a052391822b772b4e27f2594526cf1eb103cab
> git format-patch bf202e654bfa57fb8cf9d93d4c6855890b70b9c4^..bf202e654bfa57fb8cf9d93d4c6855890b70b9c4
> 
> Results:
> 
> Linux redact 6.6.48 #1-NixOS SMP PREEMPT_DYNAMIC Tue Jan  1 00:00:00 UTC 1980 x86_64 GNU/Linux
> 
> analyzing CPU 56:
>    driver: amd-pstate-epp
>    CPUs which run at the same hardware frequency: 56
>    CPUs which need to have their frequency coordinated by software: 56
>    maximum transition latency:  Cannot determine or is not supported.
>    hardware limits: 400 MHz - 3.35 GHz
>    available cpufreq governors: performance powersave
>    current policy: frequency should be within 400 MHz and 3.35 GHz.
>                    The governor "performance" may decide which speed to use
>                    within this range.
>    current CPU frequency: Unable to call hardware
>    current CPU frequency: 2.09 GHz (asserted by call to kernel)
>    boost state support:
>      Supported: yes
>      Active: yes
>      AMD PSTATE Highest Performance: 255. Maximum Frequency: 3.35 GHz.
>      AMD PSTATE Nominal Performance: 152. Nominal Frequency: 2.00 GHz.
>      AMD PSTATE Lowest Non-linear Performance: 115. Lowest Non-linear Frequency: 1.51 GHz.
>      AMD PSTATE Lowest Performance: 31. Lowest Frequency: 400 MHz.
> 
> And our builds are back to being fast with `amd_pstate=active amd_prefcore=enable amd_pstate.shared_mem=1`.
> 
> Morgan
> 
> -----Original Message-----
> From: Mario Limonciello <mario.limonciello@amd.com>
> Sent: Thursday, September 5, 2024 8:12 AM
> To: Jones, Morgan <Morgan.Jones@viasat.com>
> Cc: linux-pm@vger.kernel.org; linux-kernel@vger.kernel.org; David Arcari <darcari@redhat.com>; Dhananjay Ugwekar <Dhananjay.Ugwekar@amd.com>; rafael@kernel.org; viresh.kumar@linaro.org; gautham.shenoy@amd.com; perry.yuan@amd.com; skhan@linuxfoundation.org; li.meng@amd.com; ray.huang@amd.com
> Subject: Re: [EXTERNAL] Re: [PATCH v2 2/2] cpufreq/amd-pstate: Fix the scaling_max_freq setting on shared memory CPPC systems
> 
> Hi Morgan,
> 
> Please apply these 3 commits:
> 
> commit 12753d71e8c5 ("ACPI: CPPC: Add helper to get the highest performance value") commit ed429c686b79 ("cpufreq: amd-pstate: Enable amd-pstate preferred core support") commit 3d291fe47fe1 ("cpufreq: amd-pstate: fix the highest frequency issue which limits performance")
> 
> The first two should help your system, the third will prevent introducing a regression on a different one.
> 
> Assuming that works we should ask @stable to pull all 3 in to fix this regression.
> 
> Thanks,
> 
> On 9/4/2024 08:57, Mario Limonciello wrote:
>> Morgan,
>>
>> I was referring specfiically to the version that landed in Linus' tree:
>> https://urldefense.us/v3/__https://git.kernel.org/torvalds/c/8164f7433
>> 264__;!!C5Asm8uRnZQmlRln!aIZEDEbIUKD7OrxN0b0KjoqKYDL2yMkwk4EK7x_oSnyHQ
>> 6MEq7yt6JHjd0TD9DgEYEWDcF58OKL8c7G11bT3dSqL8eM$
>>
>> But yeah it's effectively the same thing.  In any case, it's not the
>> solution.
>>
>> We had some internal discussion and suspect this is due to missing
>> prefcore patches in 6.6 as that feature landed in 6.9.  We'll try to
>> reproduce this on a Rome system and come back with our findings and
>> suggestions what to do.
>>
>> Thanks,
>>
> 

Re: linux-6.6.y regression on amd-pstate

[Greg Kroah-Hartman]

On Thu, Sep 05, 2024 at 04:14:26PM -0500, Mario Limonciello wrote:
> + stable
> + regressions
> New subject
> 
> Great news.
> 
> Greg, Sasha,
> 
> Can you please pull in these 3 commits specifically to 6.6.y to fix a
> regression that was reported by Morgan in 6.6.y:
> 
> commit 12753d71e8c5 ("ACPI: CPPC: Add helper to get the highest performance
> value")

This is fine, but:

> commit ed429c686b79 ("cpufreq: amd-pstate: Enable amd-pstate preferred core
> support")

This is not a valid git id in Linus's tree :(

> commit 3d291fe47fe1 ("cpufreq: amd-pstate: fix the highest frequency issue
> which limits performance")

And neither is this :(

So perhaps you got them wrong?

thanks,

greg k-h

Re: linux-6.6.y regression on amd-pstate

[Christian Heusel]

[-- Attachment #1: Type: text/plain, Size: 1129 bytes --]

Hey Greg,

On 24/09/08 04:05PM, Greg Kroah-Hartman wrote:
> On Thu, Sep 05, 2024 at 04:14:26PM -0500, Mario Limonciello wrote:
> > + stable
> > + regressions
> > New subject
> > 
> > Great news.
> > 
> > Greg, Sasha,
> > 
> > Can you please pull in these 3 commits specifically to 6.6.y to fix a
> > regression that was reported by Morgan in 6.6.y:
> > 
> > commit 12753d71e8c5 ("ACPI: CPPC: Add helper to get the highest performance
> > value")
> 
> This is fine, but:
> 
> > commit ed429c686b79 ("cpufreq: amd-pstate: Enable amd-pstate preferred core
> > support")
> 
> This is not a valid git id in Linus's tree :(

f3a052391822 ("cpufreq: amd-pstate: Enable amd-pstate preferred core support")

> 
> > commit 3d291fe47fe1 ("cpufreq: amd-pstate: fix the highest frequency issue
> > which limits performance")
> 
> And neither is this :(

bf202e654bfa ("cpufreq: amd-pstate: fix the highest frequency issue which limits performance")

> So perhaps you got them wrong?

I have added the ID's of the matching commits from Linus' tree above! :)

> thanks,
> 
> greg k-h

Cheers,
Chris

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: linux-6.6.y regression on amd-pstate

[Greg Kroah-Hartman]

On Sun, Sep 08, 2024 at 04:12:28PM +0200, Christian Heusel wrote:
> Hey Greg,
> 
> On 24/09/08 04:05PM, Greg Kroah-Hartman wrote:
> > On Thu, Sep 05, 2024 at 04:14:26PM -0500, Mario Limonciello wrote:
> > > + stable
> > > + regressions
> > > New subject
> > > 
> > > Great news.
> > > 
> > > Greg, Sasha,
> > > 
> > > Can you please pull in these 3 commits specifically to 6.6.y to fix a
> > > regression that was reported by Morgan in 6.6.y:
> > > 
> > > commit 12753d71e8c5 ("ACPI: CPPC: Add helper to get the highest performance
> > > value")
> > 
> > This is fine, but:
> > 
> > > commit ed429c686b79 ("cpufreq: amd-pstate: Enable amd-pstate preferred core
> > > support")
> > 
> > This is not a valid git id in Linus's tree :(
> 
> f3a052391822 ("cpufreq: amd-pstate: Enable amd-pstate preferred core support")
> 
> > 
> > > commit 3d291fe47fe1 ("cpufreq: amd-pstate: fix the highest frequency issue
> > > which limits performance")
> > 
> > And neither is this :(
> 
> bf202e654bfa ("cpufreq: amd-pstate: fix the highest frequency issue which limits performance")
> 
> > So perhaps you got them wrong?
> 
> I have added the ID's of the matching commits from Linus' tree above! :)
> 

Thanks, that works, all now queued up.

greg k-h

RE: [EXTERNAL] Re: linux-6.6.y regression on amd-pstate

[Jones, Morgan]

Hey all,

I think some form of this is back between 6.12 and 6.15 on our fractious AMD EPYC 7702. The symptom appears to be that the core will not boost past 2 GHz (the nominal frequency), so we lose out on 1.36 GHz of boost frequency. Downgrade from 6.15.7 to LTS (6.12.39) seems to fix it.

Keeping an eye out for other threads reporting similar symptoms on recent kernels:

[    0.000000] Linux version 6.15.7-xanmod1 (nixbld@localhost) (gcc (GCC) 14.2.1 20250322, GNU ld (GNU Binutils) 2.44) #1-NixOS SMP PREEMPT_DYNAMIC Tue Jan  1 00:00:00 UTC 1980

# cat /proc/cmdline
[snip] amd_pstate=active amd_prefcore=enable amd_pstate.shared_mem=1

# cat /proc/cpuinfo
[snip]
processor       : 127
vendor_id       : AuthenticAMD
cpu family      : 23
model           : 49
model name      : AMD EPYC 7702 64-Core Processor
stepping        : 0
microcode       : 0x830107d
cpu MHz         : 400.000
cache size      : 512 KB
physical id     : 0
siblings        : 128
core id         : 63
cpu cores       : 64
apicid          : 127
initial apicid  : 127
fpu             : yes
fpu_exception   : yes
cpuid level     : 16
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl xtopology nonstop_tsc cpuid extd_apicid aperfmperf rapl pni pclmulqdq monitor ssse3 fma cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw ibs skinit wdt tce topoext perfctr_core perfctr_nb bpext perfctr_llc mwaitx cpb cat_l3 cdp_l3 hw_pstate ssbd mba ibrs ibpb stibp vmmcall fsgsbase bmi1 avx2 smep bmi2 cqm rdt_a rdseed adx smap clflushopt clwb sha_ni xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local clzero irperf xsaveerptr rdpru wbnoinvd amd_ppin arat npt lbrv svm_lock nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold avic v_vmsave_vmload vgif v_spec_ctrl umip rdpid overflow_recov succor smca sev sev_es
bugs            : sysret_ss_attrs spectre_v1 spectre_v2 spec_store_bypass retbleed smt_rsb srso ibpb_no_ret
bogomips        : 3992.75
TLB size        : 3072 4K pages
clflush size    : 64
cache_alignment : 64
address sizes   : 43 bits physical, 48 bits virtual
power management: ts ttp tm hwpstate cpb eff_freq_ro [13] [14]

# cpupower frequency-info
analyzing CPU 76:
  driver: amd-pstate-epp
  CPUs which run at the same hardware frequency: 76
  CPUs which need to have their frequency coordinated by software: 76
  energy performance preference: performance
  hardware limits: 408 MHz - 3.36 GHz
  available cpufreq governors: performance powersave
  current policy: frequency should be within 1.51 GHz and 3.36 GHz.
                  The governor "performance" may decide which speed to use
                  within this range.
  current CPU frequency: 1.98 GHz (asserted by call to kernel)
  boost state support:
    Supported: yes
    Active: yes
  amd-pstate limits:
    Highest Performance: 255. Maximum Frequency: 3.36 GHz.
    Nominal Performance: 152. Nominal Frequency: 2.00 GHz.
    Lowest Non-linear Performance: 115. Lowest Non-linear Frequency: 1.51 GHz.
    Lowest Performance: 31. Lowest Frequency: 400 MHz.
    Preferred Core Support: 0. Preferred Core Ranking: 255.

Regards,
Morgan

-----Original Message-----
From: Greg Kroah-Hartman <gregkh@linuxfoundation.org> 
Sent: Sunday, September 8, 2024 7:30 AM
To: Christian Heusel <christian@heusel.eu>
Cc: Mario Limonciello <mario.limonciello@amd.com>; Jones, Morgan <Morgan.Jones@viasat.com>; Sasha Levin <sashal@kernel.org>; linux-pm@vger.kernel.org; linux-kernel@vger.kernel.org; David Arcari <darcari@redhat.com>; Dhananjay Ugwekar <Dhananjay.Ugwekar@amd.com>; rafael@kernel.org; viresh.kumar@linaro.org; gautham.shenoy@amd.com; perry.yuan@amd.com; skhan@linuxfoundation.org; li.meng@amd.com; ray.huang@amd.com; stable@vger.kernel.org; Linux kernel regressions list <regressions@lists.linux.dev>
Subject: [EXTERNAL] Re: linux-6.6.y regression on amd-pstate

On Sun, Sep 08, 2024 at 04:12:28PM +0200, Christian Heusel wrote:
> Hey Greg,
> 
> On 24/09/08 04:05PM, Greg Kroah-Hartman wrote:
> > On Thu, Sep 05, 2024 at 04:14:26PM -0500, Mario Limonciello wrote:
> > > + stable
> > > + regressions
> > > New subject
> > > 
> > > Great news.
> > > 
> > > Greg, Sasha,
> > > 
> > > Can you please pull in these 3 commits specifically to 6.6.y to 
> > > fix a regression that was reported by Morgan in 6.6.y:
> > > 
> > > commit 12753d71e8c5 ("ACPI: CPPC: Add helper to get the highest 
> > > performance
> > > value")
> > 
> > This is fine, but:
> > 
> > > commit ed429c686b79 ("cpufreq: amd-pstate: Enable amd-pstate 
> > > preferred core
> > > support")
> > 
> > This is not a valid git id in Linus's tree :(
> 
> f3a052391822 ("cpufreq: amd-pstate: Enable amd-pstate preferred core 
> support")
> 
> > 
> > > commit 3d291fe47fe1 ("cpufreq: amd-pstate: fix the highest 
> > > frequency issue which limits performance")
> > 
> > And neither is this :(
> 
> bf202e654bfa ("cpufreq: amd-pstate: fix the highest frequency issue 
> which limits performance")
> 
> > So perhaps you got them wrong?
> 
> I have added the ID's of the matching commits from Linus' tree above! 
> :)
> 

Thanks, that works, all now queued up.

greg k-h

Re: [EXTERNAL] Re: linux-6.6.y regression on amd-pstate

[Mario Limonciello]

On 8/1/2025 2:14 AM, Jones, Morgan wrote:
> Hey all,
> 
> I think some form of this is back between 6.12 and 6.15 on our fractious AMD EPYC 7702. The symptom appears to be that the core will not boost past 2 GHz (the nominal frequency), so we lose out on 1.36 GHz of boost frequency. Downgrade from 6.15.7 to LTS (6.12.39) seems to fix it.
> 
> Keeping an eye out for other threads reporting similar symptoms on recent kernels:
> 
> [    0.000000] Linux version 6.15.7-xanmod1 (nixbld@localhost) (gcc (GCC) 14.2.1 20250322, GNU ld (GNU Binutils) 2.44) #1-NixOS SMP PREEMPT_DYNAMIC Tue Jan  1 00:00:00 UTC 1980
> 
> # cat /proc/cmdline
> [snip] amd_pstate=active amd_prefcore=enable amd_pstate.shared_mem=1
> 
> # cat /proc/cpuinfo
> [snip]
> processor       : 127
> vendor_id       : AuthenticAMD
> cpu family      : 23
> model           : 49
> model name      : AMD EPYC 7702 64-Core Processor
> stepping        : 0
> microcode       : 0x830107d
> cpu MHz         : 400.000
> cache size      : 512 KB
> physical id     : 0
> siblings        : 128
> core id         : 63
> cpu cores       : 64
> apicid          : 127
> initial apicid  : 127
> fpu             : yes
> fpu_exception   : yes
> cpuid level     : 16
> wp              : yes
> flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl xtopology nonstop_tsc cpuid extd_apicid aperfmperf rapl pni pclmulqdq monitor ssse3 fma cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw ibs skinit wdt tce topoext perfctr_core perfctr_nb bpext perfctr_llc mwaitx cpb cat_l3 cdp_l3 hw_pstate ssbd mba ibrs ibpb stibp vmmcall fsgsbase bmi1 avx2 smep bmi2 cqm rdt_a rdseed adx smap clflushopt clwb sha_ni xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local clzero irperf xsaveerptr rdpru wbnoinvd amd_ppin arat npt lbrv svm_lock nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold avic v_vmsave_vmload vgif v_spec_ctrl umip rdpid overflow_recov succor smca sev sev_es
> bugs            : sysret_ss_attrs spectre_v1 spectre_v2 spec_store_bypass retbleed smt_rsb srso ibpb_no_ret
> bogomips        : 3992.75
> TLB size        : 3072 4K pages
> clflush size    : 64
> cache_alignment : 64
> address sizes   : 43 bits physical, 48 bits virtual
> power management: ts ttp tm hwpstate cpb eff_freq_ro [13] [14]
> 
> # cpupower frequency-info
> analyzing CPU 76:
>    driver: amd-pstate-epp
>    CPUs which run at the same hardware frequency: 76
>    CPUs which need to have their frequency coordinated by software: 76
>    energy performance preference: performance
>    hardware limits: 408 MHz - 3.36 GHz
>    available cpufreq governors: performance powersave
>    current policy: frequency should be within 1.51 GHz and 3.36 GHz.
>                    The governor "performance" may decide which speed to use
>                    within this range.
>    current CPU frequency: 1.98 GHz (asserted by call to kernel)
>    boost state support:
>      Supported: yes
>      Active: yes
>    amd-pstate limits:
>      Highest Performance: 255. Maximum Frequency: 3.36 GHz.
>      Nominal Performance: 152. Nominal Frequency: 2.00 GHz.
>      Lowest Non-linear Performance: 115. Lowest Non-linear Frequency: 1.51 GHz.
>      Lowest Performance: 31. Lowest Frequency: 400 MHz.
>      Preferred Core Support: 0. Preferred Core Ranking: 255.
> 
> Regards,
> Morgan
> 

Hello Morgan,

6.12 to 6.15 unfortunately includes a pretty big overhaul to the 
amd-pstate driver.  But I'm pretty surprised to hear this regression as 
we have had a lot of mileage on it across a very wide variety of hardware.

That being said:
1) Please capture a report using amd-pstate from amd-debug-tools 
(https://git.kernel.org/pub/scm/linux/kernel/git/superm1/amd-debug-tools.git/about/) 
both on a good and bad kernel and share them.

2) Can you reproduce on mainline 6.16?

If 1 and 2 don't lead an obvious answer:

3) Can you please bisect?

Thanks,