[Bug 219431] New: [6.12] amd-pstate / Ryzen 5xxx (Zen 3, Vermeer): Could not retrieve highest performance (-19)

[Bug 219431] New: [6.12] amd-pstate / Ryzen 5xxx (Zen 3, Vermeer): Could not retrieve highest performance (-19)

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=219431

            Bug ID: 219431
           Summary: [6.12] amd-pstate / Ryzen 5xxx (Zen 3, Vermeer): Could
                    not retrieve highest performance (-19)
           Product: Power Management
           Version: 2.5
          Hardware: AMD
                OS: Linux
            Status: NEW
          Severity: normal
          Priority: P3
         Component: cpufreq
          Assignee: linux-pm@vger.kernel.org
          Reporter: intelfx@intelfx.name
        Regression: No

Created attachment 307075
  --> https://bugzilla.kernel.org/attachment.cgi?id=307075&action=edit
dmesg

With https://lore.kernel.org/lkml/20240905163007.1350840-1-superm1@kernel.org/
applied (or on 6.12-rc3+), booting on a Ryzen 9 5950X CPU (Zen 3, Vermeer)
yields the newly-promoted warning:

```
[    0.623843] Could not retrieve highest performance (-19)
```

-- 
You may reply to this email to add a comment.

You are receiving this mail because:
You are the assignee for the bug.

[Bug 219431] [6.12] amd-pstate / Ryzen 5xxx (Zen 3, Vermeer): Could not retrieve highest performance (-19)

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=219431

--- Comment #1 from Ivan Shapovalov (intelfx@intelfx.name) ---
Created attachment 307076
  --> https://bugzilla.kernel.org/attachment.cgi?id=307076&action=edit
/proc/cpuinfo

-- 
You may reply to this email to add a comment.

You are receiving this mail because:
You are the assignee for the bug.

[Bug 219431] [6.12] amd-pstate / Ryzen 5xxx (Zen 3, Vermeer): Could not retrieve highest performance (-19)

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=219431

--- Comment #2 from Ivan Shapovalov (intelfx@intelfx.name) ---
Created attachment 307077
  --> https://bugzilla.kernel.org/attachment.cgi?id=307077&action=edit
/proc/config.gz

-- 
You may reply to this email to add a comment.

You are receiving this mail because:
You are the assignee for the bug.

[Bug 219431] [6.12] amd-pstate / Ryzen 5xxx (Zen 3, Vermeer): Could not retrieve highest performance (-19)

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=219431

--- Comment #3 from Mario Limonciello (AMD) (mario.limonciello@amd.com) ---
Can you repro on 6.12-rc5? Can I please see kernel log with AMD pstate dyndbg
on the kernel command line for your system on 6.12rc5?

-- 
You may reply to this email to add a comment.

You are receiving this mail because:
You are the assignee for the bug.

[Bug 219431] [6.12] amd-pstate / Ryzen 5xxx (Zen 3, Vermeer): Could not retrieve highest performance (-19)

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=219431

Ivan Shapovalov (intelfx@intelfx.name) changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
 Attachment #307075|0                           |1
        is obsolete|                            |

--- Comment #4 from Ivan Shapovalov (intelfx@intelfx.name) ---
Created attachment 307079
  --> https://bugzilla.kernel.org/attachment.cgi?id=307079&action=edit
dmesg w/ dynamic debug on amd-pstate.c/cppc.c/cppc_acpi.c

(In reply to Mario Limonciello (AMD) from comment #3)
> Can you repro on 6.12-rc5? Can I please see kernel log with AMD pstate
> dyndbg on the kernel command line for your system on 6.12rc5?

Reproduced on 6.12-rc5. There was nothing from amd_pstate.c before the warning
message, so I also enabled dynamic debug on cppc.c and cppc_acpi.c.

-- 
You may reply to this email to add a comment.

You are receiving this mail because:
You are the assignee for the bug.

[Bug 219431] [6.12] amd-pstate / Ryzen 5xxx (Zen 3, Vermeer): Could not retrieve highest performance (-19)

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=219431

--- Comment #5 from Mario Limonciello (AMD) (mario.limonciello@amd.com) ---
It looks like this message is being emitted specifically from the calls in
amd_set_max_freq_ratio().

You system is a shared memory design not a MSR design.
"[  +0.000016] amd_pstate: AMD CPPC shared memory based functionality is
supported"

This looks like the calltrace:
amd_set_max_freq_ratio
->amd_get_boost_ratio_numerator
->->amd_detect_prefcore
->->->amd_get_highest_perf
->->->->cppc_get_highest_perf
->->->->->cppc_get_perf

The first error you see is "No CPC descriptor for CPU:1" which comes from
cppc_get_perf().

Can you please share your acpidump?  It sure seems like a BIOS bug to me.

-- 
You may reply to this email to add a comment.

You are receiving this mail because:
You are the assignee for the bug.

[Bug 219431] [6.12] amd-pstate / Ryzen 5xxx (Zen 3, Vermeer): Could not retrieve highest performance (-19)

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=219431

--- Comment #6 from Oleksandr Natalenko (oleksandr@natalenko.name) ---
Created attachment 307080
  --> https://bugzilla.kernel.org/attachment.cgi?id=307080&action=edit
acpidump, 5950X, ASUS Pro WS X570-ACE

I'm experiencing this issue too. Attaching acpidump output for 5950X and ASUS
Pro WS X570-ACE motherboard with BIOS version 4805.

The latest BIOS for my M/B is 4902, but I haven't got a chance to update just
yet.

-- 
You may reply to this email to add a comment.

You are receiving this mail because:
You are the assignee for the bug.

[Bug 219431] [6.12] amd-pstate / Ryzen 5xxx (Zen 3, Vermeer): Could not retrieve highest performance (-19)

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=219431

--- Comment #7 from Mario Limonciello (AMD) (mario.limonciello@amd.com) ---
Please also share the same dyndbg output.  Let's make sure we're talking about
the same thing.

Are you guys seeing any functional issues you can elaborate upon?

-- 
You may reply to this email to add a comment.

You are receiving this mail because:
You are the assignee for the bug.

[Bug 219431] [6.12] amd-pstate / Ryzen 5xxx (Zen 3, Vermeer): Could not retrieve highest performance (-19)

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=219431

--- Comment #8 from Oleksandr Natalenko (oleksandr@natalenko.name) ---
Created attachment 307081
  --> https://bugzilla.kernel.org/attachment.cgi?id=307081&action=edit
dmesg dyndbg, 5950X, ASUS Pro WS X570-ACE

Attached dmesg with dyndbg.

I do not think I can describe any functional limitations I face.

-- 
You may reply to this email to add a comment.

You are receiving this mail because:
You are the assignee for the bug.

[Bug 219431] [6.12] amd-pstate / Ryzen 5xxx (Zen 3, Vermeer): Could not retrieve highest performance (-19)

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=219431

--- Comment #9 from Ivan Shapovalov (intelfx@intelfx.name) ---
Created attachment 307082
  --> https://bugzilla.kernel.org/attachment.cgi?id=307082&action=edit
`acpidump -b`, 5950X, ASRock X570M-Pro4, BIOS P3.60

Attaching `acpidump -b` output for my system, which is a 5950X on an ASRock
X570M-Pro4 motherboard. The BIOS is also quite old here.

I'm not observing any specific functional limitations *that I can track to this
issue*.

However, there is this thing that basically nothing I do to the EPP knobs or
the cpufreq governor has any effect on the CPU frequency management behavior.
Also, sometimes under ~75% load (cgroup limited) some cores get stuck at <4GHz
while others boost to expected ~4.7-ish, but I've no idea how to reproduce this
one reliably.

-- 
You may reply to this email to add a comment.

You are receiving this mail because:
You are the assignee for the bug.

[Bug 219431] [6.12] amd-pstate / Ryzen 5xxx (Zen 3, Vermeer): Could not retrieve highest performance (-19)

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=219431

--- Comment #10 from Ivan Shapovalov (intelfx@intelfx.name) ---
(In reply to Mario Limonciello (AMD) from comment #5)
> It looks like this message is being emitted specifically from the calls in
> amd_set_max_freq_ratio().
> 
> You system is a shared memory design not a MSR design.
> "[  +0.000016] amd_pstate: AMD CPPC shared memory based functionality is
> supported"

I assumed that all Zen 3 desktop parts are ACPI CPPC-based rather than
MSR-based? That's why I mentioned Vermeer in both my original mail and this bug
report.

> This looks like the calltrace:
> amd_set_max_freq_ratio
> ->amd_get_boost_ratio_numerator
> ->->amd_detect_prefcore
> ->->->amd_get_highest_perf
> ->->->->cppc_get_highest_perf
> ->->->->->cppc_get_perf
> 
> The first error you see is "No CPC descriptor for CPU:1" which comes from
> cppc_get_perf().
> 
> Can you please share your acpidump?  It sure seems like a BIOS bug to me.

There is also an interesting observation, that at some point *after* this
warning we get a pr_debug() from the end of amd_detect_prefcore(), which
indicates that the latter is called again and this error does not happen
anymore:

```
Oct 28 17:13:08 kernel: ACPI CPPC: CPPC v2 _OSC not acked
Oct 28 17:13:08 kernel: ACPI CPPC: Parsed CPC struct for CPU: 0
Oct 28 17:13:08 kernel: ACPI CPPC: TX completed. CMD sent:0, ret:0
Oct 28 17:13:08 kernel: ACPI CPPC: TX completed. CMD sent:0, ret:0
Oct 28 17:13:08 kernel: ACPI CPPC: No CPC descriptor for CPU:1
Oct 28 17:13:08 kernel: Could not retrieve highest performance (-19)
<...>
Oct 28 17:13:08 kernel: amd_pstate: AMD CPPC shared memory based functionality
is supported
Oct 28 17:13:08 kernel: ACPI CPPC: TX completed. CMD sent:0, ret:0
Oct 28 17:13:08 kernel: ACPI CPPC: TX completed. CMD sent:0, ret:0
Oct 28 17:13:08 kernel: AMD CPPC preferred core is supported (highest perf:
0xe7)
```

It's even more interesting that the debug logs from the amd-pstate flow you
described earlier appear to be interleaved with debug logs from
acpi_cppc_processor_probe():

```
Oct 28 17:13:08 kernel: ACPI CPPC: CPPC v2 _OSC not acked
Oct 28 17:13:08 kernel: ACPI CPPC: Parsed CPC struct for CPU: 0
Oct 28 17:13:08 kernel: ACPI CPPC: TX completed. CMD sent:0, ret:0
Oct 28 17:13:08 kernel: ACPI CPPC: TX completed. CMD sent:0, ret:0
Oct 28 17:13:08 kernel: ACPI CPPC: No CPC descriptor for CPU:1
Oct 28 17:13:08 kernel: Could not retrieve highest performance (-19)
Oct 28 17:13:08 kernel: Monitor-Mwait will be used to enter C-1 state
Oct 28 17:13:08 kernel: ACPI CPPC: CPPC v2 _OSC not acked
Oct 28 17:13:08 kernel: ACPI CPPC: Parsed CPC struct for CPU: 1 <-- THIS
```

I'm not familiar with control flow in these subsystems, but isn't it possible
that we are simply racing with ACPI (and thus CPPC) subsystem initialization?

-- 
You may reply to this email to add a comment.

You are receiving this mail because:
You are the assignee for the bug.

[Bug 219431] [6.12] amd-pstate / Ryzen 5xxx (Zen 3, Vermeer): Could not retrieve highest performance (-19)

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=219431

--- Comment #11 from Mario Limonciello (AMD) (mario.limonciello@amd.com) ---
>I assumed that all Zen 3 desktop parts are ACPI CPPC-based rather than
>MSR-based? That's why I mentioned Vermeer in both my original mail and this
>bug report.

Vermeer is a shared memory based design.  All the newer designs are MSR based.

> There is also an interesting observation, that at some point *after* this
> warning we get a pr_debug() from the end of amd_detect_prefcore(), which
> indicates that the latter is called again and this error does not happen
> anymore:

Yeah that's why I was wondering if there was a functional issue. It seems like
this is going to influence some of the frequency invariance code that's set up
earlier on.

> I'm not familiar with control flow in these subsystems, but isn't it possible
> that we are simply racing with ACPI (and thus CPPC) subsystem initialization?

acpi_init() calls 
acpi_bus_init()() calls 
acpi_bus_osc_negotiate_platform_control() 
which sets osc_sb_cppc2_support_acked

Here is the more full call path that fails:

acpi_init() calls
acpi_scan_init() calls
acpi_processor_init() 

which loads "processor" kernel module

From "processor" kernel module:

acpi_processor_driver_init() calls
acpi_soft_cpu_online() calls
__acpi_processor_start() calls 
acpi_cppc_processor_probe() calls 
arch_init_invariance_cppc() which is a macro to call 
init_freq_invariance_cppc().


I'm not sure I understand why CPPC v2 _OSC isn't acked though at the right
time.  

Can you use ftrace or annotate the functions with some printk's to figure it
out perhaps from what I've shared?

-- 
You may reply to this email to add a comment.

You are receiving this mail because:
You are the assignee for the bug.

[Bug 219431] [6.12] amd-pstate / Ryzen 5xxx (Zen 3, Vermeer): Could not retrieve highest performance (-19)

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=219431

--- Comment #12 from Mario Limonciello (AMD) (mario.limonciello@amd.com) ---
Something else that would be really useful to understand is whether this is
only showing up now because the error level was raised
(IE : https://git.kernel.org/superm1/c/21fb59ab4b976) or if it was there in
past kernels too, but because it was debug no one ever saw it.

Try turning up dynamic debug on arch/x86/kernel/acpi/cppc.c on an older kernel
that doesn't have that commit and see if you see it there too.

I suspect the answer is yes.

-- 
You may reply to this email to add a comment.

You are receiving this mail because:
You are the assignee for the bug.

[Bug 219431] [6.12] amd-pstate / Ryzen 5xxx (Zen 3, Vermeer): Could not retrieve highest performance (-19)

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=219431

--- Comment #13 from Mario Limonciello (AMD) (mario.limonciello@amd.com) ---
> I suspect the answer is yes.

Actually; no it should be new, but it's because the frequency calculation was
wrong before.

I found a system with shared mem and I have a theory what's going on is that
frequency invariance is attempting to get setup before all CPUs have probed.

Before the preferred cores code frequency invariance would be calculated from
the first CPU only as _CPC is probed.  But with preferred cores code in place
you need ALL _CPC values present to know whether you have preferred cores or
not.

I added some dump stack calls to acpi_cppc_processor_probe when the struct is
parsed:

ACPI CPPC: Parsed CPC struct for CPU: 0
CPU: 0 UID: 0 PID: 23 Comm: cpuhp/0 Not tainted 6.12.0-rc5-00026-g0c7b883226dd
#268
Hardware name: LENOVO 30E1S68903/1046, BIOS S07KT5BA 01/26/2024
Call Trace:
 <TASK>
 dump_stack_lvl+0x60/0x80
 acpi_cppc_processor_probe+0x9fc/0xe50
 acpi_soft_cpu_online+0x69/0x160
 ? __pfx_acpi_soft_cpu_online+0x10/0x10
 cpuhp_invoke_callback+0x308/0x4c0
 ? srso_return_thunk+0x5/0x5f
 ? __pfx_smpboot_thread_fn+0x10/0x10
 cpuhp_thread_fun+0x131/0x210
 smpboot_thread_fn+0x187/0x220
 kthread+0xdd/0x110
 ? __pfx_kthread+0x10/0x10
 ret_from_fork+0x30/0x50
 ? __pfx_kthread+0x10/0x10
 ret_from_fork_asm+0x1a/0x30
 </TASK>

Then I added one to the error sequence.

ACPI CPPC: No CPC descriptor for CPU:1
CPU: 0 UID: 0 PID: 23 Comm: cpuhp/0 Not tainted 6.12.0-rc5-00026-g0c7b883226dd
#268
Hardware name: LENOVO 30E1S68903/1046, BIOS S07KT5BA 01/26/2024
Call Trace:
 <TASK>
 dump_stack_lvl+0x60/0x80
 cppc_get_perf+0x127/0x1c0
 amd_get_highest_perf+0x5e/0x80
 amd_detect_prefcore+0x7a/0x130
 amd_get_boost_ratio_numerator+0x28/0x90
 init_freq_invariance_cppc+0xad/0x140
 acpi_cppc_processor_probe+0x7ce/0xe50
 acpi_soft_cpu_online+0x69/0x160
 ? __pfx_acpi_soft_cpu_online+0x10/0x10
 cpuhp_invoke_callback+0x308/0x4c0
 ? srso_return_thunk+0x5/0x5f
 ? __pfx_smpboot_thread_fn+0x10/0x10
 cpuhp_thread_fun+0x131/0x210
 smpboot_thread_fn+0x187/0x220
 kthread+0xdd/0x110
 ? __pfx_kthread+0x10/0x10
 ret_from_fork+0x30/0x50
 ? __pfx_kthread+0x10/0x10
 ret_from_fork_asm+0x1a/0x30
 </TASK>
Could not retrieve highest performance (-19)

You can see that in both cases the code runs on CPU0.  So it's trying to get
values that won't be initialized until CPU1 is ready.

I think this will help "prevent" the error from coming up:

diff --git a/arch/x86/kernel/acpi/cppc.c b/arch/x86/kernel/acpi/cppc.c
index 5e9325273a9fb..a21c26d1fe65a 100644
--- a/arch/x86/kernel/acpi/cppc.c
+++ b/arch/x86/kernel/acpi/cppc.c
@@ -190,7 +190,7 @@ int amd_detect_prefcore(bool *detected)
                break;
        }

-       for_each_present_cpu(cpu) {
+       for_each_online_cpu(cpu) {
                u32 tmp;
                int ret;

But it's going to probably break preferred cores detection, so it's not a
correct solution.

-- 
You may reply to this email to add a comment.

You are receiving this mail because:
You are the assignee for the bug.

[Bug 219431] [6.12] amd-pstate / Ryzen 5xxx (Zen 3, Vermeer): Could not retrieve highest performance (-19)

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=219431

--- Comment #14 from Mario Limonciello (AMD) (mario.limonciello@amd.com) ---
Created attachment 307084
  --> https://bugzilla.kernel.org/attachment.cgi?id=307084&action=edit
possible patch (v1)

Here's a potential patch.  This helps on the system I can reproduce this issue,
please see if helps you too.

Also if it does help, would you mind if I used your email address in the commit
message for the Reported-and-tested-by tag?  I can get it from Bugzilla, but I
would like your permission first.

-- 
You may reply to this email to add a comment.

You are receiving this mail because:
You are the assignee for the bug.

[Bug 219431] [6.12] amd-pstate / Ryzen 5xxx (Zen 3, Vermeer): Could not retrieve highest performance (-19)

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=219431

--- Comment #15 from Ivan Shapovalov (intelfx@intelfx.name) ---
I'll test this later today, thanks. (I also want to try to dig into _OSC CPCv2
flag not being set, just to definitively confirm if this is a firmware issue or
an ordering issue.)

-- 
You may reply to this email to add a comment.

You are receiving this mail because:
You are the assignee for the bug.

[Bug 219431] [6.12] amd-pstate / Ryzen 5xxx (Zen 3, Vermeer): Could not retrieve highest performance (-19)

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=219431

--- Comment #16 from Oleksandr Natalenko (oleksandr@natalenko.name) ---
The message disappears with this patch for me. Should I check something else
too?

-- 
You may reply to this email to add a comment.

You are receiving this mail because:
You are the assignee for the bug.

[Bug 219431] [6.12] amd-pstate / Ryzen 5xxx (Zen 3, Vermeer): Could not retrieve highest performance (-19)

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=219431

--- Comment #17 from Mario Limonciello (AMD) (mario.limonciello@amd.com) ---
Thanks, you can check whether the CPU cores are reporting correct maximum
frequencies.

Can I use your email when I post it?

-- 
You may reply to this email to add a comment.

You are receiving this mail because:
You are the assignee for the bug.

[Bug 219431] [6.12] amd-pstate / Ryzen 5xxx (Zen 3, Vermeer): Could not retrieve highest performance (-19)

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=219431

--- Comment #18 from Oleksandr Natalenko (oleksandr@natalenko.name) ---
> you can check whether the CPU cores are reporting correct maximum
> frequencies.

In `amd_pstate_max_freq` it's 5084000 on all cores. Running `turbostat`
parallel to `xmrig` I see ~4.4 GHz on all cores under heavy load. In this
regard, for my setup, nothing changed comparing to an unpatched kernel, I
guess.

> Can I use your email when I post it?

Sure.

-- 
You may reply to this email to add a comment.

You are receiving this mail because:
You are the assignee for the bug.

[Bug 219431] [6.12] amd-pstate / Ryzen 5xxx (Zen 3, Vermeer): Could not retrieve highest performance (-19)

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=219431

Mario Limonciello (AMD) (mario.limonciello@amd.com) changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|NEW                         |ASSIGNED
           Assignee|linux-pm@vger.kernel.org    |mario.limonciello@amd.com

--- Comment #19 from Mario Limonciello (AMD) (mario.limonciello@amd.com) ---
https://lore.kernel.org/linux-acpi/20241029174910.600482-1-superm1@kernel.org/

-- 
You may reply to this email to add a comment.

You are receiving this mail because:
You are the assignee for the bug.