[PATCH v2 0/2] AMD Pstate driver fixes

[PATCH v2 0/2] AMD Pstate driver fixes

[Dhananjay Ugwekar]

1. Handle the nominal freq units inconsistency in amd-pstate-ut, which was
leading to the below error on inserting the amd-pstate-ut module.

[ 4982.498864] amd_pstate_ut: 1    amd_pstate_ut_acpi_cpc_valid  success!
[ 4982.498873] amd_pstate_ut: 2    amd_pstate_ut_check_enabled   success!
[ 4982.509151] amd_pstate_ut: 3    amd_pstate_ut_check_perf      success!
[ 4982.509155] amd_pstate_ut: amd_pstate_ut_check_freq cpu0 max=3709000  >= nominal=2401 > lowest_nonlinear=1903000 > min=400000 > 0, the formula is incorrect!
[ 4982.509157] amd_pstate_ut: 4    amd_pstate_ut_check_freq      fail!

2. Setting the scaling_max_freq on shared memory CPPC systems was
broken in amd-pstate-epp driver(amd_pstate=active mode). The
scaling_max_freq value was not being propagated to the shared memory area, 
so the frequency capping was not being honored.

Tested on a AMD Zen3 Milan machine(shared memory CPPC): 

stress-ng is running on the system to keep the CPU utilization at 100%
to test the scaling_max_freq capping.

Before the patch:
We can see below, setting the scaling_max_freq is not taking effect.
 
linux/tools/power/x86/turbostat# ./turbostat --Summary
turbostat version 2023.11.07 - Len Brown <lenb@kernel.org>
[Snip]
cpu0: cpufreq driver: amd-pstate-epp
cpu0: cpufreq governor: performance
[Snip]
Avg_MHz Busy%   Bzy_MHz TSC_MHz IPC     IRQ     POLL    C1      C2      POLL%   C1%     C2%     CorWatt PkgWatt
2620    100.00  2620    2026    0.80    164935  0       0       0       0.00    0.00    0.00    176.07  249.18
2580    100.00  2580    1995    0.80    162208  0       0       0       0.00    0.00    0.00    173.27  245.37
2584    100.00  2584    1998    0.79    162379  0       0       0       0.00    0.00    0.00    173.42  245.68
2577    100.00  2577    1996    0.79    162146  0       0       0       0.00    0.00    0.00    173.15  245.41
2578    100.00  2578    1996    0.80    162025  0       0       0       0.00    0.00    0.00    173.07  245.46
2575    100.00  2575    1996    0.80    162115  0       0       0       0.00    0.00    0.00    172.96  245.41
2576    100.00  2576    1996    0.79    161998  0       0       0       0.00    0.00    0.00    172.87  245.32
linux/tools/power/x86/turbostat# echo 2000000 | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_max_freq
2000000
linux/tools/power/x86/turbostat# cat /sys/devices/system/cpu/cpu*/cpufreq/scaling_max_freq | uniq
2000000
linux/tools/power/x86/turbostat# ./turbostat --Summary
turbostat version 2023.11.07 - Len Brown <lenb@kernel.org>
[Snip]
cpu0: cpufreq driver: amd-pstate-epp
cpu0: cpufreq governor: performance
[Snip]
Avg_MHz Busy%   Bzy_MHz TSC_MHz IPC     IRQ     POLL    C1      C2      POLL%   C1%     C2%     CorWatt PkgWatt
2620    100.00  2620    2038    0.79    166103  0       0       2       0.00    0.00    0.00    175.44  250.96
2566    100.00  2566    1996    0.79    162038  0       0       0       0.00    0.00    0.00    171.79  245.23
2566    100.00  2566    1996    0.79    162289  0       0       0       0.00    0.00    0.00    171.76  245.59
2571    100.00  2571    1996    0.80    162034  0       0       0       0.00    0.00    0.00    171.69  245.44
2566    100.00  2566    1996    0.79    162179  0       0       0       0.00    0.00    0.00    171.62  245.41
2567    100.00  2567    1996    0.79    162028  0       0       0       0.00    0.00    0.00    171.57  245.46
2567    100.00  2567    1996    0.80    162037  0       0       0       0.00    0.00    0.00    171.53  245.41

After applying the patch:
On setting scaling_max_freq at 2GHz, the CPU frequency gets capped at 2GHz.

linux/tools/power/x86/turbostat# ./turbostat --Summary
turbostat version 2023.11.07 - Len Brown <lenb@kernel.org>
[Snip]
cpu0: cpufreq driver: amd-pstate-epp
cpu0: cpufreq governor: performance
[Snip]
Avg_MHz Busy%   Bzy_MHz TSC_MHz IPC     IRQ     POLL    C1      C2      POLL%   C1%     C2%     CorWatt PkgWatt
2551    100.00  2551    1956    0.80    165998  0       0       0       0.00    0.00    0.00    171.34  231.22
2713    100.00  2713    2078    0.79    175801  0       0       0       0.00    0.00    0.00    181.92  266.13
2594    100.00  2594    1991    0.79    162183  0       0       1       0.00    0.00    0.00    173.99  244.50
2606    100.00  2606    2003    0.79    162632  0       0       0       0.00    0.00    0.00    174.81  246.51
2599    100.00  2599    1996    0.79    162168  0       0       0       0.00    0.00    0.00    174.05  245.46
linux/tools/power/x86/turbostat# echo 2000000 | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_max_freq
2000000
linux/tools/power/x86/turbostat# cat /sys/devices/system/cpu/cpu*/cpufreq/scaling_max_freq | uniq
2000000
linux/tools/power/x86/turbostat# ./turbostat --Summary
turbostat version 2023.11.07 - Len Brown <lenb@kernel.org>
[Snip]
cpu0: cpufreq driver: amd-pstate-epp
cpu0: cpufreq governor: performance
[Snip]
Avg_MHz Busy%   Bzy_MHz TSC_MHz IPC     IRQ     POLL    C1      C2      POLL%   C1%     C2%     CorWatt PkgWatt
2010    100.00  2010    2030    0.80    165565  0       0       0       0.00    0.00    0.00    101.22  173.52
1975    100.00  1975    1995    0.80    162042  0       0       0       0.00    0.00    0.00    99.03   169.88
1977    100.00  1977    1998    0.80    162559  0       0       0       0.00    0.00    0.00    99.16   170.20
1976    100.00  1976    1996    0.80    162243  0       0       0       0.00    0.00    0.00    99.09   170.08
1976    100.00  1976    1996    0.80    162490  0       0       0       0.00    0.00    0.00    99.17   170.16
1976    100.00  1976    1996    0.80    162056  0       0       0       0.00    0.00    0.00    99.11   170.17

Dhananjay Ugwekar (2):
  cpufreq/amd-pstate-ut: Convert nominal_freq to khz during comparisons
  cpufreq/amd-pstate: Fix the scaling_max_freq setting on shared memory
    CPPC systems

 drivers/cpufreq/amd-pstate-ut.c | 12 +++++----
 drivers/cpufreq/amd-pstate.c    | 43 ++++++++++++++++++---------------
 2 files changed, 30 insertions(+), 25 deletions(-)

-- 
2.34.1

[PATCH v2 1/2] cpufreq/amd-pstate-ut: Convert nominal_freq to khz during comparisons

[Dhananjay Ugwekar]

cpudata->nominal_freq being in MHz whereas other frequencies being in
KHz breaks the amd-pstate-ut frequency sanity check. This fixes it.

Fixes: e4731baaf294 ("cpufreq: amd-pstate: Fix the inconsistency in max frequency units")
Reported-by: David Arcari <darcari@redhat.com>
Signed-off-by: Dhananjay Ugwekar <Dhananjay.Ugwekar@amd.com>
Reviewed-by: Mario Limonciello <mario.limonciello@amd.com>
Reviewed-by: Gautham R. Shenoy <gautham.shenoy@amd.com>
---
 drivers/cpufreq/amd-pstate-ut.c | 12 +++++++-----
 1 file changed, 7 insertions(+), 5 deletions(-)

diff --git a/drivers/cpufreq/amd-pstate-ut.c b/drivers/cpufreq/amd-pstate-ut.c
index fc275d41d51e..66b73c308ce6 100644
--- a/drivers/cpufreq/amd-pstate-ut.c
+++ b/drivers/cpufreq/amd-pstate-ut.c
@@ -202,6 +202,7 @@ static void amd_pstate_ut_check_freq(u32 index)
 	int cpu = 0;
 	struct cpufreq_policy *policy = NULL;
 	struct amd_cpudata *cpudata = NULL;
+	u32 nominal_freq_khz;
 
 	for_each_possible_cpu(cpu) {
 		policy = cpufreq_cpu_get(cpu);
@@ -209,13 +210,14 @@ static void amd_pstate_ut_check_freq(u32 index)
 			break;
 		cpudata = policy->driver_data;
 
-		if (!((cpudata->max_freq >= cpudata->nominal_freq) &&
-			(cpudata->nominal_freq > cpudata->lowest_nonlinear_freq) &&
+		nominal_freq_khz = cpudata->nominal_freq*1000;
+		if (!((cpudata->max_freq >= nominal_freq_khz) &&
+			(nominal_freq_khz > cpudata->lowest_nonlinear_freq) &&
 			(cpudata->lowest_nonlinear_freq > cpudata->min_freq) &&
 			(cpudata->min_freq > 0))) {
 			amd_pstate_ut_cases[index].result = AMD_PSTATE_UT_RESULT_FAIL;
 			pr_err("%s cpu%d max=%d >= nominal=%d > lowest_nonlinear=%d > min=%d > 0, the formula is incorrect!\n",
-				__func__, cpu, cpudata->max_freq, cpudata->nominal_freq,
+				__func__, cpu, cpudata->max_freq, nominal_freq_khz,
 				cpudata->lowest_nonlinear_freq, cpudata->min_freq);
 			goto skip_test;
 		}
@@ -229,13 +231,13 @@ static void amd_pstate_ut_check_freq(u32 index)
 
 		if (cpudata->boost_supported) {
 			if ((policy->max == cpudata->max_freq) ||
-					(policy->max == cpudata->nominal_freq))
+					(policy->max == nominal_freq_khz))
 				amd_pstate_ut_cases[index].result = AMD_PSTATE_UT_RESULT_PASS;
 			else {
 				amd_pstate_ut_cases[index].result = AMD_PSTATE_UT_RESULT_FAIL;
 				pr_err("%s cpu%d policy_max=%d should be equal cpu_max=%d or cpu_nominal=%d !\n",
 					__func__, cpu, policy->max, cpudata->max_freq,
-					cpudata->nominal_freq);
+					nominal_freq_khz);
 				goto skip_test;
 			}
 		} else {
-- 
2.34.1

[PATCH v2 2/2] cpufreq/amd-pstate: Fix the scaling_max_freq setting on shared memory CPPC systems

[Dhananjay Ugwekar]

On shared memory CPPC systems, with amd_pstate=active mode, the change
in scaling_max_freq doesn't get written to the shared memory
region. Due to this, the writes to the scaling_max_freq sysfs file
don't take effect. Fix this by propagating the scaling_max_freq
changes to the shared memory region.

Fixes: ffa5096a7c33 ("cpufreq: amd-pstate: implement Pstate EPP support for the AMD processors")
Reported-by: David Arcari <darcari@redhat.com>
Signed-off-by: Dhananjay Ugwekar <Dhananjay.Ugwekar@amd.com>
---
 drivers/cpufreq/amd-pstate.c | 43 +++++++++++++++++++-----------------
 1 file changed, 23 insertions(+), 20 deletions(-)

diff --git a/drivers/cpufreq/amd-pstate.c b/drivers/cpufreq/amd-pstate.c
index 9ad62dbe8bfb..a092b13ffbc2 100644
--- a/drivers/cpufreq/amd-pstate.c
+++ b/drivers/cpufreq/amd-pstate.c
@@ -247,6 +247,26 @@ static int amd_pstate_get_energy_pref_index(struct amd_cpudata *cpudata)
 	return index;
 }
 
+static void pstate_update_perf(struct amd_cpudata *cpudata, u32 min_perf,
+			       u32 des_perf, u32 max_perf, bool fast_switch)
+{
+	if (fast_switch)
+		wrmsrl(MSR_AMD_CPPC_REQ, READ_ONCE(cpudata->cppc_req_cached));
+	else
+		wrmsrl_on_cpu(cpudata->cpu, MSR_AMD_CPPC_REQ,
+			      READ_ONCE(cpudata->cppc_req_cached));
+}
+
+DEFINE_STATIC_CALL(amd_pstate_update_perf, pstate_update_perf);
+
+static inline void amd_pstate_update_perf(struct amd_cpudata *cpudata,
+					  u32 min_perf, u32 des_perf,
+					  u32 max_perf, bool fast_switch)
+{
+	static_call(amd_pstate_update_perf)(cpudata, min_perf, des_perf,
+					    max_perf, fast_switch);
+}
+
 static int amd_pstate_set_epp(struct amd_cpudata *cpudata, u32 epp)
 {
 	int ret;
@@ -263,6 +283,9 @@ static int amd_pstate_set_epp(struct amd_cpudata *cpudata, u32 epp)
 		if (!ret)
 			cpudata->epp_cached = epp;
 	} else {
+		amd_pstate_update_perf(cpudata, cpudata->min_limit_perf, 0U,
+					     cpudata->max_limit_perf, false);
+
 		perf_ctrls.energy_perf = epp;
 		ret = cppc_set_epp_perf(cpudata->cpu, &perf_ctrls, 1);
 		if (ret) {
@@ -452,16 +475,6 @@ static inline int amd_pstate_init_perf(struct amd_cpudata *cpudata)
 	return static_call(amd_pstate_init_perf)(cpudata);
 }
 
-static void pstate_update_perf(struct amd_cpudata *cpudata, u32 min_perf,
-			       u32 des_perf, u32 max_perf, bool fast_switch)
-{
-	if (fast_switch)
-		wrmsrl(MSR_AMD_CPPC_REQ, READ_ONCE(cpudata->cppc_req_cached));
-	else
-		wrmsrl_on_cpu(cpudata->cpu, MSR_AMD_CPPC_REQ,
-			      READ_ONCE(cpudata->cppc_req_cached));
-}
-
 static void cppc_update_perf(struct amd_cpudata *cpudata,
 			     u32 min_perf, u32 des_perf,
 			     u32 max_perf, bool fast_switch)
@@ -475,16 +488,6 @@ static void cppc_update_perf(struct amd_cpudata *cpudata,
 	cppc_set_perf(cpudata->cpu, &perf_ctrls);
 }
 
-DEFINE_STATIC_CALL(amd_pstate_update_perf, pstate_update_perf);
-
-static inline void amd_pstate_update_perf(struct amd_cpudata *cpudata,
-					  u32 min_perf, u32 des_perf,
-					  u32 max_perf, bool fast_switch)
-{
-	static_call(amd_pstate_update_perf)(cpudata, min_perf, des_perf,
-					    max_perf, fast_switch);
-}
-
 static inline bool amd_pstate_sample(struct amd_cpudata *cpudata)
 {
 	u64 aperf, mperf, tsc;
-- 
2.34.1

Re: [PATCH v2 0/2] AMD Pstate driver fixes

[Dhananjay Ugwekar]

Forgot to mention the v2 changes,

v2 changes:
* Add reported-by tags (Gautham)
* Modify patch 2 to use amd_pstate_update_perf (Mario) 
* Modify commit name for patch 1 (Gautham)
* Modify commit message for patch 2, from scaling_min/max_freq to 
  just scaling_max_freq, as scaling_min_freq still needs some debugging
  to work correctly with active mode.

Regards,
Dhananjay

On 7/2/2024 1:44 PM, Dhananjay Ugwekar wrote:
> 1. Handle the nominal freq units inconsistency in amd-pstate-ut, which was
> leading to the below error on inserting the amd-pstate-ut module.
> 
> [ 4982.498864] amd_pstate_ut: 1    amd_pstate_ut_acpi_cpc_valid  success!
> [ 4982.498873] amd_pstate_ut: 2    amd_pstate_ut_check_enabled   success!
> [ 4982.509151] amd_pstate_ut: 3    amd_pstate_ut_check_perf      success!
> [ 4982.509155] amd_pstate_ut: amd_pstate_ut_check_freq cpu0 max=3709000  >= nominal=2401 > lowest_nonlinear=1903000 > min=400000 > 0, the formula is incorrect!
> [ 4982.509157] amd_pstate_ut: 4    amd_pstate_ut_check_freq      fail!
> 
> 2. Setting the scaling_max_freq on shared memory CPPC systems was
> broken in amd-pstate-epp driver(amd_pstate=active mode). The
> scaling_max_freq value was not being propagated to the shared memory area, 
> so the frequency capping was not being honored.
> 
> Tested on a AMD Zen3 Milan machine(shared memory CPPC): 
> 
> stress-ng is running on the system to keep the CPU utilization at 100%
> to test the scaling_max_freq capping.
> 
> Before the patch:
> We can see below, setting the scaling_max_freq is not taking effect.
>  
> linux/tools/power/x86/turbostat# ./turbostat --Summary
> turbostat version 2023.11.07 - Len Brown <lenb@kernel.org>
> [Snip]
> cpu0: cpufreq driver: amd-pstate-epp
> cpu0: cpufreq governor: performance
> [Snip]
> Avg_MHz Busy%   Bzy_MHz TSC_MHz IPC     IRQ     POLL    C1      C2      POLL%   C1%     C2%     CorWatt PkgWatt
> 2620    100.00  2620    2026    0.80    164935  0       0       0       0.00    0.00    0.00    176.07  249.18
> 2580    100.00  2580    1995    0.80    162208  0       0       0       0.00    0.00    0.00    173.27  245.37
> 2584    100.00  2584    1998    0.79    162379  0       0       0       0.00    0.00    0.00    173.42  245.68
> 2577    100.00  2577    1996    0.79    162146  0       0       0       0.00    0.00    0.00    173.15  245.41
> 2578    100.00  2578    1996    0.80    162025  0       0       0       0.00    0.00    0.00    173.07  245.46
> 2575    100.00  2575    1996    0.80    162115  0       0       0       0.00    0.00    0.00    172.96  245.41
> 2576    100.00  2576    1996    0.79    161998  0       0       0       0.00    0.00    0.00    172.87  245.32
> linux/tools/power/x86/turbostat# echo 2000000 | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_max_freq
> 2000000
> linux/tools/power/x86/turbostat# cat /sys/devices/system/cpu/cpu*/cpufreq/scaling_max_freq | uniq
> 2000000
> linux/tools/power/x86/turbostat# ./turbostat --Summary
> turbostat version 2023.11.07 - Len Brown <lenb@kernel.org>
> [Snip]
> cpu0: cpufreq driver: amd-pstate-epp
> cpu0: cpufreq governor: performance
> [Snip]
> Avg_MHz Busy%   Bzy_MHz TSC_MHz IPC     IRQ     POLL    C1      C2      POLL%   C1%     C2%     CorWatt PkgWatt
> 2620    100.00  2620    2038    0.79    166103  0       0       2       0.00    0.00    0.00    175.44  250.96
> 2566    100.00  2566    1996    0.79    162038  0       0       0       0.00    0.00    0.00    171.79  245.23
> 2566    100.00  2566    1996    0.79    162289  0       0       0       0.00    0.00    0.00    171.76  245.59
> 2571    100.00  2571    1996    0.80    162034  0       0       0       0.00    0.00    0.00    171.69  245.44
> 2566    100.00  2566    1996    0.79    162179  0       0       0       0.00    0.00    0.00    171.62  245.41
> 2567    100.00  2567    1996    0.79    162028  0       0       0       0.00    0.00    0.00    171.57  245.46
> 2567    100.00  2567    1996    0.80    162037  0       0       0       0.00    0.00    0.00    171.53  245.41
> 
> After applying the patch:
> On setting scaling_max_freq at 2GHz, the CPU frequency gets capped at 2GHz.
> 
> linux/tools/power/x86/turbostat# ./turbostat --Summary
> turbostat version 2023.11.07 - Len Brown <lenb@kernel.org>
> [Snip]
> cpu0: cpufreq driver: amd-pstate-epp
> cpu0: cpufreq governor: performance
> [Snip]
> Avg_MHz Busy%   Bzy_MHz TSC_MHz IPC     IRQ     POLL    C1      C2      POLL%   C1%     C2%     CorWatt PkgWatt
> 2551    100.00  2551    1956    0.80    165998  0       0       0       0.00    0.00    0.00    171.34  231.22
> 2713    100.00  2713    2078    0.79    175801  0       0       0       0.00    0.00    0.00    181.92  266.13
> 2594    100.00  2594    1991    0.79    162183  0       0       1       0.00    0.00    0.00    173.99  244.50
> 2606    100.00  2606    2003    0.79    162632  0       0       0       0.00    0.00    0.00    174.81  246.51
> 2599    100.00  2599    1996    0.79    162168  0       0       0       0.00    0.00    0.00    174.05  245.46
> linux/tools/power/x86/turbostat# echo 2000000 | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_max_freq
> 2000000
> linux/tools/power/x86/turbostat# cat /sys/devices/system/cpu/cpu*/cpufreq/scaling_max_freq | uniq
> 2000000
> linux/tools/power/x86/turbostat# ./turbostat --Summary
> turbostat version 2023.11.07 - Len Brown <lenb@kernel.org>
> [Snip]
> cpu0: cpufreq driver: amd-pstate-epp
> cpu0: cpufreq governor: performance
> [Snip]
> Avg_MHz Busy%   Bzy_MHz TSC_MHz IPC     IRQ     POLL    C1      C2      POLL%   C1%     C2%     CorWatt PkgWatt
> 2010    100.00  2010    2030    0.80    165565  0       0       0       0.00    0.00    0.00    101.22  173.52
> 1975    100.00  1975    1995    0.80    162042  0       0       0       0.00    0.00    0.00    99.03   169.88
> 1977    100.00  1977    1998    0.80    162559  0       0       0       0.00    0.00    0.00    99.16   170.20
> 1976    100.00  1976    1996    0.80    162243  0       0       0       0.00    0.00    0.00    99.09   170.08
> 1976    100.00  1976    1996    0.80    162490  0       0       0       0.00    0.00    0.00    99.17   170.16
> 1976    100.00  1976    1996    0.80    162056  0       0       0       0.00    0.00    0.00    99.11   170.17
> 
> Dhananjay Ugwekar (2):
>   cpufreq/amd-pstate-ut: Convert nominal_freq to khz during comparisons
>   cpufreq/amd-pstate: Fix the scaling_max_freq setting on shared memory
>     CPPC systems
> 
>  drivers/cpufreq/amd-pstate-ut.c | 12 +++++----
>  drivers/cpufreq/amd-pstate.c    | 43 ++++++++++++++++++---------------
>  2 files changed, 30 insertions(+), 25 deletions(-)
> 

Re: [PATCH v2 2/2] cpufreq/amd-pstate: Fix the scaling_max_freq setting on shared memory CPPC systems

[Mario Limonciello]

On 7/2/2024 3:14, Dhananjay Ugwekar wrote:
> On shared memory CPPC systems, with amd_pstate=active mode, the change
> in scaling_max_freq doesn't get written to the shared memory
> region. Due to this, the writes to the scaling_max_freq sysfs file
> don't take effect. Fix this by propagating the scaling_max_freq
> changes to the shared memory region.
> 
> Fixes: ffa5096a7c33 ("cpufreq: amd-pstate: implement Pstate EPP support for the AMD processors")
> Reported-by: David Arcari <darcari@redhat.com>
> Signed-off-by: Dhananjay Ugwekar <Dhananjay.Ugwekar@amd.com>
Reviewed-by: Mario Limonciello <mario.limonciello@amd.com>
> ---
>   drivers/cpufreq/amd-pstate.c | 43 +++++++++++++++++++-----------------
>   1 file changed, 23 insertions(+), 20 deletions(-)
> 
> diff --git a/drivers/cpufreq/amd-pstate.c b/drivers/cpufreq/amd-pstate.c
> index 9ad62dbe8bfb..a092b13ffbc2 100644
> --- a/drivers/cpufreq/amd-pstate.c
> +++ b/drivers/cpufreq/amd-pstate.c
> @@ -247,6 +247,26 @@ static int amd_pstate_get_energy_pref_index(struct amd_cpudata *cpudata)
>   	return index;
>   }
>   
> +static void pstate_update_perf(struct amd_cpudata *cpudata, u32 min_perf,
> +			       u32 des_perf, u32 max_perf, bool fast_switch)
> +{
> +	if (fast_switch)
> +		wrmsrl(MSR_AMD_CPPC_REQ, READ_ONCE(cpudata->cppc_req_cached));
> +	else
> +		wrmsrl_on_cpu(cpudata->cpu, MSR_AMD_CPPC_REQ,
> +			      READ_ONCE(cpudata->cppc_req_cached));
> +}
> +
> +DEFINE_STATIC_CALL(amd_pstate_update_perf, pstate_update_perf);
> +
> +static inline void amd_pstate_update_perf(struct amd_cpudata *cpudata,
> +					  u32 min_perf, u32 des_perf,
> +					  u32 max_perf, bool fast_switch)
> +{
> +	static_call(amd_pstate_update_perf)(cpudata, min_perf, des_perf,
> +					    max_perf, fast_switch);
> +}
> +
>   static int amd_pstate_set_epp(struct amd_cpudata *cpudata, u32 epp)
>   {
>   	int ret;
> @@ -263,6 +283,9 @@ static int amd_pstate_set_epp(struct amd_cpudata *cpudata, u32 epp)
>   		if (!ret)
>   			cpudata->epp_cached = epp;
>   	} else {
> +		amd_pstate_update_perf(cpudata, cpudata->min_limit_perf, 0U,
> +					     cpudata->max_limit_perf, false);
> +
>   		perf_ctrls.energy_perf = epp;
>   		ret = cppc_set_epp_perf(cpudata->cpu, &perf_ctrls, 1);
>   		if (ret) {
> @@ -452,16 +475,6 @@ static inline int amd_pstate_init_perf(struct amd_cpudata *cpudata)
>   	return static_call(amd_pstate_init_perf)(cpudata);
>   }
>   
> -static void pstate_update_perf(struct amd_cpudata *cpudata, u32 min_perf,
> -			       u32 des_perf, u32 max_perf, bool fast_switch)
> -{
> -	if (fast_switch)
> -		wrmsrl(MSR_AMD_CPPC_REQ, READ_ONCE(cpudata->cppc_req_cached));
> -	else
> -		wrmsrl_on_cpu(cpudata->cpu, MSR_AMD_CPPC_REQ,
> -			      READ_ONCE(cpudata->cppc_req_cached));
> -}
> -
>   static void cppc_update_perf(struct amd_cpudata *cpudata,
>   			     u32 min_perf, u32 des_perf,
>   			     u32 max_perf, bool fast_switch)
> @@ -475,16 +488,6 @@ static void cppc_update_perf(struct amd_cpudata *cpudata,
>   	cppc_set_perf(cpudata->cpu, &perf_ctrls);
>   }
>   
> -DEFINE_STATIC_CALL(amd_pstate_update_perf, pstate_update_perf);
> -
> -static inline void amd_pstate_update_perf(struct amd_cpudata *cpudata,
> -					  u32 min_perf, u32 des_perf,
> -					  u32 max_perf, bool fast_switch)
> -{
> -	static_call(amd_pstate_update_perf)(cpudata, min_perf, des_perf,
> -					    max_perf, fast_switch);
> -}
> -
>   static inline bool amd_pstate_sample(struct amd_cpudata *cpudata)
>   {
>   	u64 aperf, mperf, tsc;

Re: [PATCH v2 0/2] AMD Pstate driver fixes

[Mario Limonciello]

On 7/2/2024 3:24, Dhananjay Ugwekar wrote:
> Forgot to mention the v2 changes,
> 
> v2 changes:
> * Add reported-by tags (Gautham)
> * Modify patch 2 to use amd_pstate_update_perf (Mario)
> * Modify commit name for patch 1 (Gautham)
> * Modify commit message for patch 2, from scaling_min/max_freq to
>    just scaling_max_freq, as scaling_min_freq still needs some debugging
>    to work correctly with active mode.
> 
> Regards,
> Dhananjay
> 

Thanks!  On it's own it looks good.  I've added it to 
superm1/bleeding-edge for some more testing.

While testing I did uncover some other issues that I dropped some other 
patches for review if you can please look them over.

https://lore.kernel.org/linux-pm/20240702171515.6780-1-mario.limonciello@amd.com/T/#m1304e6bfe80b7a76e41d551784542fc1038e03e8

Thanks,

Re: [PATCH v2 2/2] cpufreq/amd-pstate: Fix the scaling_max_freq setting on shared memory CPPC systems

[Gautham R.Shenoy]

Dhananjay Ugwekar <Dhananjay.Ugwekar@amd.com> writes:

> On shared memory CPPC systems, with amd_pstate=active mode, the change
> in scaling_max_freq doesn't get written to the shared memory
> region. Due to this, the writes to the scaling_max_freq sysfs file
> don't take effect. Fix this by propagating the scaling_max_freq
> changes to the shared memory region.
>
> Fixes: ffa5096a7c33 ("cpufreq: amd-pstate: implement Pstate EPP support for the AMD processors")
> Reported-by: David Arcari <darcari@redhat.com>
> Signed-off-by: Dhananjay Ugwekar <Dhananjay.Ugwekar@amd.com>
> ---
>  drivers/cpufreq/amd-pstate.c | 43 +++++++++++++++++++-----------------
>  1 file changed, 23 insertions(+), 20 deletions(-)
>
> diff --git a/drivers/cpufreq/amd-pstate.c b/drivers/cpufreq/amd-pstate.c
> index 9ad62dbe8bfb..a092b13ffbc2 100644
> --- a/drivers/cpufreq/amd-pstate.c
> +++ b/drivers/cpufreq/amd-pstate.c
> @@ -247,6 +247,26 @@ static int amd_pstate_get_energy_pref_index(struct amd_cpudata *cpudata)
>  	return index;
>  }
>  
> +static void pstate_update_perf(struct amd_cpudata *cpudata, u32 min_perf,
> +			       u32 des_perf, u32 max_perf, bool fast_switch)
> +{
> +	if (fast_switch)
> +		wrmsrl(MSR_AMD_CPPC_REQ, READ_ONCE(cpudata->cppc_req_cached));
> +	else
> +		wrmsrl_on_cpu(cpudata->cpu, MSR_AMD_CPPC_REQ,
> +			      READ_ONCE(cpudata->cppc_req_cached));
> +}
> +
> +DEFINE_STATIC_CALL(amd_pstate_update_perf, pstate_update_perf);
> +
> +static inline void amd_pstate_update_perf(struct amd_cpudata *cpudata,
> +					  u32 min_perf, u32 des_perf,
> +					  u32 max_perf, bool fast_switch)
> +{
> +	static_call(amd_pstate_update_perf)(cpudata, min_perf, des_perf,
> +					    max_perf, fast_switch);
> +}
> +
>  static int amd_pstate_set_epp(struct amd_cpudata *cpudata, u32 epp)
>  {
>  	int ret;
> @@ -263,6 +283,9 @@ static int amd_pstate_set_epp(struct amd_cpudata *cpudata, u32 epp)
>  		if (!ret)
>  			cpudata->epp_cached = epp;
>  	} else {
> +		amd_pstate_update_perf(cpudata, cpudata->min_limit_perf, 0U,
> +					     cpudata->max_limit_perf, false);
> +

Looks good to me.

Reviewed-by: Gautham R. Shenoy <gautham.shenoy@amd.com>



>  		perf_ctrls.energy_perf = epp;
>  		ret = cppc_set_epp_perf(cpudata->cpu, &perf_ctrls, 1);
>  		if (ret) {
> @@ -452,16 +475,6 @@ static inline int amd_pstate_init_perf(struct amd_cpudata *cpudata)
>  	return static_call(amd_pstate_init_perf)(cpudata);
>  }
>  
> -static void pstate_update_perf(struct amd_cpudata *cpudata, u32 min_perf,
> -			       u32 des_perf, u32 max_perf, bool fast_switch)
> -{
> -	if (fast_switch)
> -		wrmsrl(MSR_AMD_CPPC_REQ, READ_ONCE(cpudata->cppc_req_cached));
> -	else
> -		wrmsrl_on_cpu(cpudata->cpu, MSR_AMD_CPPC_REQ,
> -			      READ_ONCE(cpudata->cppc_req_cached));
> -}
> -
>  static void cppc_update_perf(struct amd_cpudata *cpudata,
>  			     u32 min_perf, u32 des_perf,
>  			     u32 max_perf, bool fast_switch)
> @@ -475,16 +488,6 @@ static void cppc_update_perf(struct amd_cpudata *cpudata,
>  	cppc_set_perf(cpudata->cpu, &perf_ctrls);
>  }
>  
> -DEFINE_STATIC_CALL(amd_pstate_update_perf, pstate_update_perf);
> -
> -static inline void amd_pstate_update_perf(struct amd_cpudata *cpudata,
> -					  u32 min_perf, u32 des_perf,
> -					  u32 max_perf, bool fast_switch)
> -{
> -	static_call(amd_pstate_update_perf)(cpudata, min_perf, des_perf,
> -					    max_perf, fast_switch);
> -}
> -
>  static inline bool amd_pstate_sample(struct amd_cpudata *cpudata)
>  {
>  	u64 aperf, mperf, tsc;
> -- 
> 2.34.1

RE: [PATCH v2 2/2] cpufreq/amd-pstate: Fix the scaling_max_freq setting on shared memory CPPC systems

[Jones, Morgan]

Hi there,

We are experiencing a ~35% performance regression on an AMD EPYC 7702 64 core machine after applying this patch. We observed Linux 6.6.44 starting to cause the issue, and performed a bisect involving rebooting the machine over 15 times. (Note that kexec didn't successfully identify the problem, since the PM memory mailbox is never reset).

It appears that we are getting a max of 2.18 GHz when this CPU can boost to 3.35 GHz, explaining the slowdown. Blacklisting amd-pstate solves the issue at the expense of the performance increase we used to get by using it.

Is it possible that the upper limits were implicitly at the max CPU power before, and setting the upper limit to something other than the boost frequency can reduce performance now?

# bad: [7213910600667c51c978e577bf5454d3f7b313b7] Linux 6.6.44
# good: [58b0425ff5df680d0b67f64ae1f3f1ebdf1c4de9] Linux 6.6.43
git bisect start '7213910600667c51c978e577bf5454d3f7b313b7' '58b0425ff5df680d0b67f64ae1f3f1ebdf1c4de9'
# good: [72ff9d26964a3a80f7650df719df139f5c1f965d] arm64: dts: qcom: sm6350: Add missing qcom,non-secure-domain property
git bisect good 72ff9d26964a3a80f7650df719df139f5c1f965d
# good: [0fffc2e1bf40a2220ef5a38f834ea063dba832d3] ARM: dts: sunxi: remove duplicated entries in makefile
git bisect good 0fffc2e1bf40a2220ef5a38f834ea063dba832d3
# bad: [8cdbe6ebfd1763a5c41a2a3058497c0a9163311c] pinctrl: renesas: r8a779g0: Fix CANFD5 suffix
git bisect bad 8cdbe6ebfd1763a5c41a2a3058497c0a9163311c
# bad: [5dbb98e7fa42bebc1325899193d8f13f0705a148] drm/mediatek: Turn off the layers with zero width or height
git bisect bad 5dbb98e7fa42bebc1325899193d8f13f0705a148
# bad: [691ec7043122c9c8c46d84f6e6cd85d13d50cd93] selftests/bpf: Null checks for links in bpf_tcp_ca
git bisect bad 691ec7043122c9c8c46d84f6e6cd85d13d50cd93
# bad: [a1359e085d75d7393a250054e66c0a7bc6c3dbfa] perf/x86: Serialize set_attr_rdpmc()
git bisect bad a1359e085d75d7393a250054e66c0a7bc6c3dbfa
# bad: [e99d9b16ff153de9540073239d24adc3b0a3a997] wifi: ath12k: change DMA direction while mapping reinjected packets
git bisect bad e99d9b16ff153de9540073239d24adc3b0a3a997
# bad: [d027ac4a08541beb2a89563d3e034da7085050ba] firmware: turris-mox-rwtm: Initialize completion before mailbox
git bisect bad d027ac4a08541beb2a89563d3e034da7085050ba
# bad: [e6c9eca327e6a41a81e7eba0d0ddc13da37f82a1] ARM: spitz: fix GPIO assignment for backlight
git bisect bad e6c9eca327e6a41a81e7eba0d0ddc13da37f82a1
# bad: [b8cdefdaa555bbfc269c2198803f8791a8923960] m68k: cmpxchg: Fix return value for default case in __arch_xchg()
git bisect bad b8cdefdaa555bbfc269c2198803f8791a8923960
# bad: [13a71384ae6a8779da809b00c6f378dcead10427] cpufreq/amd-pstate: Fix the scaling_max_freq setting on shared memory CPPC systems
git bisect bad 13a71384ae6a8779da809b00c6f378dcead10427
# first bad commit: [13a71384ae6a8779da809b00c6f378dcead10427] cpufreq/amd-pstate: Fix the scaling_max_freq setting on shared memory CPPC systems  

cpupower output:

analyzing CPU 47:
  driver: amd-pstate-epp
  CPUs which run at the same hardware frequency: 47
  CPUs which need to have their frequency coordinated by software: 47
  maximum transition latency:  Cannot determine or is not supported.
  hardware limits: 400 MHz - 2.18 GHz
  available cpufreq governors: performance powersave
  current policy: frequency should be within 400 MHz and 2.18 GHz.
                  The governor "performance" may decide which speed to use
                  within this range.
  current CPU frequency: Unable to call hardware
  current CPU frequency: 2.17 GHz (asserted by call to kernel)
  boost state support:
    Supported: yes
    Active: yes
    AMD PSTATE Highest Performance: 166. Maximum Frequency: 2.18 GHz.
    AMD PSTATE Nominal Performance: 152. Nominal Frequency: 2.00 GHz.
    AMD PSTATE Lowest Non-linear Performance: 115. Lowest Non-linear Frequency: 1.51 GHz.
    AMD PSTATE Lowest Performance: 31. Lowest Frequency: 400 MHz.

Thanks,
Morgan

-----Original Message-----
From: Mario Limonciello <mario.limonciello@amd.com> 
Sent: Tuesday, July 2, 2024 10:49 AM
To: Dhananjay Ugwekar <Dhananjay.Ugwekar@amd.com>; rafael@kernel.org; viresh.kumar@linaro.org; gautham.shenoy@amd.com; perry.yuan@amd.com; skhan@linuxfoundation.org; li.meng@amd.com; ray.huang@amd.com
Cc: linux-pm@vger.kernel.org; linux-kernel@vger.kernel.org; David Arcari <darcari@redhat.com>
Subject: Re: [PATCH v2 2/2] cpufreq/amd-pstate: Fix the scaling_max_freq setting on shared memory CPPC systems

On 7/2/2024 3:14, Dhananjay Ugwekar wrote:
> On shared memory CPPC systems, with amd_pstate=active mode, the change 
> in scaling_max_freq doesn't get written to the shared memory region. 
> Due to this, the writes to the scaling_max_freq sysfs file don't take 
> effect. Fix this by propagating the scaling_max_freq changes to the 
> shared memory region.
> 
> Fixes: ffa5096a7c33 ("cpufreq: amd-pstate: implement Pstate EPP 
> support for the AMD processors")
> Reported-by: David Arcari <darcari@redhat.com>
> Signed-off-by: Dhananjay Ugwekar <Dhananjay.Ugwekar@amd.com>
Reviewed-by: Mario Limonciello <mario.limonciello@amd.com>
> ---
>   drivers/cpufreq/amd-pstate.c | 43 +++++++++++++++++++-----------------
>   1 file changed, 23 insertions(+), 20 deletions(-)
> 
> diff --git a/drivers/cpufreq/amd-pstate.c 
> b/drivers/cpufreq/amd-pstate.c index 9ad62dbe8bfb..a092b13ffbc2 100644
> --- a/drivers/cpufreq/amd-pstate.c
> +++ b/drivers/cpufreq/amd-pstate.c
> @@ -247,6 +247,26 @@ static int amd_pstate_get_energy_pref_index(struct amd_cpudata *cpudata)
>   	return index;
>   }
>   
> +static void pstate_update_perf(struct amd_cpudata *cpudata, u32 min_perf,
> +			       u32 des_perf, u32 max_perf, bool fast_switch) {
> +	if (fast_switch)
> +		wrmsrl(MSR_AMD_CPPC_REQ, READ_ONCE(cpudata->cppc_req_cached));
> +	else
> +		wrmsrl_on_cpu(cpudata->cpu, MSR_AMD_CPPC_REQ,
> +			      READ_ONCE(cpudata->cppc_req_cached));
> +}
> +
> +DEFINE_STATIC_CALL(amd_pstate_update_perf, pstate_update_perf);
> +
> +static inline void amd_pstate_update_perf(struct amd_cpudata *cpudata,
> +					  u32 min_perf, u32 des_perf,
> +					  u32 max_perf, bool fast_switch) {
> +	static_call(amd_pstate_update_perf)(cpudata, min_perf, des_perf,
> +					    max_perf, fast_switch);
> +}
> +
>   static int amd_pstate_set_epp(struct amd_cpudata *cpudata, u32 epp)
>   {
>   	int ret;
> @@ -263,6 +283,9 @@ static int amd_pstate_set_epp(struct amd_cpudata *cpudata, u32 epp)
>   		if (!ret)
>   			cpudata->epp_cached = epp;
>   	} else {
> +		amd_pstate_update_perf(cpudata, cpudata->min_limit_perf, 0U,
> +					     cpudata->max_limit_perf, false);
> +
>   		perf_ctrls.energy_perf = epp;
>   		ret = cppc_set_epp_perf(cpudata->cpu, &perf_ctrls, 1);
>   		if (ret) {
> @@ -452,16 +475,6 @@ static inline int amd_pstate_init_perf(struct amd_cpudata *cpudata)
>   	return static_call(amd_pstate_init_perf)(cpudata);
>   }
>   
> -static void pstate_update_perf(struct amd_cpudata *cpudata, u32 min_perf,
> -			       u32 des_perf, u32 max_perf, bool fast_switch)
> -{
> -	if (fast_switch)
> -		wrmsrl(MSR_AMD_CPPC_REQ, READ_ONCE(cpudata->cppc_req_cached));
> -	else
> -		wrmsrl_on_cpu(cpudata->cpu, MSR_AMD_CPPC_REQ,
> -			      READ_ONCE(cpudata->cppc_req_cached));
> -}
> -
>   static void cppc_update_perf(struct amd_cpudata *cpudata,
>   			     u32 min_perf, u32 des_perf,
>   			     u32 max_perf, bool fast_switch) @@ -475,16 +488,6 @@ static 
> void cppc_update_perf(struct amd_cpudata *cpudata,
>   	cppc_set_perf(cpudata->cpu, &perf_ctrls);
>   }
>   
> -DEFINE_STATIC_CALL(amd_pstate_update_perf, pstate_update_perf);
> -
> -static inline void amd_pstate_update_perf(struct amd_cpudata *cpudata,
> -					  u32 min_perf, u32 des_perf,
> -					  u32 max_perf, bool fast_switch)
> -{
> -	static_call(amd_pstate_update_perf)(cpudata, min_perf, des_perf,
> -					    max_perf, fast_switch);
> -}
> -
>   static inline bool amd_pstate_sample(struct amd_cpudata *cpudata)
>   {
>   	u64 aperf, mperf, tsc;

Re: [PATCH v2 2/2] cpufreq/amd-pstate: Fix the scaling_max_freq setting on shared memory CPPC systems

[Mario Limonciello]

Hi Morgan,

Can you please cross reference 6.11-rc6 to see if you're still having a 
problem?  I would like to understand if we're missing a backport to 
stable or this is still an upstream problem.

Thanks,

On 9/3/2024 12:51, Jones, Morgan wrote:
> Hi there,
> 
> We are experiencing a ~35% performance regression on an AMD EPYC 7702 64 core machine after applying this patch. We observed Linux 6.6.44 starting to cause the issue, and performed a bisect involving rebooting the machine over 15 times. (Note that kexec didn't successfully identify the problem, since the PM memory mailbox is never reset).
> 
> It appears that we are getting a max of 2.18 GHz when this CPU can boost to 3.35 GHz, explaining the slowdown. Blacklisting amd-pstate solves the issue at the expense of the performance increase we used to get by using it.
> 
> Is it possible that the upper limits were implicitly at the max CPU power before, and setting the upper limit to something other than the boost frequency can reduce performance now?
> 
> # bad: [7213910600667c51c978e577bf5454d3f7b313b7] Linux 6.6.44
> # good: [58b0425ff5df680d0b67f64ae1f3f1ebdf1c4de9] Linux 6.6.43
> git bisect start '7213910600667c51c978e577bf5454d3f7b313b7' '58b0425ff5df680d0b67f64ae1f3f1ebdf1c4de9'
> # good: [72ff9d26964a3a80f7650df719df139f5c1f965d] arm64: dts: qcom: sm6350: Add missing qcom,non-secure-domain property
> git bisect good 72ff9d26964a3a80f7650df719df139f5c1f965d
> # good: [0fffc2e1bf40a2220ef5a38f834ea063dba832d3] ARM: dts: sunxi: remove duplicated entries in makefile
> git bisect good 0fffc2e1bf40a2220ef5a38f834ea063dba832d3
> # bad: [8cdbe6ebfd1763a5c41a2a3058497c0a9163311c] pinctrl: renesas: r8a779g0: Fix CANFD5 suffix
> git bisect bad 8cdbe6ebfd1763a5c41a2a3058497c0a9163311c
> # bad: [5dbb98e7fa42bebc1325899193d8f13f0705a148] drm/mediatek: Turn off the layers with zero width or height
> git bisect bad 5dbb98e7fa42bebc1325899193d8f13f0705a148
> # bad: [691ec7043122c9c8c46d84f6e6cd85d13d50cd93] selftests/bpf: Null checks for links in bpf_tcp_ca
> git bisect bad 691ec7043122c9c8c46d84f6e6cd85d13d50cd93
> # bad: [a1359e085d75d7393a250054e66c0a7bc6c3dbfa] perf/x86: Serialize set_attr_rdpmc()
> git bisect bad a1359e085d75d7393a250054e66c0a7bc6c3dbfa
> # bad: [e99d9b16ff153de9540073239d24adc3b0a3a997] wifi: ath12k: change DMA direction while mapping reinjected packets
> git bisect bad e99d9b16ff153de9540073239d24adc3b0a3a997
> # bad: [d027ac4a08541beb2a89563d3e034da7085050ba] firmware: turris-mox-rwtm: Initialize completion before mailbox
> git bisect bad d027ac4a08541beb2a89563d3e034da7085050ba
> # bad: [e6c9eca327e6a41a81e7eba0d0ddc13da37f82a1] ARM: spitz: fix GPIO assignment for backlight
> git bisect bad e6c9eca327e6a41a81e7eba0d0ddc13da37f82a1
> # bad: [b8cdefdaa555bbfc269c2198803f8791a8923960] m68k: cmpxchg: Fix return value for default case in __arch_xchg()
> git bisect bad b8cdefdaa555bbfc269c2198803f8791a8923960
> # bad: [13a71384ae6a8779da809b00c6f378dcead10427] cpufreq/amd-pstate: Fix the scaling_max_freq setting on shared memory CPPC systems
> git bisect bad 13a71384ae6a8779da809b00c6f378dcead10427
> # first bad commit: [13a71384ae6a8779da809b00c6f378dcead10427] cpufreq/amd-pstate: Fix the scaling_max_freq setting on shared memory CPPC systems
> 
> cpupower output:
> 
> analyzing CPU 47:
>    driver: amd-pstate-epp
>    CPUs which run at the same hardware frequency: 47
>    CPUs which need to have their frequency coordinated by software: 47
>    maximum transition latency:  Cannot determine or is not supported.
>    hardware limits: 400 MHz - 2.18 GHz
>    available cpufreq governors: performance powersave
>    current policy: frequency should be within 400 MHz and 2.18 GHz.
>                    The governor "performance" may decide which speed to use
>                    within this range.
>    current CPU frequency: Unable to call hardware
>    current CPU frequency: 2.17 GHz (asserted by call to kernel)
>    boost state support:
>      Supported: yes
>      Active: yes
>      AMD PSTATE Highest Performance: 166. Maximum Frequency: 2.18 GHz.
>      AMD PSTATE Nominal Performance: 152. Nominal Frequency: 2.00 GHz.
>      AMD PSTATE Lowest Non-linear Performance: 115. Lowest Non-linear Frequency: 1.51 GHz.
>      AMD PSTATE Lowest Performance: 31. Lowest Frequency: 400 MHz.
> 
> Thanks,
> Morgan
> 
> -----Original Message-----
> From: Mario Limonciello <mario.limonciello@amd.com>
> Sent: Tuesday, July 2, 2024 10:49 AM
> To: Dhananjay Ugwekar <Dhananjay.Ugwekar@amd.com>; rafael@kernel.org; viresh.kumar@linaro.org; gautham.shenoy@amd.com; perry.yuan@amd.com; skhan@linuxfoundation.org; li.meng@amd.com; ray.huang@amd.com
> Cc: linux-pm@vger.kernel.org; linux-kernel@vger.kernel.org; David Arcari <darcari@redhat.com>
> Subject: Re: [PATCH v2 2/2] cpufreq/amd-pstate: Fix the scaling_max_freq setting on shared memory CPPC systems
> 
> On 7/2/2024 3:14, Dhananjay Ugwekar wrote:
>> On shared memory CPPC systems, with amd_pstate=active mode, the change
>> in scaling_max_freq doesn't get written to the shared memory region.
>> Due to this, the writes to the scaling_max_freq sysfs file don't take
>> effect. Fix this by propagating the scaling_max_freq changes to the
>> shared memory region.
>>
>> Fixes: ffa5096a7c33 ("cpufreq: amd-pstate: implement Pstate EPP
>> support for the AMD processors")
>> Reported-by: David Arcari <darcari@redhat.com>
>> Signed-off-by: Dhananjay Ugwekar <Dhananjay.Ugwekar@amd.com>
> Reviewed-by: Mario Limonciello <mario.limonciello@amd.com>
>> ---
>>    drivers/cpufreq/amd-pstate.c | 43 +++++++++++++++++++-----------------
>>    1 file changed, 23 insertions(+), 20 deletions(-)
>>
>> diff --git a/drivers/cpufreq/amd-pstate.c
>> b/drivers/cpufreq/amd-pstate.c index 9ad62dbe8bfb..a092b13ffbc2 100644
>> --- a/drivers/cpufreq/amd-pstate.c
>> +++ b/drivers/cpufreq/amd-pstate.c
>> @@ -247,6 +247,26 @@ static int amd_pstate_get_energy_pref_index(struct amd_cpudata *cpudata)
>>    	return index;
>>    }
>>    
>> +static void pstate_update_perf(struct amd_cpudata *cpudata, u32 min_perf,
>> +			       u32 des_perf, u32 max_perf, bool fast_switch) {
>> +	if (fast_switch)
>> +		wrmsrl(MSR_AMD_CPPC_REQ, READ_ONCE(cpudata->cppc_req_cached));
>> +	else
>> +		wrmsrl_on_cpu(cpudata->cpu, MSR_AMD_CPPC_REQ,
>> +			      READ_ONCE(cpudata->cppc_req_cached));
>> +}
>> +
>> +DEFINE_STATIC_CALL(amd_pstate_update_perf, pstate_update_perf);
>> +
>> +static inline void amd_pstate_update_perf(struct amd_cpudata *cpudata,
>> +					  u32 min_perf, u32 des_perf,
>> +					  u32 max_perf, bool fast_switch) {
>> +	static_call(amd_pstate_update_perf)(cpudata, min_perf, des_perf,
>> +					    max_perf, fast_switch);
>> +}
>> +
>>    static int amd_pstate_set_epp(struct amd_cpudata *cpudata, u32 epp)
>>    {
>>    	int ret;
>> @@ -263,6 +283,9 @@ static int amd_pstate_set_epp(struct amd_cpudata *cpudata, u32 epp)
>>    		if (!ret)
>>    			cpudata->epp_cached = epp;
>>    	} else {
>> +		amd_pstate_update_perf(cpudata, cpudata->min_limit_perf, 0U,
>> +					     cpudata->max_limit_perf, false);
>> +
>>    		perf_ctrls.energy_perf = epp;
>>    		ret = cppc_set_epp_perf(cpudata->cpu, &perf_ctrls, 1);
>>    		if (ret) {
>> @@ -452,16 +475,6 @@ static inline int amd_pstate_init_perf(struct amd_cpudata *cpudata)
>>    	return static_call(amd_pstate_init_perf)(cpudata);
>>    }
>>    
>> -static void pstate_update_perf(struct amd_cpudata *cpudata, u32 min_perf,
>> -			       u32 des_perf, u32 max_perf, bool fast_switch)
>> -{
>> -	if (fast_switch)
>> -		wrmsrl(MSR_AMD_CPPC_REQ, READ_ONCE(cpudata->cppc_req_cached));
>> -	else
>> -		wrmsrl_on_cpu(cpudata->cpu, MSR_AMD_CPPC_REQ,
>> -			      READ_ONCE(cpudata->cppc_req_cached));
>> -}
>> -
>>    static void cppc_update_perf(struct amd_cpudata *cpudata,
>>    			     u32 min_perf, u32 des_perf,
>>    			     u32 max_perf, bool fast_switch) @@ -475,16 +488,6 @@ static
>> void cppc_update_perf(struct amd_cpudata *cpudata,
>>    	cppc_set_perf(cpudata->cpu, &perf_ctrls);
>>    }
>>    
>> -DEFINE_STATIC_CALL(amd_pstate_update_perf, pstate_update_perf);
>> -
>> -static inline void amd_pstate_update_perf(struct amd_cpudata *cpudata,
>> -					  u32 min_perf, u32 des_perf,
>> -					  u32 max_perf, bool fast_switch)
>> -{
>> -	static_call(amd_pstate_update_perf)(cpudata, min_perf, des_perf,
>> -					    max_perf, fast_switch);
>> -}
>> -
>>    static inline bool amd_pstate_sample(struct amd_cpudata *cpudata)
>>    {
>>    	u64 aperf, mperf, tsc;
> 
> 

RE: [EXTERNAL] Re: [PATCH v2 2/2] cpufreq/amd-pstate: Fix the scaling_max_freq setting on shared memory CPPC systems

[Jones, Morgan]

Hey Mario,

Smoking gun here, the max frequency is incorrect on 6.6.44+ but is correct on 6.11.0-rc6.

Linux redact 6.11.0-rc6 #1-NixOS SMP PREEMPT_DYNAMIC Tue Jan  1 00:00:00 UTC 1980 x86_64 GNU/Linux

analyzing CPU 12:
  driver: amd-pstate-epp
  CPUs which run at the same hardware frequency: 12
  CPUs which need to have their frequency coordinated by software: 12
  maximum transition latency:  Cannot determine or is not supported.
  hardware limits: 400 MHz - 3.35 GHz
  available cpufreq governors: performance powersave
  current policy: frequency should be within 400 MHz and 3.35 GHz.
                  The governor "performance" may decide which speed to use
                  within this range.
  current CPU frequency: Unable to call hardware
  current CPU frequency: 3.34 GHz (asserted by call to kernel)
  boost state support:
    Supported: yes
    Active: yes
    AMD PSTATE Highest Performance: 255. Maximum Frequency: 3.35 GHz.
    AMD PSTATE Nominal Performance: 152. Nominal Frequency: 2.00 GHz.
    AMD PSTATE Lowest Non-linear Performance: 115. Lowest Non-linear Frequency: 1.51 GHz.
    AMD PSTATE Lowest Performance: 31. Lowest Frequency: 400 MHz.

We're running amd_pstate=active and amd_pstate.shared_mem=1, and our workloads are back to normal performance on 6.11.0-rc6.

Morgan

-----Original Message-----
From: Mario Limonciello <mario.limonciello@amd.com> 
Sent: Tuesday, September 3, 2024 10:55 AM
To: Jones, Morgan <Morgan.Jones@viasat.com>; Dhananjay Ugwekar <Dhananjay.Ugwekar@amd.com>; rafael@kernel.org; viresh.kumar@linaro.org; gautham.shenoy@amd.com; perry.yuan@amd.com; skhan@linuxfoundation.org; li.meng@amd.com; ray.huang@amd.com
Cc: linux-pm@vger.kernel.org; linux-kernel@vger.kernel.org; David Arcari <darcari@redhat.com>
Subject: [EXTERNAL] Re: [PATCH v2 2/2] cpufreq/amd-pstate: Fix the scaling_max_freq setting on shared memory CPPC systems

Hi Morgan,

Can you please cross reference 6.11-rc6 to see if you're still having a problem?  I would like to understand if we're missing a backport to stable or this is still an upstream problem.

Thanks,

On 9/3/2024 12:51, Jones, Morgan wrote:
> Hi there,
> 
> We are experiencing a ~35% performance regression on an AMD EPYC 7702 64 core machine after applying this patch. We observed Linux 6.6.44 starting to cause the issue, and performed a bisect involving rebooting the machine over 15 times. (Note that kexec didn't successfully identify the problem, since the PM memory mailbox is never reset).
> 
> It appears that we are getting a max of 2.18 GHz when this CPU can boost to 3.35 GHz, explaining the slowdown. Blacklisting amd-pstate solves the issue at the expense of the performance increase we used to get by using it.
> 
> Is it possible that the upper limits were implicitly at the max CPU power before, and setting the upper limit to something other than the boost frequency can reduce performance now?
> 
> # bad: [7213910600667c51c978e577bf5454d3f7b313b7] Linux 6.6.44 # good: 
> [58b0425ff5df680d0b67f64ae1f3f1ebdf1c4de9] Linux 6.6.43 git bisect 
> start '7213910600667c51c978e577bf5454d3f7b313b7' '58b0425ff5df680d0b67f64ae1f3f1ebdf1c4de9'
> # good: [72ff9d26964a3a80f7650df719df139f5c1f965d] arm64: dts: qcom: 
> sm6350: Add missing qcom,non-secure-domain property git bisect good 
> 72ff9d26964a3a80f7650df719df139f5c1f965d
> # good: [0fffc2e1bf40a2220ef5a38f834ea063dba832d3] ARM: dts: sunxi: 
> remove duplicated entries in makefile git bisect good 
> 0fffc2e1bf40a2220ef5a38f834ea063dba832d3
> # bad: [8cdbe6ebfd1763a5c41a2a3058497c0a9163311c] pinctrl: renesas: 
> r8a779g0: Fix CANFD5 suffix git bisect bad 
> 8cdbe6ebfd1763a5c41a2a3058497c0a9163311c
> # bad: [5dbb98e7fa42bebc1325899193d8f13f0705a148] drm/mediatek: Turn 
> off the layers with zero width or height git bisect bad 
> 5dbb98e7fa42bebc1325899193d8f13f0705a148
> # bad: [691ec7043122c9c8c46d84f6e6cd85d13d50cd93] selftests/bpf: Null 
> checks for links in bpf_tcp_ca git bisect bad 
> 691ec7043122c9c8c46d84f6e6cd85d13d50cd93
> # bad: [a1359e085d75d7393a250054e66c0a7bc6c3dbfa] perf/x86: Serialize 
> set_attr_rdpmc() git bisect bad 
> a1359e085d75d7393a250054e66c0a7bc6c3dbfa
> # bad: [e99d9b16ff153de9540073239d24adc3b0a3a997] wifi: ath12k: change 
> DMA direction while mapping reinjected packets git bisect bad 
> e99d9b16ff153de9540073239d24adc3b0a3a997
> # bad: [d027ac4a08541beb2a89563d3e034da7085050ba] firmware: 
> turris-mox-rwtm: Initialize completion before mailbox git bisect bad 
> d027ac4a08541beb2a89563d3e034da7085050ba
> # bad: [e6c9eca327e6a41a81e7eba0d0ddc13da37f82a1] ARM: spitz: fix GPIO 
> assignment for backlight git bisect bad 
> e6c9eca327e6a41a81e7eba0d0ddc13da37f82a1
> # bad: [b8cdefdaa555bbfc269c2198803f8791a8923960] m68k: cmpxchg: Fix 
> return value for default case in __arch_xchg() git bisect bad 
> b8cdefdaa555bbfc269c2198803f8791a8923960
> # bad: [13a71384ae6a8779da809b00c6f378dcead10427] cpufreq/amd-pstate: 
> Fix the scaling_max_freq setting on shared memory CPPC systems git 
> bisect bad 13a71384ae6a8779da809b00c6f378dcead10427
> # first bad commit: [13a71384ae6a8779da809b00c6f378dcead10427] 
> cpufreq/amd-pstate: Fix the scaling_max_freq setting on shared memory 
> CPPC systems
> 
> cpupower output:
> 
> analyzing CPU 47:
>    driver: amd-pstate-epp
>    CPUs which run at the same hardware frequency: 47
>    CPUs which need to have their frequency coordinated by software: 47
>    maximum transition latency:  Cannot determine or is not supported.
>    hardware limits: 400 MHz - 2.18 GHz
>    available cpufreq governors: performance powersave
>    current policy: frequency should be within 400 MHz and 2.18 GHz.
>                    The governor "performance" may decide which speed to use
>                    within this range.
>    current CPU frequency: Unable to call hardware
>    current CPU frequency: 2.17 GHz (asserted by call to kernel)
>    boost state support:
>      Supported: yes
>      Active: yes
>      AMD PSTATE Highest Performance: 166. Maximum Frequency: 2.18 GHz.
>      AMD PSTATE Nominal Performance: 152. Nominal Frequency: 2.00 GHz.
>      AMD PSTATE Lowest Non-linear Performance: 115. Lowest Non-linear Frequency: 1.51 GHz.
>      AMD PSTATE Lowest Performance: 31. Lowest Frequency: 400 MHz.
> 
> Thanks,
> Morgan
> 
> -----Original Message-----
> From: Mario Limonciello <mario.limonciello@amd.com>
> Sent: Tuesday, July 2, 2024 10:49 AM
> To: Dhananjay Ugwekar <Dhananjay.Ugwekar@amd.com>; rafael@kernel.org; 
> viresh.kumar@linaro.org; gautham.shenoy@amd.com; perry.yuan@amd.com; 
> skhan@linuxfoundation.org; li.meng@amd.com; ray.huang@amd.com
> Cc: linux-pm@vger.kernel.org; linux-kernel@vger.kernel.org; David 
> Arcari <darcari@redhat.com>
> Subject: Re: [PATCH v2 2/2] cpufreq/amd-pstate: Fix the 
> scaling_max_freq setting on shared memory CPPC systems
> 
> On 7/2/2024 3:14, Dhananjay Ugwekar wrote:
>> On shared memory CPPC systems, with amd_pstate=active mode, the 
>> change in scaling_max_freq doesn't get written to the shared memory region.
>> Due to this, the writes to the scaling_max_freq sysfs file don't take 
>> effect. Fix this by propagating the scaling_max_freq changes to the 
>> shared memory region.
>>
>> Fixes: ffa5096a7c33 ("cpufreq: amd-pstate: implement Pstate EPP 
>> support for the AMD processors")
>> Reported-by: David Arcari <darcari@redhat.com>
>> Signed-off-by: Dhananjay Ugwekar <Dhananjay.Ugwekar@amd.com>
> Reviewed-by: Mario Limonciello <mario.limonciello@amd.com>
>> ---
>>    drivers/cpufreq/amd-pstate.c | 43 +++++++++++++++++++-----------------
>>    1 file changed, 23 insertions(+), 20 deletions(-)
>>
>> diff --git a/drivers/cpufreq/amd-pstate.c 
>> b/drivers/cpufreq/amd-pstate.c index 9ad62dbe8bfb..a092b13ffbc2 
>> 100644
>> --- a/drivers/cpufreq/amd-pstate.c
>> +++ b/drivers/cpufreq/amd-pstate.c
>> @@ -247,6 +247,26 @@ static int amd_pstate_get_energy_pref_index(struct amd_cpudata *cpudata)
>>    	return index;
>>    }
>>    
>> +static void pstate_update_perf(struct amd_cpudata *cpudata, u32 min_perf,
>> +			       u32 des_perf, u32 max_perf, bool fast_switch) {
>> +	if (fast_switch)
>> +		wrmsrl(MSR_AMD_CPPC_REQ, READ_ONCE(cpudata->cppc_req_cached));
>> +	else
>> +		wrmsrl_on_cpu(cpudata->cpu, MSR_AMD_CPPC_REQ,
>> +			      READ_ONCE(cpudata->cppc_req_cached));
>> +}
>> +
>> +DEFINE_STATIC_CALL(amd_pstate_update_perf, pstate_update_perf);
>> +
>> +static inline void amd_pstate_update_perf(struct amd_cpudata *cpudata,
>> +					  u32 min_perf, u32 des_perf,
>> +					  u32 max_perf, bool fast_switch) {
>> +	static_call(amd_pstate_update_perf)(cpudata, min_perf, des_perf,
>> +					    max_perf, fast_switch);
>> +}
>> +
>>    static int amd_pstate_set_epp(struct amd_cpudata *cpudata, u32 epp)
>>    {
>>    	int ret;
>> @@ -263,6 +283,9 @@ static int amd_pstate_set_epp(struct amd_cpudata *cpudata, u32 epp)
>>    		if (!ret)
>>    			cpudata->epp_cached = epp;
>>    	} else {
>> +		amd_pstate_update_perf(cpudata, cpudata->min_limit_perf, 0U,
>> +					     cpudata->max_limit_perf, false);
>> +
>>    		perf_ctrls.energy_perf = epp;
>>    		ret = cppc_set_epp_perf(cpudata->cpu, &perf_ctrls, 1);
>>    		if (ret) {
>> @@ -452,16 +475,6 @@ static inline int amd_pstate_init_perf(struct amd_cpudata *cpudata)
>>    	return static_call(amd_pstate_init_perf)(cpudata);
>>    }
>>    
>> -static void pstate_update_perf(struct amd_cpudata *cpudata, u32 min_perf,
>> -			       u32 des_perf, u32 max_perf, bool fast_switch)
>> -{
>> -	if (fast_switch)
>> -		wrmsrl(MSR_AMD_CPPC_REQ, READ_ONCE(cpudata->cppc_req_cached));
>> -	else
>> -		wrmsrl_on_cpu(cpudata->cpu, MSR_AMD_CPPC_REQ,
>> -			      READ_ONCE(cpudata->cppc_req_cached));
>> -}
>> -
>>    static void cppc_update_perf(struct amd_cpudata *cpudata,
>>    			     u32 min_perf, u32 des_perf,
>>    			     u32 max_perf, bool fast_switch) @@ -475,16 +488,6 @@ 
>> static void cppc_update_perf(struct amd_cpudata *cpudata,
>>    	cppc_set_perf(cpudata->cpu, &perf_ctrls);
>>    }
>>    
>> -DEFINE_STATIC_CALL(amd_pstate_update_perf, pstate_update_perf);
>> -
>> -static inline void amd_pstate_update_perf(struct amd_cpudata *cpudata,
>> -					  u32 min_perf, u32 des_perf,
>> -					  u32 max_perf, bool fast_switch)
>> -{
>> -	static_call(amd_pstate_update_perf)(cpudata, min_perf, des_perf,
>> -					    max_perf, fast_switch);
>> -}
>> -
>>    static inline bool amd_pstate_sample(struct amd_cpudata *cpudata)
>>    {
>>    	u64 aperf, mperf, tsc;
> 
> 

Re: [EXTERNAL] Re: [PATCH v2 2/2] cpufreq/amd-pstate: Fix the scaling_max_freq setting on shared memory CPPC systems

[Mario Limonciello]

Morgan,

OK that's great news that it's just a backport effort.  That same commit 
also backported to 6.10.3.  Can you see if 6.10.y is affected?

Ugwekar,

Any thoughts on what else needs to come back to 6.6.y off hand?

Thanks,

On 9/3/2024 15:07, Jones, Morgan wrote:
> Hey Mario,
> 
> Smoking gun here, the max frequency is incorrect on 6.6.44+ but is correct on 6.11.0-rc6.
> 
> Linux redact 6.11.0-rc6 #1-NixOS SMP PREEMPT_DYNAMIC Tue Jan  1 00:00:00 UTC 1980 x86_64 GNU/Linux
> 
> analyzing CPU 12:
>    driver: amd-pstate-epp
>    CPUs which run at the same hardware frequency: 12
>    CPUs which need to have their frequency coordinated by software: 12
>    maximum transition latency:  Cannot determine or is not supported.
>    hardware limits: 400 MHz - 3.35 GHz
>    available cpufreq governors: performance powersave
>    current policy: frequency should be within 400 MHz and 3.35 GHz.
>                    The governor "performance" may decide which speed to use
>                    within this range.
>    current CPU frequency: Unable to call hardware
>    current CPU frequency: 3.34 GHz (asserted by call to kernel)
>    boost state support:
>      Supported: yes
>      Active: yes
>      AMD PSTATE Highest Performance: 255. Maximum Frequency: 3.35 GHz.
>      AMD PSTATE Nominal Performance: 152. Nominal Frequency: 2.00 GHz.
>      AMD PSTATE Lowest Non-linear Performance: 115. Lowest Non-linear Frequency: 1.51 GHz.
>      AMD PSTATE Lowest Performance: 31. Lowest Frequency: 400 MHz.
> 
> We're running amd_pstate=active and amd_pstate.shared_mem=1, and our workloads are back to normal performance on 6.11.0-rc6.
> 
> Morgan
> 
> -----Original Message-----
> From: Mario Limonciello <mario.limonciello@amd.com>
> Sent: Tuesday, September 3, 2024 10:55 AM
> To: Jones, Morgan <Morgan.Jones@viasat.com>; Dhananjay Ugwekar <Dhananjay.Ugwekar@amd.com>; rafael@kernel.org; viresh.kumar@linaro.org; gautham.shenoy@amd.com; perry.yuan@amd.com; skhan@linuxfoundation.org; li.meng@amd.com; ray.huang@amd.com
> Cc: linux-pm@vger.kernel.org; linux-kernel@vger.kernel.org; David Arcari <darcari@redhat.com>
> Subject: [EXTERNAL] Re: [PATCH v2 2/2] cpufreq/amd-pstate: Fix the scaling_max_freq setting on shared memory CPPC systems
> 
> Hi Morgan,
> 
> Can you please cross reference 6.11-rc6 to see if you're still having a problem?  I would like to understand if we're missing a backport to stable or this is still an upstream problem.
> 
> Thanks,
> 
> On 9/3/2024 12:51, Jones, Morgan wrote:
>> Hi there,
>>
>> We are experiencing a ~35% performance regression on an AMD EPYC 7702 64 core machine after applying this patch. We observed Linux 6.6.44 starting to cause the issue, and performed a bisect involving rebooting the machine over 15 times. (Note that kexec didn't successfully identify the problem, since the PM memory mailbox is never reset).
>>
>> It appears that we are getting a max of 2.18 GHz when this CPU can boost to 3.35 GHz, explaining the slowdown. Blacklisting amd-pstate solves the issue at the expense of the performance increase we used to get by using it.
>>
>> Is it possible that the upper limits were implicitly at the max CPU power before, and setting the upper limit to something other than the boost frequency can reduce performance now?
>>
>> # bad: [7213910600667c51c978e577bf5454d3f7b313b7] Linux 6.6.44 # good:
>> [58b0425ff5df680d0b67f64ae1f3f1ebdf1c4de9] Linux 6.6.43 git bisect
>> start '7213910600667c51c978e577bf5454d3f7b313b7' '58b0425ff5df680d0b67f64ae1f3f1ebdf1c4de9'
>> # good: [72ff9d26964a3a80f7650df719df139f5c1f965d] arm64: dts: qcom:
>> sm6350: Add missing qcom,non-secure-domain property git bisect good
>> 72ff9d26964a3a80f7650df719df139f5c1f965d
>> # good: [0fffc2e1bf40a2220ef5a38f834ea063dba832d3] ARM: dts: sunxi:
>> remove duplicated entries in makefile git bisect good
>> 0fffc2e1bf40a2220ef5a38f834ea063dba832d3
>> # bad: [8cdbe6ebfd1763a5c41a2a3058497c0a9163311c] pinctrl: renesas:
>> r8a779g0: Fix CANFD5 suffix git bisect bad
>> 8cdbe6ebfd1763a5c41a2a3058497c0a9163311c
>> # bad: [5dbb98e7fa42bebc1325899193d8f13f0705a148] drm/mediatek: Turn
>> off the layers with zero width or height git bisect bad
>> 5dbb98e7fa42bebc1325899193d8f13f0705a148
>> # bad: [691ec7043122c9c8c46d84f6e6cd85d13d50cd93] selftests/bpf: Null
>> checks for links in bpf_tcp_ca git bisect bad
>> 691ec7043122c9c8c46d84f6e6cd85d13d50cd93
>> # bad: [a1359e085d75d7393a250054e66c0a7bc6c3dbfa] perf/x86: Serialize
>> set_attr_rdpmc() git bisect bad
>> a1359e085d75d7393a250054e66c0a7bc6c3dbfa
>> # bad: [e99d9b16ff153de9540073239d24adc3b0a3a997] wifi: ath12k: change
>> DMA direction while mapping reinjected packets git bisect bad
>> e99d9b16ff153de9540073239d24adc3b0a3a997
>> # bad: [d027ac4a08541beb2a89563d3e034da7085050ba] firmware:
>> turris-mox-rwtm: Initialize completion before mailbox git bisect bad
>> d027ac4a08541beb2a89563d3e034da7085050ba
>> # bad: [e6c9eca327e6a41a81e7eba0d0ddc13da37f82a1] ARM: spitz: fix GPIO
>> assignment for backlight git bisect bad
>> e6c9eca327e6a41a81e7eba0d0ddc13da37f82a1
>> # bad: [b8cdefdaa555bbfc269c2198803f8791a8923960] m68k: cmpxchg: Fix
>> return value for default case in __arch_xchg() git bisect bad
>> b8cdefdaa555bbfc269c2198803f8791a8923960
>> # bad: [13a71384ae6a8779da809b00c6f378dcead10427] cpufreq/amd-pstate:
>> Fix the scaling_max_freq setting on shared memory CPPC systems git
>> bisect bad 13a71384ae6a8779da809b00c6f378dcead10427
>> # first bad commit: [13a71384ae6a8779da809b00c6f378dcead10427]
>> cpufreq/amd-pstate: Fix the scaling_max_freq setting on shared memory
>> CPPC systems
>>
>> cpupower output:
>>
>> analyzing CPU 47:
>>     driver: amd-pstate-epp
>>     CPUs which run at the same hardware frequency: 47
>>     CPUs which need to have their frequency coordinated by software: 47
>>     maximum transition latency:  Cannot determine or is not supported.
>>     hardware limits: 400 MHz - 2.18 GHz
>>     available cpufreq governors: performance powersave
>>     current policy: frequency should be within 400 MHz and 2.18 GHz.
>>                     The governor "performance" may decide which speed to use
>>                     within this range.
>>     current CPU frequency: Unable to call hardware
>>     current CPU frequency: 2.17 GHz (asserted by call to kernel)
>>     boost state support:
>>       Supported: yes
>>       Active: yes
>>       AMD PSTATE Highest Performance: 166. Maximum Frequency: 2.18 GHz.
>>       AMD PSTATE Nominal Performance: 152. Nominal Frequency: 2.00 GHz.
>>       AMD PSTATE Lowest Non-linear Performance: 115. Lowest Non-linear Frequency: 1.51 GHz.
>>       AMD PSTATE Lowest Performance: 31. Lowest Frequency: 400 MHz.
>>
>> Thanks,
>> Morgan
>>
>> -----Original Message-----
>> From: Mario Limonciello <mario.limonciello@amd.com>
>> Sent: Tuesday, July 2, 2024 10:49 AM
>> To: Dhananjay Ugwekar <Dhananjay.Ugwekar@amd.com>; rafael@kernel.org;
>> viresh.kumar@linaro.org; gautham.shenoy@amd.com; perry.yuan@amd.com;
>> skhan@linuxfoundation.org; li.meng@amd.com; ray.huang@amd.com
>> Cc: linux-pm@vger.kernel.org; linux-kernel@vger.kernel.org; David
>> Arcari <darcari@redhat.com>
>> Subject: Re: [PATCH v2 2/2] cpufreq/amd-pstate: Fix the
>> scaling_max_freq setting on shared memory CPPC systems
>>
>> On 7/2/2024 3:14, Dhananjay Ugwekar wrote:
>>> On shared memory CPPC systems, with amd_pstate=active mode, the
>>> change in scaling_max_freq doesn't get written to the shared memory region.
>>> Due to this, the writes to the scaling_max_freq sysfs file don't take
>>> effect. Fix this by propagating the scaling_max_freq changes to the
>>> shared memory region.
>>>
>>> Fixes: ffa5096a7c33 ("cpufreq: amd-pstate: implement Pstate EPP
>>> support for the AMD processors")
>>> Reported-by: David Arcari <darcari@redhat.com>
>>> Signed-off-by: Dhananjay Ugwekar <Dhananjay.Ugwekar@amd.com>
>> Reviewed-by: Mario Limonciello <mario.limonciello@amd.com>
>>> ---
>>>     drivers/cpufreq/amd-pstate.c | 43 +++++++++++++++++++-----------------
>>>     1 file changed, 23 insertions(+), 20 deletions(-)
>>>
>>> diff --git a/drivers/cpufreq/amd-pstate.c
>>> b/drivers/cpufreq/amd-pstate.c index 9ad62dbe8bfb..a092b13ffbc2
>>> 100644
>>> --- a/drivers/cpufreq/amd-pstate.c
>>> +++ b/drivers/cpufreq/amd-pstate.c
>>> @@ -247,6 +247,26 @@ static int amd_pstate_get_energy_pref_index(struct amd_cpudata *cpudata)
>>>     	return index;
>>>     }
>>>     
>>> +static void pstate_update_perf(struct amd_cpudata *cpudata, u32 min_perf,
>>> +			       u32 des_perf, u32 max_perf, bool fast_switch) {
>>> +	if (fast_switch)
>>> +		wrmsrl(MSR_AMD_CPPC_REQ, READ_ONCE(cpudata->cppc_req_cached));
>>> +	else
>>> +		wrmsrl_on_cpu(cpudata->cpu, MSR_AMD_CPPC_REQ,
>>> +			      READ_ONCE(cpudata->cppc_req_cached));
>>> +}
>>> +
>>> +DEFINE_STATIC_CALL(amd_pstate_update_perf, pstate_update_perf);
>>> +
>>> +static inline void amd_pstate_update_perf(struct amd_cpudata *cpudata,
>>> +					  u32 min_perf, u32 des_perf,
>>> +					  u32 max_perf, bool fast_switch) {
>>> +	static_call(amd_pstate_update_perf)(cpudata, min_perf, des_perf,
>>> +					    max_perf, fast_switch);
>>> +}
>>> +
>>>     static int amd_pstate_set_epp(struct amd_cpudata *cpudata, u32 epp)
>>>     {
>>>     	int ret;
>>> @@ -263,6 +283,9 @@ static int amd_pstate_set_epp(struct amd_cpudata *cpudata, u32 epp)
>>>     		if (!ret)
>>>     			cpudata->epp_cached = epp;
>>>     	} else {
>>> +		amd_pstate_update_perf(cpudata, cpudata->min_limit_perf, 0U,
>>> +					     cpudata->max_limit_perf, false);
>>> +
>>>     		perf_ctrls.energy_perf = epp;
>>>     		ret = cppc_set_epp_perf(cpudata->cpu, &perf_ctrls, 1);
>>>     		if (ret) {
>>> @@ -452,16 +475,6 @@ static inline int amd_pstate_init_perf(struct amd_cpudata *cpudata)
>>>     	return static_call(amd_pstate_init_perf)(cpudata);
>>>     }
>>>     
>>> -static void pstate_update_perf(struct amd_cpudata *cpudata, u32 min_perf,
>>> -			       u32 des_perf, u32 max_perf, bool fast_switch)
>>> -{
>>> -	if (fast_switch)
>>> -		wrmsrl(MSR_AMD_CPPC_REQ, READ_ONCE(cpudata->cppc_req_cached));
>>> -	else
>>> -		wrmsrl_on_cpu(cpudata->cpu, MSR_AMD_CPPC_REQ,
>>> -			      READ_ONCE(cpudata->cppc_req_cached));
>>> -}
>>> -
>>>     static void cppc_update_perf(struct amd_cpudata *cpudata,
>>>     			     u32 min_perf, u32 des_perf,
>>>     			     u32 max_perf, bool fast_switch) @@ -475,16 +488,6 @@
>>> static void cppc_update_perf(struct amd_cpudata *cpudata,
>>>     	cppc_set_perf(cpudata->cpu, &perf_ctrls);
>>>     }
>>>     
>>> -DEFINE_STATIC_CALL(amd_pstate_update_perf, pstate_update_perf);
>>> -
>>> -static inline void amd_pstate_update_perf(struct amd_cpudata *cpudata,
>>> -					  u32 min_perf, u32 des_perf,
>>> -					  u32 max_perf, bool fast_switch)
>>> -{
>>> -	static_call(amd_pstate_update_perf)(cpudata, min_perf, des_perf,
>>> -					    max_perf, fast_switch);
>>> -}
>>> -
>>>     static inline bool amd_pstate_sample(struct amd_cpudata *cpudata)
>>>     {
>>>     	u64 aperf, mperf, tsc;
>>
>>
> 

Re: [EXTERNAL] Re: [PATCH v2 2/2] cpufreq/amd-pstate: Fix the scaling_max_freq setting on shared memory CPPC systems

[Mario Limonciello]

Morgan,

This does remind me of a clamping issue that touches some of the same 
variables.  Can you please see if backporting

commit 8164f7433264 ("cpufreq: amd-pstate: adjust min/max limit perf")

to 6.6.y helps for you?

Thanks,

On 9/3/2024 15:09, Mario Limonciello wrote:
> Morgan,
> 
> OK that's great news that it's just a backport effort.  That same commit 
> also backported to 6.10.3.  Can you see if 6.10.y is affected?
> 
> Ugwekar,
> 
> Any thoughts on what else needs to come back to 6.6.y off hand?
> 
> Thanks,
> 
> On 9/3/2024 15:07, Jones, Morgan wrote:
>> Hey Mario,
>>
>> Smoking gun here, the max frequency is incorrect on 6.6.44+ but is 
>> correct on 6.11.0-rc6.
>>
>> Linux redact 6.11.0-rc6 #1-NixOS SMP PREEMPT_DYNAMIC Tue Jan  1 
>> 00:00:00 UTC 1980 x86_64 GNU/Linux
>>
>> analyzing CPU 12:
>>    driver: amd-pstate-epp
>>    CPUs which run at the same hardware frequency: 12
>>    CPUs which need to have their frequency coordinated by software: 12
>>    maximum transition latency:  Cannot determine or is not supported.
>>    hardware limits: 400 MHz - 3.35 GHz
>>    available cpufreq governors: performance powersave
>>    current policy: frequency should be within 400 MHz and 3.35 GHz.
>>                    The governor "performance" may decide which speed 
>> to use
>>                    within this range.
>>    current CPU frequency: Unable to call hardware
>>    current CPU frequency: 3.34 GHz (asserted by call to kernel)
>>    boost state support:
>>      Supported: yes
>>      Active: yes
>>      AMD PSTATE Highest Performance: 255. Maximum Frequency: 3.35 GHz.
>>      AMD PSTATE Nominal Performance: 152. Nominal Frequency: 2.00 GHz.
>>      AMD PSTATE Lowest Non-linear Performance: 115. Lowest Non-linear 
>> Frequency: 1.51 GHz.
>>      AMD PSTATE Lowest Performance: 31. Lowest Frequency: 400 MHz.
>>
>> We're running amd_pstate=active and amd_pstate.shared_mem=1, and our 
>> workloads are back to normal performance on 6.11.0-rc6.
>>
>> Morgan
>>
>> -----Original Message-----
>> From: Mario Limonciello <mario.limonciello@amd.com>
>> Sent: Tuesday, September 3, 2024 10:55 AM
>> To: Jones, Morgan <Morgan.Jones@viasat.com>; Dhananjay Ugwekar 
>> <Dhananjay.Ugwekar@amd.com>; rafael@kernel.org; 
>> viresh.kumar@linaro.org; gautham.shenoy@amd.com; perry.yuan@amd.com; 
>> skhan@linuxfoundation.org; li.meng@amd.com; ray.huang@amd.com
>> Cc: linux-pm@vger.kernel.org; linux-kernel@vger.kernel.org; David 
>> Arcari <darcari@redhat.com>
>> Subject: [EXTERNAL] Re: [PATCH v2 2/2] cpufreq/amd-pstate: Fix the 
>> scaling_max_freq setting on shared memory CPPC systems
>>
>> Hi Morgan,
>>
>> Can you please cross reference 6.11-rc6 to see if you're still having 
>> a problem?  I would like to understand if we're missing a backport to 
>> stable or this is still an upstream problem.
>>
>> Thanks,
>>
>> On 9/3/2024 12:51, Jones, Morgan wrote:
>>> Hi there,
>>>
>>> We are experiencing a ~35% performance regression on an AMD EPYC 7702 
>>> 64 core machine after applying this patch. We observed Linux 6.6.44 
>>> starting to cause the issue, and performed a bisect involving 
>>> rebooting the machine over 15 times. (Note that kexec didn't 
>>> successfully identify the problem, since the PM memory mailbox is 
>>> never reset).
>>>
>>> It appears that we are getting a max of 2.18 GHz when this CPU can 
>>> boost to 3.35 GHz, explaining the slowdown. Blacklisting amd-pstate 
>>> solves the issue at the expense of the performance increase we used 
>>> to get by using it.
>>>
>>> Is it possible that the upper limits were implicitly at the max CPU 
>>> power before, and setting the upper limit to something other than the 
>>> boost frequency can reduce performance now?
>>>
>>> # bad: [7213910600667c51c978e577bf5454d3f7b313b7] Linux 6.6.44 # good:
>>> [58b0425ff5df680d0b67f64ae1f3f1ebdf1c4de9] Linux 6.6.43 git bisect
>>> start '7213910600667c51c978e577bf5454d3f7b313b7' 
>>> '58b0425ff5df680d0b67f64ae1f3f1ebdf1c4de9'
>>> # good: [72ff9d26964a3a80f7650df719df139f5c1f965d] arm64: dts: qcom:
>>> sm6350: Add missing qcom,non-secure-domain property git bisect good
>>> 72ff9d26964a3a80f7650df719df139f5c1f965d
>>> # good: [0fffc2e1bf40a2220ef5a38f834ea063dba832d3] ARM: dts: sunxi:
>>> remove duplicated entries in makefile git bisect good
>>> 0fffc2e1bf40a2220ef5a38f834ea063dba832d3
>>> # bad: [8cdbe6ebfd1763a5c41a2a3058497c0a9163311c] pinctrl: renesas:
>>> r8a779g0: Fix CANFD5 suffix git bisect bad
>>> 8cdbe6ebfd1763a5c41a2a3058497c0a9163311c
>>> # bad: [5dbb98e7fa42bebc1325899193d8f13f0705a148] drm/mediatek: Turn
>>> off the layers with zero width or height git bisect bad
>>> 5dbb98e7fa42bebc1325899193d8f13f0705a148
>>> # bad: [691ec7043122c9c8c46d84f6e6cd85d13d50cd93] selftests/bpf: Null
>>> checks for links in bpf_tcp_ca git bisect bad
>>> 691ec7043122c9c8c46d84f6e6cd85d13d50cd93
>>> # bad: [a1359e085d75d7393a250054e66c0a7bc6c3dbfa] perf/x86: Serialize
>>> set_attr_rdpmc() git bisect bad
>>> a1359e085d75d7393a250054e66c0a7bc6c3dbfa
>>> # bad: [e99d9b16ff153de9540073239d24adc3b0a3a997] wifi: ath12k: change
>>> DMA direction while mapping reinjected packets git bisect bad
>>> e99d9b16ff153de9540073239d24adc3b0a3a997
>>> # bad: [d027ac4a08541beb2a89563d3e034da7085050ba] firmware:
>>> turris-mox-rwtm: Initialize completion before mailbox git bisect bad
>>> d027ac4a08541beb2a89563d3e034da7085050ba
>>> # bad: [e6c9eca327e6a41a81e7eba0d0ddc13da37f82a1] ARM: spitz: fix GPIO
>>> assignment for backlight git bisect bad
>>> e6c9eca327e6a41a81e7eba0d0ddc13da37f82a1
>>> # bad: [b8cdefdaa555bbfc269c2198803f8791a8923960] m68k: cmpxchg: Fix
>>> return value for default case in __arch_xchg() git bisect bad
>>> b8cdefdaa555bbfc269c2198803f8791a8923960
>>> # bad: [13a71384ae6a8779da809b00c6f378dcead10427] cpufreq/amd-pstate:
>>> Fix the scaling_max_freq setting on shared memory CPPC systems git
>>> bisect bad 13a71384ae6a8779da809b00c6f378dcead10427
>>> # first bad commit: [13a71384ae6a8779da809b00c6f378dcead10427]
>>> cpufreq/amd-pstate: Fix the scaling_max_freq setting on shared memory
>>> CPPC systems
>>>
>>> cpupower output:
>>>
>>> analyzing CPU 47:
>>>     driver: amd-pstate-epp
>>>     CPUs which run at the same hardware frequency: 47
>>>     CPUs which need to have their frequency coordinated by software: 47
>>>     maximum transition latency:  Cannot determine or is not supported.
>>>     hardware limits: 400 MHz - 2.18 GHz
>>>     available cpufreq governors: performance powersave
>>>     current policy: frequency should be within 400 MHz and 2.18 GHz.
>>>                     The governor "performance" may decide which speed 
>>> to use
>>>                     within this range.
>>>     current CPU frequency: Unable to call hardware
>>>     current CPU frequency: 2.17 GHz (asserted by call to kernel)
>>>     boost state support:
>>>       Supported: yes
>>>       Active: yes
>>>       AMD PSTATE Highest Performance: 166. Maximum Frequency: 2.18 GHz.
>>>       AMD PSTATE Nominal Performance: 152. Nominal Frequency: 2.00 GHz.
>>>       AMD PSTATE Lowest Non-linear Performance: 115. Lowest 
>>> Non-linear Frequency: 1.51 GHz.
>>>       AMD PSTATE Lowest Performance: 31. Lowest Frequency: 400 MHz.
>>>
>>> Thanks,
>>> Morgan
>>>
>>> -----Original Message-----
>>> From: Mario Limonciello <mario.limonciello@amd.com>
>>> Sent: Tuesday, July 2, 2024 10:49 AM
>>> To: Dhananjay Ugwekar <Dhananjay.Ugwekar@amd.com>; rafael@kernel.org;
>>> viresh.kumar@linaro.org; gautham.shenoy@amd.com; perry.yuan@amd.com;
>>> skhan@linuxfoundation.org; li.meng@amd.com; ray.huang@amd.com
>>> Cc: linux-pm@vger.kernel.org; linux-kernel@vger.kernel.org; David
>>> Arcari <darcari@redhat.com>
>>> Subject: Re: [PATCH v2 2/2] cpufreq/amd-pstate: Fix the
>>> scaling_max_freq setting on shared memory CPPC systems
>>>
>>> On 7/2/2024 3:14, Dhananjay Ugwekar wrote:
>>>> On shared memory CPPC systems, with amd_pstate=active mode, the
>>>> change in scaling_max_freq doesn't get written to the shared memory 
>>>> region.
>>>> Due to this, the writes to the scaling_max_freq sysfs file don't take
>>>> effect. Fix this by propagating the scaling_max_freq changes to the
>>>> shared memory region.
>>>>
>>>> Fixes: ffa5096a7c33 ("cpufreq: amd-pstate: implement Pstate EPP
>>>> support for the AMD processors")
>>>> Reported-by: David Arcari <darcari@redhat.com>
>>>> Signed-off-by: Dhananjay Ugwekar <Dhananjay.Ugwekar@amd.com>
>>> Reviewed-by: Mario Limonciello <mario.limonciello@amd.com>
>>>> ---
>>>>     drivers/cpufreq/amd-pstate.c | 43 
>>>> +++++++++++++++++++-----------------
>>>>     1 file changed, 23 insertions(+), 20 deletions(-)
>>>>
>>>> diff --git a/drivers/cpufreq/amd-pstate.c
>>>> b/drivers/cpufreq/amd-pstate.c index 9ad62dbe8bfb..a092b13ffbc2
>>>> 100644
>>>> --- a/drivers/cpufreq/amd-pstate.c
>>>> +++ b/drivers/cpufreq/amd-pstate.c
>>>> @@ -247,6 +247,26 @@ static int 
>>>> amd_pstate_get_energy_pref_index(struct amd_cpudata *cpudata)
>>>>         return index;
>>>>     }
>>>> +static void pstate_update_perf(struct amd_cpudata *cpudata, u32 
>>>> min_perf,
>>>> +                   u32 des_perf, u32 max_perf, bool fast_switch) {
>>>> +    if (fast_switch)
>>>> +        wrmsrl(MSR_AMD_CPPC_REQ, READ_ONCE(cpudata->cppc_req_cached));
>>>> +    else
>>>> +        wrmsrl_on_cpu(cpudata->cpu, MSR_AMD_CPPC_REQ,
>>>> +                  READ_ONCE(cpudata->cppc_req_cached));
>>>> +}
>>>> +
>>>> +DEFINE_STATIC_CALL(amd_pstate_update_perf, pstate_update_perf);
>>>> +
>>>> +static inline void amd_pstate_update_perf(struct amd_cpudata *cpudata,
>>>> +                      u32 min_perf, u32 des_perf,
>>>> +                      u32 max_perf, bool fast_switch) {
>>>> +    static_call(amd_pstate_update_perf)(cpudata, min_perf, des_perf,
>>>> +                        max_perf, fast_switch);
>>>> +}
>>>> +
>>>>     static int amd_pstate_set_epp(struct amd_cpudata *cpudata, u32 epp)
>>>>     {
>>>>         int ret;
>>>> @@ -263,6 +283,9 @@ static int amd_pstate_set_epp(struct amd_cpudata 
>>>> *cpudata, u32 epp)
>>>>             if (!ret)
>>>>                 cpudata->epp_cached = epp;
>>>>         } else {
>>>> +        amd_pstate_update_perf(cpudata, cpudata->min_limit_perf, 0U,
>>>> +                         cpudata->max_limit_perf, false);
>>>> +
>>>>             perf_ctrls.energy_perf = epp;
>>>>             ret = cppc_set_epp_perf(cpudata->cpu, &perf_ctrls, 1);
>>>>             if (ret) {
>>>> @@ -452,16 +475,6 @@ static inline int amd_pstate_init_perf(struct 
>>>> amd_cpudata *cpudata)
>>>>         return static_call(amd_pstate_init_perf)(cpudata);
>>>>     }
>>>> -static void pstate_update_perf(struct amd_cpudata *cpudata, u32 
>>>> min_perf,
>>>> -                   u32 des_perf, u32 max_perf, bool fast_switch)
>>>> -{
>>>> -    if (fast_switch)
>>>> -        wrmsrl(MSR_AMD_CPPC_REQ, READ_ONCE(cpudata->cppc_req_cached));
>>>> -    else
>>>> -        wrmsrl_on_cpu(cpudata->cpu, MSR_AMD_CPPC_REQ,
>>>> -                  READ_ONCE(cpudata->cppc_req_cached));
>>>> -}
>>>> -
>>>>     static void cppc_update_perf(struct amd_cpudata *cpudata,
>>>>                      u32 min_perf, u32 des_perf,
>>>>                      u32 max_perf, bool fast_switch) @@ -475,16 
>>>> +488,6 @@
>>>> static void cppc_update_perf(struct amd_cpudata *cpudata,
>>>>         cppc_set_perf(cpudata->cpu, &perf_ctrls);
>>>>     }
>>>> -DEFINE_STATIC_CALL(amd_pstate_update_perf, pstate_update_perf);
>>>> -
>>>> -static inline void amd_pstate_update_perf(struct amd_cpudata *cpudata,
>>>> -                      u32 min_perf, u32 des_perf,
>>>> -                      u32 max_perf, bool fast_switch)
>>>> -{
>>>> -    static_call(amd_pstate_update_perf)(cpudata, min_perf, des_perf,
>>>> -                        max_perf, fast_switch);
>>>> -}
>>>> -
>>>>     static inline bool amd_pstate_sample(struct amd_cpudata *cpudata)
>>>>     {
>>>>         u64 aperf, mperf, tsc;
>>>
>>>
>>
> 

RE: [EXTERNAL] Re: [PATCH v2 2/2] cpufreq/amd-pstate: Fix the scaling_max_freq setting on shared memory CPPC systems

[Jones, Morgan]

Seems like 6.10 is OK but 6.6 is not.

Linux redact 6.10.7 #1-NixOS SMP PREEMPT_DYNAMIC Tue Jan  1 00:00:00 UTC 1980 x86_64 GNU/Linux

analyzing CPU 50:
  driver: amd-pstate-epp
  CPUs which run at the same hardware frequency: 50
  CPUs which need to have their frequency coordinated by software: 50
  maximum transition latency:  Cannot determine or is not supported.
  hardware limits: 400 MHz - 3.35 GHz
  available cpufreq governors: performance powersave
  current policy: frequency should be within 400 MHz and 3.35 GHz.
                  The governor "performance" may decide which speed to use
                  within this range.
  current CPU frequency: Unable to call hardware
  current CPU frequency: 3.32 GHz (asserted by call to kernel)
  boost state support:
    Supported: yes
    Active: yes
    AMD PSTATE Highest Performance: 255. Maximum Frequency: 3.35 GHz.
    AMD PSTATE Nominal Performance: 152. Nominal Frequency: 2.00 GHz.
    AMD PSTATE Lowest Non-linear Performance: 115. Lowest Non-linear Frequency: 1.51 GHz.
    AMD PSTATE Lowest Performance: 31. Lowest Frequency: 400 MHz.


-----Original Message-----
From: Mario Limonciello <mario.limonciello@amd.com> 
Sent: Tuesday, September 3, 2024 1:10 PM
To: Jones, Morgan <Morgan.Jones@viasat.com>; Dhananjay Ugwekar <Dhananjay.Ugwekar@amd.com>; rafael@kernel.org; viresh.kumar@linaro.org; gautham.shenoy@amd.com; perry.yuan@amd.com; skhan@linuxfoundation.org; li.meng@amd.com; ray.huang@amd.com
Cc: linux-pm@vger.kernel.org; linux-kernel@vger.kernel.org; David Arcari <darcari@redhat.com>
Subject: Re: [EXTERNAL] Re: [PATCH v2 2/2] cpufreq/amd-pstate: Fix the scaling_max_freq setting on shared memory CPPC systems

Morgan,

OK that's great news that it's just a backport effort.  That same commit also backported to 6.10.3.  Can you see if 6.10.y is affected?

Ugwekar,

Any thoughts on what else needs to come back to 6.6.y off hand?

Thanks,

On 9/3/2024 15:07, Jones, Morgan wrote:
> Hey Mario,
> 
> Smoking gun here, the max frequency is incorrect on 6.6.44+ but is correct on 6.11.0-rc6.
> 
> Linux redact 6.11.0-rc6 #1-NixOS SMP PREEMPT_DYNAMIC Tue Jan  1 
> 00:00:00 UTC 1980 x86_64 GNU/Linux
> 
> analyzing CPU 12:
>    driver: amd-pstate-epp
>    CPUs which run at the same hardware frequency: 12
>    CPUs which need to have their frequency coordinated by software: 12
>    maximum transition latency:  Cannot determine or is not supported.
>    hardware limits: 400 MHz - 3.35 GHz
>    available cpufreq governors: performance powersave
>    current policy: frequency should be within 400 MHz and 3.35 GHz.
>                    The governor "performance" may decide which speed to use
>                    within this range.
>    current CPU frequency: Unable to call hardware
>    current CPU frequency: 3.34 GHz (asserted by call to kernel)
>    boost state support:
>      Supported: yes
>      Active: yes
>      AMD PSTATE Highest Performance: 255. Maximum Frequency: 3.35 GHz.
>      AMD PSTATE Nominal Performance: 152. Nominal Frequency: 2.00 GHz.
>      AMD PSTATE Lowest Non-linear Performance: 115. Lowest Non-linear Frequency: 1.51 GHz.
>      AMD PSTATE Lowest Performance: 31. Lowest Frequency: 400 MHz.
> 
> We're running amd_pstate=active and amd_pstate.shared_mem=1, and our workloads are back to normal performance on 6.11.0-rc6.
> 
> Morgan
> 
> -----Original Message-----
> From: Mario Limonciello <mario.limonciello@amd.com>
> Sent: Tuesday, September 3, 2024 10:55 AM
> To: Jones, Morgan <Morgan.Jones@viasat.com>; Dhananjay Ugwekar 
> <Dhananjay.Ugwekar@amd.com>; rafael@kernel.org; 
> viresh.kumar@linaro.org; gautham.shenoy@amd.com; perry.yuan@amd.com; 
> skhan@linuxfoundation.org; li.meng@amd.com; ray.huang@amd.com
> Cc: linux-pm@vger.kernel.org; linux-kernel@vger.kernel.org; David 
> Arcari <darcari@redhat.com>
> Subject: [EXTERNAL] Re: [PATCH v2 2/2] cpufreq/amd-pstate: Fix the 
> scaling_max_freq setting on shared memory CPPC systems
> 
> Hi Morgan,
> 
> Can you please cross reference 6.11-rc6 to see if you're still having a problem?  I would like to understand if we're missing a backport to stable or this is still an upstream problem.
> 
> Thanks,
> 
> On 9/3/2024 12:51, Jones, Morgan wrote:
>> Hi there,
>>
>> We are experiencing a ~35% performance regression on an AMD EPYC 7702 64 core machine after applying this patch. We observed Linux 6.6.44 starting to cause the issue, and performed a bisect involving rebooting the machine over 15 times. (Note that kexec didn't successfully identify the problem, since the PM memory mailbox is never reset).
>>
>> It appears that we are getting a max of 2.18 GHz when this CPU can boost to 3.35 GHz, explaining the slowdown. Blacklisting amd-pstate solves the issue at the expense of the performance increase we used to get by using it.
>>
>> Is it possible that the upper limits were implicitly at the max CPU power before, and setting the upper limit to something other than the boost frequency can reduce performance now?
>>
>> # bad: [7213910600667c51c978e577bf5454d3f7b313b7] Linux 6.6.44 # good:
>> [58b0425ff5df680d0b67f64ae1f3f1ebdf1c4de9] Linux 6.6.43 git bisect 
>> start '7213910600667c51c978e577bf5454d3f7b313b7' '58b0425ff5df680d0b67f64ae1f3f1ebdf1c4de9'
>> # good: [72ff9d26964a3a80f7650df719df139f5c1f965d] arm64: dts: qcom:
>> sm6350: Add missing qcom,non-secure-domain property git bisect good 
>> 72ff9d26964a3a80f7650df719df139f5c1f965d
>> # good: [0fffc2e1bf40a2220ef5a38f834ea063dba832d3] ARM: dts: sunxi:
>> remove duplicated entries in makefile git bisect good
>> 0fffc2e1bf40a2220ef5a38f834ea063dba832d3
>> # bad: [8cdbe6ebfd1763a5c41a2a3058497c0a9163311c] pinctrl: renesas:
>> r8a779g0: Fix CANFD5 suffix git bisect bad 
>> 8cdbe6ebfd1763a5c41a2a3058497c0a9163311c
>> # bad: [5dbb98e7fa42bebc1325899193d8f13f0705a148] drm/mediatek: Turn 
>> off the layers with zero width or height git bisect bad
>> 5dbb98e7fa42bebc1325899193d8f13f0705a148
>> # bad: [691ec7043122c9c8c46d84f6e6cd85d13d50cd93] selftests/bpf: Null 
>> checks for links in bpf_tcp_ca git bisect bad
>> 691ec7043122c9c8c46d84f6e6cd85d13d50cd93
>> # bad: [a1359e085d75d7393a250054e66c0a7bc6c3dbfa] perf/x86: Serialize
>> set_attr_rdpmc() git bisect bad
>> a1359e085d75d7393a250054e66c0a7bc6c3dbfa
>> # bad: [e99d9b16ff153de9540073239d24adc3b0a3a997] wifi: ath12k: 
>> change DMA direction while mapping reinjected packets git bisect bad
>> e99d9b16ff153de9540073239d24adc3b0a3a997
>> # bad: [d027ac4a08541beb2a89563d3e034da7085050ba] firmware:
>> turris-mox-rwtm: Initialize completion before mailbox git bisect bad 
>> d027ac4a08541beb2a89563d3e034da7085050ba
>> # bad: [e6c9eca327e6a41a81e7eba0d0ddc13da37f82a1] ARM: spitz: fix 
>> GPIO assignment for backlight git bisect bad
>> e6c9eca327e6a41a81e7eba0d0ddc13da37f82a1
>> # bad: [b8cdefdaa555bbfc269c2198803f8791a8923960] m68k: cmpxchg: Fix 
>> return value for default case in __arch_xchg() git bisect bad
>> b8cdefdaa555bbfc269c2198803f8791a8923960
>> # bad: [13a71384ae6a8779da809b00c6f378dcead10427] cpufreq/amd-pstate:
>> Fix the scaling_max_freq setting on shared memory CPPC systems git 
>> bisect bad 13a71384ae6a8779da809b00c6f378dcead10427
>> # first bad commit: [13a71384ae6a8779da809b00c6f378dcead10427]
>> cpufreq/amd-pstate: Fix the scaling_max_freq setting on shared memory 
>> CPPC systems
>>
>> cpupower output:
>>
>> analyzing CPU 47:
>>     driver: amd-pstate-epp
>>     CPUs which run at the same hardware frequency: 47
>>     CPUs which need to have their frequency coordinated by software: 47
>>     maximum transition latency:  Cannot determine or is not supported.
>>     hardware limits: 400 MHz - 2.18 GHz
>>     available cpufreq governors: performance powersave
>>     current policy: frequency should be within 400 MHz and 2.18 GHz.
>>                     The governor "performance" may decide which speed to use
>>                     within this range.
>>     current CPU frequency: Unable to call hardware
>>     current CPU frequency: 2.17 GHz (asserted by call to kernel)
>>     boost state support:
>>       Supported: yes
>>       Active: yes
>>       AMD PSTATE Highest Performance: 166. Maximum Frequency: 2.18 GHz.
>>       AMD PSTATE Nominal Performance: 152. Nominal Frequency: 2.00 GHz.
>>       AMD PSTATE Lowest Non-linear Performance: 115. Lowest Non-linear Frequency: 1.51 GHz.
>>       AMD PSTATE Lowest Performance: 31. Lowest Frequency: 400 MHz.
>>
>> Thanks,
>> Morgan
>>
>> -----Original Message-----
>> From: Mario Limonciello <mario.limonciello@amd.com>
>> Sent: Tuesday, July 2, 2024 10:49 AM
>> To: Dhananjay Ugwekar <Dhananjay.Ugwekar@amd.com>; rafael@kernel.org; 
>> viresh.kumar@linaro.org; gautham.shenoy@amd.com; perry.yuan@amd.com; 
>> skhan@linuxfoundation.org; li.meng@amd.com; ray.huang@amd.com
>> Cc: linux-pm@vger.kernel.org; linux-kernel@vger.kernel.org; David 
>> Arcari <darcari@redhat.com>
>> Subject: Re: [PATCH v2 2/2] cpufreq/amd-pstate: Fix the 
>> scaling_max_freq setting on shared memory CPPC systems
>>
>> On 7/2/2024 3:14, Dhananjay Ugwekar wrote:
>>> On shared memory CPPC systems, with amd_pstate=active mode, the 
>>> change in scaling_max_freq doesn't get written to the shared memory region.
>>> Due to this, the writes to the scaling_max_freq sysfs file don't 
>>> take effect. Fix this by propagating the scaling_max_freq changes to 
>>> the shared memory region.
>>>
>>> Fixes: ffa5096a7c33 ("cpufreq: amd-pstate: implement Pstate EPP 
>>> support for the AMD processors")
>>> Reported-by: David Arcari <darcari@redhat.com>
>>> Signed-off-by: Dhananjay Ugwekar <Dhananjay.Ugwekar@amd.com>
>> Reviewed-by: Mario Limonciello <mario.limonciello@amd.com>
>>> ---
>>>     drivers/cpufreq/amd-pstate.c | 43 +++++++++++++++++++-----------------
>>>     1 file changed, 23 insertions(+), 20 deletions(-)
>>>
>>> diff --git a/drivers/cpufreq/amd-pstate.c 
>>> b/drivers/cpufreq/amd-pstate.c index 9ad62dbe8bfb..a092b13ffbc2
>>> 100644
>>> --- a/drivers/cpufreq/amd-pstate.c
>>> +++ b/drivers/cpufreq/amd-pstate.c
>>> @@ -247,6 +247,26 @@ static int amd_pstate_get_energy_pref_index(struct amd_cpudata *cpudata)
>>>     	return index;
>>>     }
>>>     
>>> +static void pstate_update_perf(struct amd_cpudata *cpudata, u32 min_perf,
>>> +			       u32 des_perf, u32 max_perf, bool fast_switch) {
>>> +	if (fast_switch)
>>> +		wrmsrl(MSR_AMD_CPPC_REQ, READ_ONCE(cpudata->cppc_req_cached));
>>> +	else
>>> +		wrmsrl_on_cpu(cpudata->cpu, MSR_AMD_CPPC_REQ,
>>> +			      READ_ONCE(cpudata->cppc_req_cached));
>>> +}
>>> +
>>> +DEFINE_STATIC_CALL(amd_pstate_update_perf, pstate_update_perf);
>>> +
>>> +static inline void amd_pstate_update_perf(struct amd_cpudata *cpudata,
>>> +					  u32 min_perf, u32 des_perf,
>>> +					  u32 max_perf, bool fast_switch) {
>>> +	static_call(amd_pstate_update_perf)(cpudata, min_perf, des_perf,
>>> +					    max_perf, fast_switch);
>>> +}
>>> +
>>>     static int amd_pstate_set_epp(struct amd_cpudata *cpudata, u32 epp)
>>>     {
>>>     	int ret;
>>> @@ -263,6 +283,9 @@ static int amd_pstate_set_epp(struct amd_cpudata *cpudata, u32 epp)
>>>     		if (!ret)
>>>     			cpudata->epp_cached = epp;
>>>     	} else {
>>> +		amd_pstate_update_perf(cpudata, cpudata->min_limit_perf, 0U,
>>> +					     cpudata->max_limit_perf, false);
>>> +
>>>     		perf_ctrls.energy_perf = epp;
>>>     		ret = cppc_set_epp_perf(cpudata->cpu, &perf_ctrls, 1);
>>>     		if (ret) {
>>> @@ -452,16 +475,6 @@ static inline int amd_pstate_init_perf(struct amd_cpudata *cpudata)
>>>     	return static_call(amd_pstate_init_perf)(cpudata);
>>>     }
>>>     
>>> -static void pstate_update_perf(struct amd_cpudata *cpudata, u32 min_perf,
>>> -			       u32 des_perf, u32 max_perf, bool fast_switch)
>>> -{
>>> -	if (fast_switch)
>>> -		wrmsrl(MSR_AMD_CPPC_REQ, READ_ONCE(cpudata->cppc_req_cached));
>>> -	else
>>> -		wrmsrl_on_cpu(cpudata->cpu, MSR_AMD_CPPC_REQ,
>>> -			      READ_ONCE(cpudata->cppc_req_cached));
>>> -}
>>> -
>>>     static void cppc_update_perf(struct amd_cpudata *cpudata,
>>>     			     u32 min_perf, u32 des_perf,
>>>     			     u32 max_perf, bool fast_switch) @@ -475,16 +488,6 @@ 
>>> static void cppc_update_perf(struct amd_cpudata *cpudata,
>>>     	cppc_set_perf(cpudata->cpu, &perf_ctrls);
>>>     }
>>>     
>>> -DEFINE_STATIC_CALL(amd_pstate_update_perf, pstate_update_perf);
>>> -
>>> -static inline void amd_pstate_update_perf(struct amd_cpudata *cpudata,
>>> -					  u32 min_perf, u32 des_perf,
>>> -					  u32 max_perf, bool fast_switch)
>>> -{
>>> -	static_call(amd_pstate_update_perf)(cpudata, min_perf, des_perf,
>>> -					    max_perf, fast_switch);
>>> -}
>>> -
>>>     static inline bool amd_pstate_sample(struct amd_cpudata *cpudata)
>>>     {
>>>     	u64 aperf, mperf, tsc;
>>
>>
> 

RE: [EXTERNAL] Re: [PATCH v2 2/2] cpufreq/amd-pstate: Fix the scaling_max_freq setting on shared memory CPPC systems

[Jones, Morgan]

On it.

-----Original Message-----
From: Mario Limonciello <mario.limonciello@amd.com> 
Sent: Tuesday, September 3, 2024 1:52 PM
To: Jones, Morgan <Morgan.Jones@viasat.com>; Dhananjay Ugwekar <Dhananjay.Ugwekar@amd.com>; rafael@kernel.org; viresh.kumar@linaro.org; gautham.shenoy@amd.com; perry.yuan@amd.com; skhan@linuxfoundation.org; li.meng@amd.com; ray.huang@amd.com
Cc: linux-pm@vger.kernel.org; linux-kernel@vger.kernel.org; David Arcari <darcari@redhat.com>
Subject: Re: [EXTERNAL] Re: [PATCH v2 2/2] cpufreq/amd-pstate: Fix the scaling_max_freq setting on shared memory CPPC systems

Morgan,

This does remind me of a clamping issue that touches some of the same variables.  Can you please see if backporting

commit 8164f7433264 ("cpufreq: amd-pstate: adjust min/max limit perf")

to 6.6.y helps for you?

Thanks,

On 9/3/2024 15:09, Mario Limonciello wrote:
> Morgan,
> 
> OK that's great news that it's just a backport effort.  That same 
> commit also backported to 6.10.3.  Can you see if 6.10.y is affected?
> 
> Ugwekar,
> 
> Any thoughts on what else needs to come back to 6.6.y off hand?
> 
> Thanks,
> 
> On 9/3/2024 15:07, Jones, Morgan wrote:
>> Hey Mario,
>>
>> Smoking gun here, the max frequency is incorrect on 6.6.44+ but is 
>> correct on 6.11.0-rc6.
>>
>> Linux redact 6.11.0-rc6 #1-NixOS SMP PREEMPT_DYNAMIC Tue Jan  1
>> 00:00:00 UTC 1980 x86_64 GNU/Linux
>>
>> analyzing CPU 12:
>>    driver: amd-pstate-epp
>>    CPUs which run at the same hardware frequency: 12
>>    CPUs which need to have their frequency coordinated by software: 
>> 12
>>    maximum transition latency:  Cannot determine or is not supported.
>>    hardware limits: 400 MHz - 3.35 GHz
>>    available cpufreq governors: performance powersave
>>    current policy: frequency should be within 400 MHz and 3.35 GHz.
>>                    The governor "performance" may decide which speed 
>> to use
>>                    within this range.
>>    current CPU frequency: Unable to call hardware
>>    current CPU frequency: 3.34 GHz (asserted by call to kernel)
>>    boost state support:
>>      Supported: yes
>>      Active: yes
>>      AMD PSTATE Highest Performance: 255. Maximum Frequency: 3.35 GHz.
>>      AMD PSTATE Nominal Performance: 152. Nominal Frequency: 2.00 GHz.
>>      AMD PSTATE Lowest Non-linear Performance: 115. Lowest Non-linear
>> Frequency: 1.51 GHz.
>>      AMD PSTATE Lowest Performance: 31. Lowest Frequency: 400 MHz.
>>
>> We're running amd_pstate=active and amd_pstate.shared_mem=1, and our 
>> workloads are back to normal performance on 6.11.0-rc6.
>>
>> Morgan
>>
>> -----Original Message-----
>> From: Mario Limonciello <mario.limonciello@amd.com>
>> Sent: Tuesday, September 3, 2024 10:55 AM
>> To: Jones, Morgan <Morgan.Jones@viasat.com>; Dhananjay Ugwekar 
>> <Dhananjay.Ugwekar@amd.com>; rafael@kernel.org; 
>> viresh.kumar@linaro.org; gautham.shenoy@amd.com; perry.yuan@amd.com; 
>> skhan@linuxfoundation.org; li.meng@amd.com; ray.huang@amd.com
>> Cc: linux-pm@vger.kernel.org; linux-kernel@vger.kernel.org; David 
>> Arcari <darcari@redhat.com>
>> Subject: [EXTERNAL] Re: [PATCH v2 2/2] cpufreq/amd-pstate: Fix the 
>> scaling_max_freq setting on shared memory CPPC systems
>>
>> Hi Morgan,
>>
>> Can you please cross reference 6.11-rc6 to see if you're still having 
>> a problem?  I would like to understand if we're missing a backport to 
>> stable or this is still an upstream problem.
>>
>> Thanks,
>>
>> On 9/3/2024 12:51, Jones, Morgan wrote:
>>> Hi there,
>>>
>>> We are experiencing a ~35% performance regression on an AMD EPYC 
>>> 7702
>>> 64 core machine after applying this patch. We observed Linux 6.6.44 
>>> starting to cause the issue, and performed a bisect involving 
>>> rebooting the machine over 15 times. (Note that kexec didn't 
>>> successfully identify the problem, since the PM memory mailbox is 
>>> never reset).
>>>
>>> It appears that we are getting a max of 2.18 GHz when this CPU can 
>>> boost to 3.35 GHz, explaining the slowdown. Blacklisting amd-pstate 
>>> solves the issue at the expense of the performance increase we used 
>>> to get by using it.
>>>
>>> Is it possible that the upper limits were implicitly at the max CPU 
>>> power before, and setting the upper limit to something other than 
>>> the boost frequency can reduce performance now?
>>>
>>> # bad: [7213910600667c51c978e577bf5454d3f7b313b7] Linux 6.6.44 # good:
>>> [58b0425ff5df680d0b67f64ae1f3f1ebdf1c4de9] Linux 6.6.43 git bisect 
>>> start '7213910600667c51c978e577bf5454d3f7b313b7'
>>> '58b0425ff5df680d0b67f64ae1f3f1ebdf1c4de9'
>>> # good: [72ff9d26964a3a80f7650df719df139f5c1f965d] arm64: dts: qcom:
>>> sm6350: Add missing qcom,non-secure-domain property git bisect good 
>>> 72ff9d26964a3a80f7650df719df139f5c1f965d
>>> # good: [0fffc2e1bf40a2220ef5a38f834ea063dba832d3] ARM: dts: sunxi:
>>> remove duplicated entries in makefile git bisect good
>>> 0fffc2e1bf40a2220ef5a38f834ea063dba832d3
>>> # bad: [8cdbe6ebfd1763a5c41a2a3058497c0a9163311c] pinctrl: renesas:
>>> r8a779g0: Fix CANFD5 suffix git bisect bad 
>>> 8cdbe6ebfd1763a5c41a2a3058497c0a9163311c
>>> # bad: [5dbb98e7fa42bebc1325899193d8f13f0705a148] drm/mediatek: Turn 
>>> off the layers with zero width or height git bisect bad
>>> 5dbb98e7fa42bebc1325899193d8f13f0705a148
>>> # bad: [691ec7043122c9c8c46d84f6e6cd85d13d50cd93] selftests/bpf: 
>>> Null checks for links in bpf_tcp_ca git bisect bad
>>> 691ec7043122c9c8c46d84f6e6cd85d13d50cd93
>>> # bad: [a1359e085d75d7393a250054e66c0a7bc6c3dbfa] perf/x86: 
>>> Serialize
>>> set_attr_rdpmc() git bisect bad
>>> a1359e085d75d7393a250054e66c0a7bc6c3dbfa
>>> # bad: [e99d9b16ff153de9540073239d24adc3b0a3a997] wifi: ath12k: 
>>> change DMA direction while mapping reinjected packets git bisect bad
>>> e99d9b16ff153de9540073239d24adc3b0a3a997
>>> # bad: [d027ac4a08541beb2a89563d3e034da7085050ba] firmware:
>>> turris-mox-rwtm: Initialize completion before mailbox git bisect bad 
>>> d027ac4a08541beb2a89563d3e034da7085050ba
>>> # bad: [e6c9eca327e6a41a81e7eba0d0ddc13da37f82a1] ARM: spitz: fix 
>>> GPIO assignment for backlight git bisect bad
>>> e6c9eca327e6a41a81e7eba0d0ddc13da37f82a1
>>> # bad: [b8cdefdaa555bbfc269c2198803f8791a8923960] m68k: cmpxchg: Fix 
>>> return value for default case in __arch_xchg() git bisect bad
>>> b8cdefdaa555bbfc269c2198803f8791a8923960
>>> # bad: [13a71384ae6a8779da809b00c6f378dcead10427] cpufreq/amd-pstate:
>>> Fix the scaling_max_freq setting on shared memory CPPC systems git 
>>> bisect bad 13a71384ae6a8779da809b00c6f378dcead10427
>>> # first bad commit: [13a71384ae6a8779da809b00c6f378dcead10427]
>>> cpufreq/amd-pstate: Fix the scaling_max_freq setting on shared 
>>> memory CPPC systems
>>>
>>> cpupower output:
>>>
>>> analyzing CPU 47:
>>>     driver: amd-pstate-epp
>>>     CPUs which run at the same hardware frequency: 47
>>>     CPUs which need to have their frequency coordinated by software: 
>>> 47
>>>     maximum transition latency:  Cannot determine or is not supported.
>>>     hardware limits: 400 MHz - 2.18 GHz
>>>     available cpufreq governors: performance powersave
>>>     current policy: frequency should be within 400 MHz and 2.18 GHz.
>>>                     The governor "performance" may decide which 
>>> speed to use
>>>                     within this range.
>>>     current CPU frequency: Unable to call hardware
>>>     current CPU frequency: 2.17 GHz (asserted by call to kernel)
>>>     boost state support:
>>>       Supported: yes
>>>       Active: yes
>>>       AMD PSTATE Highest Performance: 166. Maximum Frequency: 2.18 GHz.
>>>       AMD PSTATE Nominal Performance: 152. Nominal Frequency: 2.00 GHz.
>>>       AMD PSTATE Lowest Non-linear Performance: 115. Lowest 
>>> Non-linear Frequency: 1.51 GHz.
>>>       AMD PSTATE Lowest Performance: 31. Lowest Frequency: 400 MHz.
>>>
>>> Thanks,
>>> Morgan
>>>
>>> -----Original Message-----
>>> From: Mario Limonciello <mario.limonciello@amd.com>
>>> Sent: Tuesday, July 2, 2024 10:49 AM
>>> To: Dhananjay Ugwekar <Dhananjay.Ugwekar@amd.com>; 
>>> rafael@kernel.org; viresh.kumar@linaro.org; gautham.shenoy@amd.com; 
>>> perry.yuan@amd.com; skhan@linuxfoundation.org; li.meng@amd.com; 
>>> ray.huang@amd.com
>>> Cc: linux-pm@vger.kernel.org; linux-kernel@vger.kernel.org; David 
>>> Arcari <darcari@redhat.com>
>>> Subject: Re: [PATCH v2 2/2] cpufreq/amd-pstate: Fix the 
>>> scaling_max_freq setting on shared memory CPPC systems
>>>
>>> On 7/2/2024 3:14, Dhananjay Ugwekar wrote:
>>>> On shared memory CPPC systems, with amd_pstate=active mode, the 
>>>> change in scaling_max_freq doesn't get written to the shared memory 
>>>> region.
>>>> Due to this, the writes to the scaling_max_freq sysfs file don't 
>>>> take effect. Fix this by propagating the scaling_max_freq changes 
>>>> to the shared memory region.
>>>>
>>>> Fixes: ffa5096a7c33 ("cpufreq: amd-pstate: implement Pstate EPP 
>>>> support for the AMD processors")
>>>> Reported-by: David Arcari <darcari@redhat.com>
>>>> Signed-off-by: Dhananjay Ugwekar <Dhananjay.Ugwekar@amd.com>
>>> Reviewed-by: Mario Limonciello <mario.limonciello@amd.com>
>>>> ---
>>>>     drivers/cpufreq/amd-pstate.c | 43
>>>> +++++++++++++++++++-----------------
>>>>     1 file changed, 23 insertions(+), 20 deletions(-)
>>>>
>>>> diff --git a/drivers/cpufreq/amd-pstate.c 
>>>> b/drivers/cpufreq/amd-pstate.c index 9ad62dbe8bfb..a092b13ffbc2
>>>> 100644
>>>> --- a/drivers/cpufreq/amd-pstate.c
>>>> +++ b/drivers/cpufreq/amd-pstate.c
>>>> @@ -247,6 +247,26 @@ static int
>>>> amd_pstate_get_energy_pref_index(struct amd_cpudata *cpudata)
>>>>         return index;
>>>>     }
>>>> +static void pstate_update_perf(struct amd_cpudata *cpudata, u32
>>>> min_perf,
>>>> +                   u32 des_perf, u32 max_perf, bool fast_switch) {
>>>> +    if (fast_switch)
>>>> +        wrmsrl(MSR_AMD_CPPC_REQ, 
>>>> +READ_ONCE(cpudata->cppc_req_cached));
>>>> +    else
>>>> +        wrmsrl_on_cpu(cpudata->cpu, MSR_AMD_CPPC_REQ,
>>>> +                  READ_ONCE(cpudata->cppc_req_cached));
>>>> +}
>>>> +
>>>> +DEFINE_STATIC_CALL(amd_pstate_update_perf, pstate_update_perf);
>>>> +
>>>> +static inline void amd_pstate_update_perf(struct amd_cpudata 
>>>> +*cpudata,
>>>> +                      u32 min_perf, u32 des_perf,
>>>> +                      u32 max_perf, bool fast_switch) {
>>>> +    static_call(amd_pstate_update_perf)(cpudata, min_perf, 
>>>> +des_perf,
>>>> +                        max_perf, fast_switch); }
>>>> +
>>>>     static int amd_pstate_set_epp(struct amd_cpudata *cpudata, u32 
>>>> epp)
>>>>     {
>>>>         int ret;
>>>> @@ -263,6 +283,9 @@ static int amd_pstate_set_epp(struct 
>>>> amd_cpudata *cpudata, u32 epp)
>>>>             if (!ret)
>>>>                 cpudata->epp_cached = epp;
>>>>         } else {
>>>> +        amd_pstate_update_perf(cpudata, cpudata->min_limit_perf, 
>>>> +0U,
>>>> +                         cpudata->max_limit_perf, false);
>>>> +
>>>>             perf_ctrls.energy_perf = epp;
>>>>             ret = cppc_set_epp_perf(cpudata->cpu, &perf_ctrls, 1);
>>>>             if (ret) {
>>>> @@ -452,16 +475,6 @@ static inline int amd_pstate_init_perf(struct 
>>>> amd_cpudata *cpudata)
>>>>         return static_call(amd_pstate_init_perf)(cpudata);
>>>>     }
>>>> -static void pstate_update_perf(struct amd_cpudata *cpudata, u32 
>>>> min_perf,
>>>> -                   u32 des_perf, u32 max_perf, bool fast_switch) 
>>>> -{
>>>> -    if (fast_switch)
>>>> -        wrmsrl(MSR_AMD_CPPC_REQ, 
>>>> READ_ONCE(cpudata->cppc_req_cached));
>>>> -    else
>>>> -        wrmsrl_on_cpu(cpudata->cpu, MSR_AMD_CPPC_REQ,
>>>> -                  READ_ONCE(cpudata->cppc_req_cached));
>>>> -}
>>>> -
>>>>     static void cppc_update_perf(struct amd_cpudata *cpudata,
>>>>                      u32 min_perf, u32 des_perf,
>>>>                      u32 max_perf, bool fast_switch) @@ -475,16
>>>> +488,6 @@
>>>> static void cppc_update_perf(struct amd_cpudata *cpudata,
>>>>         cppc_set_perf(cpudata->cpu, &perf_ctrls);
>>>>     }
>>>> -DEFINE_STATIC_CALL(amd_pstate_update_perf, pstate_update_perf);
>>>> -
>>>> -static inline void amd_pstate_update_perf(struct amd_cpudata 
>>>> *cpudata,
>>>> -                      u32 min_perf, u32 des_perf,
>>>> -                      u32 max_perf, bool fast_switch) -{
>>>> -    static_call(amd_pstate_update_perf)(cpudata, min_perf, 
>>>> des_perf,
>>>> -                        max_perf, fast_switch); -}
>>>> -
>>>>     static inline bool amd_pstate_sample(struct amd_cpudata 
>>>> *cpudata)
>>>>     {
>>>>         u64 aperf, mperf, tsc;
>>>
>>>
>>
> 

RE: [EXTERNAL] Re: [PATCH v2 2/2] cpufreq/amd-pstate: Fix the scaling_max_freq setting on shared memory CPPC systems

[Jones, Morgan]

With and without the patch, I get the same:

Linux redact 6.6.48 #1-NixOS SMP PREEMPT_DYNAMIC Tue Jan  1 00:00:00 UTC 1980 x86_64 GNU/Linux

analyzing CPU 31:
  driver: amd-pstate-epp
  CPUs which run at the same hardware frequency: 31
  CPUs which need to have their frequency coordinated by software: 31
  maximum transition latency:  Cannot determine or is not supported.
  hardware limits: 400 MHz - 2.18 GHz
  available cpufreq governors: performance powersave
  current policy: frequency should be within 400 MHz and 2.18 GHz.
                  The governor "performance" may decide which speed to use
                  within this range.
  current CPU frequency: Unable to call hardware
  current CPU frequency: 2.17 GHz (asserted by call to kernel)
  boost state support:
    Supported: yes
    Active: yes
    AMD PSTATE Highest Performance: 166. Maximum Frequency: 2.18 GHz.
    AMD PSTATE Nominal Performance: 152. Nominal Frequency: 2.00 GHz.
    AMD PSTATE Lowest Non-linear Performance: 115. Lowest Non-linear Frequency: 1.51 GHz.
    AMD PSTATE Lowest Performance: 31. Lowest Frequency: 400 MHz.


-----Original Message-----
From: Mario Limonciello <mario.limonciello@amd.com> 
Sent: Tuesday, September 3, 2024 1:52 PM
To: Jones, Morgan <Morgan.Jones@viasat.com>; Dhananjay Ugwekar <Dhananjay.Ugwekar@amd.com>; rafael@kernel.org; viresh.kumar@linaro.org; gautham.shenoy@amd.com; perry.yuan@amd.com; skhan@linuxfoundation.org; li.meng@amd.com; ray.huang@amd.com
Cc: linux-pm@vger.kernel.org; linux-kernel@vger.kernel.org; David Arcari <darcari@redhat.com>
Subject: Re: [EXTERNAL] Re: [PATCH v2 2/2] cpufreq/amd-pstate: Fix the scaling_max_freq setting on shared memory CPPC systems

Morgan,

This does remind me of a clamping issue that touches some of the same variables.  Can you please see if backporting

commit 8164f7433264 ("cpufreq: amd-pstate: adjust min/max limit perf")

to 6.6.y helps for you?

Thanks,

On 9/3/2024 15:09, Mario Limonciello wrote:
> Morgan,
> 
> OK that's great news that it's just a backport effort.  That same 
> commit also backported to 6.10.3.  Can you see if 6.10.y is affected?
> 
> Ugwekar,
> 
> Any thoughts on what else needs to come back to 6.6.y off hand?
> 
> Thanks,
> 
> On 9/3/2024 15:07, Jones, Morgan wrote:
>> Hey Mario,
>>
>> Smoking gun here, the max frequency is incorrect on 6.6.44+ but is 
>> correct on 6.11.0-rc6.
>>
>> Linux redact 6.11.0-rc6 #1-NixOS SMP PREEMPT_DYNAMIC Tue Jan  1
>> 00:00:00 UTC 1980 x86_64 GNU/Linux
>>
>> analyzing CPU 12:
>>    driver: amd-pstate-epp
>>    CPUs which run at the same hardware frequency: 12
>>    CPUs which need to have their frequency coordinated by software: 
>> 12
>>    maximum transition latency:  Cannot determine or is not supported.
>>    hardware limits: 400 MHz - 3.35 GHz
>>    available cpufreq governors: performance powersave
>>    current policy: frequency should be within 400 MHz and 3.35 GHz.
>>                    The governor "performance" may decide which speed 
>> to use
>>                    within this range.
>>    current CPU frequency: Unable to call hardware
>>    current CPU frequency: 3.34 GHz (asserted by call to kernel)
>>    boost state support:
>>      Supported: yes
>>      Active: yes
>>      AMD PSTATE Highest Performance: 255. Maximum Frequency: 3.35 GHz.
>>      AMD PSTATE Nominal Performance: 152. Nominal Frequency: 2.00 GHz.
>>      AMD PSTATE Lowest Non-linear Performance: 115. Lowest Non-linear
>> Frequency: 1.51 GHz.
>>      AMD PSTATE Lowest Performance: 31. Lowest Frequency: 400 MHz.
>>
>> We're running amd_pstate=active and amd_pstate.shared_mem=1, and our 
>> workloads are back to normal performance on 6.11.0-rc6.
>>
>> Morgan
>>
>> -----Original Message-----
>> From: Mario Limonciello <mario.limonciello@amd.com>
>> Sent: Tuesday, September 3, 2024 10:55 AM
>> To: Jones, Morgan <Morgan.Jones@viasat.com>; Dhananjay Ugwekar 
>> <Dhananjay.Ugwekar@amd.com>; rafael@kernel.org; 
>> viresh.kumar@linaro.org; gautham.shenoy@amd.com; perry.yuan@amd.com; 
>> skhan@linuxfoundation.org; li.meng@amd.com; ray.huang@amd.com
>> Cc: linux-pm@vger.kernel.org; linux-kernel@vger.kernel.org; David 
>> Arcari <darcari@redhat.com>
>> Subject: [EXTERNAL] Re: [PATCH v2 2/2] cpufreq/amd-pstate: Fix the 
>> scaling_max_freq setting on shared memory CPPC systems
>>
>> Hi Morgan,
>>
>> Can you please cross reference 6.11-rc6 to see if you're still having 
>> a problem?  I would like to understand if we're missing a backport to 
>> stable or this is still an upstream problem.
>>
>> Thanks,
>>
>> On 9/3/2024 12:51, Jones, Morgan wrote:
>>> Hi there,
>>>
>>> We are experiencing a ~35% performance regression on an AMD EPYC 
>>> 7702
>>> 64 core machine after applying this patch. We observed Linux 6.6.44 
>>> starting to cause the issue, and performed a bisect involving 
>>> rebooting the machine over 15 times. (Note that kexec didn't 
>>> successfully identify the problem, since the PM memory mailbox is 
>>> never reset).
>>>
>>> It appears that we are getting a max of 2.18 GHz when this CPU can 
>>> boost to 3.35 GHz, explaining the slowdown. Blacklisting amd-pstate 
>>> solves the issue at the expense of the performance increase we used 
>>> to get by using it.
>>>
>>> Is it possible that the upper limits were implicitly at the max CPU 
>>> power before, and setting the upper limit to something other than 
>>> the boost frequency can reduce performance now?
>>>
>>> # bad: [7213910600667c51c978e577bf5454d3f7b313b7] Linux 6.6.44 # good:
>>> [58b0425ff5df680d0b67f64ae1f3f1ebdf1c4de9] Linux 6.6.43 git bisect 
>>> start '7213910600667c51c978e577bf5454d3f7b313b7'
>>> '58b0425ff5df680d0b67f64ae1f3f1ebdf1c4de9'
>>> # good: [72ff9d26964a3a80f7650df719df139f5c1f965d] arm64: dts: qcom:
>>> sm6350: Add missing qcom,non-secure-domain property git bisect good 
>>> 72ff9d26964a3a80f7650df719df139f5c1f965d
>>> # good: [0fffc2e1bf40a2220ef5a38f834ea063dba832d3] ARM: dts: sunxi:
>>> remove duplicated entries in makefile git bisect good
>>> 0fffc2e1bf40a2220ef5a38f834ea063dba832d3
>>> # bad: [8cdbe6ebfd1763a5c41a2a3058497c0a9163311c] pinctrl: renesas:
>>> r8a779g0: Fix CANFD5 suffix git bisect bad 
>>> 8cdbe6ebfd1763a5c41a2a3058497c0a9163311c
>>> # bad: [5dbb98e7fa42bebc1325899193d8f13f0705a148] drm/mediatek: Turn 
>>> off the layers with zero width or height git bisect bad
>>> 5dbb98e7fa42bebc1325899193d8f13f0705a148
>>> # bad: [691ec7043122c9c8c46d84f6e6cd85d13d50cd93] selftests/bpf: 
>>> Null checks for links in bpf_tcp_ca git bisect bad
>>> 691ec7043122c9c8c46d84f6e6cd85d13d50cd93
>>> # bad: [a1359e085d75d7393a250054e66c0a7bc6c3dbfa] perf/x86: 
>>> Serialize
>>> set_attr_rdpmc() git bisect bad
>>> a1359e085d75d7393a250054e66c0a7bc6c3dbfa
>>> # bad: [e99d9b16ff153de9540073239d24adc3b0a3a997] wifi: ath12k: 
>>> change DMA direction while mapping reinjected packets git bisect bad
>>> e99d9b16ff153de9540073239d24adc3b0a3a997
>>> # bad: [d027ac4a08541beb2a89563d3e034da7085050ba] firmware:
>>> turris-mox-rwtm: Initialize completion before mailbox git bisect bad 
>>> d027ac4a08541beb2a89563d3e034da7085050ba
>>> # bad: [e6c9eca327e6a41a81e7eba0d0ddc13da37f82a1] ARM: spitz: fix 
>>> GPIO assignment for backlight git bisect bad
>>> e6c9eca327e6a41a81e7eba0d0ddc13da37f82a1
>>> # bad: [b8cdefdaa555bbfc269c2198803f8791a8923960] m68k: cmpxchg: Fix 
>>> return value for default case in __arch_xchg() git bisect bad
>>> b8cdefdaa555bbfc269c2198803f8791a8923960
>>> # bad: [13a71384ae6a8779da809b00c6f378dcead10427] cpufreq/amd-pstate:
>>> Fix the scaling_max_freq setting on shared memory CPPC systems git 
>>> bisect bad 13a71384ae6a8779da809b00c6f378dcead10427
>>> # first bad commit: [13a71384ae6a8779da809b00c6f378dcead10427]
>>> cpufreq/amd-pstate: Fix the scaling_max_freq setting on shared 
>>> memory CPPC systems
>>>
>>> cpupower output:
>>>
>>> analyzing CPU 47:
>>>     driver: amd-pstate-epp
>>>     CPUs which run at the same hardware frequency: 47
>>>     CPUs which need to have their frequency coordinated by software: 
>>> 47
>>>     maximum transition latency:  Cannot determine or is not supported.
>>>     hardware limits: 400 MHz - 2.18 GHz
>>>     available cpufreq governors: performance powersave
>>>     current policy: frequency should be within 400 MHz and 2.18 GHz.
>>>                     The governor "performance" may decide which 
>>> speed to use
>>>                     within this range.
>>>     current CPU frequency: Unable to call hardware
>>>     current CPU frequency: 2.17 GHz (asserted by call to kernel)
>>>     boost state support:
>>>       Supported: yes
>>>       Active: yes
>>>       AMD PSTATE Highest Performance: 166. Maximum Frequency: 2.18 GHz.
>>>       AMD PSTATE Nominal Performance: 152. Nominal Frequency: 2.00 GHz.
>>>       AMD PSTATE Lowest Non-linear Performance: 115. Lowest 
>>> Non-linear Frequency: 1.51 GHz.
>>>       AMD PSTATE Lowest Performance: 31. Lowest Frequency: 400 MHz.
>>>
>>> Thanks,
>>> Morgan
>>>
>>> -----Original Message-----
>>> From: Mario Limonciello <mario.limonciello@amd.com>
>>> Sent: Tuesday, July 2, 2024 10:49 AM
>>> To: Dhananjay Ugwekar <Dhananjay.Ugwekar@amd.com>; 
>>> rafael@kernel.org; viresh.kumar@linaro.org; gautham.shenoy@amd.com; 
>>> perry.yuan@amd.com; skhan@linuxfoundation.org; li.meng@amd.com; 
>>> ray.huang@amd.com
>>> Cc: linux-pm@vger.kernel.org; linux-kernel@vger.kernel.org; David 
>>> Arcari <darcari@redhat.com>
>>> Subject: Re: [PATCH v2 2/2] cpufreq/amd-pstate: Fix the 
>>> scaling_max_freq setting on shared memory CPPC systems
>>>
>>> On 7/2/2024 3:14, Dhananjay Ugwekar wrote:
>>>> On shared memory CPPC systems, with amd_pstate=active mode, the 
>>>> change in scaling_max_freq doesn't get written to the shared memory 
>>>> region.
>>>> Due to this, the writes to the scaling_max_freq sysfs file don't 
>>>> take effect. Fix this by propagating the scaling_max_freq changes 
>>>> to the shared memory region.
>>>>
>>>> Fixes: ffa5096a7c33 ("cpufreq: amd-pstate: implement Pstate EPP 
>>>> support for the AMD processors")
>>>> Reported-by: David Arcari <darcari@redhat.com>
>>>> Signed-off-by: Dhananjay Ugwekar <Dhananjay.Ugwekar@amd.com>
>>> Reviewed-by: Mario Limonciello <mario.limonciello@amd.com>
>>>> ---
>>>>     drivers/cpufreq/amd-pstate.c | 43
>>>> +++++++++++++++++++-----------------
>>>>     1 file changed, 23 insertions(+), 20 deletions(-)
>>>>
>>>> diff --git a/drivers/cpufreq/amd-pstate.c 
>>>> b/drivers/cpufreq/amd-pstate.c index 9ad62dbe8bfb..a092b13ffbc2
>>>> 100644
>>>> --- a/drivers/cpufreq/amd-pstate.c
>>>> +++ b/drivers/cpufreq/amd-pstate.c
>>>> @@ -247,6 +247,26 @@ static int
>>>> amd_pstate_get_energy_pref_index(struct amd_cpudata *cpudata)
>>>>         return index;
>>>>     }
>>>> +static void pstate_update_perf(struct amd_cpudata *cpudata, u32
>>>> min_perf,
>>>> +                   u32 des_perf, u32 max_perf, bool fast_switch) {
>>>> +    if (fast_switch)
>>>> +        wrmsrl(MSR_AMD_CPPC_REQ, 
>>>> +READ_ONCE(cpudata->cppc_req_cached));
>>>> +    else
>>>> +        wrmsrl_on_cpu(cpudata->cpu, MSR_AMD_CPPC_REQ,
>>>> +                  READ_ONCE(cpudata->cppc_req_cached));
>>>> +}
>>>> +
>>>> +DEFINE_STATIC_CALL(amd_pstate_update_perf, pstate_update_perf);
>>>> +
>>>> +static inline void amd_pstate_update_perf(struct amd_cpudata 
>>>> +*cpudata,
>>>> +                      u32 min_perf, u32 des_perf,
>>>> +                      u32 max_perf, bool fast_switch) {
>>>> +    static_call(amd_pstate_update_perf)(cpudata, min_perf, 
>>>> +des_perf,
>>>> +                        max_perf, fast_switch); }
>>>> +
>>>>     static int amd_pstate_set_epp(struct amd_cpudata *cpudata, u32 
>>>> epp)
>>>>     {
>>>>         int ret;
>>>> @@ -263,6 +283,9 @@ static int amd_pstate_set_epp(struct 
>>>> amd_cpudata *cpudata, u32 epp)
>>>>             if (!ret)
>>>>                 cpudata->epp_cached = epp;
>>>>         } else {
>>>> +        amd_pstate_update_perf(cpudata, cpudata->min_limit_perf, 
>>>> +0U,
>>>> +                         cpudata->max_limit_perf, false);
>>>> +
>>>>             perf_ctrls.energy_perf = epp;
>>>>             ret = cppc_set_epp_perf(cpudata->cpu, &perf_ctrls, 1);
>>>>             if (ret) {
>>>> @@ -452,16 +475,6 @@ static inline int amd_pstate_init_perf(struct 
>>>> amd_cpudata *cpudata)
>>>>         return static_call(amd_pstate_init_perf)(cpudata);
>>>>     }
>>>> -static void pstate_update_perf(struct amd_cpudata *cpudata, u32 
>>>> min_perf,
>>>> -                   u32 des_perf, u32 max_perf, bool fast_switch) 
>>>> -{
>>>> -    if (fast_switch)
>>>> -        wrmsrl(MSR_AMD_CPPC_REQ, 
>>>> READ_ONCE(cpudata->cppc_req_cached));
>>>> -    else
>>>> -        wrmsrl_on_cpu(cpudata->cpu, MSR_AMD_CPPC_REQ,
>>>> -                  READ_ONCE(cpudata->cppc_req_cached));
>>>> -}
>>>> -
>>>>     static void cppc_update_perf(struct amd_cpudata *cpudata,
>>>>                      u32 min_perf, u32 des_perf,
>>>>                      u32 max_perf, bool fast_switch) @@ -475,16
>>>> +488,6 @@
>>>> static void cppc_update_perf(struct amd_cpudata *cpudata,
>>>>         cppc_set_perf(cpudata->cpu, &perf_ctrls);
>>>>     }
>>>> -DEFINE_STATIC_CALL(amd_pstate_update_perf, pstate_update_perf);
>>>> -
>>>> -static inline void amd_pstate_update_perf(struct amd_cpudata 
>>>> *cpudata,
>>>> -                      u32 min_perf, u32 des_perf,
>>>> -                      u32 max_perf, bool fast_switch) -{
>>>> -    static_call(amd_pstate_update_perf)(cpudata, min_perf, 
>>>> des_perf,
>>>> -                        max_perf, fast_switch); -}
>>>> -
>>>>     static inline bool amd_pstate_sample(struct amd_cpudata 
>>>> *cpudata)
>>>>     {
>>>>         u64 aperf, mperf, tsc;
>>>
>>>
>>
> 

RE: [EXTERNAL] Re: [PATCH v2 2/2] cpufreq/amd-pstate: Fix the scaling_max_freq setting on shared memory CPPC systems

[Jones, Morgan]

Just to verify, this was the patch in question, correct? I had some trouble finding it on patchwork.

https://patches.linaro.org/project/linux-acpi/patch/20240227073924.3573398-1-li.meng@amd.com/

-----Original Message-----
From: Mario Limonciello <mario.limonciello@amd.com> 
Sent: Tuesday, September 3, 2024 1:52 PM
To: Jones, Morgan <Morgan.Jones@viasat.com>; Dhananjay Ugwekar <Dhananjay.Ugwekar@amd.com>; rafael@kernel.org; viresh.kumar@linaro.org; gautham.shenoy@amd.com; perry.yuan@amd.com; skhan@linuxfoundation.org; li.meng@amd.com; ray.huang@amd.com
Cc: linux-pm@vger.kernel.org; linux-kernel@vger.kernel.org; David Arcari <darcari@redhat.com>
Subject: Re: [EXTERNAL] Re: [PATCH v2 2/2] cpufreq/amd-pstate: Fix the scaling_max_freq setting on shared memory CPPC systems

Morgan,

This does remind me of a clamping issue that touches some of the same variables.  Can you please see if backporting

commit 8164f7433264 ("cpufreq: amd-pstate: adjust min/max limit perf")

to 6.6.y helps for you?

Thanks,

On 9/3/2024 15:09, Mario Limonciello wrote:
> Morgan,
> 
> OK that's great news that it's just a backport effort.  That same 
> commit also backported to 6.10.3.  Can you see if 6.10.y is affected?
> 
> Ugwekar,
> 
> Any thoughts on what else needs to come back to 6.6.y off hand?
> 
> Thanks,
> 
> On 9/3/2024 15:07, Jones, Morgan wrote:
>> Hey Mario,
>>
>> Smoking gun here, the max frequency is incorrect on 6.6.44+ but is 
>> correct on 6.11.0-rc6.
>>
>> Linux redact 6.11.0-rc6 #1-NixOS SMP PREEMPT_DYNAMIC Tue Jan  1
>> 00:00:00 UTC 1980 x86_64 GNU/Linux
>>
>> analyzing CPU 12:
>>    driver: amd-pstate-epp
>>    CPUs which run at the same hardware frequency: 12
>>    CPUs which need to have their frequency coordinated by software: 
>> 12
>>    maximum transition latency:  Cannot determine or is not supported.
>>    hardware limits: 400 MHz - 3.35 GHz
>>    available cpufreq governors: performance powersave
>>    current policy: frequency should be within 400 MHz and 3.35 GHz.
>>                    The governor "performance" may decide which speed 
>> to use
>>                    within this range.
>>    current CPU frequency: Unable to call hardware
>>    current CPU frequency: 3.34 GHz (asserted by call to kernel)
>>    boost state support:
>>      Supported: yes
>>      Active: yes
>>      AMD PSTATE Highest Performance: 255. Maximum Frequency: 3.35 GHz.
>>      AMD PSTATE Nominal Performance: 152. Nominal Frequency: 2.00 GHz.
>>      AMD PSTATE Lowest Non-linear Performance: 115. Lowest Non-linear
>> Frequency: 1.51 GHz.
>>      AMD PSTATE Lowest Performance: 31. Lowest Frequency: 400 MHz.
>>
>> We're running amd_pstate=active and amd_pstate.shared_mem=1, and our 
>> workloads are back to normal performance on 6.11.0-rc6.
>>
>> Morgan
>>
>> -----Original Message-----
>> From: Mario Limonciello <mario.limonciello@amd.com>
>> Sent: Tuesday, September 3, 2024 10:55 AM
>> To: Jones, Morgan <Morgan.Jones@viasat.com>; Dhananjay Ugwekar 
>> <Dhananjay.Ugwekar@amd.com>; rafael@kernel.org; 
>> viresh.kumar@linaro.org; gautham.shenoy@amd.com; perry.yuan@amd.com; 
>> skhan@linuxfoundation.org; li.meng@amd.com; ray.huang@amd.com
>> Cc: linux-pm@vger.kernel.org; linux-kernel@vger.kernel.org; David 
>> Arcari <darcari@redhat.com>
>> Subject: [EXTERNAL] Re: [PATCH v2 2/2] cpufreq/amd-pstate: Fix the 
>> scaling_max_freq setting on shared memory CPPC systems
>>
>> Hi Morgan,
>>
>> Can you please cross reference 6.11-rc6 to see if you're still having 
>> a problem?  I would like to understand if we're missing a backport to 
>> stable or this is still an upstream problem.
>>
>> Thanks,
>>
>> On 9/3/2024 12:51, Jones, Morgan wrote:
>>> Hi there,
>>>
>>> We are experiencing a ~35% performance regression on an AMD EPYC 
>>> 7702
>>> 64 core machine after applying this patch. We observed Linux 6.6.44 
>>> starting to cause the issue, and performed a bisect involving 
>>> rebooting the machine over 15 times. (Note that kexec didn't 
>>> successfully identify the problem, since the PM memory mailbox is 
>>> never reset).
>>>
>>> It appears that we are getting a max of 2.18 GHz when this CPU can 
>>> boost to 3.35 GHz, explaining the slowdown. Blacklisting amd-pstate 
>>> solves the issue at the expense of the performance increase we used 
>>> to get by using it.
>>>
>>> Is it possible that the upper limits were implicitly at the max CPU 
>>> power before, and setting the upper limit to something other than 
>>> the boost frequency can reduce performance now?
>>>
>>> # bad: [7213910600667c51c978e577bf5454d3f7b313b7] Linux 6.6.44 # good:
>>> [58b0425ff5df680d0b67f64ae1f3f1ebdf1c4de9] Linux 6.6.43 git bisect 
>>> start '7213910600667c51c978e577bf5454d3f7b313b7'
>>> '58b0425ff5df680d0b67f64ae1f3f1ebdf1c4de9'
>>> # good: [72ff9d26964a3a80f7650df719df139f5c1f965d] arm64: dts: qcom:
>>> sm6350: Add missing qcom,non-secure-domain property git bisect good 
>>> 72ff9d26964a3a80f7650df719df139f5c1f965d
>>> # good: [0fffc2e1bf40a2220ef5a38f834ea063dba832d3] ARM: dts: sunxi:
>>> remove duplicated entries in makefile git bisect good
>>> 0fffc2e1bf40a2220ef5a38f834ea063dba832d3
>>> # bad: [8cdbe6ebfd1763a5c41a2a3058497c0a9163311c] pinctrl: renesas:
>>> r8a779g0: Fix CANFD5 suffix git bisect bad 
>>> 8cdbe6ebfd1763a5c41a2a3058497c0a9163311c
>>> # bad: [5dbb98e7fa42bebc1325899193d8f13f0705a148] drm/mediatek: Turn 
>>> off the layers with zero width or height git bisect bad
>>> 5dbb98e7fa42bebc1325899193d8f13f0705a148
>>> # bad: [691ec7043122c9c8c46d84f6e6cd85d13d50cd93] selftests/bpf: 
>>> Null checks for links in bpf_tcp_ca git bisect bad
>>> 691ec7043122c9c8c46d84f6e6cd85d13d50cd93
>>> # bad: [a1359e085d75d7393a250054e66c0a7bc6c3dbfa] perf/x86: 
>>> Serialize
>>> set_attr_rdpmc() git bisect bad
>>> a1359e085d75d7393a250054e66c0a7bc6c3dbfa
>>> # bad: [e99d9b16ff153de9540073239d24adc3b0a3a997] wifi: ath12k: 
>>> change DMA direction while mapping reinjected packets git bisect bad
>>> e99d9b16ff153de9540073239d24adc3b0a3a997
>>> # bad: [d027ac4a08541beb2a89563d3e034da7085050ba] firmware:
>>> turris-mox-rwtm: Initialize completion before mailbox git bisect bad 
>>> d027ac4a08541beb2a89563d3e034da7085050ba
>>> # bad: [e6c9eca327e6a41a81e7eba0d0ddc13da37f82a1] ARM: spitz: fix 
>>> GPIO assignment for backlight git bisect bad
>>> e6c9eca327e6a41a81e7eba0d0ddc13da37f82a1
>>> # bad: [b8cdefdaa555bbfc269c2198803f8791a8923960] m68k: cmpxchg: Fix 
>>> return value for default case in __arch_xchg() git bisect bad
>>> b8cdefdaa555bbfc269c2198803f8791a8923960
>>> # bad: [13a71384ae6a8779da809b00c6f378dcead10427] cpufreq/amd-pstate:
>>> Fix the scaling_max_freq setting on shared memory CPPC systems git 
>>> bisect bad 13a71384ae6a8779da809b00c6f378dcead10427
>>> # first bad commit: [13a71384ae6a8779da809b00c6f378dcead10427]
>>> cpufreq/amd-pstate: Fix the scaling_max_freq setting on shared 
>>> memory CPPC systems
>>>
>>> cpupower output:
>>>
>>> analyzing CPU 47:
>>>     driver: amd-pstate-epp
>>>     CPUs which run at the same hardware frequency: 47
>>>     CPUs which need to have their frequency coordinated by software: 
>>> 47
>>>     maximum transition latency:  Cannot determine or is not supported.
>>>     hardware limits: 400 MHz - 2.18 GHz
>>>     available cpufreq governors: performance powersave
>>>     current policy: frequency should be within 400 MHz and 2.18 GHz.
>>>                     The governor "performance" may decide which 
>>> speed to use
>>>                     within this range.
>>>     current CPU frequency: Unable to call hardware
>>>     current CPU frequency: 2.17 GHz (asserted by call to kernel)
>>>     boost state support:
>>>       Supported: yes
>>>       Active: yes
>>>       AMD PSTATE Highest Performance: 166. Maximum Frequency: 2.18 GHz.
>>>       AMD PSTATE Nominal Performance: 152. Nominal Frequency: 2.00 GHz.
>>>       AMD PSTATE Lowest Non-linear Performance: 115. Lowest 
>>> Non-linear Frequency: 1.51 GHz.
>>>       AMD PSTATE Lowest Performance: 31. Lowest Frequency: 400 MHz.
>>>
>>> Thanks,
>>> Morgan
>>>
>>> -----Original Message-----
>>> From: Mario Limonciello <mario.limonciello@amd.com>
>>> Sent: Tuesday, July 2, 2024 10:49 AM
>>> To: Dhananjay Ugwekar <Dhananjay.Ugwekar@amd.com>; 
>>> rafael@kernel.org; viresh.kumar@linaro.org; gautham.shenoy@amd.com; 
>>> perry.yuan@amd.com; skhan@linuxfoundation.org; li.meng@amd.com; 
>>> ray.huang@amd.com
>>> Cc: linux-pm@vger.kernel.org; linux-kernel@vger.kernel.org; David 
>>> Arcari <darcari@redhat.com>
>>> Subject: Re: [PATCH v2 2/2] cpufreq/amd-pstate: Fix the 
>>> scaling_max_freq setting on shared memory CPPC systems
>>>
>>> On 7/2/2024 3:14, Dhananjay Ugwekar wrote:
>>>> On shared memory CPPC systems, with amd_pstate=active mode, the 
>>>> change in scaling_max_freq doesn't get written to the shared memory 
>>>> region.
>>>> Due to this, the writes to the scaling_max_freq sysfs file don't 
>>>> take effect. Fix this by propagating the scaling_max_freq changes 
>>>> to the shared memory region.
>>>>
>>>> Fixes: ffa5096a7c33 ("cpufreq: amd-pstate: implement Pstate EPP 
>>>> support for the AMD processors")
>>>> Reported-by: David Arcari <darcari@redhat.com>
>>>> Signed-off-by: Dhananjay Ugwekar <Dhananjay.Ugwekar@amd.com>
>>> Reviewed-by: Mario Limonciello <mario.limonciello@amd.com>
>>>> ---
>>>>     drivers/cpufreq/amd-pstate.c | 43
>>>> +++++++++++++++++++-----------------
>>>>     1 file changed, 23 insertions(+), 20 deletions(-)
>>>>
>>>> diff --git a/drivers/cpufreq/amd-pstate.c 
>>>> b/drivers/cpufreq/amd-pstate.c index 9ad62dbe8bfb..a092b13ffbc2
>>>> 100644
>>>> --- a/drivers/cpufreq/amd-pstate.c
>>>> +++ b/drivers/cpufreq/amd-pstate.c
>>>> @@ -247,6 +247,26 @@ static int
>>>> amd_pstate_get_energy_pref_index(struct amd_cpudata *cpudata)
>>>>         return index;
>>>>     }
>>>> +static void pstate_update_perf(struct amd_cpudata *cpudata, u32
>>>> min_perf,
>>>> +                   u32 des_perf, u32 max_perf, bool fast_switch) {
>>>> +    if (fast_switch)
>>>> +        wrmsrl(MSR_AMD_CPPC_REQ, 
>>>> +READ_ONCE(cpudata->cppc_req_cached));
>>>> +    else
>>>> +        wrmsrl_on_cpu(cpudata->cpu, MSR_AMD_CPPC_REQ,
>>>> +                  READ_ONCE(cpudata->cppc_req_cached));
>>>> +}
>>>> +
>>>> +DEFINE_STATIC_CALL(amd_pstate_update_perf, pstate_update_perf);
>>>> +
>>>> +static inline void amd_pstate_update_perf(struct amd_cpudata 
>>>> +*cpudata,
>>>> +                      u32 min_perf, u32 des_perf,
>>>> +                      u32 max_perf, bool fast_switch) {
>>>> +    static_call(amd_pstate_update_perf)(cpudata, min_perf, 
>>>> +des_perf,
>>>> +                        max_perf, fast_switch); }
>>>> +
>>>>     static int amd_pstate_set_epp(struct amd_cpudata *cpudata, u32 
>>>> epp)
>>>>     {
>>>>         int ret;
>>>> @@ -263,6 +283,9 @@ static int amd_pstate_set_epp(struct 
>>>> amd_cpudata *cpudata, u32 epp)
>>>>             if (!ret)
>>>>                 cpudata->epp_cached = epp;
>>>>         } else {
>>>> +        amd_pstate_update_perf(cpudata, cpudata->min_limit_perf, 
>>>> +0U,
>>>> +                         cpudata->max_limit_perf, false);
>>>> +
>>>>             perf_ctrls.energy_perf = epp;
>>>>             ret = cppc_set_epp_perf(cpudata->cpu, &perf_ctrls, 1);
>>>>             if (ret) {
>>>> @@ -452,16 +475,6 @@ static inline int amd_pstate_init_perf(struct 
>>>> amd_cpudata *cpudata)
>>>>         return static_call(amd_pstate_init_perf)(cpudata);
>>>>     }
>>>> -static void pstate_update_perf(struct amd_cpudata *cpudata, u32 
>>>> min_perf,
>>>> -                   u32 des_perf, u32 max_perf, bool fast_switch) 
>>>> -{
>>>> -    if (fast_switch)
>>>> -        wrmsrl(MSR_AMD_CPPC_REQ, 
>>>> READ_ONCE(cpudata->cppc_req_cached));
>>>> -    else
>>>> -        wrmsrl_on_cpu(cpudata->cpu, MSR_AMD_CPPC_REQ,
>>>> -                  READ_ONCE(cpudata->cppc_req_cached));
>>>> -}
>>>> -
>>>>     static void cppc_update_perf(struct amd_cpudata *cpudata,
>>>>                      u32 min_perf, u32 des_perf,
>>>>                      u32 max_perf, bool fast_switch) @@ -475,16
>>>> +488,6 @@
>>>> static void cppc_update_perf(struct amd_cpudata *cpudata,
>>>>         cppc_set_perf(cpudata->cpu, &perf_ctrls);
>>>>     }
>>>> -DEFINE_STATIC_CALL(amd_pstate_update_perf, pstate_update_perf);
>>>> -
>>>> -static inline void amd_pstate_update_perf(struct amd_cpudata 
>>>> *cpudata,
>>>> -                      u32 min_perf, u32 des_perf,
>>>> -                      u32 max_perf, bool fast_switch) -{
>>>> -    static_call(amd_pstate_update_perf)(cpudata, min_perf, 
>>>> des_perf,
>>>> -                        max_perf, fast_switch); -}
>>>> -
>>>>     static inline bool amd_pstate_sample(struct amd_cpudata 
>>>> *cpudata)
>>>>     {
>>>>         u64 aperf, mperf, tsc;
>>>
>>>
>>
> 

Re: [EXTERNAL] Re: [PATCH v2 2/2] cpufreq/amd-pstate: Fix the scaling_max_freq setting on shared memory CPPC systems

[Mario Limonciello]

Morgan,

I was referring specfiically to the version that landed in Linus' tree:
https://git.kernel.org/torvalds/c/8164f7433264

But yeah it's effectively the same thing.  In any case, it's not the 
solution.

We had some internal discussion and suspect this is due to missing 
prefcore patches in 6.6 as that feature landed in 6.9.  We'll try to 
reproduce this on a Rome system and come back with our findings and 
suggestions what to do.

Thanks,

On 9/3/2024 17:24, Jones, Morgan wrote:
> Just to verify, this was the patch in question, correct? I had some trouble finding it on patchwork.
> 
> https://patches.linaro.org/project/linux-acpi/patch/20240227073924.3573398-1-li.meng@amd.com/
> 
> -----Original Message-----
> From: Mario Limonciello <mario.limonciello@amd.com>
> Sent: Tuesday, September 3, 2024 1:52 PM
> To: Jones, Morgan <Morgan.Jones@viasat.com>; Dhananjay Ugwekar <Dhananjay.Ugwekar@amd.com>; rafael@kernel.org; viresh.kumar@linaro.org; gautham.shenoy@amd.com; perry.yuan@amd.com; skhan@linuxfoundation.org; li.meng@amd.com; ray.huang@amd.com
> Cc: linux-pm@vger.kernel.org; linux-kernel@vger.kernel.org; David Arcari <darcari@redhat.com>
> Subject: Re: [EXTERNAL] Re: [PATCH v2 2/2] cpufreq/amd-pstate: Fix the scaling_max_freq setting on shared memory CPPC systems
> 
> Morgan,
> 
> This does remind me of a clamping issue that touches some of the same variables.  Can you please see if backporting
> 
> commit 8164f7433264 ("cpufreq: amd-pstate: adjust min/max limit perf")
> 
> to 6.6.y helps for you?
> 
> Thanks,
> 
> On 9/3/2024 15:09, Mario Limonciello wrote:
>> Morgan,
>>
>> OK that's great news that it's just a backport effort.  That same
>> commit also backported to 6.10.3.  Can you see if 6.10.y is affected?
>>
>> Ugwekar,
>>
>> Any thoughts on what else needs to come back to 6.6.y off hand?
>>
>> Thanks,
>>
>> On 9/3/2024 15:07, Jones, Morgan wrote:
>>> Hey Mario,
>>>
>>> Smoking gun here, the max frequency is incorrect on 6.6.44+ but is
>>> correct on 6.11.0-rc6.
>>>
>>> Linux redact 6.11.0-rc6 #1-NixOS SMP PREEMPT_DYNAMIC Tue Jan  1
>>> 00:00:00 UTC 1980 x86_64 GNU/Linux
>>>
>>> analyzing CPU 12:
>>>     driver: amd-pstate-epp
>>>     CPUs which run at the same hardware frequency: 12
>>>     CPUs which need to have their frequency coordinated by software:
>>> 12
>>>     maximum transition latency:  Cannot determine or is not supported.
>>>     hardware limits: 400 MHz - 3.35 GHz
>>>     available cpufreq governors: performance powersave
>>>     current policy: frequency should be within 400 MHz and 3.35 GHz.
>>>                     The governor "performance" may decide which speed
>>> to use
>>>                     within this range.
>>>     current CPU frequency: Unable to call hardware
>>>     current CPU frequency: 3.34 GHz (asserted by call to kernel)
>>>     boost state support:
>>>       Supported: yes
>>>       Active: yes
>>>       AMD PSTATE Highest Performance: 255. Maximum Frequency: 3.35 GHz.
>>>       AMD PSTATE Nominal Performance: 152. Nominal Frequency: 2.00 GHz.
>>>       AMD PSTATE Lowest Non-linear Performance: 115. Lowest Non-linear
>>> Frequency: 1.51 GHz.
>>>       AMD PSTATE Lowest Performance: 31. Lowest Frequency: 400 MHz.
>>>
>>> We're running amd_pstate=active and amd_pstate.shared_mem=1, and our
>>> workloads are back to normal performance on 6.11.0-rc6.
>>>
>>> Morgan
>>>
>>> -----Original Message-----
>>> From: Mario Limonciello <mario.limonciello@amd.com>
>>> Sent: Tuesday, September 3, 2024 10:55 AM
>>> To: Jones, Morgan <Morgan.Jones@viasat.com>; Dhananjay Ugwekar
>>> <Dhananjay.Ugwekar@amd.com>; rafael@kernel.org;
>>> viresh.kumar@linaro.org; gautham.shenoy@amd.com; perry.yuan@amd.com;
>>> skhan@linuxfoundation.org; li.meng@amd.com; ray.huang@amd.com
>>> Cc: linux-pm@vger.kernel.org; linux-kernel@vger.kernel.org; David
>>> Arcari <darcari@redhat.com>
>>> Subject: [EXTERNAL] Re: [PATCH v2 2/2] cpufreq/amd-pstate: Fix the
>>> scaling_max_freq setting on shared memory CPPC systems
>>>
>>> Hi Morgan,
>>>
>>> Can you please cross reference 6.11-rc6 to see if you're still having
>>> a problem?  I would like to understand if we're missing a backport to
>>> stable or this is still an upstream problem.
>>>
>>> Thanks,
>>>
>>> On 9/3/2024 12:51, Jones, Morgan wrote:
>>>> Hi there,
>>>>
>>>> We are experiencing a ~35% performance regression on an AMD EPYC
>>>> 7702
>>>> 64 core machine after applying this patch. We observed Linux 6.6.44
>>>> starting to cause the issue, and performed a bisect involving
>>>> rebooting the machine over 15 times. (Note that kexec didn't
>>>> successfully identify the problem, since the PM memory mailbox is
>>>> never reset).
>>>>
>>>> It appears that we are getting a max of 2.18 GHz when this CPU can
>>>> boost to 3.35 GHz, explaining the slowdown. Blacklisting amd-pstate
>>>> solves the issue at the expense of the performance increase we used
>>>> to get by using it.
>>>>
>>>> Is it possible that the upper limits were implicitly at the max CPU
>>>> power before, and setting the upper limit to something other than
>>>> the boost frequency can reduce performance now?
>>>>
>>>> # bad: [7213910600667c51c978e577bf5454d3f7b313b7] Linux 6.6.44 # good:
>>>> [58b0425ff5df680d0b67f64ae1f3f1ebdf1c4de9] Linux 6.6.43 git bisect
>>>> start '7213910600667c51c978e577bf5454d3f7b313b7'
>>>> '58b0425ff5df680d0b67f64ae1f3f1ebdf1c4de9'
>>>> # good: [72ff9d26964a3a80f7650df719df139f5c1f965d] arm64: dts: qcom:
>>>> sm6350: Add missing qcom,non-secure-domain property git bisect good
>>>> 72ff9d26964a3a80f7650df719df139f5c1f965d
>>>> # good: [0fffc2e1bf40a2220ef5a38f834ea063dba832d3] ARM: dts: sunxi:
>>>> remove duplicated entries in makefile git bisect good
>>>> 0fffc2e1bf40a2220ef5a38f834ea063dba832d3
>>>> # bad: [8cdbe6ebfd1763a5c41a2a3058497c0a9163311c] pinctrl: renesas:
>>>> r8a779g0: Fix CANFD5 suffix git bisect bad
>>>> 8cdbe6ebfd1763a5c41a2a3058497c0a9163311c
>>>> # bad: [5dbb98e7fa42bebc1325899193d8f13f0705a148] drm/mediatek: Turn
>>>> off the layers with zero width or height git bisect bad
>>>> 5dbb98e7fa42bebc1325899193d8f13f0705a148
>>>> # bad: [691ec7043122c9c8c46d84f6e6cd85d13d50cd93] selftests/bpf:
>>>> Null checks for links in bpf_tcp_ca git bisect bad
>>>> 691ec7043122c9c8c46d84f6e6cd85d13d50cd93
>>>> # bad: [a1359e085d75d7393a250054e66c0a7bc6c3dbfa] perf/x86:
>>>> Serialize
>>>> set_attr_rdpmc() git bisect bad
>>>> a1359e085d75d7393a250054e66c0a7bc6c3dbfa
>>>> # bad: [e99d9b16ff153de9540073239d24adc3b0a3a997] wifi: ath12k:
>>>> change DMA direction while mapping reinjected packets git bisect bad
>>>> e99d9b16ff153de9540073239d24adc3b0a3a997
>>>> # bad: [d027ac4a08541beb2a89563d3e034da7085050ba] firmware:
>>>> turris-mox-rwtm: Initialize completion before mailbox git bisect bad
>>>> d027ac4a08541beb2a89563d3e034da7085050ba
>>>> # bad: [e6c9eca327e6a41a81e7eba0d0ddc13da37f82a1] ARM: spitz: fix
>>>> GPIO assignment for backlight git bisect bad
>>>> e6c9eca327e6a41a81e7eba0d0ddc13da37f82a1
>>>> # bad: [b8cdefdaa555bbfc269c2198803f8791a8923960] m68k: cmpxchg: Fix
>>>> return value for default case in __arch_xchg() git bisect bad
>>>> b8cdefdaa555bbfc269c2198803f8791a8923960
>>>> # bad: [13a71384ae6a8779da809b00c6f378dcead10427] cpufreq/amd-pstate:
>>>> Fix the scaling_max_freq setting on shared memory CPPC systems git
>>>> bisect bad 13a71384ae6a8779da809b00c6f378dcead10427
>>>> # first bad commit: [13a71384ae6a8779da809b00c6f378dcead10427]
>>>> cpufreq/amd-pstate: Fix the scaling_max_freq setting on shared
>>>> memory CPPC systems
>>>>
>>>> cpupower output:
>>>>
>>>> analyzing CPU 47:
>>>>      driver: amd-pstate-epp
>>>>      CPUs which run at the same hardware frequency: 47
>>>>      CPUs which need to have their frequency coordinated by software:
>>>> 47
>>>>      maximum transition latency:  Cannot determine or is not supported.
>>>>      hardware limits: 400 MHz - 2.18 GHz
>>>>      available cpufreq governors: performance powersave
>>>>      current policy: frequency should be within 400 MHz and 2.18 GHz.
>>>>                      The governor "performance" may decide which
>>>> speed to use
>>>>                      within this range.
>>>>      current CPU frequency: Unable to call hardware
>>>>      current CPU frequency: 2.17 GHz (asserted by call to kernel)
>>>>      boost state support:
>>>>        Supported: yes
>>>>        Active: yes
>>>>        AMD PSTATE Highest Performance: 166. Maximum Frequency: 2.18 GHz.
>>>>        AMD PSTATE Nominal Performance: 152. Nominal Frequency: 2.00 GHz.
>>>>        AMD PSTATE Lowest Non-linear Performance: 115. Lowest
>>>> Non-linear Frequency: 1.51 GHz.
>>>>        AMD PSTATE Lowest Performance: 31. Lowest Frequency: 400 MHz.
>>>>
>>>> Thanks,
>>>> Morgan
>>>>
>>>> -----Original Message-----
>>>> From: Mario Limonciello <mario.limonciello@amd.com>
>>>> Sent: Tuesday, July 2, 2024 10:49 AM
>>>> To: Dhananjay Ugwekar <Dhananjay.Ugwekar@amd.com>;
>>>> rafael@kernel.org; viresh.kumar@linaro.org; gautham.shenoy@amd.com;
>>>> perry.yuan@amd.com; skhan@linuxfoundation.org; li.meng@amd.com;
>>>> ray.huang@amd.com
>>>> Cc: linux-pm@vger.kernel.org; linux-kernel@vger.kernel.org; David
>>>> Arcari <darcari@redhat.com>
>>>> Subject: Re: [PATCH v2 2/2] cpufreq/amd-pstate: Fix the
>>>> scaling_max_freq setting on shared memory CPPC systems
>>>>
>>>> On 7/2/2024 3:14, Dhananjay Ugwekar wrote:
>>>>> On shared memory CPPC systems, with amd_pstate=active mode, the
>>>>> change in scaling_max_freq doesn't get written to the shared memory
>>>>> region.
>>>>> Due to this, the writes to the scaling_max_freq sysfs file don't
>>>>> take effect. Fix this by propagating the scaling_max_freq changes
>>>>> to the shared memory region.
>>>>>
>>>>> Fixes: ffa5096a7c33 ("cpufreq: amd-pstate: implement Pstate EPP
>>>>> support for the AMD processors")
>>>>> Reported-by: David Arcari <darcari@redhat.com>
>>>>> Signed-off-by: Dhananjay Ugwekar <Dhananjay.Ugwekar@amd.com>
>>>> Reviewed-by: Mario Limonciello <mario.limonciello@amd.com>
>>>>> ---
>>>>>      drivers/cpufreq/amd-pstate.c | 43
>>>>> +++++++++++++++++++-----------------
>>>>>      1 file changed, 23 insertions(+), 20 deletions(-)
>>>>>
>>>>> diff --git a/drivers/cpufreq/amd-pstate.c
>>>>> b/drivers/cpufreq/amd-pstate.c index 9ad62dbe8bfb..a092b13ffbc2
>>>>> 100644
>>>>> --- a/drivers/cpufreq/amd-pstate.c
>>>>> +++ b/drivers/cpufreq/amd-pstate.c
>>>>> @@ -247,6 +247,26 @@ static int
>>>>> amd_pstate_get_energy_pref_index(struct amd_cpudata *cpudata)
>>>>>          return index;
>>>>>      }
>>>>> +static void pstate_update_perf(struct amd_cpudata *cpudata, u32
>>>>> min_perf,
>>>>> +                   u32 des_perf, u32 max_perf, bool fast_switch) {
>>>>> +    if (fast_switch)
>>>>> +        wrmsrl(MSR_AMD_CPPC_REQ,
>>>>> +READ_ONCE(cpudata->cppc_req_cached));
>>>>> +    else
>>>>> +        wrmsrl_on_cpu(cpudata->cpu, MSR_AMD_CPPC_REQ,
>>>>> +                  READ_ONCE(cpudata->cppc_req_cached));
>>>>> +}
>>>>> +
>>>>> +DEFINE_STATIC_CALL(amd_pstate_update_perf, pstate_update_perf);
>>>>> +
>>>>> +static inline void amd_pstate_update_perf(struct amd_cpudata
>>>>> +*cpudata,
>>>>> +                      u32 min_perf, u32 des_perf,
>>>>> +                      u32 max_perf, bool fast_switch) {
>>>>> +    static_call(amd_pstate_update_perf)(cpudata, min_perf,
>>>>> +des_perf,
>>>>> +                        max_perf, fast_switch); }
>>>>> +
>>>>>      static int amd_pstate_set_epp(struct amd_cpudata *cpudata, u32
>>>>> epp)
>>>>>      {
>>>>>          int ret;
>>>>> @@ -263,6 +283,9 @@ static int amd_pstate_set_epp(struct
>>>>> amd_cpudata *cpudata, u32 epp)
>>>>>              if (!ret)
>>>>>                  cpudata->epp_cached = epp;
>>>>>          } else {
>>>>> +        amd_pstate_update_perf(cpudata, cpudata->min_limit_perf,
>>>>> +0U,
>>>>> +                         cpudata->max_limit_perf, false);
>>>>> +
>>>>>              perf_ctrls.energy_perf = epp;
>>>>>              ret = cppc_set_epp_perf(cpudata->cpu, &perf_ctrls, 1);
>>>>>              if (ret) {
>>>>> @@ -452,16 +475,6 @@ static inline int amd_pstate_init_perf(struct
>>>>> amd_cpudata *cpudata)
>>>>>          return static_call(amd_pstate_init_perf)(cpudata);
>>>>>      }
>>>>> -static void pstate_update_perf(struct amd_cpudata *cpudata, u32
>>>>> min_perf,
>>>>> -                   u32 des_perf, u32 max_perf, bool fast_switch)
>>>>> -{
>>>>> -    if (fast_switch)
>>>>> -        wrmsrl(MSR_AMD_CPPC_REQ,
>>>>> READ_ONCE(cpudata->cppc_req_cached));
>>>>> -    else
>>>>> -        wrmsrl_on_cpu(cpudata->cpu, MSR_AMD_CPPC_REQ,
>>>>> -                  READ_ONCE(cpudata->cppc_req_cached));
>>>>> -}
>>>>> -
>>>>>      static void cppc_update_perf(struct amd_cpudata *cpudata,
>>>>>                       u32 min_perf, u32 des_perf,
>>>>>                       u32 max_perf, bool fast_switch) @@ -475,16
>>>>> +488,6 @@
>>>>> static void cppc_update_perf(struct amd_cpudata *cpudata,
>>>>>          cppc_set_perf(cpudata->cpu, &perf_ctrls);
>>>>>      }
>>>>> -DEFINE_STATIC_CALL(amd_pstate_update_perf, pstate_update_perf);
>>>>> -
>>>>> -static inline void amd_pstate_update_perf(struct amd_cpudata
>>>>> *cpudata,
>>>>> -                      u32 min_perf, u32 des_perf,
>>>>> -                      u32 max_perf, bool fast_switch) -{
>>>>> -    static_call(amd_pstate_update_perf)(cpudata, min_perf,
>>>>> des_perf,
>>>>> -                        max_perf, fast_switch); -}
>>>>> -
>>>>>      static inline bool amd_pstate_sample(struct amd_cpudata
>>>>> *cpudata)
>>>>>      {
>>>>>          u64 aperf, mperf, tsc;
>>>>
>>>>
>>>
>>
> 

Re: [EXTERNAL] Re: [PATCH v2 2/2] cpufreq/amd-pstate: Fix the scaling_max_freq setting on shared memory CPPC systems

[Mario Limonciello]

Hi Morgan,

Please apply these 3 commits:

commit 12753d71e8c5 ("ACPI: CPPC: Add helper to get the highest 
performance value")
commit ed429c686b79 ("cpufreq: amd-pstate: Enable amd-pstate preferred 
core support")
commit 3d291fe47fe1 ("cpufreq: amd-pstate: fix the highest frequency 
issue which limits performance")

The first two should help your system, the third will prevent 
introducing a regression on a different one.

Assuming that works we should ask @stable to pull all 3 in to fix this 
regression.

Thanks,

On 9/4/2024 08:57, Mario Limonciello wrote:
> Morgan,
> 
> I was referring specfiically to the version that landed in Linus' tree:
> https://git.kernel.org/torvalds/c/8164f7433264
> 
> But yeah it's effectively the same thing.  In any case, it's not the 
> solution.
> 
> We had some internal discussion and suspect this is due to missing 
> prefcore patches in 6.6 as that feature landed in 6.9.  We'll try to 
> reproduce this on a Rome system and come back with our findings and 
> suggestions what to do.
> 
> Thanks,
> 

RE: [EXTERNAL] Re: [PATCH v2 2/2] cpufreq/amd-pstate: Fix the scaling_max_freq setting on shared memory CPPC systems

[Jones, Morgan]

Mario,

Confirmed. Thank you for the help! Slightly different refs on my end:

Remotes:

next    https://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git (fetch)
next    https://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git (push)
origin  git://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git (fetch)
origin  git://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git (push)
superm1 https://git.kernel.org/pub/scm/linux/kernel/git/superm1/linux.git/ (fetch)
superm1 https://git.kernel.org/pub/scm/linux/kernel/git/superm1/linux.git/ (push)
torvalds        git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git (fetch)
torvalds        git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git (push)

Patches:

git format-patch 12753d71e8c5^..12753d71e8c5
git format-patch f3a052391822b772b4e27f2594526cf1eb103cab^..f3a052391822b772b4e27f2594526cf1eb103cab
git format-patch bf202e654bfa57fb8cf9d93d4c6855890b70b9c4^..bf202e654bfa57fb8cf9d93d4c6855890b70b9c4

Results:

Linux redact 6.6.48 #1-NixOS SMP PREEMPT_DYNAMIC Tue Jan  1 00:00:00 UTC 1980 x86_64 GNU/Linux

analyzing CPU 56:
  driver: amd-pstate-epp
  CPUs which run at the same hardware frequency: 56
  CPUs which need to have their frequency coordinated by software: 56
  maximum transition latency:  Cannot determine or is not supported.
  hardware limits: 400 MHz - 3.35 GHz
  available cpufreq governors: performance powersave
  current policy: frequency should be within 400 MHz and 3.35 GHz.
                  The governor "performance" may decide which speed to use
                  within this range.
  current CPU frequency: Unable to call hardware
  current CPU frequency: 2.09 GHz (asserted by call to kernel)
  boost state support:
    Supported: yes
    Active: yes
    AMD PSTATE Highest Performance: 255. Maximum Frequency: 3.35 GHz.
    AMD PSTATE Nominal Performance: 152. Nominal Frequency: 2.00 GHz.
    AMD PSTATE Lowest Non-linear Performance: 115. Lowest Non-linear Frequency: 1.51 GHz.
    AMD PSTATE Lowest Performance: 31. Lowest Frequency: 400 MHz.

And our builds are back to being fast with `amd_pstate=active amd_prefcore=enable amd_pstate.shared_mem=1`.

Morgan

-----Original Message-----
From: Mario Limonciello <mario.limonciello@amd.com> 
Sent: Thursday, September 5, 2024 8:12 AM
To: Jones, Morgan <Morgan.Jones@viasat.com>
Cc: linux-pm@vger.kernel.org; linux-kernel@vger.kernel.org; David Arcari <darcari@redhat.com>; Dhananjay Ugwekar <Dhananjay.Ugwekar@amd.com>; rafael@kernel.org; viresh.kumar@linaro.org; gautham.shenoy@amd.com; perry.yuan@amd.com; skhan@linuxfoundation.org; li.meng@amd.com; ray.huang@amd.com
Subject: Re: [EXTERNAL] Re: [PATCH v2 2/2] cpufreq/amd-pstate: Fix the scaling_max_freq setting on shared memory CPPC systems

Hi Morgan,

Please apply these 3 commits:

commit 12753d71e8c5 ("ACPI: CPPC: Add helper to get the highest performance value") commit ed429c686b79 ("cpufreq: amd-pstate: Enable amd-pstate preferred core support") commit 3d291fe47fe1 ("cpufreq: amd-pstate: fix the highest frequency issue which limits performance")

The first two should help your system, the third will prevent introducing a regression on a different one.

Assuming that works we should ask @stable to pull all 3 in to fix this regression.

Thanks,

On 9/4/2024 08:57, Mario Limonciello wrote:
> Morgan,
> 
> I was referring specfiically to the version that landed in Linus' tree:
> https://urldefense.us/v3/__https://git.kernel.org/torvalds/c/8164f7433
> 264__;!!C5Asm8uRnZQmlRln!aIZEDEbIUKD7OrxN0b0KjoqKYDL2yMkwk4EK7x_oSnyHQ
> 6MEq7yt6JHjd0TD9DgEYEWDcF58OKL8c7G11bT3dSqL8eM$
> 
> But yeah it's effectively the same thing.  In any case, it's not the 
> solution.
> 
> We had some internal discussion and suspect this is due to missing 
> prefcore patches in 6.6 as that feature landed in 6.9.  We'll try to 
> reproduce this on a Rome system and come back with our findings and 
> suggestions what to do.
> 
> Thanks,
> 

linux-6.6.y regression on amd-pstate

[Mario Limonciello]

+ stable
+ regressions
New subject

Great news.

Greg, Sasha,

Can you please pull in these 3 commits specifically to 6.6.y to fix a 
regression that was reported by Morgan in 6.6.y:

commit 12753d71e8c5 ("ACPI: CPPC: Add helper to get the highest 
performance value")
commit ed429c686b79 ("cpufreq: amd-pstate: Enable amd-pstate preferred 
core support")
commit 3d291fe47fe1 ("cpufreq: amd-pstate: fix the highest frequency 
issue which limits performance")

Further details are below.

Thanks!

On 9/5/2024 16:09, Jones, Morgan wrote:
> Mario,
> 
> Confirmed. Thank you for the help! Slightly different refs on my end:
> 
> Remotes:
> 
> next    https://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git (fetch)
> next    https://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git (push)
> origin  git://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git (fetch)
> origin  git://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git (push)
> superm1 https://git.kernel.org/pub/scm/linux/kernel/git/superm1/linux.git/ (fetch)
> superm1 https://git.kernel.org/pub/scm/linux/kernel/git/superm1/linux.git/ (push)
> torvalds        git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git (fetch)
> torvalds        git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git (push)
> 
> Patches:
> 
> git format-patch 12753d71e8c5^..12753d71e8c5
> git format-patch f3a052391822b772b4e27f2594526cf1eb103cab^..f3a052391822b772b4e27f2594526cf1eb103cab
> git format-patch bf202e654bfa57fb8cf9d93d4c6855890b70b9c4^..bf202e654bfa57fb8cf9d93d4c6855890b70b9c4
> 
> Results:
> 
> Linux redact 6.6.48 #1-NixOS SMP PREEMPT_DYNAMIC Tue Jan  1 00:00:00 UTC 1980 x86_64 GNU/Linux
> 
> analyzing CPU 56:
>    driver: amd-pstate-epp
>    CPUs which run at the same hardware frequency: 56
>    CPUs which need to have their frequency coordinated by software: 56
>    maximum transition latency:  Cannot determine or is not supported.
>    hardware limits: 400 MHz - 3.35 GHz
>    available cpufreq governors: performance powersave
>    current policy: frequency should be within 400 MHz and 3.35 GHz.
>                    The governor "performance" may decide which speed to use
>                    within this range.
>    current CPU frequency: Unable to call hardware
>    current CPU frequency: 2.09 GHz (asserted by call to kernel)
>    boost state support:
>      Supported: yes
>      Active: yes
>      AMD PSTATE Highest Performance: 255. Maximum Frequency: 3.35 GHz.
>      AMD PSTATE Nominal Performance: 152. Nominal Frequency: 2.00 GHz.
>      AMD PSTATE Lowest Non-linear Performance: 115. Lowest Non-linear Frequency: 1.51 GHz.
>      AMD PSTATE Lowest Performance: 31. Lowest Frequency: 400 MHz.
> 
> And our builds are back to being fast with `amd_pstate=active amd_prefcore=enable amd_pstate.shared_mem=1`.
> 
> Morgan
> 
> -----Original Message-----
> From: Mario Limonciello <mario.limonciello@amd.com>
> Sent: Thursday, September 5, 2024 8:12 AM
> To: Jones, Morgan <Morgan.Jones@viasat.com>
> Cc: linux-pm@vger.kernel.org; linux-kernel@vger.kernel.org; David Arcari <darcari@redhat.com>; Dhananjay Ugwekar <Dhananjay.Ugwekar@amd.com>; rafael@kernel.org; viresh.kumar@linaro.org; gautham.shenoy@amd.com; perry.yuan@amd.com; skhan@linuxfoundation.org; li.meng@amd.com; ray.huang@amd.com
> Subject: Re: [EXTERNAL] Re: [PATCH v2 2/2] cpufreq/amd-pstate: Fix the scaling_max_freq setting on shared memory CPPC systems
> 
> Hi Morgan,
> 
> Please apply these 3 commits:
> 
> commit 12753d71e8c5 ("ACPI: CPPC: Add helper to get the highest performance value") commit ed429c686b79 ("cpufreq: amd-pstate: Enable amd-pstate preferred core support") commit 3d291fe47fe1 ("cpufreq: amd-pstate: fix the highest frequency issue which limits performance")
> 
> The first two should help your system, the third will prevent introducing a regression on a different one.
> 
> Assuming that works we should ask @stable to pull all 3 in to fix this regression.
> 
> Thanks,
> 
> On 9/4/2024 08:57, Mario Limonciello wrote:
>> Morgan,
>>
>> I was referring specfiically to the version that landed in Linus' tree:
>> https://urldefense.us/v3/__https://git.kernel.org/torvalds/c/8164f7433
>> 264__;!!C5Asm8uRnZQmlRln!aIZEDEbIUKD7OrxN0b0KjoqKYDL2yMkwk4EK7x_oSnyHQ
>> 6MEq7yt6JHjd0TD9DgEYEWDcF58OKL8c7G11bT3dSqL8eM$
>>
>> But yeah it's effectively the same thing.  In any case, it's not the
>> solution.
>>
>> We had some internal discussion and suspect this is due to missing
>> prefcore patches in 6.6 as that feature landed in 6.9.  We'll try to
>> reproduce this on a Rome system and come back with our findings and
>> suggestions what to do.
>>
>> Thanks,
>>
> 

Re: linux-6.6.y regression on amd-pstate

[Greg Kroah-Hartman]

On Thu, Sep 05, 2024 at 04:14:26PM -0500, Mario Limonciello wrote:
> + stable
> + regressions
> New subject
> 
> Great news.
> 
> Greg, Sasha,
> 
> Can you please pull in these 3 commits specifically to 6.6.y to fix a
> regression that was reported by Morgan in 6.6.y:
> 
> commit 12753d71e8c5 ("ACPI: CPPC: Add helper to get the highest performance
> value")

This is fine, but:

> commit ed429c686b79 ("cpufreq: amd-pstate: Enable amd-pstate preferred core
> support")

This is not a valid git id in Linus's tree :(

> commit 3d291fe47fe1 ("cpufreq: amd-pstate: fix the highest frequency issue
> which limits performance")

And neither is this :(

So perhaps you got them wrong?

thanks,

greg k-h

Re: linux-6.6.y regression on amd-pstate

[Christian Heusel]

[-- Attachment #1: Type: text/plain, Size: 1129 bytes --]

Hey Greg,

On 24/09/08 04:05PM, Greg Kroah-Hartman wrote:
> On Thu, Sep 05, 2024 at 04:14:26PM -0500, Mario Limonciello wrote:
> > + stable
> > + regressions
> > New subject
> > 
> > Great news.
> > 
> > Greg, Sasha,
> > 
> > Can you please pull in these 3 commits specifically to 6.6.y to fix a
> > regression that was reported by Morgan in 6.6.y:
> > 
> > commit 12753d71e8c5 ("ACPI: CPPC: Add helper to get the highest performance
> > value")
> 
> This is fine, but:
> 
> > commit ed429c686b79 ("cpufreq: amd-pstate: Enable amd-pstate preferred core
> > support")
> 
> This is not a valid git id in Linus's tree :(

f3a052391822 ("cpufreq: amd-pstate: Enable amd-pstate preferred core support")

> 
> > commit 3d291fe47fe1 ("cpufreq: amd-pstate: fix the highest frequency issue
> > which limits performance")
> 
> And neither is this :(

bf202e654bfa ("cpufreq: amd-pstate: fix the highest frequency issue which limits performance")

> So perhaps you got them wrong?

I have added the ID's of the matching commits from Linus' tree above! :)

> thanks,
> 
> greg k-h

Cheers,
Chris

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: linux-6.6.y regression on amd-pstate

[Greg Kroah-Hartman]

On Sun, Sep 08, 2024 at 04:12:28PM +0200, Christian Heusel wrote:
> Hey Greg,
> 
> On 24/09/08 04:05PM, Greg Kroah-Hartman wrote:
> > On Thu, Sep 05, 2024 at 04:14:26PM -0500, Mario Limonciello wrote:
> > > + stable
> > > + regressions
> > > New subject
> > > 
> > > Great news.
> > > 
> > > Greg, Sasha,
> > > 
> > > Can you please pull in these 3 commits specifically to 6.6.y to fix a
> > > regression that was reported by Morgan in 6.6.y:
> > > 
> > > commit 12753d71e8c5 ("ACPI: CPPC: Add helper to get the highest performance
> > > value")
> > 
> > This is fine, but:
> > 
> > > commit ed429c686b79 ("cpufreq: amd-pstate: Enable amd-pstate preferred core
> > > support")
> > 
> > This is not a valid git id in Linus's tree :(
> 
> f3a052391822 ("cpufreq: amd-pstate: Enable amd-pstate preferred core support")
> 
> > 
> > > commit 3d291fe47fe1 ("cpufreq: amd-pstate: fix the highest frequency issue
> > > which limits performance")
> > 
> > And neither is this :(
> 
> bf202e654bfa ("cpufreq: amd-pstate: fix the highest frequency issue which limits performance")
> 
> > So perhaps you got them wrong?
> 
> I have added the ID's of the matching commits from Linus' tree above! :)
> 

Thanks, that works, all now queued up.

greg k-h

RE: [EXTERNAL] Re: linux-6.6.y regression on amd-pstate

[Jones, Morgan]

Hey all,

I think some form of this is back between 6.12 and 6.15 on our fractious AMD EPYC 7702. The symptom appears to be that the core will not boost past 2 GHz (the nominal frequency), so we lose out on 1.36 GHz of boost frequency. Downgrade from 6.15.7 to LTS (6.12.39) seems to fix it.

Keeping an eye out for other threads reporting similar symptoms on recent kernels:

[    0.000000] Linux version 6.15.7-xanmod1 (nixbld@localhost) (gcc (GCC) 14.2.1 20250322, GNU ld (GNU Binutils) 2.44) #1-NixOS SMP PREEMPT_DYNAMIC Tue Jan  1 00:00:00 UTC 1980

# cat /proc/cmdline
[snip] amd_pstate=active amd_prefcore=enable amd_pstate.shared_mem=1

# cat /proc/cpuinfo
[snip]
processor       : 127
vendor_id       : AuthenticAMD
cpu family      : 23
model           : 49
model name      : AMD EPYC 7702 64-Core Processor
stepping        : 0
microcode       : 0x830107d
cpu MHz         : 400.000
cache size      : 512 KB
physical id     : 0
siblings        : 128
core id         : 63
cpu cores       : 64
apicid          : 127
initial apicid  : 127
fpu             : yes
fpu_exception   : yes
cpuid level     : 16
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl xtopology nonstop_tsc cpuid extd_apicid aperfmperf rapl pni pclmulqdq monitor ssse3 fma cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw ibs skinit wdt tce topoext perfctr_core perfctr_nb bpext perfctr_llc mwaitx cpb cat_l3 cdp_l3 hw_pstate ssbd mba ibrs ibpb stibp vmmcall fsgsbase bmi1 avx2 smep bmi2 cqm rdt_a rdseed adx smap clflushopt clwb sha_ni xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local clzero irperf xsaveerptr rdpru wbnoinvd amd_ppin arat npt lbrv svm_lock nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold avic v_vmsave_vmload vgif v_spec_ctrl umip rdpid overflow_recov succor smca sev sev_es
bugs            : sysret_ss_attrs spectre_v1 spectre_v2 spec_store_bypass retbleed smt_rsb srso ibpb_no_ret
bogomips        : 3992.75
TLB size        : 3072 4K pages
clflush size    : 64
cache_alignment : 64
address sizes   : 43 bits physical, 48 bits virtual
power management: ts ttp tm hwpstate cpb eff_freq_ro [13] [14]

# cpupower frequency-info
analyzing CPU 76:
  driver: amd-pstate-epp
  CPUs which run at the same hardware frequency: 76
  CPUs which need to have their frequency coordinated by software: 76
  energy performance preference: performance
  hardware limits: 408 MHz - 3.36 GHz
  available cpufreq governors: performance powersave
  current policy: frequency should be within 1.51 GHz and 3.36 GHz.
                  The governor "performance" may decide which speed to use
                  within this range.
  current CPU frequency: 1.98 GHz (asserted by call to kernel)
  boost state support:
    Supported: yes
    Active: yes
  amd-pstate limits:
    Highest Performance: 255. Maximum Frequency: 3.36 GHz.
    Nominal Performance: 152. Nominal Frequency: 2.00 GHz.
    Lowest Non-linear Performance: 115. Lowest Non-linear Frequency: 1.51 GHz.
    Lowest Performance: 31. Lowest Frequency: 400 MHz.
    Preferred Core Support: 0. Preferred Core Ranking: 255.

Regards,
Morgan

-----Original Message-----
From: Greg Kroah-Hartman <gregkh@linuxfoundation.org> 
Sent: Sunday, September 8, 2024 7:30 AM
To: Christian Heusel <christian@heusel.eu>
Cc: Mario Limonciello <mario.limonciello@amd.com>; Jones, Morgan <Morgan.Jones@viasat.com>; Sasha Levin <sashal@kernel.org>; linux-pm@vger.kernel.org; linux-kernel@vger.kernel.org; David Arcari <darcari@redhat.com>; Dhananjay Ugwekar <Dhananjay.Ugwekar@amd.com>; rafael@kernel.org; viresh.kumar@linaro.org; gautham.shenoy@amd.com; perry.yuan@amd.com; skhan@linuxfoundation.org; li.meng@amd.com; ray.huang@amd.com; stable@vger.kernel.org; Linux kernel regressions list <regressions@lists.linux.dev>
Subject: [EXTERNAL] Re: linux-6.6.y regression on amd-pstate

On Sun, Sep 08, 2024 at 04:12:28PM +0200, Christian Heusel wrote:
> Hey Greg,
> 
> On 24/09/08 04:05PM, Greg Kroah-Hartman wrote:
> > On Thu, Sep 05, 2024 at 04:14:26PM -0500, Mario Limonciello wrote:
> > > + stable
> > > + regressions
> > > New subject
> > > 
> > > Great news.
> > > 
> > > Greg, Sasha,
> > > 
> > > Can you please pull in these 3 commits specifically to 6.6.y to 
> > > fix a regression that was reported by Morgan in 6.6.y:
> > > 
> > > commit 12753d71e8c5 ("ACPI: CPPC: Add helper to get the highest 
> > > performance
> > > value")
> > 
> > This is fine, but:
> > 
> > > commit ed429c686b79 ("cpufreq: amd-pstate: Enable amd-pstate 
> > > preferred core
> > > support")
> > 
> > This is not a valid git id in Linus's tree :(
> 
> f3a052391822 ("cpufreq: amd-pstate: Enable amd-pstate preferred core 
> support")
> 
> > 
> > > commit 3d291fe47fe1 ("cpufreq: amd-pstate: fix the highest 
> > > frequency issue which limits performance")
> > 
> > And neither is this :(
> 
> bf202e654bfa ("cpufreq: amd-pstate: fix the highest frequency issue 
> which limits performance")
> 
> > So perhaps you got them wrong?
> 
> I have added the ID's of the matching commits from Linus' tree above! 
> :)
> 

Thanks, that works, all now queued up.

greg k-h

Re: [EXTERNAL] Re: linux-6.6.y regression on amd-pstate

[Mario Limonciello]

On 8/1/2025 2:14 AM, Jones, Morgan wrote:
> Hey all,
> 
> I think some form of this is back between 6.12 and 6.15 on our fractious AMD EPYC 7702. The symptom appears to be that the core will not boost past 2 GHz (the nominal frequency), so we lose out on 1.36 GHz of boost frequency. Downgrade from 6.15.7 to LTS (6.12.39) seems to fix it.
> 
> Keeping an eye out for other threads reporting similar symptoms on recent kernels:
> 
> [    0.000000] Linux version 6.15.7-xanmod1 (nixbld@localhost) (gcc (GCC) 14.2.1 20250322, GNU ld (GNU Binutils) 2.44) #1-NixOS SMP PREEMPT_DYNAMIC Tue Jan  1 00:00:00 UTC 1980
> 
> # cat /proc/cmdline
> [snip] amd_pstate=active amd_prefcore=enable amd_pstate.shared_mem=1
> 
> # cat /proc/cpuinfo
> [snip]
> processor       : 127
> vendor_id       : AuthenticAMD
> cpu family      : 23
> model           : 49
> model name      : AMD EPYC 7702 64-Core Processor
> stepping        : 0
> microcode       : 0x830107d
> cpu MHz         : 400.000
> cache size      : 512 KB
> physical id     : 0
> siblings        : 128
> core id         : 63
> cpu cores       : 64
> apicid          : 127
> initial apicid  : 127
> fpu             : yes
> fpu_exception   : yes
> cpuid level     : 16
> wp              : yes
> flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl xtopology nonstop_tsc cpuid extd_apicid aperfmperf rapl pni pclmulqdq monitor ssse3 fma cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw ibs skinit wdt tce topoext perfctr_core perfctr_nb bpext perfctr_llc mwaitx cpb cat_l3 cdp_l3 hw_pstate ssbd mba ibrs ibpb stibp vmmcall fsgsbase bmi1 avx2 smep bmi2 cqm rdt_a rdseed adx smap clflushopt clwb sha_ni xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local clzero irperf xsaveerptr rdpru wbnoinvd amd_ppin arat npt lbrv svm_lock nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold avic v_vmsave_vmload vgif v_spec_ctrl umip rdpid overflow_recov succor smca sev sev_es
> bugs            : sysret_ss_attrs spectre_v1 spectre_v2 spec_store_bypass retbleed smt_rsb srso ibpb_no_ret
> bogomips        : 3992.75
> TLB size        : 3072 4K pages
> clflush size    : 64
> cache_alignment : 64
> address sizes   : 43 bits physical, 48 bits virtual
> power management: ts ttp tm hwpstate cpb eff_freq_ro [13] [14]
> 
> # cpupower frequency-info
> analyzing CPU 76:
>    driver: amd-pstate-epp
>    CPUs which run at the same hardware frequency: 76
>    CPUs which need to have their frequency coordinated by software: 76
>    energy performance preference: performance
>    hardware limits: 408 MHz - 3.36 GHz
>    available cpufreq governors: performance powersave
>    current policy: frequency should be within 1.51 GHz and 3.36 GHz.
>                    The governor "performance" may decide which speed to use
>                    within this range.
>    current CPU frequency: 1.98 GHz (asserted by call to kernel)
>    boost state support:
>      Supported: yes
>      Active: yes
>    amd-pstate limits:
>      Highest Performance: 255. Maximum Frequency: 3.36 GHz.
>      Nominal Performance: 152. Nominal Frequency: 2.00 GHz.
>      Lowest Non-linear Performance: 115. Lowest Non-linear Frequency: 1.51 GHz.
>      Lowest Performance: 31. Lowest Frequency: 400 MHz.
>      Preferred Core Support: 0. Preferred Core Ranking: 255.
> 
> Regards,
> Morgan
> 

Hello Morgan,

6.12 to 6.15 unfortunately includes a pretty big overhaul to the 
amd-pstate driver.  But I'm pretty surprised to hear this regression as 
we have had a lot of mileage on it across a very wide variety of hardware.

That being said:
1) Please capture a report using amd-pstate from amd-debug-tools 
(https://git.kernel.org/pub/scm/linux/kernel/git/superm1/amd-debug-tools.git/about/) 
both on a good and bad kernel and share them.

2) Can you reproduce on mainline 6.16?

If 1 and 2 don't lead an obvious answer:

3) Can you please bisect?

Thanks,