[RFC PATCH 0/4] cpufreq/amd-pstate: Per-core EPP boost for recently-busy CPUs

[RFC PATCH 0/4] cpufreq/amd-pstate: Per-core EPP boost for recently-busy CPUs

[David Vernet]

In active (EPP) mode the platform autonomously picks the operating point
between min_perf and max_perf, biased by the EPP hint, and the kernel
only rewrites the CPPC request on policy or limit changes. A workload
dominated by one mostly-busy thread that takes frequent short sleeps
(common in gaming workloads, for example) can fare poorly under this
strategy. Each sleep decays the hardware's performance signal, causing
post-wakeup bursts to start at a low operating point and inflating tail
latency even though the CPU is essentially fully busy while work is
available.

As mentioned above, the motivating case here is gaming. A game's main or
render thread typically blocks briefly on a futex or a GPU fence every
frame, and the resulting frequency droop shows up directly as stale
frames and inflated frame-time percentiles.

Solving this at the cpufreq layer requires walking a fairly narrow
path. Globally forcing EPP=performance fixes the tail but burns power
on every core for every workload, which matters on handhelds where the
CPU and GPU share a power budget. Raising min_perf on the busy core
seems like the obvious surgical fix, but it pins the core at or above
nominal even during micro-idle and vsync waits. On Van Gogh (Steam
Deck) experiments that I ran, that perturbs the SMU's shared CPU/GPU
boost management enough to regress the frame-time tail relative to doing
nothing (numbers below).

This series instead adds an opt-in, per-core EPP boost. When the
epp_boost module parameter is enabled, an update-util hook samples each
core's C0 residency (delta MPERF over delta TSC) at most once every
10 ms. If a sample shows the core at least 50% busy, the EPP field of
its MSR_AMD_CPPC_REQ is set to performance (0) and held there until
300 ms pass without another busy sample, at which point the hook
restores the request that policy management last stored in
cppc_req_cached. Both writes happen only on the busy and idle edges, so
the CPPC_REQ write rate matches that of a global EPP=performance
setting. min, max and desired perf are never touched. The mechanism is
only available in active mode on MSR (X86_FEATURE_CPPC) systems, since
the hook does local MSR accesses from scheduler context which the
shared memory interface cannot do. It composes with dynamic_epp, which
selects the policy EPP from the platform profile and power source.
epp_boost temporarily overrides whatever policy EPP is installed and
restores it when the core goes idle.

Precedents
==========

The closest precedent is intel_pstate's hwp_boost. It has the same
overall shape as this feature. It is an opt-in update-util hook that
temporarily rewrites the HWP request from scheduler context on a boost
edge and restores the unboosted request after a hardcoded hold time
(hwp_boost_hold_time_ns). It differs in two ways, both deliberate:

1. Trigger. hwp_boost activates on SCHED_CPUFREQ_IOWAIT. The waits
   that matter here are futex waits and amdgpu fence waits, which do
   not set the iowait flag, so a C0 residency trigger is used instead.
   Residency also naturally covers the "mostly busy with short gaps"
   pattern rather than only the wakeup instant.

2. Knob being boosted. hwp_boost raises the HWP min. As described
   above, a min_perf floor measurably regressed the tail on Van Gogh,
   so this feature biases only the EPP hint and leaves the platform
   free to drop the operating point during the idle portions of the
   frame.

On the thresholds themselves, the sample period, busy threshold and
decay window are hardcoded rather than exposed as tunables. I'm not sure
if this is appropriate or not, but it seemed like it followed existing
contours.

hwp_boost_hold_time_ns for example is a hardcoded 3 ms, and schedutil's
iowait boost decay is tied to TICK_NSEC, with no knobs for either. The
300 ms decay is sized so that a render thread which is only 50-80% busy
from periodic vsync and GPU-fence waits holds the boost across its whole
busy period at a couple of CPPC_REQ writes total, while an idle core
sheds the boost well before it can matter. The energy exposure of a wide
window is small because EPP only influences behavior in C0 and an idle
core sits in CC6 regardless. If folks want me to make these tunable I am
happy to expose them.

Testing methodology
===================

All numbers are from a Steam Deck LCD (Van Gogh APU) running in active
mode at EPP=balance_performance, using the Civilization VI graphics
benchmark as a single-thread CPU-bound workload with a repeatable
built-in benchmark pass.

Comparisons were run as interleaved A/B tests with 6 iterations per
configuration. For each run I collected per-frame frame times and the
busy core's frequency, and derived average fps, 1%-low fps and the p99
and p999 frame-time percentiles. Deltas were evaluated with Welch's
t-test, and I report the p-values alongside the deltas below.

Results:

Default settings
----------------
The busy core's median frequency sat at 2.43 GHz despite 98%
utilization, which is the frequency droop described above.

Global EPP=performance
----------------------
Globally forcing EPP=performance lifts the median to 3.5 GHz, cuts
frame-time p999 by ~40% and raises 1%-low fps by ~16%, but does so on
every core and for every workload.

Raising min_perf to nominal on the busy core
--------------------------------------------
A variant of this patch that instead raised min_perf to nominal on the
busy core reached the same 3.5 GHz median yet regressed p999 by 13-21%
by perturbing the SMU boost management as described above.

epp_boost enabled
-----------------
With epp_boost enabled, the busy core running the Civ VI main thread
runs at a 3.5 GHz median and the benchmark gains 31.8% in 1%-low fps
(p=0.014) and 4.1% in frame-time p99 (p=0.015), with p999 and average
fps unchanged.

David Vernet (4):
  cpufreq/amd-pstate: Document missing kernel-doc members
  cpufreq/amd-pstate: Update cppc_req_cached before writing the MSR
  cpufreq/amd-pstate: Add per-core EPP boost for recently-busy CPUs
  Documentation: amd-pstate: Document the epp_boost parameter

 Documentation/admin-guide/pm/amd-pstate.rst |  16 ++
 drivers/cpufreq/amd-pstate.c                | 230 ++++++++++++++++++++++++++--
 drivers/cpufreq/amd-pstate.h                |  20 +++
 3 files changed, 255 insertions(+), 11 deletions(-)


base-commit: 92bf086d086f7cfe0d6f807dfd6b8d11bc61b626
-- 
2.53.0

[RFC PATCH 1/4] cpufreq/amd-pstate: Document missing kernel-doc members

[David Vernet]

kernel-doc warns about five undescribed members in amd-pstate.h:
union perf_cached's @val and struct amd_cpudata's @raw_epp,
@current_profile, @ppdev and @profile_name. Describe them.

Signed-off-by: David Vernet <void@manifault.com>
---
 drivers/cpufreq/amd-pstate.h | 6 ++++++
 1 file changed, 6 insertions(+)

diff --git a/drivers/cpufreq/amd-pstate.h b/drivers/cpufreq/amd-pstate.h
index 23e8baa05849..99532595150e 100644
--- a/drivers/cpufreq/amd-pstate.h
+++ b/drivers/cpufreq/amd-pstate.h
@@ -32,6 +32,7 @@
  * @min_limit_perf: Cached value of the performance corresponding to policy->min
  * @max_limit_perf: Cached value of the performance corresponding to policy->max
  * @bios_min_perf: Cached perf value corresponding to the "Requested CPU Min Frequency" BIOS option
+ * @val: Raw 64-bit value for atomic access via READ_ONCE()/WRITE_ONCE()
  */
 union perf_cached {
 	struct {
@@ -89,7 +90,12 @@ struct amd_aperf_mperf {
  * @epp_default_ac: Default EPP value for AC power source
  * @epp_default_dc: Default EPP value for DC power source
  * @dynamic_epp: Whether dynamic EPP is enabled
+ * @raw_epp: Whether the last EPP write was a raw numeric value rather than a
+ *	     named preference
  * @power_nb: Notifier block for power events
+ * @current_profile: Currently selected platform profile option
+ * @ppdev: Device registered with the platform profile handler
+ * @profile_name: Name under which @ppdev is registered
  *
  * The amd_cpudata is key private data for each CPU thread in AMD P-State, and
  * represents all the attributes and goals that AMD P-State requests at runtime.
-- 
2.53.0

[RFC PATCH 2/4] cpufreq/amd-pstate: Update cppc_req_cached before writing the MSR

[David Vernet]

msr_update_perf() and msr_set_epp() currently write MSR_AMD_CPPC_REQ
first and update cppc_req_cached only after the write succeeds. This
leaves a window in which the MSR holds the new request while the cache
holds the previous one. That's fine right now, but a subsequent patch
will add a per-core EPP boost that runs from scheduling context and
mutates that MSR from the contents of the cached value. If we update the
cache after writing the MSR, the sched callback could run between the
wrmsrq and the cache being updated, and accidentally overwrite the
intended value of the MSR by issuing a wrmsrq on the stale cached value.

To avoid this, let's update the cache prior to the MSR write, as is done
in intel_pstate_set_epp(). This should be a functional no-op.

Signed-off-by: David Vernet <void@manifault.com>
---
 drivers/cpufreq/amd-pstate.c | 16 +++++++++-------
 1 file changed, 9 insertions(+), 7 deletions(-)

diff --git a/drivers/cpufreq/amd-pstate.c b/drivers/cpufreq/amd-pstate.c
index 6e255a22a0b8..5d7debb5a35c 100644
--- a/drivers/cpufreq/amd-pstate.c
+++ b/drivers/cpufreq/amd-pstate.c
@@ -262,17 +262,19 @@ static int msr_update_perf(struct cpufreq_policy *policy, u8 min_perf,
 	if (value == prev)
 		return 0;
 
+	WRITE_ONCE(cpudata->cppc_req_cached, value);
+
 	if (fast_switch) {
 		wrmsrq(MSR_AMD_CPPC_REQ, value);
 	} else {
 		int ret = wrmsrq_on_cpu(cpudata->cpu, MSR_AMD_CPPC_REQ, value);
 
-		if (ret)
+		if (ret) {
+			WRITE_ONCE(cpudata->cppc_req_cached, prev);
 			return ret;
+		}
 	}
 
-	WRITE_ONCE(cpudata->cppc_req_cached, value);
-
 	return 0;
 }
 
@@ -312,16 +314,16 @@ static int msr_set_epp(struct cpufreq_policy *policy, u8 epp)
 	if (value == prev)
 		return 0;
 
+	WRITE_ONCE(cpudata->cppc_req_cached, value);
+
 	ret = wrmsrq_on_cpu(cpudata->cpu, MSR_AMD_CPPC_REQ, value);
 	if (ret) {
+		WRITE_ONCE(cpudata->cppc_req_cached, prev);
 		pr_err("failed to set energy perf value (%d)\n", ret);
 		return ret;
 	}
 
-	/* update both so that msr_update_perf() can effectively check */
-	WRITE_ONCE(cpudata->cppc_req_cached, value);
-
-	return ret;
+	return 0;
 }
 
 DEFINE_STATIC_CALL(amd_pstate_set_epp, msr_set_epp);
-- 
2.53.0

[RFC PATCH 3/4] cpufreq/amd-pstate: Add per-core EPP boost for recently-busy CPUs

[David Vernet]

In active (EPP) mode the platform autonomously picks the operating point
between min_perf and max_perf, biased by the EPP hint, and the kernel
only rewrites the CPPC request on policy/limit changes. A workload
dominated by one mostly-busy thread that takes frequent short sleeps
(e.g. a game render thread blocking on a futex waiting for the GPU or a
worker thread on every frame) can defeat the platform's utilization
tracking. Each sleep decays the hardware's performance signal, so
post-wakeup bursts start at a low operating point and the frame-time
tail inflates even though the CPU is ~100% busy during the frame itself.

This is a common problem in gaming workloads, and is not easy to solve
at the cpufreq layer. We have to toe the line between running at a high
voltage unnecessarily and draining battery (and potentially contending
with e.g. the GPU for power draw), and lowering frequency on the core
that's running the game's main and/or render threads (thus materially
increasing frame latency and causing a tail increase in stale frames).

Unity, for example, has a main thread and a render thread which run
(mostly) serialized on the frame-rendering pipeline, and Unity games'
fps typically experience much lower stale frame rates when CPU frequency
is higher on cores while they're running those threads. On the other
hand, for e.g. a 60 fps game (a ~16.7 ms frame deadline), the CPU may be
idle for some portion of that window when the frame completes ahead of
the deadline. We also have to contend with the fact that leaving the
core boosted unnecessarily can be actively harmful, even aside from
battery consumption. For example, keeping a high min_perf floor in place
even during short idle and vsync waits can regress performance by
perturbing SMU boost management.

We thus need a cpufreq solution that boosts a high-priority core as soon
as it becomes utilized, and then restores the requested policy value
when the core goes idle. This diff implements a new mechanism called
"per-core EPP boost", which achieves this by adding an opt-in per-core
mechanism that temporarily applies the EPP=performance policy to a core
that stays at least 50% busy over some sampled window.

The mechanism works as follows: When the epp_boost module parameter is
enabled, an update-util hook samples each core's C0 residency (delta
MPERF over delta TSC) at most once every 10 ms. If a sample shows the
core at least 50% busy, the EPP field of its MSR_AMD_CPPC_REQ is set to
performance (0) and held there until 300 ms pass without another busy
sample. The hook then writes back the request that policy management
last stored in cppc_req_cached. Both writes happen only on the busy and
idle edges, so the CPPC_REQ write rate matches that of a global
EPP=performance setting. min, max and desired perf are never touched.
This mirrors intel_pstate's hwp_boost, except that hwp_boost triggers on
SCHED_CPUFREQ_IOWAIT, which the waits at issue here (futex waits and
amdgpu fence waits) do not set, so a residency trigger is used instead.

Note that this mechanism is only available in active mode on MSR
(X86_FEATURE_CPPC) systems, as the hook does local MSR accesses from
scheduler context which the shared memory interface cannot do. It
composes with dynamic_epp, which selects the policy EPP from the
platform profile and power source. epp_boost temporarily overrides
whatever policy EPP is installed and restores it when the core goes
idle.

I ran a series of benchmarks on a Steam Deck (Van Gogh APU,
EPP=balance_performance) which illustrate the patch:

*Civ VI graphics benchmark (single-thread CPU-bound)*

Default settings
================
The busy core's median frequency sat at 2.43 GHz despite 98% utilization.

Global EPP=performance
======================
Globally forcing EPP=performance lifts the median to 3.5 GHz, cuts
frame-time p999 by ~40% and raises 1%-low fps by ~16%, but does so on
every core and every workload.

Raising min_perf to nominal on the busy core
============================================
A variant of this patch that instead raised min_perf to nominal on the
busy core reached the same 3.5 GHz median yet regressed p999 by 13-21%
for the reasons described above.

epp_boost enabled
=================
With epp_boost enabled (interleaved A/B tests with 6 iterations, using
Welch's t-test to arrive at the deltas), the busy core running the Civ VI
main thread runs at a 3.5 GHz median and the benchmark gains 31.8% in
1%-low fps (p=0.014) and 4.1% in frame-time p99 (p=0.015), with p999 and
average fps unchanged.

Signed-off-by: David Vernet <void@manifault.com>
---
 drivers/cpufreq/amd-pstate.c | 214 ++++++++++++++++++++++++++++++++++-
 drivers/cpufreq/amd-pstate.h |  14 +++
 2 files changed, 224 insertions(+), 4 deletions(-)

diff --git a/drivers/cpufreq/amd-pstate.c b/drivers/cpufreq/amd-pstate.c
index 5d7debb5a35c..771a74e04f86 100644
--- a/drivers/cpufreq/amd-pstate.c
+++ b/drivers/cpufreq/amd-pstate.c
@@ -27,6 +27,7 @@
 #include <linux/module.h>
 #include <linux/init.h>
 #include <linux/smp.h>
+#include <linux/cpu.h>
 #include <linux/sched.h>
 #include <linux/cpufreq.h>
 #include <linux/compiler.h>
@@ -262,6 +263,10 @@ static int msr_update_perf(struct cpufreq_policy *policy, u8 min_perf,
 	if (value == prev)
 		return 0;
 
+	/*
+	 * Must come before MSR write to ensure amd_pstate_epp_boost_release()
+	 * doesn't write a stale value.
+	 */
 	WRITE_ONCE(cpudata->cppc_req_cached, value);
 
 	if (fast_switch) {
@@ -314,6 +319,7 @@ static int msr_set_epp(struct cpufreq_policy *policy, u8 epp)
 	if (value == prev)
 		return 0;
 
+	/* Must update cached value prior to wrmsrq -- see msr_update_perf() */
 	WRITE_ONCE(cpudata->cppc_req_cached, value);
 
 	ret = wrmsrq_on_cpu(cpudata->cpu, MSR_AMD_CPPC_REQ, value);
@@ -1877,6 +1883,187 @@ static bool amd_pstate_acpi_pm_profile_undefined(void)
 	return false;
 }
 
+/*
+ * Per-core EPP boost.
+ *
+ * While the epp_boost module parameter is enabled, each CPU in active (EPP)
+ * mode on an MSR (X86_FEATURE_CPPC) system has a cpufreq update-util hook
+ * registered which puts the core in EPP=performance mode when the core is
+ * sufficiently utilized.
+ *
+ * Enables more efficient frequency scaling on cores that have strict latency
+ * requirements, such as e.g. gaming workloads where render threads use a CPU
+ * for some period during frame windows, and then go idle until the next frame.
+ */
+static bool epp_boost __read_mostly;
+
+#define AMD_PSTATE_EPP_BOOST_SAMPLE_NS	(10 * NSEC_PER_MSEC)
+/*
+ * Busy threshold is deliberately low and the decay window wide so that a
+ * render thread which is only ~50-80% busy (periodic vsync and GPU-fence
+ * waits) holds the boost across its whole busy period at a couple of CPPC_REQ
+ * writes total.
+ */
+#define AMD_PSTATE_EPP_BOOST_DECAY_NS	(300 * NSEC_PER_MSEC)
+#define AMD_PSTATE_EPP_BOOST_BUSY_PCT	50
+
+static void amd_pstate_epp_boost_apply(struct amd_cpudata *cpudata, u64 cached)
+{
+	union perf_cached perf = READ_ONCE(cpudata->perf);
+
+	/* never boost a request whose limits are not populated */
+	if (!perf.max_limit_perf)
+		return;
+
+	cached &= ~AMD_CPPC_EPP_PERF_MASK;
+	cached |= FIELD_PREP(AMD_CPPC_EPP_PERF_MASK, AMD_CPPC_EPP_PERFORMANCE);
+
+	wrmsrq(MSR_AMD_CPPC_REQ, cached);
+}
+
+static void amd_pstate_epp_boost_release(struct amd_cpudata *cpudata)
+{
+	wrmsrq(MSR_AMD_CPPC_REQ, READ_ONCE(cpudata->cppc_req_cached));
+}
+
+static void amd_pstate_epp_boost_update_util(struct update_util_data *data,
+					     u64 time, unsigned int flags)
+{
+	struct amd_cpudata *cpudata = container_of(data, struct amd_cpudata,
+						   epp_boost_update_util);
+	union perf_cached perf;
+	bool first_sample;
+	u64 busy_pct;
+	bool active;
+
+	if (smp_processor_id() != cpudata->cpu)
+		return;
+
+	first_sample = !cpudata->epp_boost_last_sample;
+	if (!first_sample &&
+	    time - cpudata->epp_boost_last_sample < AMD_PSTATE_EPP_BOOST_SAMPLE_NS)
+		return;
+	cpudata->epp_boost_last_sample = time;
+
+	/*
+	 * Counters can reset across suspend or hotplug (so their values would
+	 * be garbage), so just recalculate baselines.
+	 */
+	if (amd_pstate_sample(cpudata) && !first_sample) {
+		busy_pct = div64_u64(cpudata->cur.mperf * 100, cpudata->cur.tsc);
+		if (busy_pct >= AMD_PSTATE_EPP_BOOST_BUSY_PCT)
+			cpudata->epp_boost_last_busy = time;
+	}
+
+	perf = READ_ONCE(cpudata->perf);
+	active = perf.max_limit_perf &&
+		 (time - cpudata->epp_boost_last_busy < AMD_PSTATE_EPP_BOOST_DECAY_NS);
+
+	if (active && !cpudata->epp_boost_active)
+		amd_pstate_epp_boost_apply(cpudata,
+					   READ_ONCE(cpudata->cppc_req_cached));
+	else if (!active && cpudata->epp_boost_active)
+		amd_pstate_epp_boost_release(cpudata);
+
+	cpudata->epp_boost_active = active;
+}
+
+static void amd_pstate_epp_boost_enable(struct cpufreq_policy *policy)
+{
+	struct amd_cpudata *cpudata = policy->driver_data;
+
+	if (!READ_ONCE(epp_boost))
+		return;
+
+	if (cppc_state != AMD_PSTATE_ACTIVE ||
+	    !cpu_feature_enabled(X86_FEATURE_CPPC))
+		return;
+
+	if (cpudata->epp_boost_registered)
+		return;
+
+	cpudata->epp_boost_last_sample = 0;
+	cpudata->epp_boost_last_busy = 0;
+	cpufreq_add_update_util_hook(policy->cpu, &cpudata->epp_boost_update_util,
+				     amd_pstate_epp_boost_update_util);
+	cpudata->epp_boost_registered = true;
+}
+
+static void amd_pstate_epp_boost_disable(struct cpufreq_policy *policy)
+{
+	struct amd_cpudata *cpudata = policy->driver_data;
+
+	if (cpudata->epp_boost_registered) {
+		cpufreq_remove_update_util_hook(policy->cpu);
+		synchronize_rcu();
+		cpudata->epp_boost_registered = false;
+	}
+
+	if (cpudata->epp_boost_active) {
+		u64 cached = READ_ONCE(cpudata->cppc_req_cached);
+
+		wrmsrq_on_cpu(cpudata->cpu, MSR_AMD_CPPC_REQ, cached);
+
+		/*
+		 * Cross-CPU restore is not ordered against a concurrent
+		 * limit/EPP writer that doesn't hold the policy rwsem, so
+		 * recheck the cache once to ensure that the MSR reflects the
+		 * cached value
+		 */
+		if (READ_ONCE(cpudata->cppc_req_cached) != cached)
+			wrmsrq_on_cpu(cpudata->cpu, MSR_AMD_CPPC_REQ,
+				      READ_ONCE(cpudata->cppc_req_cached));
+
+		cpudata->epp_boost_active = false;
+	}
+}
+
+static int epp_boost_param_set(const char *val, const struct kernel_param *kp)
+{
+	unsigned int cpu;
+	bool enabled;
+	int ret;
+
+	ret = kstrtobool(val, &enabled);
+	if (ret)
+		return ret;
+
+	guard(mutex)(&amd_pstate_driver_lock);
+
+	if (epp_boost == enabled)
+		return 0;
+
+	WRITE_ONCE(epp_boost, enabled);
+
+	if (!current_pstate_driver)
+		return 0;
+
+	guard(cpus_read_lock)();
+
+	for_each_online_cpu(cpu) {
+		struct cpufreq_policy *policy __free(put_cpufreq_policy) =
+							cpufreq_cpu_get(cpu);
+
+		if (!policy)
+			continue;
+
+		if (enabled)
+			amd_pstate_epp_boost_enable(policy);
+		else
+			amd_pstate_epp_boost_disable(policy);
+	}
+
+	return 0;
+}
+
+static const struct kernel_param_ops epp_boost_param_ops = {
+	.set = epp_boost_param_set,
+	.get = param_get_bool,
+};
+module_param_cb(epp_boost, &epp_boost_param_ops, &epp_boost, 0644);
+MODULE_PARM_DESC(epp_boost,
+		 "boost EPP toward performance on recently-busy CPUs (active mode, MSR systems only)");
+
 static int amd_pstate_epp_cpu_init(struct cpufreq_policy *policy)
 {
 	struct amd_cpudata *cpudata;
@@ -1970,6 +2157,8 @@ static int amd_pstate_epp_cpu_init(struct cpufreq_policy *policy)
 
 	current_pstate_driver->adjust_perf = NULL;
 
+	amd_pstate_epp_boost_enable(policy);
+
 	return 0;
 
 free_cpudata1:
@@ -1989,6 +2178,8 @@ static void amd_pstate_epp_cpu_exit(struct cpufreq_policy *policy)
 		if (cpudata->dynamic_epp)
 			amd_pstate_clear_dynamic_epp(policy);
 
+		amd_pstate_epp_boost_disable(policy);
+
 		/* Reset CPPC_REQ MSR to the BIOS value */
 		amd_pstate_update_perf(policy, perf.bios_min_perf, 0U, 0U, 0U, false);
 		amd_pstate_set_floor_perf(policy, cpudata->bios_floor_perf);
@@ -2057,7 +2248,13 @@ static int amd_pstate_cpu_online(struct cpufreq_policy *policy)
 		return ret;
 
 	cached_floor_perf = freq_to_perf(perf, cpudata->nominal_freq, cpudata->floor_freq);
-	return amd_pstate_set_floor_perf(policy, cached_floor_perf);
+	ret = amd_pstate_set_floor_perf(policy, cached_floor_perf);
+	if (ret)
+		return ret;
+
+	amd_pstate_epp_boost_enable(policy);
+
+	return 0;
 }
 
 static int amd_pstate_cpu_offline(struct cpufreq_policy *policy)
@@ -2066,6 +2263,8 @@ static int amd_pstate_cpu_offline(struct cpufreq_policy *policy)
 	union perf_cached perf = READ_ONCE(cpudata->perf);
 	int ret;
 
+	amd_pstate_epp_boost_disable(policy);
+
 	/*
 	 * Reset CPPC_REQ MSR to the BIOS value, this will allow us to retain the BIOS specified
 	 * min_perf value across kexec reboots. If this CPU is just onlined normally after this, the
@@ -2088,6 +2287,8 @@ static int amd_pstate_suspend(struct cpufreq_policy *policy)
 	union perf_cached perf = READ_ONCE(cpudata->perf);
 	int ret;
 
+	amd_pstate_epp_boost_disable(policy);
+
 	/*
 	 * Reset CPPC_REQ MSR to the BIOS value, this will allow us to retain the BIOS specified
 	 * min_perf value across kexec reboots. If this CPU is just resumed back without kexec,
@@ -2134,10 +2335,9 @@ static int amd_pstate_epp_resume(struct cpufreq_policy *policy)
 	struct amd_cpudata *cpudata = policy->driver_data;
 	union perf_cached perf = READ_ONCE(cpudata->perf);
 	u8 cached_floor_perf;
+	int ret;
 
 	if (cpudata->suspended) {
-		int ret;
-
 		/* enable amd pstate from suspend state*/
 		ret = amd_pstate_epp_update_limit(policy, false);
 		if (ret)
@@ -2147,7 +2347,13 @@ static int amd_pstate_epp_resume(struct cpufreq_policy *policy)
 	}
 
 	cached_floor_perf = freq_to_perf(perf, cpudata->nominal_freq, cpudata->floor_freq);
-	return amd_pstate_set_floor_perf(policy, cached_floor_perf);
+	ret = amd_pstate_set_floor_perf(policy, cached_floor_perf);
+	if (ret)
+		return ret;
+
+	amd_pstate_epp_boost_enable(policy);
+
+	return 0;
 }
 
 static struct cpufreq_driver amd_pstate_driver = {
diff --git a/drivers/cpufreq/amd-pstate.h b/drivers/cpufreq/amd-pstate.h
index 99532595150e..0408fc331633 100644
--- a/drivers/cpufreq/amd-pstate.h
+++ b/drivers/cpufreq/amd-pstate.h
@@ -10,6 +10,7 @@
 
 #include <linux/pm_qos.h>
 #include <linux/platform_profile.h>
+#include <linux/sched/cpufreq.h>
 
 /*********************************************************************
  *                        AMD P-state INTERFACE                       *
@@ -96,6 +97,12 @@ struct amd_aperf_mperf {
  * @current_profile: Currently selected platform profile option
  * @ppdev: Device registered with the platform profile handler
  * @profile_name: Name under which @ppdev is registered
+ * @epp_boost_update_util: update-util hook for the per-core EPP boost
+ * @epp_boost_last_sample: last C0-residency sample time (ns; 0 means the
+ *			   next callback only re-baselines the counters)
+ * @epp_boost_last_busy: last time the CPU sampled as busy (ns)
+ * @epp_boost_active: EPP boost currently applied to MSR_AMD_CPPC_REQ
+ * @epp_boost_registered: update-util hook currently registered
  *
  * The amd_cpudata is key private data for each CPU thread in AMD P-State, and
  * represents all the attributes and goals that AMD P-State requests at runtime.
@@ -139,6 +146,13 @@ struct amd_cpudata {
 	enum platform_profile_option current_profile;
 	struct device *ppdev;
 	char *profile_name;
+
+	/* per-core EPP boost (active mode, MSR systems only) */
+	struct	update_util_data epp_boost_update_util;
+	u64	epp_boost_last_sample;
+	u64	epp_boost_last_busy;
+	bool	epp_boost_active;
+	bool	epp_boost_registered;
 };
 
 /*
-- 
2.53.0

[RFC PATCH 4/4] Documentation: amd-pstate: Document the epp_boost parameter

[David Vernet]

Document the per-core EPP boost mechanism and its epp_boost module
parameter added by the previous patch.

Signed-off-by: David Vernet <void@manifault.com>
---
 Documentation/admin-guide/pm/amd-pstate.rst | 16 ++++++++++++++++
 1 file changed, 16 insertions(+)

diff --git a/Documentation/admin-guide/pm/amd-pstate.rst b/Documentation/admin-guide/pm/amd-pstate.rst
index a95e2ebce005..004b79ff925c 100644
--- a/Documentation/admin-guide/pm/amd-pstate.rst
+++ b/Documentation/admin-guide/pm/amd-pstate.rst
@@ -372,6 +372,22 @@ based on the power source and will not react to user desired power state.
 Attempting to manually write to the ``energy_performance_preference`` sysfs
 file will fail when ``dynamic_epp`` is enabled.
 
+Energy performance preference boosting
+======================================
+On MSR-based systems running in the ``active mode``, the driver can optionally
+bias the energy performance preference of a recently-busy CPU toward
+performance while leaving all other CPUs at the policy value.  C0 residency is
+sampled at most every 10 ms; while a CPU has been at least 50% busy within the
+last 300 ms, the EPP field of its CPPC request is set to ``performance`` (0).
+The policy value is restored once 300 ms pass without a busy sample.
+
+This behavior is disabled by default.  It can be enabled with the module
+parameter ``amd_pstate.epp_boost=1`` on the kernel command line, or at runtime
+by writing ``1`` to ``/sys/module/amd_pstate/parameters/epp_boost``.  The
+mechanism only runs while the parameter is enabled and has no overhead when
+disabled.  It has no effect in the ``passive`` and ``guided`` modes or on
+shared-memory (non-MSR) systems.
+
 ``amd-pstate`` vs ``acpi-cpufreq``
 ======================================
 
-- 
2.53.0

Re: [RFC PATCH 0/4] cpufreq/amd-pstate: Per-core EPP boost for recently-busy CPUs

[David Vernet]

[-- Attachment #1: Type: text/plain, Size: 757 bytes --]

On Tue, Jul 28, 2026 at 02:31:46AM -0500, David Vernet wrote:
> In active (EPP) mode the platform autonomously picks the operating point
> between min_perf and max_perf, biased by the EPP hint, and the kernel
> only rewrites the CPPC request on policy or limit changes. A workload
> dominated by one mostly-busy thread that takes frequent short sleeps
> (common in gaming workloads, for example) can fare poorly under this
> strategy. Each sleep decays the hardware's performance signal, causing
> post-wakeup bursts to start at a low operating point and inflating tail
> latency even though the CPU is essentially fully busy while work is

... Argh, sorry, got my aliases mixed up.

+cc the rest of the AMD pstate driver maintainers (and removing Gautham)

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 228 bytes --]

Re: [RFC PATCH 0/4] cpufreq/amd-pstate: Per-core EPP boost for recently-busy CPUs

[Christian Loehle]

On 7/28/26 08:31, David Vernet wrote:
> In active (EPP) mode the platform autonomously picks the operating point
> between min_perf and max_perf, biased by the EPP hint, and the kernel
> only rewrites the CPPC request on policy or limit changes. A workload
> dominated by one mostly-busy thread that takes frequent short sleeps
> (common in gaming workloads, for example) can fare poorly under this
> strategy. Each sleep decays the hardware's performance signal, causing
> post-wakeup bursts to start at a low operating point and inflating tail
> latency even though the CPU is essentially fully busy while work is
> available.
> 
> As mentioned above, the motivating case here is gaming. A game's main or
> render thread typically blocks briefly on a futex or a GPU fence every
> frame, and the resulting frequency droop shows up directly as stale
> frames and inflated frame-time percentiles.
> 
> Solving this at the cpufreq layer requires walking a fairly narrow
> path. Globally forcing EPP=performance fixes the tail but burns power
> on every core for every workload, which matters on handhelds where the
> CPU and GPU share a power budget. Raising min_perf on the busy core
> seems like the obvious surgical fix, but it pins the core at or above
> nominal even during micro-idle and vsync waits. On Van Gogh (Steam
> Deck) experiments that I ran, that perturbs the SMU's shared CPU/GPU
> boost management enough to regress the frame-time tail relative to doing
> nothing (numbers below).
> 
> This series instead adds an opt-in, per-core EPP boost. When the
> epp_boost module parameter is enabled, an update-util hook samples each
> core's C0 residency (delta MPERF over delta TSC) at most once every
> 10 ms. If a sample shows the core at least 50% busy, the EPP field of
> its MSR_AMD_CPPC_REQ is set to performance (0) and held there until
> 300 ms pass without another busy sample, at which point the hook
> restores the request that policy management last stored in
> cppc_req_cached. Both writes happen only on the busy and idle edges, so
> the CPPC_REQ write rate matches that of a global EPP=performance
> setting. min, max and desired perf are never touched. The mechanism is
> only available in active mode on MSR (X86_FEATURE_CPPC) systems, since
> the hook does local MSR accesses from scheduler context which the
> shared memory interface cannot do. It composes with dynamic_epp, which
> selects the policy EPP from the platform profile and power source.
> epp_boost temporarily overrides whatever policy EPP is installed and
> restores it when the core goes idle.
> 
> Precedents
> ==========
> 
> The closest precedent is intel_pstate's hwp_boost. It has the same
> overall shape as this feature. It is an opt-in update-util hook that
> temporarily rewrites the HWP request from scheduler context on a boost
> edge and restores the unboosted request after a hardcoded hold time
> (hwp_boost_hold_time_ns). It differs in two ways, both deliberate:
> 
> 1. Trigger. hwp_boost activates on SCHED_CPUFREQ_IOWAIT. The waits
>    that matter here are futex waits and amdgpu fence waits, which do
>    not set the iowait flag, so a C0 residency trigger is used instead.
>    Residency also naturally covers the "mostly busy with short gaps"
>    pattern rather than only the wakeup instant.
> 
> 2. Knob being boosted. hwp_boost raises the HWP min. As described
>    above, a min_perf floor measurably regressed the tail on Van Gogh,
>    so this feature biases only the EPP hint and leaves the platform
>    free to drop the operating point during the idle portions of the
>    frame.
> 
> On the thresholds themselves, the sample period, busy threshold and
> decay window are hardcoded rather than exposed as tunables. I'm not sure
> if this is appropriate or not, but it seemed like it followed existing
> contours.
> 
> hwp_boost_hold_time_ns for example is a hardcoded 3 ms, and schedutil's
> iowait boost decay is tied to TICK_NSEC, with no knobs for either. The
> 300 ms decay is sized so that a render thread which is only 50-80% busy
> from periodic vsync and GPU-fence waits holds the boost across its whole
> busy period at a couple of CPPC_REQ writes total, while an idle core
> sheds the boost well before it can matter. The energy exposure of a wide
> window is small because EPP only influences behavior in C0 and an idle
> core sits in CC6 regardless. If folks want me to make these tunable I am
> happy to expose them.
> 
> Testing methodology
> ===================
> 
> All numbers are from a Steam Deck LCD (Van Gogh APU) running in active
> mode at EPP=balance_performance, using the Civilization VI graphics
> benchmark as a single-thread CPU-bound workload with a repeatable
> built-in benchmark pass.
> 
> Comparisons were run as interleaved A/B tests with 6 iterations per
> configuration. For each run I collected per-frame frame times and the
> busy core's frequency, and derived average fps, 1%-low fps and the p99
> and p999 frame-time percentiles. Deltas were evaluated with Welch's
> t-test, and I report the p-values alongside the deltas below.
> 
> Results:
> 
> Default settings
> ----------------
> The busy core's median frequency sat at 2.43 GHz despite 98%
> utilization, which is the frequency droop described above.

Two things to confirm for my understanding:
-Median frequency is the 50th percentile when weighing the OPPs by
residency, right?
- You're also using busy_pct = delta_MPERF * 100 / delta_TSC as
utilization, right? (Not util_avg or anything like that)

Your observation looks like a firmware or SMU issue.
Can we confirm the (short) sleeps reset the demand-estimation, i.e.
by tracing cpu_idle/sched_switch and sample APERF/MPERF alongside it?
Also rt-app might be helpful to get a feel of how this behaves (and
create a similar pathological case like the single-threaded game).

> 
> Global EPP=performance
> ----------------------
> Globally forcing EPP=performance lifts the median to 3.5 GHz, cuts
> frame-time p999 by ~40% and raises 1%-low fps by ~16%, but does so on
> every core and for every workload.
> 
> Raising min_perf to nominal on the busy core
> --------------------------------------------
> A variant of this patch that instead raised min_perf to nominal on the
> busy core reached the same 3.5 GHz median yet regressed p999 by 13-21%
> by perturbing the SMU boost management as described above.
> 
> epp_boost enabled
> -----------------
> With epp_boost enabled, the busy core running the Civ VI main thread
> runs at a 3.5 GHz median and the benchmark gains 31.8% in 1%-low fps
> (p=0.014) and 4.1% in frame-time p99 (p=0.015), with p999 and average
> fps unchanged.
> 
> David Vernet (4):
>   cpufreq/amd-pstate: Document missing kernel-doc members
>   cpufreq/amd-pstate: Update cppc_req_cached before writing the MSR
>   cpufreq/amd-pstate: Add per-core EPP boost for recently-busy CPUs
>   Documentation: amd-pstate: Document the epp_boost parameter
> 
>  Documentation/admin-guide/pm/amd-pstate.rst |  16 ++
>  drivers/cpufreq/amd-pstate.c                | 230 ++++++++++++++++++++++++++--
>  drivers/cpufreq/amd-pstate.h                |  20 +++
>  3 files changed, 255 insertions(+), 11 deletions(-)
> 
> 
> base-commit: 92bf086d086f7cfe0d6f807dfd6b8d11bc61b626