[RFC PATCH 3/4] cpufreq/amd-pstate: Add per-core EPP boost for recently-busy CPUs

[David Vernet <void@manifault.com>]

In active (EPP) mode the platform autonomously picks the operating point
between min_perf and max_perf, biased by the EPP hint, and the kernel
only rewrites the CPPC request on policy/limit changes. A workload
dominated by one mostly-busy thread that takes frequent short sleeps
(e.g. a game render thread blocking on a futex waiting for the GPU or a
worker thread on every frame) can defeat the platform's utilization
tracking. Each sleep decays the hardware's performance signal, so
post-wakeup bursts start at a low operating point and the frame-time
tail inflates even though the CPU is ~100% busy during the frame itself.

This is a common problem in gaming workloads, and is not easy to solve
at the cpufreq layer. We have to toe the line between running at a high
voltage unnecessarily and draining battery (and potentially contending
with e.g. the GPU for power draw), and lowering frequency on the core
that's running the game's main and/or render threads (thus materially
increasing frame latency and causing a tail increase in stale frames).

Unity, for example, has a main thread and a render thread which run
(mostly) serialized on the frame-rendering pipeline, and Unity games'
fps typically experience much lower stale frame rates when CPU frequency
is higher on cores while they're running those threads. On the other
hand, for e.g. a 60 fps game (a ~16.7 ms frame deadline), the CPU may be
idle for some portion of that window when the frame completes ahead of
the deadline. We also have to contend with the fact that leaving the
core boosted unnecessarily can be actively harmful, even aside from
battery consumption. For example, keeping a high min_perf floor in place
even during short idle and vsync waits can regress performance by
perturbing SMU boost management.

We thus need a cpufreq solution that boosts a high-priority core as soon
as it becomes utilized, and then restores the requested policy value
when the core goes idle. This diff implements a new mechanism called
"per-core EPP boost", which achieves this by adding an opt-in per-core
mechanism that temporarily applies the EPP=performance policy to a core
that stays at least 50% busy over some sampled window.

The mechanism works as follows: When the epp_boost module parameter is
enabled, an update-util hook samples each core's C0 residency (delta
MPERF over delta TSC) at most once every 10 ms. If a sample shows the
core at least 50% busy, the EPP field of its MSR_AMD_CPPC_REQ is set to
performance (0) and held there until 300 ms pass without another busy
sample. The hook then writes back the request that policy management
last stored in cppc_req_cached. Both writes happen only on the busy and
idle edges, so the CPPC_REQ write rate matches that of a global
EPP=performance setting. min, max and desired perf are never touched.
This mirrors intel_pstate's hwp_boost, except that hwp_boost triggers on
SCHED_CPUFREQ_IOWAIT, which the waits at issue here (futex waits and
amdgpu fence waits) do not set, so a residency trigger is used instead.

Note that this mechanism is only available in active mode on MSR
(X86_FEATURE_CPPC) systems, as the hook does local MSR accesses from
scheduler context which the shared memory interface cannot do. It
composes with dynamic_epp, which selects the policy EPP from the
platform profile and power source. epp_boost temporarily overrides
whatever policy EPP is installed and restores it when the core goes
idle.

I ran a series of benchmarks on a Steam Deck (Van Gogh APU,
EPP=balance_performance) which illustrate the patch:

*Civ VI graphics benchmark (single-thread CPU-bound)*

Default settings
================
The busy core's median frequency sat at 2.43 GHz despite 98% utilization.

Global EPP=performance
======================
Globally forcing EPP=performance lifts the median to 3.5 GHz, cuts
frame-time p999 by ~40% and raises 1%-low fps by ~16%, but does so on
every core and every workload.

Raising min_perf to nominal on the busy core
============================================
A variant of this patch that instead raised min_perf to nominal on the
busy core reached the same 3.5 GHz median yet regressed p999 by 13-21%
for the reasons described above.

epp_boost enabled
=================
With epp_boost enabled (interleaved A/B tests with 6 iterations, using
Welch's t-test to arrive at the deltas), the busy core running the Civ VI
main thread runs at a 3.5 GHz median and the benchmark gains 31.8% in
1%-low fps (p=0.014) and 4.1% in frame-time p99 (p=0.015), with p999 and
average fps unchanged.

Signed-off-by: David Vernet <void@manifault.com>
---
 drivers/cpufreq/amd-pstate.c | 214 ++++++++++++++++++++++++++++++++++-
 drivers/cpufreq/amd-pstate.h |  14 +++
 2 files changed, 224 insertions(+), 4 deletions(-)

diff --git a/drivers/cpufreq/amd-pstate.c b/drivers/cpufreq/amd-pstate.c
index 5d7debb5a35c..771a74e04f86 100644
--- a/drivers/cpufreq/amd-pstate.c
+++ b/drivers/cpufreq/amd-pstate.c
@@ -27,6 +27,7 @@
 #include <linux/module.h>
 #include <linux/init.h>
 #include <linux/smp.h>
+#include <linux/cpu.h>
 #include <linux/sched.h>
 #include <linux/cpufreq.h>
 #include <linux/compiler.h>
@@ -262,6 +263,10 @@ static int msr_update_perf(struct cpufreq_policy *policy, u8 min_perf,
 	if (value == prev)
 		return 0;
 
+	/*
+	 * Must come before MSR write to ensure amd_pstate_epp_boost_release()
+	 * doesn't write a stale value.
+	 */
 	WRITE_ONCE(cpudata->cppc_req_cached, value);
 
 	if (fast_switch) {
@@ -314,6 +319,7 @@ static int msr_set_epp(struct cpufreq_policy *policy, u8 epp)
 	if (value == prev)
 		return 0;
 
+	/* Must update cached value prior to wrmsrq -- see msr_update_perf() */
 	WRITE_ONCE(cpudata->cppc_req_cached, value);
 
 	ret = wrmsrq_on_cpu(cpudata->cpu, MSR_AMD_CPPC_REQ, value);
@@ -1877,6 +1883,187 @@ static bool amd_pstate_acpi_pm_profile_undefined(void)
 	return false;
 }
 
+/*
+ * Per-core EPP boost.
+ *
+ * While the epp_boost module parameter is enabled, each CPU in active (EPP)
+ * mode on an MSR (X86_FEATURE_CPPC) system has a cpufreq update-util hook
+ * registered which puts the core in EPP=performance mode when the core is
+ * sufficiently utilized.
+ *
+ * Enables more efficient frequency scaling on cores that have strict latency
+ * requirements, such as e.g. gaming workloads where render threads use a CPU
+ * for some period during frame windows, and then go idle until the next frame.
+ */
+static bool epp_boost __read_mostly;
+
+#define AMD_PSTATE_EPP_BOOST_SAMPLE_NS	(10 * NSEC_PER_MSEC)
+/*
+ * Busy threshold is deliberately low and the decay window wide so that a
+ * render thread which is only ~50-80% busy (periodic vsync and GPU-fence
+ * waits) holds the boost across its whole busy period at a couple of CPPC_REQ
+ * writes total.
+ */
+#define AMD_PSTATE_EPP_BOOST_DECAY_NS	(300 * NSEC_PER_MSEC)
+#define AMD_PSTATE_EPP_BOOST_BUSY_PCT	50
+
+static void amd_pstate_epp_boost_apply(struct amd_cpudata *cpudata, u64 cached)
+{
+	union perf_cached perf = READ_ONCE(cpudata->perf);
+
+	/* never boost a request whose limits are not populated */
+	if (!perf.max_limit_perf)
+		return;
+
+	cached &= ~AMD_CPPC_EPP_PERF_MASK;
+	cached |= FIELD_PREP(AMD_CPPC_EPP_PERF_MASK, AMD_CPPC_EPP_PERFORMANCE);
+
+	wrmsrq(MSR_AMD_CPPC_REQ, cached);
+}
+
+static void amd_pstate_epp_boost_release(struct amd_cpudata *cpudata)
+{
+	wrmsrq(MSR_AMD_CPPC_REQ, READ_ONCE(cpudata->cppc_req_cached));
+}
+
+static void amd_pstate_epp_boost_update_util(struct update_util_data *data,
+					     u64 time, unsigned int flags)
+{
+	struct amd_cpudata *cpudata = container_of(data, struct amd_cpudata,
+						   epp_boost_update_util);
+	union perf_cached perf;
+	bool first_sample;
+	u64 busy_pct;
+	bool active;
+
+	if (smp_processor_id() != cpudata->cpu)
+		return;
+
+	first_sample = !cpudata->epp_boost_last_sample;
+	if (!first_sample &&
+	    time - cpudata->epp_boost_last_sample < AMD_PSTATE_EPP_BOOST_SAMPLE_NS)
+		return;
+	cpudata->epp_boost_last_sample = time;
+
+	/*
+	 * Counters can reset across suspend or hotplug (so their values would
+	 * be garbage), so just recalculate baselines.
+	 */
+	if (amd_pstate_sample(cpudata) && !first_sample) {
+		busy_pct = div64_u64(cpudata->cur.mperf * 100, cpudata->cur.tsc);
+		if (busy_pct >= AMD_PSTATE_EPP_BOOST_BUSY_PCT)
+			cpudata->epp_boost_last_busy = time;
+	}
+
+	perf = READ_ONCE(cpudata->perf);
+	active = perf.max_limit_perf &&
+		 (time - cpudata->epp_boost_last_busy < AMD_PSTATE_EPP_BOOST_DECAY_NS);
+
+	if (active && !cpudata->epp_boost_active)
+		amd_pstate_epp_boost_apply(cpudata,
+					   READ_ONCE(cpudata->cppc_req_cached));
+	else if (!active && cpudata->epp_boost_active)
+		amd_pstate_epp_boost_release(cpudata);
+
+	cpudata->epp_boost_active = active;
+}
+
+static void amd_pstate_epp_boost_enable(struct cpufreq_policy *policy)
+{
+	struct amd_cpudata *cpudata = policy->driver_data;
+
+	if (!READ_ONCE(epp_boost))
+		return;
+
+	if (cppc_state != AMD_PSTATE_ACTIVE ||
+	    !cpu_feature_enabled(X86_FEATURE_CPPC))
+		return;
+
+	if (cpudata->epp_boost_registered)
+		return;
+
+	cpudata->epp_boost_last_sample = 0;
+	cpudata->epp_boost_last_busy = 0;
+	cpufreq_add_update_util_hook(policy->cpu, &cpudata->epp_boost_update_util,
+				     amd_pstate_epp_boost_update_util);
+	cpudata->epp_boost_registered = true;
+}
+
+static void amd_pstate_epp_boost_disable(struct cpufreq_policy *policy)
+{
+	struct amd_cpudata *cpudata = policy->driver_data;
+
+	if (cpudata->epp_boost_registered) {
+		cpufreq_remove_update_util_hook(policy->cpu);
+		synchronize_rcu();
+		cpudata->epp_boost_registered = false;
+	}
+
+	if (cpudata->epp_boost_active) {
+		u64 cached = READ_ONCE(cpudata->cppc_req_cached);
+
+		wrmsrq_on_cpu(cpudata->cpu, MSR_AMD_CPPC_REQ, cached);
+
+		/*
+		 * Cross-CPU restore is not ordered against a concurrent
+		 * limit/EPP writer that doesn't hold the policy rwsem, so
+		 * recheck the cache once to ensure that the MSR reflects the
+		 * cached value
+		 */
+		if (READ_ONCE(cpudata->cppc_req_cached) != cached)
+			wrmsrq_on_cpu(cpudata->cpu, MSR_AMD_CPPC_REQ,
+				      READ_ONCE(cpudata->cppc_req_cached));
+
+		cpudata->epp_boost_active = false;
+	}
+}
+
+static int epp_boost_param_set(const char *val, const struct kernel_param *kp)
+{
+	unsigned int cpu;
+	bool enabled;
+	int ret;
+
+	ret = kstrtobool(val, &enabled);
+	if (ret)
+		return ret;
+
+	guard(mutex)(&amd_pstate_driver_lock);
+
+	if (epp_boost == enabled)
+		return 0;
+
+	WRITE_ONCE(epp_boost, enabled);
+
+	if (!current_pstate_driver)
+		return 0;
+
+	guard(cpus_read_lock)();
+
+	for_each_online_cpu(cpu) {
+		struct cpufreq_policy *policy __free(put_cpufreq_policy) =
+							cpufreq_cpu_get(cpu);
+
+		if (!policy)
+			continue;
+
+		if (enabled)
+			amd_pstate_epp_boost_enable(policy);
+		else
+			amd_pstate_epp_boost_disable(policy);
+	}
+
+	return 0;
+}
+
+static const struct kernel_param_ops epp_boost_param_ops = {
+	.set = epp_boost_param_set,
+	.get = param_get_bool,
+};
+module_param_cb(epp_boost, &epp_boost_param_ops, &epp_boost, 0644);
+MODULE_PARM_DESC(epp_boost,
+		 "boost EPP toward performance on recently-busy CPUs (active mode, MSR systems only)");
+
 static int amd_pstate_epp_cpu_init(struct cpufreq_policy *policy)
 {
 	struct amd_cpudata *cpudata;
@@ -1970,6 +2157,8 @@ static int amd_pstate_epp_cpu_init(struct cpufreq_policy *policy)
 
 	current_pstate_driver->adjust_perf = NULL;
 
+	amd_pstate_epp_boost_enable(policy);
+
 	return 0;
 
 free_cpudata1:
@@ -1989,6 +2178,8 @@ static void amd_pstate_epp_cpu_exit(struct cpufreq_policy *policy)
 		if (cpudata->dynamic_epp)
 			amd_pstate_clear_dynamic_epp(policy);
 
+		amd_pstate_epp_boost_disable(policy);
+
 		/* Reset CPPC_REQ MSR to the BIOS value */
 		amd_pstate_update_perf(policy, perf.bios_min_perf, 0U, 0U, 0U, false);
 		amd_pstate_set_floor_perf(policy, cpudata->bios_floor_perf);
@@ -2057,7 +2248,13 @@ static int amd_pstate_cpu_online(struct cpufreq_policy *policy)
 		return ret;
 
 	cached_floor_perf = freq_to_perf(perf, cpudata->nominal_freq, cpudata->floor_freq);
-	return amd_pstate_set_floor_perf(policy, cached_floor_perf);
+	ret = amd_pstate_set_floor_perf(policy, cached_floor_perf);
+	if (ret)
+		return ret;
+
+	amd_pstate_epp_boost_enable(policy);
+
+	return 0;
 }
 
 static int amd_pstate_cpu_offline(struct cpufreq_policy *policy)
@@ -2066,6 +2263,8 @@ static int amd_pstate_cpu_offline(struct cpufreq_policy *policy)
 	union perf_cached perf = READ_ONCE(cpudata->perf);
 	int ret;
 
+	amd_pstate_epp_boost_disable(policy);
+
 	/*
 	 * Reset CPPC_REQ MSR to the BIOS value, this will allow us to retain the BIOS specified
 	 * min_perf value across kexec reboots. If this CPU is just onlined normally after this, the
@@ -2088,6 +2287,8 @@ static int amd_pstate_suspend(struct cpufreq_policy *policy)
 	union perf_cached perf = READ_ONCE(cpudata->perf);
 	int ret;
 
+	amd_pstate_epp_boost_disable(policy);
+
 	/*
 	 * Reset CPPC_REQ MSR to the BIOS value, this will allow us to retain the BIOS specified
 	 * min_perf value across kexec reboots. If this CPU is just resumed back without kexec,
@@ -2134,10 +2335,9 @@ static int amd_pstate_epp_resume(struct cpufreq_policy *policy)
 	struct amd_cpudata *cpudata = policy->driver_data;
 	union perf_cached perf = READ_ONCE(cpudata->perf);
 	u8 cached_floor_perf;
+	int ret;
 
 	if (cpudata->suspended) {
-		int ret;
-
 		/* enable amd pstate from suspend state*/
 		ret = amd_pstate_epp_update_limit(policy, false);
 		if (ret)
@@ -2147,7 +2347,13 @@ static int amd_pstate_epp_resume(struct cpufreq_policy *policy)
 	}
 
 	cached_floor_perf = freq_to_perf(perf, cpudata->nominal_freq, cpudata->floor_freq);
-	return amd_pstate_set_floor_perf(policy, cached_floor_perf);
+	ret = amd_pstate_set_floor_perf(policy, cached_floor_perf);
+	if (ret)
+		return ret;
+
+	amd_pstate_epp_boost_enable(policy);
+
+	return 0;
 }
 
 static struct cpufreq_driver amd_pstate_driver = {
diff --git a/drivers/cpufreq/amd-pstate.h b/drivers/cpufreq/amd-pstate.h
index 99532595150e..0408fc331633 100644
--- a/drivers/cpufreq/amd-pstate.h
+++ b/drivers/cpufreq/amd-pstate.h
@@ -10,6 +10,7 @@
 
 #include <linux/pm_qos.h>
 #include <linux/platform_profile.h>
+#include <linux/sched/cpufreq.h>
 
 /*********************************************************************
  *                        AMD P-state INTERFACE                       *
@@ -96,6 +97,12 @@ struct amd_aperf_mperf {
  * @current_profile: Currently selected platform profile option
  * @ppdev: Device registered with the platform profile handler
  * @profile_name: Name under which @ppdev is registered
+ * @epp_boost_update_util: update-util hook for the per-core EPP boost
+ * @epp_boost_last_sample: last C0-residency sample time (ns; 0 means the
+ *			   next callback only re-baselines the counters)
+ * @epp_boost_last_busy: last time the CPU sampled as busy (ns)
+ * @epp_boost_active: EPP boost currently applied to MSR_AMD_CPPC_REQ
+ * @epp_boost_registered: update-util hook currently registered
  *
  * The amd_cpudata is key private data for each CPU thread in AMD P-State, and
  * represents all the attributes and goals that AMD P-State requests at runtime.
@@ -139,6 +146,13 @@ struct amd_cpudata {
 	enum platform_profile_option current_profile;
 	struct device *ppdev;
 	char *profile_name;
+
+	/* per-core EPP boost (active mode, MSR systems only) */
+	struct	update_util_data epp_boost_update_util;
+	u64	epp_boost_last_sample;
+	u64	epp_boost_last_busy;
+	bool	epp_boost_active;
+	bool	epp_boost_registered;
 };
 
 /*
-- 
2.53.0