[PATCH] x86/aperfmperf: Refresh stale sample via IPI for busy NOHZ_FULL CPUs

[PATCH] x86/aperfmperf: Refresh stale sample via IPI for busy NOHZ_FULL CPUs

[Jing Wu]

An isolated CPU covered by nohz_full stops its periodic tick once it
has only one runnable task, since sched_can_stop_tick() only checks
scheduling-class fairness and has no notion of cpufreq reporting
needs. arch_scale_freq_tick() runs only from scheduler_tick(), so
cpu_samples for that CPU is never refreshed again.

arch_freq_get_on_cpu() then permanently hits its staleness check and
falls back to cpufreq_quick_get(), which returns whatever policy->cur
was left at (typically the P-state floor). This happens even though
HWP hardware keeps running the CPU at full turbo autonomously, as
confirmed by turbostat and by directly reading APERF/MPERF.

Reproduce on an isolated, nohz_full CPU with intel_pstate/HWP by
loading it and watching scaling_cur_freq stay pinned at the floor:

    taskset -c $CPU stress --cpu 1 &
    for i in $(seq 10); do
        cat /sys/devices/system/cpu/cpu$CPU/cpufreq/scaling_cur_freq
        sleep 0.5
    done
    turbostat --cpu $CPU --interval 1 --num_iterations 5

scaling_cur_freq stays at the floor for the whole run, while
turbostat's Bzy_MHz confirms the CPU is actually at full turbo.

Refresh the stale sample with one on-demand arch_scale_freq_tick()
via IPI before falling back, but only when the target CPU is online
and not idle. APERF/MPERF both stop advancing during idle (C1+), so
a delta computed over an arbitrarily long stale window still yields
a correct busy-time frequency average.

Fixes: 7d84c1ebf9dd ("x86/aperfmperf: Replace aperfmperf_get_khz()")
Co-developed-by: Qiliang Yuan <yuanql9@chinatelecom.cn>
Signed-off-by: Qiliang Yuan <yuanql9@chinatelecom.cn>
Co-developed-by: Jian Zhang <zhangj332@chinatelecom.cn>
Signed-off-by: Jian Zhang <zhangj332@chinatelecom.cn>
Signed-off-by: Jing Wu <realwujing@gmail.com>
---
 arch/x86/kernel/cpu/aperfmperf.c | 34 +++++++++++++++++++++++++++++++++-
 1 file changed, 33 insertions(+), 1 deletion(-)

diff --git a/arch/x86/kernel/cpu/aperfmperf.c b/arch/x86/kernel/cpu/aperfmperf.c
index 7ffc78d5ebf21..e51544db9f9bb 100644
--- a/arch/x86/kernel/cpu/aperfmperf.c
+++ b/arch/x86/kernel/cpu/aperfmperf.c
@@ -12,6 +12,7 @@
 #include <linux/math64.h>
 #include <linux/percpu.h>
 #include <linux/rcupdate.h>
+#include <linux/sched.h>
 #include <linux/sched/isolation.h>
 #include <linux/sched/topology.h>
 #include <linux/smp.h>
@@ -503,16 +504,41 @@ void arch_scale_freq_tick(void)
  */
 #define MAX_SAMPLE_AGE	((unsigned long)HZ / 50)
 
+static void aperfmperf_snapshot_cpu_ipi(void *info)
+{
+	arch_scale_freq_tick();
+}
+
+/*
+ * A NOHZ_FULL CPU with a single runnable task (e.g. an isolated CPU running
+ * a pinned PMD/busy-poll workload) stops its periodic tick, so nothing ever
+ * calls arch_scale_freq_tick() for it again and cpu_samples goes stale
+ * forever, not just for one MAX_SAMPLE_AGE window. Force one on-demand
+ * sample via IPI so a deliberate frequency read doesn't report the P-state
+ * floor from before isolation took effect. Skip idle CPUs: their frequency
+ * genuinely doesn't matter and there is no point poking them with an IPI.
+ */
+static bool aperfmperf_refresh_stale_sample(int cpu)
+{
+	if (!cpu_online(cpu) || idle_cpu(cpu))
+		return false;
+
+	smp_call_function_single(cpu, aperfmperf_snapshot_cpu_ipi, NULL, 1);
+	return true;
+}
+
 int arch_freq_get_on_cpu(int cpu)
 {
 	struct aperfmperf *s = per_cpu_ptr(&cpu_samples, cpu);
 	unsigned int seq, freq;
 	unsigned long last;
+	bool refreshed = false;
 	u64 acnt, mcnt;
 
 	if (!cpu_feature_enabled(X86_FEATURE_APERFMPERF))
 		goto fallback;
 
+again:
 	do {
 		seq = raw_read_seqcount_begin(&s->seq);
 		last = s->last_update;
@@ -524,8 +550,14 @@ int arch_freq_get_on_cpu(int cpu)
 	 * Bail on invalid count and when the last update was too long ago,
 	 * which covers idle and NOHZ full CPUs.
 	 */
-	if (!mcnt || (jiffies - last) > MAX_SAMPLE_AGE)
+	if (!mcnt || (jiffies - last) > MAX_SAMPLE_AGE) {
+		if (!refreshed) {
+			refreshed = true;
+			if (aperfmperf_refresh_stale_sample(cpu))
+				goto again;
+		}
 		goto fallback;
+	}
 
 	return div64_u64((cpu_khz * acnt), mcnt);
 

---
base-commit: 502d801f0ab03e4f32f9a33d203154ce84887921
change-id: 20260728-bug-isolatecpu-cpufreq-864a5fba85b7

Best regards,
-- 
Jing Wu <realwujing@gmail.com>

Re: [PATCH] x86/aperfmperf: Refresh stale sample via IPI for busy NOHZ_FULL CPUs

[Peter Zijlstra]

On Tue, Jul 28, 2026 at 07:27:54PM +0800, Jing Wu wrote:
> An isolated CPU covered by nohz_full stops its periodic tick once it
> has only one runnable task, since sched_can_stop_tick() only checks
> scheduling-class fairness and has no notion of cpufreq reporting
> needs. arch_scale_freq_tick() runs only from scheduler_tick(), so
> cpu_samples for that CPU is never refreshed again.

Sending IPIs to NOHZ_FULL Cpus should be a no-no.

Re: [PATCH] x86/aperfmperf: Refresh stale sample via IPI for busy NOHZ_FULL CPUs

[Peter Zijlstra]

On Tue, Jul 28, 2026 at 07:27:54PM +0800, Jing Wu wrote:
> An isolated CPU covered by nohz_full stops its periodic tick once it
> has only one runnable task, since sched_can_stop_tick() only checks
> scheduling-class fairness and has no notion of cpufreq reporting
> needs. arch_scale_freq_tick() runs only from scheduler_tick(), so
> cpu_samples for that CPU is never refreshed again.
> 
> arch_freq_get_on_cpu() then permanently hits its staleness check and
> falls back to cpufreq_quick_get(), which returns whatever policy->cur
> was left at (typically the P-state floor). This happens even though
> HWP hardware keeps running the CPU at full turbo autonomously, as
> confirmed by turbostat and by directly reading APERF/MPERF.
> 
> Reproduce on an isolated, nohz_full CPU with intel_pstate/HWP by
> loading it and watching scaling_cur_freq stay pinned at the floor:
> 
>     taskset -c $CPU stress --cpu 1 &
>     for i in $(seq 10); do
>         cat /sys/devices/system/cpu/cpu$CPU/cpufreq/scaling_cur_freq
>         sleep 0.5
>     done
>     turbostat --cpu $CPU --interval 1 --num_iterations 5
> 
> scaling_cur_freq stays at the floor for the whole run, while
> turbostat's Bzy_MHz confirms the CPU is actually at full turbo.
> 
> Refresh the stale sample with one on-demand arch_scale_freq_tick()
> via IPI before falling back, but only when the target CPU is online
> and not idle. APERF/MPERF both stop advancing during idle (C1+), so
> a delta computed over an arbitrarily long stale window still yields
> a correct busy-time frequency average.

Aside from the fact that sending IPIs to NOHZ_FULL is just plain wrong,
this whole thing makes no sense.

When the CPU is isolated, nothing should care about the ratio anyway.
Just set the thing to '1' (1024) when the CPU enters NOHZ_FULL mode and
ensure it isn't ever modified.