Re: [PATCH] cpufreq, docs: (userspace governor) add that actual freq is >= scaling_setspeed

[Russell Haley <yumpusamongus@gmail.com>]

On 5/23/25 2:06 PM, Rafael J. Wysocki wrote:
> On Fri, May 23, 2025 at 6:25 AM Shashank Balaji
> <shashank.mahadasyam@sony.com> wrote:
>>
>> Hi Russell,
>>
>> On Thu, May 22, 2025 at 06:15:24AM -0500, Russell Haley wrote:
>>>> The userspace governor requests a frequency between policy->min and
>>>> policy->max on behalf of user space.  In intel_pstate this translates
>>>> to setting DESIRED_PERF to the requested value which is also the case
>>>> for the other governors.
>>>
>>> Huh.  On this Skylake box with kernel 6.14.6, it seems to be setting
>>> Minimum_Performance, and leaving desired at 0.
>>>
>>>> echo userspace | sudo tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
>>> userspace
>>>> echo 1400000 | sudo tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_setspeed
>>> 1400000
>>>> sudo x86_energy_perf_policy &| grep REQ
>>> cpu0: HWP_REQ: min 14 max 40 des 0 epp 128 window 0x0 (0*10^0us) use_pkg 0
>>
>> Oh cool, I didn't know about x86_energy_perf_policy.
>>
>> Consider the following on a Raptor Lake machine:
>>
>> 1. HWP_REQUEST MSR set by intel_pstate in active mode:
>>
>>         # echo active > intel_pstate/status
>>         # x86_energy_perf_policy -c 0 2>&1 | grep REQ
>>         cpu0: HWP_REQ: min 11 max 68 des 0 epp 128 window 0x0 (0*10^0us) use_pkg 0
>>         pkg0: HWP_REQ_PKG: min 1 max 255 des 0 epp 128 window 0x0 (0*10^0us)
>>         # echo 2000000 > cpufreq/policy0/scaling_min_freq
>>         # echo 3000000 > cpufreq/policy0/scaling_max_freq
>>         # x86_energy_perf_policy -c 0 2>&1 | grep REQ
>>         cpu0: HWP_REQ: min 26 max 39 des 0 epp 128 window 0x0 (0*10^0us) use_pkg 0
>>         pkg0: HWP_REQ_PKG: min 1 max 255 des 0 epp 128 window 0x0 (0*10^0us)
>>
>>         scaling_{min,max}_freq just affect the min and max frequencies
>>         set in HWP_REQEST. desired_freq is left at 0.
>>
>> 2. HWP_REQUEST MSR set by intel_pstate in passive mode with userspace
>> governor:
>>
>>         # echo passive > intel_pstate/status
>>         # echo userspace > cpufreq/policy0/scaling_governor
>>         # cat cpufreq/policy0/scaling_setspeed
>>         866151
>>         # x86_energy_perf_policy -c 0 2>&1 | grep REQ
>>         cpu0: HWP_REQ: min 11 max 68 des 0 epp 128 window 0x0 (0*10^0us) use_pkg 0
>>         pkg0: HWP_REQ_PKG: min 1 max 255 des 0 epp 128 window 0x0 (0*10^0us)
>>         # echo 2000000 > cpufreq/policy0/scaling_setspeed
>>         # x86_energy_perf_policy -c 0 2>&1 | grep REQ
>>         cpu0: HWP_REQ: min 26 max 68 des 0 epp 128 window 0x0 (0*10^0us) use_pkg 0
>>         pkg0: HWP_REQ_PKG: min 1 max 255 des 0 epp 128 window 0x0 (0*10^0us)
>>
>>         scaling_setspeed only changes the min frequency in HWP_REQUEST.
>>         Meaning, software is explicitly allowing the hardware to choose
>>         higher frequencies.
>>
>> 3. Same as above, except with strictuserspace governor, which is a
>> custom kernel module which is exactly the same as the userspace
>> governor, except it has the CPUFREQ_GOV_STRICT_TARGET flag set:
>>
>>         # echo strictuserspace > cpufreq/policy0/scaling_governor
>>         # x86_energy_perf_policy -c 0 2>&1 | grep REQ
>>         cpu0: HWP_REQ: min 26 max 26 des 0 epp 128 window 0x0 (0*10^0us) use_pkg 0
>>         pkg0: HWP_REQ_PKG: min 1 max 255 des 0 epp 128 window 0x0 (0*10^0us)
>>         # echo 3000000 > cpufreq/policy0/scaling_setspeed
>>         # x86_energy_perf_policy -c 0 2>&1 | grep REQ
>>         cpu0: HWP_REQ: min 39 max 39 des 0 epp 128 window 0x0 (0*10^0us) use_pkg 0
>>         pkg0: HWP_REQ_PKG: min 1 max 255 des 0 epp 128 window 0x0 (0*10^0us)
>>
>>         With the strict flag set, intel_pstate honours this by setting
>>         the min and max freq same.
>>
>> desired_perf is always 0 in the above cases. The strict flag check is done in
>> intel_cpufreq_update_pstate, which sets max_pstate to target_pstate if policy
>> has strict target, and cpu->max_perf_ratio otherwise.
>>
>> As Russell and Rafael have noted, CPU frequency is subject to hardware
>> coordination and optimizations. While I get that, shouldn't software try
>> its best with whatever interface it has available? If a user sets the
>> userspace governor, that's because they want to have manual control over
>> CPU frequency, for whatever reason. The kernel should honor this by
>> setting the min and max freq in HWP_REQUEST equal. The current behaviour
>> explicitly lets the hardware choose higher frequencies.
> 
> Well, the userspace governor ends up calling the same function,
> intel_cpufreq_target(), as other cpufreq governors except for
> schedutil.  This function needs to work for all of them and for some
> of them setting HWP_MIN_PERF to the same value as HWP_MAX_PERF would
> be too strict.  HWP_DESIRED_PERF can be set to the same value as
> HWP_MIN_PERF, though (please see the attached patch).

The other governors have been around a lot longer than HWP, though, and
and are used on non-Intel hardware, which may not have a, "this
frequency or higher subject to firmware heuristics," interface.

I tried this on a non-HWP Haswell machine, and there it works like
DESIRED=MIN. Or maybe DESIRED=MAX=MIN; I don't understand when or why
hardware would choose frequencies between DESIRED and MAX (before module
coordination).

IMO, intel_cpufreq_target() being wired up to HWP_MIN_PERF is actually
*more* strange for the other governors than for userspace, because at
least with userspace governor, the userspace program is free to write to
scaling_{min,max}_freq instead of scaling_setspeed if it wants.

The conservative governor on HWP hardware, for example, will cause
strictly higher frequencies (and typically, higher energy consumption)
than HWP powersave. But on non-HWP hardware, conservative is an
efficient, slow-ramping governor.

Changing the behavior of the old-style cpufreq governors is fraught,
because the defaults are schedutil and HWP-powersave, so users of the
other governors likely made an intentional choice, presumably after
tests on a specific platform. A change would invalidate those tests.

But on the other hand, they might *already* be invalid because of an
upgrade from non-HWP hardware. In that case, changing to DES=MIN would
move closer to the tested behavior.

And then there's churn coming from other parts of the stack. For
example, until recently [1] tuned would select conservative for its
"balanced" profile and ondemand for its "powersave" profile, based on
very old data. But that didn't matter until Redhat stopped funding work
on power-profiles-daemon, and the desktop environments' power-profile
selectors got wired up to tuned in Fedora. Hector Martin fixed that,
switching both to schedutil (unless CONFIG_CPU_FREQ_GOV_SCHEDUTIL=n,
which is rare I think). That is at least not terrible on non-HWP
hardware, but given what he was working on at the time, it might not
have been tested on x86.

[1]
https://github.com/redhat-performance/tuned/commit/e24bfef651aa7f4da95727815b2cacbf571b59af

Cheers,
Russell