powersave governor runs programs faster and uses more power than performance governor

powersave governor runs programs faster and uses more power than performance governor

[Melanie Kambadur]

Hi all,

I am trying to understand some strange results I've observed using the
3.9.11 kernel version of the powersave cpufreq governor on a Intel
Xeon machine, and hoping you all can help.

More specifically, I am using a 24 core Intel Xeon 2430 machine
running Ubuntu 12.04 and v 3.9.11 of the kernel. As I understand it,
this configuration affords me two cpufreq scaling governors, powersave
and performance. At least these are the options I get by running:
/sys/devices/system/cpu/cpu0/cpufreq/scaling_available_governors

Here's the issue. When I turn on the powersave governor, I have
measured the power consumed to INCREASE relative to the performance
governor. More surprisingly, I consistently see BETTER performance
(i.e., shorter runtimes) with the powersave governor than with the
performance governor for many applications out of a few sets of
commonly used benchmark applications.  I repeated my experiments
several times, even trying different cpuidle governors, and observed
the same trends. To toggle between governors, I have been using the
cpufreq-set -g command and confirming the updates by checking
/sys/devices/system/cpu/cpu0/cpufreq/scaling_governor.

From what I've read (e.g., here:
www.kernel.org/doc/Documentation/cpu-freq/governors.txt), the
powersave governor's policy is to statically set the cpu frequency at
the system minimum, whereas the performance governor aims to set the
cpu frequency higher (or, depending on the documentation, at maximum).
However, the fact that applications sped up for the powersave vs. the
performance governor with these policies seemed incorrect to me, so I
investigated further...

I profiled the current system frequency by periodically polling
/sys/devices/system/cpu/cpu${N}/cpufreq/cpuinfo_cur_freq
for all the available "N" values (0-23) on my machine.  I found that
not only does the powersave governor not seem to be running at the
system minimum, it actually seems to be running at a frequency higher
than the performance governor. This would explain why I am seeing
better performance and higher power for the powersave governor, but it
certainly doesn't fit the policy descriptions in the documentation.
Here are the results of my mini experiment:

-Minimum system frequency=1.20 * 10^6
-Maximum system frequency=2.70 * 10^6

Avg. cpu frequency across all CPUs for 30 seconds of...
-Powersave governor with no applications running = 1.77 * 10^6
-Powersave governor with a benchmark app running = 2.23 * 10^6
-Performance governor with no apps running = 1.23 * 10^6
-Performance governor with a benchmark app running = 1.34 * 10^6

Am I misunderstanding the policies of the powersave vs. performance
governor, or is this possibly a system bug?

Thanks for your help.

Sincerely,
Melanie Kambadur

Re: powersave governor runs programs faster and uses more power than performance governor

[Viresh Kumar]

On Thu, Oct 24, 2013 at 4:47 AM, Melanie Kambadur
<melanie@cs.columbia.edu> wrote:
> I am trying to understand some strange results I've observed using the
> 3.9.11 kernel version of the powersave cpufreq governor on a Intel
> Xeon machine, and hoping you all can help.
>
> More specifically, I am using a 24 core Intel Xeon 2430 machine
> running Ubuntu 12.04 and v 3.9.11 of the kernel. As I understand it,
> this configuration affords me two cpufreq scaling governors, powersave
> and performance. At least these are the options I get by running:
> /sys/devices/system/cpu/cpu0/cpufreq/scaling_available_governors
>
> Here's the issue. When I turn on the powersave governor, I have
> measured the power consumed to INCREASE relative to the performance
> governor. More surprisingly, I consistently see BETTER performance
> (i.e., shorter runtimes) with the powersave governor than with the
> performance governor for many applications out of a few sets of
> commonly used benchmark applications.  I repeated my experiments
> several times, even trying different cpuidle governors, and observed
> the same trends. To toggle between governors, I have been using the
> cpufreq-set -g command and confirming the updates by checking
> /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor.
>
> From what I've read (e.g., here:
> www.kernel.org/doc/Documentation/cpu-freq/governors.txt), the
> powersave governor's policy is to statically set the cpu frequency at
> the system minimum, whereas the performance governor aims to set the
> cpu frequency higher (or, depending on the documentation, at maximum).
> However, the fact that applications sped up for the powersave vs. the
> performance governor with these policies seemed incorrect to me, so I
> investigated further...
>
> I profiled the current system frequency by periodically polling
> /sys/devices/system/cpu/cpu${N}/cpufreq/cpuinfo_cur_freq
> for all the available "N" values (0-23) on my machine.  I found that
> not only does the powersave governor not seem to be running at the
> system minimum, it actually seems to be running at a frequency higher
> than the performance governor. This would explain why I am seeing
> better performance and higher power for the powersave governor, but it
> certainly doesn't fit the policy descriptions in the documentation.
> Here are the results of my mini experiment:
>
> -Minimum system frequency=1.20 * 10^6
> -Maximum system frequency=2.70 * 10^6
>
> Avg. cpu frequency across all CPUs for 30 seconds of...
> -Powersave governor with no applications running = 1.77 * 10^6
> -Powersave governor with a benchmark app running = 2.23 * 10^6
> -Performance governor with no apps running = 1.23 * 10^6
> -Performance governor with a benchmark app running = 1.34 * 10^6
>
> Am I misunderstanding the policies of the powersave vs. performance
> governor, or is this possibly a system bug?

There is something really long in your machine.. Any chance you can
try latest v3.12-rc kernel?

Also, please provide values of related and affected CPUs, for all CPUs..
Also, just to confirm, you are changing governor to powersave or
performance for all the CPUs??

Otherwise there is no valid reason why this should happen :)

Re: powersave governor runs programs faster and uses more power than performance governor

[Melanie Kambadur]

Thank you for your quick and very helpful responses.

A couple of updates. I neglected to RTFM for the cpufreq setter I was
using :)  When I started to print out the values for all the CPUs
rather than just the averages as Viresh suggested, I realized that my
governor settings were only applying to one of the cores at a time. My
apologies for the silly mistake, I thought I had verified that the
changes were applied to all of the cores.

After actually applying the frequency governor updates to all of the
cores (and triple-checking this time), the new results for my
mini-experiment are still odd. I don't know a good way to share data
on this forum, please see a snippet of the data at the end of this
note and let me know if there is a better way to share the complete
data set. As a summary, the new average frequencies across all the
cores were:
performance w/ no apps running = 1.13 * 10 ^ 6
performance w/ apps running = 1.33 * 10 ^ 6
powersave w/ no apps running = 1.38 * 10 ^ 6
powersave w/ apps running = 1.95 * 10 ^ 6

I compared these numbers (from cpufreq/cpuinfo_cur_freq) to i7z
reports and they seem to be reasonable. It's hard to compare
perfectly, because I can't get i7z to print the frequency values in
plain text as I would like, but they are definitely in the same
ballpark (look to be within 100 Mhz).

Obviously, these still aren't the frequency values we'd expect.  I
think David may be correct that the Dell firmware is somehow
overriding the linux governors. Here are some more details about my
server:
Dell Power Edge R420 with 2 sockets, both:
Intel® Xeon® E5-2430 2.20GHz, 15M Cache, 7.2GT/s
QPI, Turbo, 6C, 95W, Max Mem 1333MHz E52430
Each socket actually has 6 cores, with dual SMT to make 12 logical
cores per socket, or 24 total logical cores.

From /sys/devices/system/cpu/cpuN/cpufreq/scaling_driver I get that
the current p-state driver is called "intel_pstate". David, you
mention that the firmware governors are not very efficient, do you
suggest replacing the intel_pstate driver with a different driver? Of
the drivers listed here:
https://wiki.archlinux.org/index.php/CPU_Frequency_Scaling#CPU_frequency_driver
, I apparently only have available speedstep and p4-clockmod in my
current kernel. Is one of those better than intel_pstate or will I
need to download a new driver or even update the kernel to get another
one?

Also, by C1E do you mean idle state management? I should have given
some context for my adjustments to the power management policies,
which is that I am a grad student trying to research how system level
energy management policies compare to some specific application level
energy management policies. I would actually like to test a range of
system level policies, including different kinds of frequency and idle
state managers. The original goal was to compare a power-optimized
system version with a performance-optimized version (or a few such
versions), but I am learning that the options are not so simple. I
initially thought that on-demand would be the most power-efficient
frequency governor, but when I noticed that the on-demand governor was
missing in my available governors list, I did some digging and
discovered people writing that on-demand was deprecated for
Sandy-Bridge (e.g.,
http://www.phoronix.com/scan.php?page=news_item&px=MTM3NDQ) Is this
true? On a more general note, does anyone know what the theoretically
most power- and performance-optimized frequency governors/drivers
would be for my system setup?

Thanks again,

Melanie

P.S. I haven't yet tried the latest v3.12-rc kernel, and while it is
an option, I would prefer to get the frequency tuning working on my
existing kernel to avoid having to re-run some other relevant
experiments.
----------------
Some data snippets (I am giving 4 half-second interval samples for
each of the 24 cores/HW threads, for each of the 4 configs mentioned
above).

performance w/ no apps running
0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19,
20, 21, 22, 23
1188000, 1188000, 1188000, 1188000, 1188000, 1188000, 1188000,
1188000, 1188000, 1188000, 1188000, 1188000, 1188000, 1188000,
1188000, 1188000, 1188000, 1188000, 1188000, 1188000, 1188000,
1188000, 1188000, 1188000

1188000, 1188000, 1188000, 1188000, 1188000, 1188000, 1188000,
1188000, 1188000, 1188000, 1188000, 1188000, 1188000, 1188000,
1188000, 1188000, 1188000, 1188000, 1188000, 1188000, 1188000,
1188000, 1188000, 1188000

1188000, 1188000, 1188000, 1188000, 1188000, 1188000, 1188000,
1188000, 1188000, 1188000, 1188000, 1188000, 1188000, 1188000,
1188000, 1188000, 1188000, 1188000, 1188000, 1188000, 1188000,
1188000, 1188000, 1188000

1188000, 1188000, 1188000, 1188000, 1188000, 1188000, 1188000,
1188000, 1188000, 1188000, 1188000, 1188000, 1188000, 1188000,
1188000, 1188000, 1188000, 1188000, 1188000, 1188000, 1188000,
1188000, 1188000, 1188000

performance w/ apps running
0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19,
20, 21, 22, 23
1188000, 1188000, 1188000, 1650000, 1188000, 1188000, 1188000,
1188000, 1188000, 1188000, 1188000, 1958000, 1188000, 1188000,
1188000, 1188000, 1188000, 1188000, 1188000, 1188000, 1188000,
1188000, 1188000, 1188000

1826000, 1188000, 2486000, 1254000, 2486000, 1188000, 2486000,
1188000, 2486000, 1188000, 2486000, 1188000, 2486000, 1188000,
2486000, 1188000, 2486000, 1188000, 2486000, 1188000, 1188000,
1188000, 1188000, 1188000

1298000, 1188000, 1188000, 1188000, 1188000, 1188000, 1188000,
1188000, 1188000, 1188000, 1188000, 1188000, 1188000, 1188000,
1188000, 1188000, 1188000, 1188000, 1188000, 1188000, 1188000,
1188000, 1188000, 1188000

1188000, 1188000, 1188000, 1188000, 1188000, 1188000, 1188000,
1188000, 1188000, 1188000, 1188000, 1188000, 1188000, 1188000,
1188000, 1188000, 1188000, 1188000, 1188000, 1188000, 1188000,
1188000, 1188000, 1188000

powersave w/ no apps running
0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19,
20, 21, 22, 23
1320000, 1518000, 1232000, 1738000, 1386000, 2354000, 1210000,
2486000, 1298000, 2486000, 1958000, 2464000, 1342000, 2332000,
2310000, 2442000, 2068000, 2486000, 1474000, 2486000, 1562000,
2486000, 1452000, 2486000

1320000, 1782000, 1958000, 2486000, 1672000, 2486000, 1694000,
2486000, 1298000, 2486000, 1958000, 2354000, 1232000, 2486000,
2332000, 2420000, 2442000, 2486000, 2244000, 2486000, 1562000,
2486000, 1408000, 2486000

1914000, 2090000, 1782000, 2464000, 1914000, 2486000, 1694000,
2486000, 1826000, 2486000, 1958000, 2332000, 1232000, 2310000,
1914000, 1980000, 2332000, 2354000, 2244000, 2354000, 1562000,
2002000, 1408000, 2486000

1914000, 1606000, 1716000, 2354000, 1936000, 2420000, 1562000,
2420000, 1584000, 2486000, 1958000, 2486000, 1232000, 2486000,
2332000, 2464000, 2442000, 2442000, 2244000, 2486000, 1496000,
2486000, 1320000, 2486000

powersave w/ apps running
0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19,
20, 21, 22, 23
2178000, 1694000, 1738000, 1628000, 2178000, 1870000, 2200000,
1298000, 2178000, 1474000, 2200000, 1606000, 2112000, 2200000,
2200000, 2002000, 2178000, 2112000, 2178000, 2046000, 2024000,
1936000, 1848000, 1188000

2178000, 1694000, 1738000, 1452000, 2486000, 1188000, 2486000,
1826000, 2486000, 1188000, 2486000, 1848000, 2112000, 2200000,
2486000, 1980000, 2486000, 1782000, 2200000, 2046000, 2024000,
1936000, 1848000, 1188000

1430000, 1694000, 2354000, 1452000, 2156000, 1826000, 2156000,
1738000, 2046000, 1826000, 2090000, 1562000, 2200000, 2046000,
2156000, 2178000, 2200000, 2134000, 2046000, 2178000, 2112000,
2178000, 2200000, 2178000

2046000, 1738000, 2200000, 1826000, 2200000, 1694000, 2200000,
1716000, 2178000, 2134000, 2178000, 1760000, 2200000, 1914000,
2178000, 2178000, 2178000, 2178000, 2178000, 2200000, 2178000,
2200000, 2200000, 1782000

On Thu, Oct 24, 2013 at 1:28 AM, Viresh Kumar <viresh.kumar@linaro.org> wrote:
>
> On Thu, Oct 24, 2013 at 4:47 AM, Melanie Kambadur
> <melanie@cs.columbia.edu> wrote:
> > I am trying to understand some strange results I've observed using the
> > 3.9.11 kernel version of the powersave cpufreq governor on a Intel
> > Xeon machine, and hoping you all can help.
> >
> > More specifically, I am using a 24 core Intel Xeon 2430 machine
> > running Ubuntu 12.04 and v 3.9.11 of the kernel. As I understand it,
> > this configuration affords me two cpufreq scaling governors, powersave
> > and performance. At least these are the options I get by running:
> > /sys/devices/system/cpu/cpu0/cpufreq/scaling_available_governors
> >
> > Here's the issue. When I turn on the powersave governor, I have
> > measured the power consumed to INCREASE relative to the performance
> > governor. More surprisingly, I consistently see BETTER performance
> > (i.e., shorter runtimes) with the powersave governor than with the
> > performance governor for many applications out of a few sets of
> > commonly used benchmark applications.  I repeated my experiments
> > several times, even trying different cpuidle governors, and observed
> > the same trends. To toggle between governors, I have been using the
> > cpufreq-set -g command and confirming the updates by checking
> > /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor.
> >
> > From what I've read (e.g., here:
> > www.kernel.org/doc/Documentation/cpu-freq/governors.txt), the
> > powersave governor's policy is to statically set the cpu frequency at
> > the system minimum, whereas the performance governor aims to set the
> > cpu frequency higher (or, depending on the documentation, at maximum).
> > However, the fact that applications sped up for the powersave vs. the
> > performance governor with these policies seemed incorrect to me, so I
> > investigated further...
> >
> > I profiled the current system frequency by periodically polling
> > /sys/devices/system/cpu/cpu${N}/cpufreq/cpuinfo_cur_freq
> > for all the available "N" values (0-23) on my machine.  I found that
> > not only does the powersave governor not seem to be running at the
> > system minimum, it actually seems to be running at a frequency higher
> > than the performance governor. This would explain why I am seeing
> > better performance and higher power for the powersave governor, but it
> > certainly doesn't fit the policy descriptions in the documentation.
> > Here are the results of my mini experiment:
> >
> > -Minimum system frequency=1.20 * 10^6
> > -Maximum system frequency=2.70 * 10^6
> >
> > Avg. cpu frequency across all CPUs for 30 seconds of...
> > -Powersave governor with no applications running = 1.77 * 10^6
> > -Powersave governor with a benchmark app running = 2.23 * 10^6
> > -Performance governor with no apps running = 1.23 * 10^6
> > -Performance governor with a benchmark app running = 1.34 * 10^6
> >
> > Am I misunderstanding the policies of the powersave vs. performance
> > governor, or is this possibly a system bug?
>
> There is something really long in your machine.. Any chance you can
> try latest v3.12-rc kernel?
>
> Also, please provide values of related and affected CPUs, for all CPUs..
> Also, just to confirm, you are changing governor to powersave or
> performance for all the CPUs??
>
> Otherwise there is no valid reason why this should happen :)

Re: powersave governor runs programs faster and uses more power than performance governor

[David C Niemi]

On 10/24/13 15:42, Melanie Kambadur wrote:
> Thank you for your quick and very helpful responses.
>
> A couple of updates. I neglected to RTFM for the cpufreq setter I was
> using :)  When I started to print out the values for all the CPUs
> rather than just the averages as Viresh suggested, I realized that my
> governor settings were only applying to one of the cores at a time. My
> apologies for the silly mistake, I thought I had verified that the
> changes were applied to all of the cores.
>
> After actually applying the frequency governor updates to all of the
> cores (and triple-checking this time), the new results for my
> mini-experiment are still odd. I don't know a good way to share data
> on this forum, please see a snippet of the data at the end of this
> note and let me know if there is a better way to share the complete
> data set. As a summary, the new average frequencies across all the
> cores were:
> performance w/ no apps running = 1.13 * 10 ^ 6
> performance w/ apps running = 1.33 * 10 ^ 6
> powersave w/ no apps running = 1.38 * 10 ^ 6
> powersave w/ apps running = 1.95 * 10 ^ 6
>
> I compared these numbers (from cpufreq/cpuinfo_cur_freq) to i7z
> reports and they seem to be reasonable. It's hard to compare
> perfectly, because I can't get i7z to print the frequency values in
> plain text as I would like, but they are definitely in the same
> ballpark (look to be within 100 Mhz).
>
> Obviously, these still aren't the frequency values we'd expect.  I
> think David may be correct that the Dell firmware is somehow
> overriding the linux governors. Here are some more details about my
> server:
> Dell Power Edge R420 with 2 sockets, both:
> Intel® Xeon® E5-2430 2.20GHz, 15M Cache, 7.2GT/s
> QPI, Turbo, 6C, 95W, Max Mem 1333MHz E52430
> Each socket actually has 6 cores, with dual SMT to make 12 logical
> cores per socket, or 24 total logical cores.
You very definitely need to look at Dell's BIOS power management settings.  By default they tend to override what you are trying to do at the operating system level.  What you probably want is an "OS Control" setting.

>
> From /sys/devices/system/cpu/cpuN/cpufreq/scaling_driver I get that
> the current p-state driver is called "intel_pstate". David, you
> mention that the firmware governors are not very efficient, do you
> suggest replacing the intel_pstate driver with a different driver? Of
> the drivers listed here:
> https://wiki.archlinux.org/index.php/CPU_Frequency_Scaling#CPU_frequency_driver
> , I apparently only have available speedstep and p4-clockmod in my
> current kernel. Is one of those better than intel_pstate or will I
> need to download a new driver or even update the kernel to get another
> one?
The intel_pstate governor is quite new -- it is both a governor and a driver if you want to compare it.  The older more established approach is to use acpi-cpufreq as the driver and Ondemand as the governor, which works quite well for many use cases if properly configured.  We have people on this list who know intel_idle a lot better than I do, but if Dell is using its Active Power Controller intel_idle is not calling the shots anyway.

> Also, by C1E do you mean idle state management? I should have given
> some context for my adjustments to the power management policies,
> which is that I am a grad student trying to research how system level
> energy management policies compare to some specific application level
> energy management policies. I would actually like to test a range of
> system level policies, including different kinds of frequency and idle
> state managers. The original goal was to compare a power-optimized
> system version with a performance-optimized version (or a few such
> versions), but I am learning that the options are not so simple. I
> initially thought that on-demand would be the most power-efficient
> frequency governor, but when I noticed that the on-demand governor was
> missing in my available governors list, I did some digging and
> discovered people writing that on-demand was deprecated for
> Sandy-Bridge (e.g.,
> http://www.phoronix.com/scan.php?page=news_item&px=MTM3NDQ) Is this
> true? On a more general note, does anyone know what the theoretically
> most power- and performance-optimized frequency governors/drivers
> would be for my system setup?
>
> Thanks again,
>
> Melanie
>
> P.S. I haven't yet tried the latest v3.12-rc kernel, and while it is
> an option, I would prefer to get the frequency tuning working on my
> existing kernel to avoid having to re-run some other relevant
> experiments.
>
...
Ondemand works very well for Sandy Bridge if properly configured for your intended application.  The new Intel Pstate governor is specifically targeted to Sandy Bridge and later processors, and provides an interesting alternative to Ondemand within that scope, but that does not mean Ondemand is "deprecated".  Ondemand is the most common P-State governor across a huge variety of platforms ranging from phones to large servers and across many brands of processors besides Intel; it is silly to call it "deprecated" just because one of these platforms has an alternative to it.  In fact many phones have alternatives to Ondemand too, as well as many platform-specific variants.  Note that there are very big differences across these platforms -- on phones and other battery-powered devices, power savings are paramount, while on a server, performance under peak loads is usually paramount.  As for Ted Ts'o's observation, Ondemand was originally designed before tickless kernels and it is
obvious it needs to be adapted to not wake up an idle CPU just to assess load in a battery-powered applicaiton.  You might instead want to wake up when you get interrupts due to network activity.  But that is not to say managing clock rates is not a good idea, just that we have to adapt and rethink things.

There are two main sides of power management, P States (i.e. clock speed) and C States (i.e. what type of "halt" instruction is used).  intel_pstate is of course a manager of P States.  intel_idle and acpi_idle are C State drivers; most people use the "menu" C state governor and it is just a question of which C State driver to use and how to configure it.

Modern Intel processors rely heavily on having a very effective C1 sleep state that the scheduler calls when a core is idle for a short time.  C1E is the original standard for an enhanced C1 sleep state but Intel continues to improve on it, so you may see references to C1-NHM (Nehalem) or C1-SNB (Sandy Bridge) to distinguish feature changes between processor versions.

A few applications that require very low latency coming out of sleep may need to avoid sleep states deeper than C1/C1E or C3 (the deeper the sleep, the longer it takes to wake up and be ready for productive work).  It is almost never a good idea to turn off C1E -- latency to get out of C1E is very short and it saves a lot of power vs. "polling" (i.e. just leaving the core active and having it run a busy wait loop).

DCN

Re: powersave governor runs programs faster and uses more power than performance governor

[Dirk Brandewie]

On 10/24/2013 12:42 PM, Melanie Kambadur wrote:
>
>  From /sys/devices/system/cpu/cpuN/cpufreq/scaling_driver I get that
> the current p-state driver is called "intel_pstate". David, you
> mention that the firmware governors are not very efficient, do you
> suggest replacing the intel_pstate driver with a different driver?

I will need to look and see why changing to performance isn't working
correctly.

To get the behavior of the performance governor you can use

    echo 100 > /sys/devices/system/cpu/intel_pstate/min_perf_pct

This will force intel_pstate to select the highest P state and
leave it there.

Turbostat is useful for collecting frequency (P state) and idle (C state)
information.

--Dirk

Re: powersave governor runs programs faster and uses more power than performance governor

[Melanie Kambadur]

I appreciate you all taking the time to walk me through this.  Let me
see if I understand the new comments. Intel p-states is a HW-based
power manager, and strictly an alternative to (i.e., it cannot be
combined with) OS governors and drivers. If I want to use ondemand
with my Dell server I need to:

1) Modify the BIOS to give the OS exclusive power management control
because otherwise an OS driver won't be able to work properly. (I
think I know how to do this now after some more reading, e.g. here
http://en.community.dell.com/techcenter/power-cooling/w/wiki/best-practices-in-power-management.aspx
if anyone is curious.)
2) Set the O/S cpufreq driver to acpi_cpufreq, and
3) Set the O/S cpufreq governor to ondemand.

Is that correct?

Also, which driver should I try to use if I want to test the
performance & powersave governors again (or if I replicate the
behavior of the performance governor by manually modifying the
min_perf_pct value as Dirk suggested)? Will it still be acpi_cpufreq?

Finally, the behavior of the C-states is totally independent of
P-states and any kind of OS-based frequency tuning policy, correct?
However, David recommends that leaving C1E on rarely hurts performance
while significantly improving power.

-Melanie

On Fri, Oct 25, 2013 at 10:40 AM, Dirk Brandewie
<dirk.brandewie@gmail.com> wrote:
> On 10/24/2013 12:42 PM, Melanie Kambadur wrote:
>>
>>
>>  From /sys/devices/system/cpu/cpuN/cpufreq/scaling_driver I get that
>> the current p-state driver is called "intel_pstate". David, you
>> mention that the firmware governors are not very efficient, do you
>> suggest replacing the intel_pstate driver with a different driver?
>
>
> I will need to look and see why changing to performance isn't working
> correctly.
>
> To get the behavior of the performance governor you can use
>
>    echo 100 > /sys/devices/system/cpu/intel_pstate/min_perf_pct
>
> This will force intel_pstate to select the highest P state and
> leave it there.
>
> Turbostat is useful for collecting frequency (P state) and idle (C state)
> information.
>
> --Dirk
>
>

Re: powersave governor runs programs faster and uses more power than performance governor

[Dirk Brandewie]

On 10/25/2013 08:13 AM, Melanie Kambadur wrote:
> I appreciate you all taking the time to walk me through this.  Let me
> see if I understand the new comments. Intel p-states is a HW-based
> power manager,

Not hardware based but specific to Intel CPUs SandyBridge+

> and strictly an alternative to (i.e., it cannot be
> combined with) OS governors and drivers.

Correct

>If I want to use ondemand with my Dell server I need to:

Add intel_pstate=disable on your kernel commond line, this will take
intel_pstate out of the picture.  For the rest of the config on the
dell system I am no help sorry.

>
> 1) Modify the BIOS to give the OS exclusive power management control
> because otherwise an OS driver won't be able to work properly. (I
> think I know how to do this now after some more reading, e.g. here
> http://en.community.dell.com/techcenter/power-cooling/w/wiki/best-practices-in-power-management.aspx
> if anyone is curious.)
> 2) Set the O/S cpufreq driver to acpi_cpufreq, and
> 3) Set the O/S cpufreq governor to ondemand.
>
> Is that correct?
>
> Also, which driver should I try to use if I want to test the
> performance & powersave governors again (or if I replicate the
> behavior of the performance governor by manually modifying the
> min_perf_pct value as Dirk suggested)? Will it still be acpi_cpufreq?

If intel_pstate is being used acpi_cpufreq will not be loaded.  Setting 
performance with intel_pstate should work I will look to see where the
bug is.

>
> Finally, the behavior of the C-states is totally independent of
> P-states and any kind of OS-based frequency tuning policy, correct?

Correct

> However, David recommends that leaving C1E on rarely hurts performance
> while significantly improving power.
>
> -Melanie
>
> On Fri, Oct 25, 2013 at 10:40 AM, Dirk Brandewie
> <dirk.brandewie@gmail.com> wrote:
>> On 10/24/2013 12:42 PM, Melanie Kambadur wrote:
>>>
>>>
>>>   From /sys/devices/system/cpu/cpuN/cpufreq/scaling_driver I get that
>>> the current p-state driver is called "intel_pstate". David, you
>>> mention that the firmware governors are not very efficient, do you
>>> suggest replacing the intel_pstate driver with a different driver?
>>
>>
>> I will need to look and see why changing to performance isn't working
>> correctly.
>>
>> To get the behavior of the performance governor you can use
>>
>>     echo 100 > /sys/devices/system/cpu/intel_pstate/min_perf_pct
>>
>> This will force intel_pstate to select the highest P state and
>> leave it there.
>>
>> Turbostat is useful for collecting frequency (P state) and idle (C state)
>> information.
>>
>> --Dirk
>>
>>

Re: powersave governor runs programs faster and uses more power than performance governor

[Melanie Kambadur]

If I disable intel_pstate on my kernel command line, I end up with
access only to the speedstep and p4-clockmod drivers, and no
acpi-cpufreq driver.  Is there a way to download the acpi-cpufreq
driver? I've tried a couple of things without success.

BTW, I haven't gotten the chance to modify the BIOS yet (waiting on a
sysadmin).
-Melanie

On Fri, Oct 25, 2013 at 11:38 AM, Dirk Brandewie
<dirk.brandewie@gmail.com> wrote:
> On 10/25/2013 08:13 AM, Melanie Kambadur wrote:
>>
>> I appreciate you all taking the time to walk me through this.  Let me
>> see if I understand the new comments. Intel p-states is a HW-based
>> power manager,
>
>
> Not hardware based but specific to Intel CPUs SandyBridge+
>
>
>> and strictly an alternative to (i.e., it cannot be
>> combined with) OS governors and drivers.
>
>
> Correct
>
>
>> If I want to use ondemand with my Dell server I need to:
>
>
> Add intel_pstate=disable on your kernel commond line, this will take
> intel_pstate out of the picture.  For the rest of the config on the
> dell system I am no help sorry.
>
>
>>
>> 1) Modify the BIOS to give the OS exclusive power management control
>> because otherwise an OS driver won't be able to work properly. (I
>> think I know how to do this now after some more reading, e.g. here
>>
>> http://en.community.dell.com/techcenter/power-cooling/w/wiki/best-practices-in-power-management.aspx
>> if anyone is curious.)
>> 2) Set the O/S cpufreq driver to acpi_cpufreq, and
>> 3) Set the O/S cpufreq governor to ondemand.
>>
>> Is that correct?
>>
>> Also, which driver should I try to use if I want to test the
>> performance & powersave governors again (or if I replicate the
>> behavior of the performance governor by manually modifying the
>> min_perf_pct value as Dirk suggested)? Will it still be acpi_cpufreq?
>
>
> If intel_pstate is being used acpi_cpufreq will not be loaded.  Setting
> performance with intel_pstate should work I will look to see where the
> bug is.
>
>
>>
>> Finally, the behavior of the C-states is totally independent of
>> P-states and any kind of OS-based frequency tuning policy, correct?
>
>
> Correct
>
>
>> However, David recommends that leaving C1E on rarely hurts performance
>> while significantly improving power.
>>
>> -Melanie
>>
>> On Fri, Oct 25, 2013 at 10:40 AM, Dirk Brandewie
>> <dirk.brandewie@gmail.com> wrote:
>>>
>>> On 10/24/2013 12:42 PM, Melanie Kambadur wrote:
>>>>
>>>>
>>>>
>>>>   From /sys/devices/system/cpu/cpuN/cpufreq/scaling_driver I get that
>>>> the current p-state driver is called "intel_pstate". David, you
>>>> mention that the firmware governors are not very efficient, do you
>>>> suggest replacing the intel_pstate driver with a different driver?
>>>
>>>
>>>
>>> I will need to look and see why changing to performance isn't working
>>> correctly.
>>>
>>> To get the behavior of the performance governor you can use
>>>
>>>     echo 100 > /sys/devices/system/cpu/intel_pstate/min_perf_pct
>>>
>>> This will force intel_pstate to select the highest P state and
>>> leave it there.
>>>
>>> Turbostat is useful for collecting frequency (P state) and idle (C state)
>>> information.
>>>
>>> --Dirk
>>>
>>>
>

Re: powersave governor runs programs faster and uses more power than performance governor

[David C Niemi]

I'm pretty sure that Dell's Active Power Controller would supersede what the intel-pstate driver/governor is trying to do as well.  I doubt you will have enough control over the box to do meaningful tests until you can get the BIOS power setting changed to "OS Control".

speedstep and p4-clockmod are both ancient.  But it is all moot if the BIOS won't let the OS manage the CPU.

It is possible your distribution is a bit too stripped down.  You might want to build and install your own kernel so you can make sure all the sensible options are in it.  But it is also possible acpi-cpufreq is built in so you don't see it as a module, and it is not possible to activate it due to the Active Power Controller getting in the way.

Meanwhile, can you rmmod intel-pstate?

DCN

On 10/25/13 12:35, Melanie Kambadur wrote:
> If I disable intel_pstate on my kernel command line, I end up with
> access only to the speedstep and p4-clockmod drivers, and no
> acpi-cpufreq driver.  Is there a way to download the acpi-cpufreq
> driver? I've tried a couple of things without success.
>
> BTW, I haven't gotten the chance to modify the BIOS yet (waiting on a
> sysadmin).
> -Melanie

Re: powersave governor runs programs faster and uses more power than performance governor

[Melanie Kambadur]

After getting the chance to modify the BIOS, it finally looks like I
have the options that I want in terms of drivers/governors.

FYI, rmmod intel-pstate did not work (before or after modifying the
bios), and as I mentioned earlier, before updating the bios, setting
intel_pstate=disable erased most of my driver/governor options.  What
did work was to update the BIOS to OS control as suggested, and THEN
setting intel_pstate=disable gave me the acpi-cpufreq driver, along
with the following governors:

On Fri, Oct 25, 2013 at 2:27 PM, David C Niemi <dniemi@verisign.com> wrote:
>
> I'm pretty sure that Dell's Active Power Controller would supersede what the intel-pstate driver/governor is trying to do as well.  I doubt you will have enough control over the box to do meaningful tests until you can get the BIOS power setting changed to "OS Control".
>
> speedstep and p4-clockmod are both ancient.  But it is all moot if the BIOS won't let the OS manage the CPU.
>
> It is possible your distribution is a bit too stripped down.  You might want to build and install your own kernel so you can make sure all the sensible options are in it.  But it is also possible acpi-cpufreq is built in so you don't see it as a module, and it is not possible to activate it due to the Active Power Controller getting in the way.
>
> Meanwhile, can you rmmod intel-pstate?
>
> DCN
>
> On 10/25/13 12:35, Melanie Kambadur wrote:
>> If I disable intel_pstate on my kernel command line, I end up with
>> access only to the speedstep and p4-clockmod drivers, and no
>> acpi-cpufreq driver.  Is there a way to download the acpi-cpufreq
>> driver? I've tried a couple of things without success.
>>
>> BTW, I haven't gotten the chance to modify the BIOS yet (waiting on a
>> sysadmin).
>> -Melanie

Re: powersave governor runs programs faster and uses more power than performance governor

[Melanie Kambadur]

Apologies, I accidentally sent my last mail before finishing...

The available governors I have now are:
conservative ondemand userspace powersave performance.

Thankfully, I did not have to modify my kernel at all to get these.
Once I get the chance to run some tests, I will keep you posted as to
whether the governors behave as documented with this new
configuration.

Thanks again for the pointers and suggestions that got me this far!

Melanie

On Tue, Oct 29, 2013 at 12:25 PM, Melanie Kambadur
<melanie@cs.columbia.edu> wrote:
> After getting the chance to modify the BIOS, it finally looks like I
> have the options that I want in terms of drivers/governors.
>
> FYI, rmmod intel-pstate did not work (before or after modifying the
> bios), and as I mentioned earlier, before updating the bios, setting
> intel_pstate=disable erased most of my driver/governor options.  What
> did work was to update the BIOS to OS control as suggested, and THEN
> setting intel_pstate=disable gave me the acpi-cpufreq driver, along
> with the following governors:
>
> On Fri, Oct 25, 2013 at 2:27 PM, David C Niemi <dniemi@verisign.com> wrote:
>>
>> I'm pretty sure that Dell's Active Power Controller would supersede what the intel-pstate driver/governor is trying to do as well.  I doubt you will have enough control over the box to do meaningful tests until you can get the BIOS power setting changed to "OS Control".
>>
>> speedstep and p4-clockmod are both ancient.  But it is all moot if the BIOS won't let the OS manage the CPU.
>>
>> It is possible your distribution is a bit too stripped down.  You might want to build and install your own kernel so you can make sure all the sensible options are in it.  But it is also possible acpi-cpufreq is built in so you don't see it as a module, and it is not possible to activate it due to the Active Power Controller getting in the way.
>>
>> Meanwhile, can you rmmod intel-pstate?
>>
>> DCN
>>
>> On 10/25/13 12:35, Melanie Kambadur wrote:
>>> If I disable intel_pstate on my kernel command line, I end up with
>>> access only to the speedstep and p4-clockmod drivers, and no
>>> acpi-cpufreq driver.  Is there a way to download the acpi-cpufreq
>>> driver? I've tried a couple of things without success.
>>>
>>> BTW, I haven't gotten the chance to modify the BIOS yet (waiting on a
>>> sysadmin).
>>> -Melanie

Re: powersave governor runs programs faster and uses more power than performance governor

[Dirk Brandewie]

On 10/29/2013 09:27 AM, Melanie Kambadur wrote:
> Apologies, I accidentally sent my last mail before finishing...
>
> The available governors I have now are:
> conservative ondemand userspace powersave performance.
>

Only ondemand will have any active control of the P state requested.

performance == highest P state (reported by ACPI) is requested
powersave   == lowest P state (reported by ACPI) is requested
userspace   == Allows the user select a single P state.

With intel_pstate enabled
performance == highest P state (enumerated form CPU) is requested.
powersave   == Active control of the P state similar to ondemand.

To get userspace governor behavior set
/sys/devices/system/cpu/intel_pstate/{min, max}_perf_pct to the same
values.

/sys/devices/system/cpu/intel_pstate/no_turbo allows you to configure
whether P states in the turbo range for your CPU will be selected.


> Thankfully, I did not have to modify my kernel at all to get these.
> Once I get the chance to run some tests, I will keep you posted as to
> whether the governors behave as documented with this new
> configuration.
>
> Thanks again for the pointers and suggestions that got me this far!
>
> Melanie
>
> On Tue, Oct 29, 2013 at 12:25 PM, Melanie Kambadur
> <melanie@cs.columbia.edu> wrote:
>> After getting the chance to modify the BIOS, it finally looks like I
>> have the options that I want in terms of drivers/governors.
>>
>> FYI, rmmod intel-pstate did not work (before or after modifying the
>> bios),

intel_pstate is built-in only and not a module :-)

>> and as I mentioned earlier, before updating the bios, setting
>> intel_pstate=disable erased most of my driver/governor options.

The cpufreq core creates the sysfs files as governors are registered
with the core.  With intel_pstate=disable the driver will not register
with the core so if no other drivers/governors register with the core
the sysfs files will be missing.

>> What
>> did work was to update the BIOS to OS control as suggested, and THEN
>> setting intel_pstate=disable gave me the acpi-cpufreq driver, along
>> with the following governors:
>>
>> On Fri, Oct 25, 2013 at 2:27 PM, David C Niemi <dniemi@verisign.com> wrote:
>>>
>>> I'm pretty sure that Dell's Active Power Controller would supersede what the intel-pstate driver/governor is trying to do as well.  I doubt you will have enough control over the box to do meaningful tests until you can get the BIOS power setting changed to "OS Control".
>>>
>>> speedstep and p4-clockmod are both ancient.  But it is all moot if the BIOS won't let the OS manage the CPU.
>>>
>>> It is possible your distribution is a bit too stripped down.  You might want to build and install your own kernel so you can make sure all the sensible options are in it.  But it is also possible acpi-cpufreq is built in so you don't see it as a module, and it is not possible to activate it due to the Active Power Controller getting in the way.
>>>
>>> Meanwhile, can you rmmod intel-pstate?
>>>
>>> DCN
>>>
>>> On 10/25/13 12:35, Melanie Kambadur wrote:
>>>> If I disable intel_pstate on my kernel command line, I end up with
>>>> access only to the speedstep and p4-clockmod drivers, and no
>>>> acpi-cpufreq driver.  Is there a way to download the acpi-cpufreq
>>>> driver? I've tried a couple of things without success.
>>>>
>>>> BTW, I haven't gotten the chance to modify the BIOS yet (waiting on a
>>>> sysadmin).
>>>> -Melanie

Re: powersave governor runs programs faster and uses more power than performance governor

[David C Niemi]

Melanie,

By "Xeon 2430", I take it you mean dual E5-2430 CPUs in a 2-socket server or workstation?  What brand and model of machine is this in?

There is a good chance your machine's BIOS is overriding the intentions of your p-state governor and doing whatever it feels like.  Both Dell and HP default to firmware-based governors vaguely like "ondemand" but less effective.  If the firmware is calling the shots, it could explain what you are seeing.  It would be worth seeing what your power management settings are in the BIOS, looking at what p-state driver you are using (e.g. acpi-cpufreq), and see what frequency your CPUs are really running at (the utility i7z may be able to help you if, like on some Dell BIOS settings, cpufreq cannot see the current frequency reliably).

Your kernel is also rather odd if it has only the "powersave" and "performance" governors, which are not even really governors but rather static clock speed settings.  Ondemand is usually a good choice for servers, and can be configured to match the performance of the "performance" governor while consuming less power at idle.  Note that "performance" is sometimes not a very good way to get max performance, as it forces idle cores to run fast which eats up your power envelope unproductively, giving your busy cores less ability to go into boost.  Preventing the use of C1E is likewise usually a bad idea for the same reason, and typically makes power consumption go through the roof.

DCN

On 10/23/13 19:17, Melanie Kambadur wrote:
> Hi all,
>
> I am trying to understand some strange results I've observed using the
> 3.9.11 kernel version of the powersave cpufreq governor on a Intel
> Xeon machine, and hoping you all can help.
>
> More specifically, I am using a 24 core Intel Xeon 2430 machine
> running Ubuntu 12.04 and v 3.9.11 of the kernel. As I understand it,
> this configuration affords me two cpufreq scaling governors, powersave
> and performance. At least these are the options I get by running:
> /sys/devices/system/cpu/cpu0/cpufreq/scaling_available_governors
>
> Here's the issue. When I turn on the powersave governor, I have
> measured the power consumed to INCREASE relative to the performance
> governor. More surprisingly, I consistently see BETTER performance
> (i.e., shorter runtimes) with the powersave governor than with the
> performance governor for many applications out of a few sets of
> commonly used benchmark applications.  I repeated my experiments
> several times, even trying different cpuidle governors, and observed
> the same trends. To toggle between governors, I have been using the
> cpufreq-set -g command and confirming the updates by checking
> /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor.
>
> From what I've read (e.g., here:
> www.kernel.org/doc/Documentation/cpu-freq/governors.txt), the
> powersave governor's policy is to statically set the cpu frequency at
> the system minimum, whereas the performance governor aims to set the
> cpu frequency higher (or, depending on the documentation, at maximum).
> However, the fact that applications sped up for the powersave vs. the
> performance governor with these policies seemed incorrect to me, so I
> investigated further...
>
> I profiled the current system frequency by periodically polling
> /sys/devices/system/cpu/cpu${N}/cpufreq/cpuinfo_cur_freq
> for all the available "N" values (0-23) on my machine.  I found that
> not only does the powersave governor not seem to be running at the
> system minimum, it actually seems to be running at a frequency higher
> than the performance governor. This would explain why I am seeing
> better performance and higher power for the powersave governor, but it
> certainly doesn't fit the policy descriptions in the documentation.
> Here are the results of my mini experiment:
>
> -Minimum system frequency=1.20 * 10^6
> -Maximum system frequency=2.70 * 10^6
>
> Avg. cpu frequency across all CPUs for 30 seconds of...
> -Powersave governor with no applications running = 1.77 * 10^6
> -Powersave governor with a benchmark app running = 2.23 * 10^6
> -Performance governor with no apps running = 1.23 * 10^6
> -Performance governor with a benchmark app running = 1.34 * 10^6
>
> Am I misunderstanding the policies of the powersave vs. performance
> governor, or is this possibly a system bug?
>
> Thanks for your help.
>
> Sincerely,
> Melanie Kambadur