How many frequencies would cpufreq optimally like to manage?

How many frequencies would cpufreq optimally like to manage?

[Mason]

Hello everyone,

I'm running kernel 3.14 on an ARM Cortex-A9 based SoC.
The baseline frequency for the ARM CPU is 999 MHz.

The SoC provides a way to dynamically change the CPU frequency,
dividing it by N/16 (N=32..4095) [Actually, I think there are
also ways to divide by 1.x, but I need to read the docs again.]

I'm writing the platform-specific cpufreq driver.
I could expose hundreds of frequencies to cpufreq, but I don't
think that would be very productive. Correct?
(Note: I can't offer ANY frequency.)

My question is: how many frequencies should I expose for "optimal"
behavior of cpufreq?

I'm thinking I would only expose div={2,3,5} meaning the available
scaled frequencies would be {500,333,200} MHz. Are these enough?
Should there be more? Should I go lower than 200 MHz?

I'm also wondering about the cpuinfo_transition_latency variable.
It seems to be a constant. On my SoC, the latency depends on the
"width" of the frequency jump. Specifically, the CPU will change
frequency at 15 MHz/us. So if we're jumping from 1000 to 500 MHz,
it will take 33 us. From 500 to 333 will take 11 us.

Should I document the max latency? (i.e. for the "widest" jump)
Can the governor request to jump from cpuinfo_min_freq to
cpuinfo_max_freq directly, skipping intermediate frequencies?
If so, should I just program

cpuinfo_transition_latency =
  (((cpuinfo_max_freq - cpuinfo_min_freq) / 1000) / 15) * 1000

= (cpuinfo_max_freq - cpuinfo_min_freq) / 15 (in ns)
= 53280 ns in my example

Since governors.txt suggests a sampling rate of 10 ms, I suppose
a transition_latency of 50 us is acceptable?

Regards.

Re: How many frequencies would cpufreq optimally like to manage?

[Viresh Kumar]

On Thu, Nov 20, 2014 at 4:54 AM, Mason <mpeg.blue@free.fr> wrote:
> Hello everyone,

Hi,

The right list is linux-pm and try to cc maintainers from MAINTAINERS
file as they can respond quickly then.

> I'm running kernel 3.14 on an ARM Cortex-A9 based SoC.
> The baseline frequency for the ARM CPU is 999 MHz.
>
> The SoC provides a way to dynamically change the CPU frequency,
> dividing it by N/16 (N=32..4095) [Actually, I think there are
> also ways to divide by 1.x, but I need to read the docs again.]
>
> I'm writing the platform-specific cpufreq driver.
> I could expose hundreds of frequencies to cpufreq, but I don't
> think that would be very productive. Correct?
> (Note: I can't offer ANY frequency.)
>
> My question is: how many frequencies should I expose for "optimal"
> behavior of cpufreq?

You can specify as many number of frequencies as you want. The framework
doesn't have any upper cap on that. But there is no point specifying
500 MHz and 510 MHz, you wouldn't save much power with 500 :)

So, just gap them smartly. Normally people gap them to 100 MHz. But
if there is a voltage change required at some specific freq, suppose 550 MHz,
then its better you specific that as well..

> I'm thinking I would only expose div={2,3,5} meaning the available
> scaled frequencies would be {500,333,200} MHz. Are these enough?

I thought you can goto 1 GHz.. Why not the upper ones then?

> Should there be more? Should I go lower than 200 MHz?

If that really saves power.

> I'm also wondering about the cpuinfo_transition_latency variable.
> It seems to be a constant. On my SoC, the latency depends on the
> "width" of the frequency jump. Specifically, the CPU will change
> frequency at 15 MHz/us. So if we're jumping from 1000 to 500 MHz,
> it will take 33 us. From 500 to 333 will take 11 us.
>
> Should I document the max latency? (i.e. for the "widest" jump)

Yes.

> Can the governor request to jump from cpuinfo_min_freq to
> cpuinfo_max_freq directly, skipping intermediate frequencies?

Yes, its a normal case for ondemand governor.

> If so, should I just program
>
> cpuinfo_transition_latency =
>   (((cpuinfo_max_freq - cpuinfo_min_freq) / 1000) / 15) * 1000
>
> = (cpuinfo_max_freq - cpuinfo_min_freq) / 15 (in ns)
> = 53280 ns in my example
>
> Since governors.txt suggests a sampling rate of 10 ms, I suppose
> a transition_latency of 50 us is acceptable?

Yes.

Re: How many frequencies would cpufreq optimally like to manage?

[Mason]

On 20/11/2014 10:13, Viresh Kumar wrote:

> On Thu, Nov 20, 2014 at 4:54 AM, Mason <mpeg.blue@free.fr> wrote:
>
>> Hello everyone,
>
> The right list is linux-pm and try to cc maintainers from MAINTAINERS
> file as they can respond quickly then.

Sorry about that. I wasn't even aware that an index like MAINTAINERS
existed AND was kept up-to-date.

>> I'm running kernel 3.14 on an ARM Cortex-A9 based SoC.
>> The baseline frequency for the ARM CPU is 999 MHz.
>>
>> The SoC provides a way to dynamically change the CPU frequency,
>> dividing it by N/16 (N=32..4095) [Actually, I think there are
>> also ways to divide by 1.x, but I need to read the docs again.]
>>
>> I'm writing the platform-specific cpufreq driver.
>> I could expose hundreds of frequencies to cpufreq, but I don't
>> think that would be very productive. Correct?
>> (Note: I can't offer ANY frequency.)
>>
>> My question is: how many frequencies should I expose for "optimal"
>> behavior of cpufreq?
>
> You can specify as many number of frequencies as you want. The framework
> doesn't have any upper cap on that. But there is no point specifying
> 500 MHz and 510 MHz, you wouldn't save much power with 500 :)

Suppose I expose 100 frequencies. Aside from wasting memory, isn't
there also a risk that cpufreq will take time, ramping up/down
through similar frequencies? (Or does it just say "jump to THAT
frequency" skipping useless intermediate frequencies?)

> So, just gap them smartly. Normally people gap them to 100 MHz. But
> if there is a voltage change required at some specific freq, suppose 550 MHz,
> then its better you specific that as well..

Point taken.

>> I'm thinking I would only expose div={2,3,5} meaning the available
>> scaled frequencies would be {500,333,200} MHz. Are these enough?
>
> I thought you can goto 1 GHz.. Why not the upper ones then?

The actual formula for dividers is
div(I,F) = if I > 1 then I+F/16 else I+F/(32-F)
I=1..255 F=0..15

So, I can use (for example) div={1, 1.33, 2, 4, 8}
to offer freq={999,750,500,250,125}

>> Should there be more? Should I go lower than 200 MHz?
>
> If that really saves power.

There is a trade-off, tough.

My concern is this: suppose the CPU cores are set to e.g. 54 MHz.
Suddenly, the driver for an important subsystem needs to speak with
the corresponding device, with hard deadlines in the comms protocol.
Will the CPU ramp up fast enough (assuming ondemand governor).

And the end of this message is as good a place as any to thank you
for having answered my many questions.

Regards.

Re: How many frequencies would cpufreq optimally like to manage?

[Viresh Kumar]

On 20 November 2014 19:50, Mason <mpeg.blue@free.fr> wrote:
> On 20/11/2014 10:13, Viresh Kumar wrote:
> Suppose I expose 100 frequencies. Aside from wasting memory, isn't
> there also a risk that cpufreq will take time, ramping up/down
> through similar frequencies? (Or does it just say "jump to THAT
> frequency" skipping useless intermediate frequencies?)

Some time might be wasted for similar frequencies, moreover for every
change we have to traverse 100 frequencies to see which one fits best.

> There is a trade-off, tough.
>
> My concern is this: suppose the CPU cores are set to e.g. 54 MHz.
> Suddenly, the driver for an important subsystem needs to speak with
> the corresponding device, with hard deadlines in the comms protocol.
> Will the CPU ramp up fast enough (assuming ondemand governor).

All current governors depend on background timers for their functionality.
These timers run at some sampling rate (in ms) and that time we change
CPU's frequency depending on existing load on system.. So, that might
not be fast enough.

> And the end of this message is as good a place as any to thank you
> for having answered my many questions.

Your welcome :)

Re: How many frequencies would cpufreq optimally like to manage?

[Mason]

On 21/11/2014 04:36, Viresh Kumar wrote:

> All current governors depend on background timers for their functionality.
> These timers run at some sampling rate (in ms) and that time we change
> CPU's frequency depending on existing load on system.. So, that might
> not be fast enough.

As far as I can tell, on my SoC, the timer runs at 27 MHz.
But I have no idea how often it fires an interrupt.

I've been studying other cpufreq drivers, especially the OMAP driver.
I noticed that there is a lot of generic infrastructure that our driver
wasn't using, such as...

   cpufreq_generic_init
   cpufreq_generic_exit
   cpufreq_generic_frequency_table_verify
   cpufreq_generic_attr

I've changed our driver to use those.

I'm still confused about cpufreq_generic_get.
This is not a typical get/put type function, right?

What is it supposed to get?
Apparently, the actual frequency of the 'cpu-th' CPU?
I see that it calls clk_get_rate(policy->clk)

This 'struct clk' is an elusive beast.
Where is it defined? I only run into forward declarations.

On my platform, clk_get_accuracy returns -524 unconditionally,
and clk_prepare just returns 0.

of_clk_get and of_clk_get_by_name both return ERR_PTR(-2)

I suppose all this means I can't use this infrastructure "as-is",
correct? Where can I read more about it?

How is the kernel supposed to know what frequency each core is running at?
I imagine that's what the .get method override is for?

One more question (for now). Is the .get method supposed to return numbers
that match in freq_table, or can they be slightly different? For example,
I've defined my freq_table like this:

static struct cpufreq_frequency_table freq_table[] = {
	{ .driver_data = REG_VAL(1,0), .frequency = 999000 },
	{ .driver_data = REG_VAL(1,8), .frequency = 750000 },
	{ .driver_data = REG_VAL(2,0), .frequency = 500000 },
	{ .driver_data = REG_VAL(4,0), .frequency = 250000 },
	{ .driver_data = REG_VAL(8,0), .frequency = 125000 },
	{ .frequency = CPUFREQ_TABLE_END },
};

but the 3rd frequency is actually 999/2 MHz, not 500 MHz, etc.
Will that be a problem?

Regards.

Re: How many frequencies would cpufreq optimally like to manage?

[Viresh Kumar]

On 25 November 2014 at 18:32, Mason <mpeg.blue@free.fr> wrote:
> As far as I can tell, on my SoC, the timer runs at 27 MHz.
> But I have no idea how often it fires an interrupt.

27 MHz is the freq it runs at, but we get interrupt based on the value
programmed in the timer counter.

Look for CONFIG_HZ in your .config, that will tell you number of ticks
you are getting per second.

But cpufreq works in another level and so it doesn't depend on both
these.

Look for: /sys/devices/system/cpu/cpufreq/ondemand/sampling_rate

this is the rate at which cpufreq core checks if cpu's freq require revisiting.

> I've been studying other cpufreq drivers, especially the OMAP driver.
> I noticed that there is a lot of generic infrastructure that our driver
> wasn't using, such as...
>
>   cpufreq_generic_init
>   cpufreq_generic_exit
>   cpufreq_generic_frequency_table_verify
>   cpufreq_generic_attr
>
> I've changed our driver to use those.

Gud.

> I'm still confused about cpufreq_generic_get.
> This is not a typical get/put type function, right?

Correct, this should return the current frequency the CPU is running at.

> What is it supposed to get?
> Apparently, the actual frequency of the 'cpu-th' CPU?
> I see that it calls clk_get_rate(policy->clk)

Yes.

> This 'struct clk' is an elusive beast.
> Where is it defined? I only run into forward declarations.

Its architecture specific. What's the architecture you are working on?
You may implement your own routine instead of cpufreq_generic_get().

> On my platform, clk_get_accuracy returns -524 unconditionally,

That might be an error.

> and clk_prepare just returns 0.
>
> of_clk_get and of_clk_get_by_name both return ERR_PTR(-2)
>
> I suppose all this means I can't use this infrastructure "as-is",
> correct? Where can I read more about it?

Yes. Documentation/clk.txt.

> How is the kernel supposed to know what frequency each core is running at?
> I imagine that's what the .get method override is for?

Yeah. You should be able to get some call from your architecture.

> One more question (for now). Is the .get method supposed to return numbers
> that match in freq_table, or can they be slightly different? For example,

They should match.

> I've defined my freq_table like this:
>
> static struct cpufreq_frequency_table freq_table[] = {
>         { .driver_data = REG_VAL(1,0), .frequency = 999000 },
>         { .driver_data = REG_VAL(1,8), .frequency = 750000 },
>         { .driver_data = REG_VAL(2,0), .frequency = 500000 },
>         { .driver_data = REG_VAL(4,0), .frequency = 250000 },
>         { .driver_data = REG_VAL(8,0), .frequency = 125000 },
>         { .frequency = CPUFREQ_TABLE_END },
> };
>
> but the 3rd frequency is actually 999/2 MHz, not 500 MHz, etc.
> Will that be a problem?

Maybe, but it depends on what your ->get() returns.

Re: How many frequencies would cpufreq optimally like to manage?

[Mason]

On 25/11/2014 16:19, Viresh Kumar wrote:

> On 25 November 2014 at 18:32, Mason <mpeg.blue@free.fr> wrote:
>
>> As far as I can tell, on my SoC, the timer runs at 27 MHz.
>> But I have no idea how often it fires an interrupt.
> 
> 27 MHz is the freq it runs at, but we get interrupt based on the value
> programmed in the timer counter.
> 
> Look for CONFIG_HZ in your .config, that will tell you number of ticks
> you are getting per second.
> 
> But cpufreq works in another level and so it doesn't depend on both
> these.

If I remember correctly (please correct me if I am mistaken),
the kernel keeps a list of timers, ordered by deadline, and
dynamically configures the system timer to fire at the next
deadline.

> Look for: /sys/devices/system/cpu/cpufreq/ondemand/sampling_rate
> this is the rate at which cpufreq core checks if cpu's freq require revisiting.

I think setting the sampling rate to 50 ms should make the
system reactive enough to load swings?

>> This 'struct clk' is an elusive beast.
>> Where is it defined? I only run into forward declarations.
> 
> Its architecture specific. What's the architecture you are working on?
> You may implement your own routine instead of cpufreq_generic_get().

The SoC is made by Sigma Designs. It is based on the ARM Cortex-A9.

> Yes. Documentation/clk.txt.

Oh boy! platform, device tree, cpu-freq, cpuidle, clk, ... so much
to learn, so few hours in a day.

> Yeah. You should be able to get some call from your architecture.

OK, I have to fix that function along with the cpufreq update.

Regards.

Re: How many frequencies would cpufreq optimally like to manage?

[Viresh Kumar]

On 26 November 2014 at 03:22, Mason <mpeg.blue@free.fr> wrote:
> If I remember correctly (please correct me if I am mistaken),
> the kernel keeps a list of timers, ordered by deadline, and
> dynamically configures the system timer to fire at the next
> deadline.

Yes.

>> Look for: /sys/devices/system/cpu/cpufreq/ondemand/sampling_rate
>> this is the rate at which cpufreq core checks if cpu's freq require revisiting.
>
> I think setting the sampling rate to 50 ms should make the
> system reactive enough to load swings?

Depends on the kind of load system is under. If load changes quite frequently
then 50 ms can be a very big value. If it doesn't then 50 ms might be good.
But it has to be tuned properly based on how much load the system has
and how much power it consumes.

Some information about that tuning can be found here:
https://www.youtube.com/watch?v=he1UO__K9ao

This is an presentation given by Mathieu Poirier (Linaro) about the work
done on ARM's big LITTLE architecture.

Slides: http://events.linuxfoundation.org/images/stories/slides/elc2013_poirier.pdf

>>> This 'struct clk' is an elusive beast.
>>> Where is it defined? I only run into forward declarations.
>>
>> Its architecture specific. What's the architecture you are working on?
>> You may implement your own routine instead of cpufreq_generic_get().
>
> The SoC is made by Sigma Designs. It is based on the ARM Cortex-A9.

Okay, life should be easy then. Normally there will be few programmable
PLLs on your platform. You need to write a clock driver for them similar
to the ones present here: drivers/clk/

And then you can reuse cpufreq-dt.c cpufreq driver, which is generic enough
for everybody :)

>> Yes. Documentation/clk.txt.
>
> Oh boy! platform, device tree, cpu-freq, cpuidle, clk, ... so much
> to learn, so few hours in a day.

:)