Re: Delayed acpi frequency governor call

Re: Delayed acpi frequency governor call

[Viresh Kumar]

+ PM list.

On 20-01-17, 10:48, Philipp Miedl wrote:
> Hi!
> 
> I'm using the kernel version 4.4.0-59-generic and 4.4.0 with acpi cpufreq
> drivers (cpufrequtils 008) on a Lenovo T440p laptop. Now I've bumped into
> some weird behaviour:
> it can happen that the call to the governor is delayed for so long, that in
> the governor the if-statement in cpufreq_governor.c line 138
> is true, which means the governor thinks the cpu was idle. This can lead to
> the case that the frequency is still scaled up when it should in
> fact be scaled down.
> 
> Now I've written a small programm which ensures that the cpu is never idle
> for longer than 850us, but does not generate a cpu utilization higher
> than 5% - still the behaviour is the same. I've looked a bit closer at this
> and I figured out that if I increase the utilization of the observed CPU,
> at utilzations higher than 80% the maximum delay between two governor calls
> increases and can rise almost to 10x sampling rate
> (/sys/drivers/system/cpu/cpufreq/sampling_rate).
> Weirdly enough, after a long time (>1s) the average call time of the
> governor is always around the specified sampling rate.
> 
> I could not fully understand how the governor is call and this work queue
> works and therefore I also have a hard time making sense of this behaviour.
> Can anyone please help me out?

Have a look at below commit:

commit 00bfe05889e9 ("cpufreq: conservative: Decrease frequency faster
for deferred updates")

It may be related to what you are observing.

-- 
viresh

Re: Delayed acpi frequency governor call

[Philipp Miedl]

On 20.01.2017 11:07, Viresh Kumar wrote:
> + PM list.
>
> On 20-01-17, 10:48, Philipp Miedl wrote:
>> Hi!
>>
>> I'm using the kernel version 4.4.0-59-generic and 4.4.0 with acpi cpufreq
>> drivers (cpufrequtils 008) on a Lenovo T440p laptop. Now I've bumped into
>> some weird behaviour:
>> it can happen that the call to the governor is delayed for so long, that in
>> the governor the if-statement in cpufreq_governor.c line 138
>> is true, which means the governor thinks the cpu was idle. This can lead to
>> the case that the frequency is still scaled up when it should in
>> fact be scaled down.
>>
>> Now I've written a small programm which ensures that the cpu is never idle
>> for longer than 850us, but does not generate a cpu utilization higher
>> than 5% - still the behaviour is the same. I've looked a bit closer at this
>> and I figured out that if I increase the utilization of the observed CPU,
>> at utilzations higher than 80% the maximum delay between two governor calls
>> increases and can rise almost to 10x sampling rate
>> (/sys/drivers/system/cpu/cpufreq/sampling_rate).
>> Weirdly enough, after a long time (>1s) the average call time of the
>> governor is always around the specified sampling rate.
>>
>> I could not fully understand how the governor is call and this work queue
>> works and therefore I also have a hard time making sense of this behaviour.
>> Can anyone please help me out?
> Have a look at below commit:
>
> commit 00bfe05889e9 ("cpufreq: conservative: Decrease frequency faster
> for deferred updates")
>
> It may be related to what you are observing.
>

This is not my issue. The code I'm referring to can be found in 
cpufreq_governor.c:
<-->
if (unlikely(wall_time > (2 * sampling_rate) &&
        j_cdbs->prev_load)) {
   load = j_cdbs->prev_load;

   /*
    * Perform a destructive copy, to ensure that we copy
    * the previous load only once, upon the first wake-up
    * from idle.
    */
   j_cdbs->prev_load = 0;
} else {
   load = 100 * (wall_time - idle_time) / wall_time;
   j_cdbs->prev_load = load;
}
<-->

How can it happen that "unlikely(wall_time > (2 * sampling_rate)" is 
true although the CPU utilization was 5%?

Thanks

-- 
--------------------------------------------------------------------
|  Philipp Miedl                                                   |
|  PhD Student @ Computer Engineering and Networks Laboratory      |
|  ETH Zurich, ETZG76, Gloriastrasse 35, 8092 Zurich, Switzerland  |
|  mail: philipp.miedl@ee.tik.ethz.ch      phone: +41 44 632 70 61 |
--------------------------------------------------------------------

Re: Delayed acpi frequency governor call

[Viresh Kumar]

On 20-01-17, 16:27, Philipp Miedl wrote:
> This is not my issue. The code I'm referring to can be found in
> cpufreq_governor.c:
> <-->
> if (unlikely(wall_time > (2 * sampling_rate) &&
>        j_cdbs->prev_load)) {

This only confirms that the CPU was idle for sometime.

>   load = j_cdbs->prev_load;
> 
>   /*
>    * Perform a destructive copy, to ensure that we copy
>    * the previous load only once, upon the first wake-up
>    * from idle.
>    */
>   j_cdbs->prev_load = 0;
> } else {
>   load = 100 * (wall_time - idle_time) / wall_time;
>   j_cdbs->prev_load = load;
> }
> <-->
> 
> How can it happen that "unlikely(wall_time > (2 * sampling_rate)" is true
> although the CPU utilization was 5%?

I am not sure how your test works, etc. Maybe try to generate traces for your
CPUs to see if they are going into idle or not. Maybe you are having a hard time
because of SMP platform ?

-- 
viresh

Re: Delayed acpi frequency governor call

[Philipp Miedl]

On 23.01.2017 11:11, Viresh Kumar wrote:
> On 20-01-17, 16:27, Philipp Miedl wrote:
>> This is not my issue. The code I'm referring to can be found in
>> cpufreq_governor.c:
>> <-->
>> if (unlikely(wall_time > (2 * sampling_rate) &&
>>         j_cdbs->prev_load)) {
> This only confirms that the CPU was idle for sometime.
I would say it only checks whether the elapsed time since the last 
governor call. My question is whether the call to the governor can be 
delayed for such a long time without the CPU going into idle? If it can 
be guaranteed that the governor call cannot be delayed without the CPU 
going to idle than it is true - otherwise not.
>>    load = j_cdbs->prev_load;
>>
>>    /*
>>     * Perform a destructive copy, to ensure that we copy
>>     * the previous load only once, upon the first wake-up
>>     * from idle.
>>     */
>>    j_cdbs->prev_load = 0;
>> } else {
>>    load = 100 * (wall_time - idle_time) / wall_time;
>>    j_cdbs->prev_load = load;
>> }
>> <-->
>>
>> How can it happen that "unlikely(wall_time > (2 * sampling_rate)" is true
>> although the CPU utilization was 5%?
> I am not sure how your test works, etc. Maybe try to generate traces for your
> CPUs to see if they are going into idle or not. Maybe you are having a hard time
> because of SMP platform ?
Possible that I'm having problems with the platforms. Its a quadcore 
with 2 hyperthreads. I have two applications running, one should keep 
the CPU its pinned to busy at around 5% utilization by doing 
calculations. The second one is pinned to another core (its also another 
physical core) and records all the timing values etc. The rest of the 
CPUs are not utilzed by me and besides my stuff there is just a bare 
Linux running without GUI.
I checked the values I get from get_cpu_idle_time(), which gets its 
values from get_cpu_idle_time_us() (tick-sched.c) or if that is not 
possible assembles it from the cpustat, and the according to these the 
CPU is never idle longer than sampling_rate - so the if statement from 
above should never be true.

Do I interpret the values wrong or can it be that the call to the 
governor gets delayed by something else?

-- 
--------------------------------------------------------------------
|  Philipp Miedl                                                   |
|  PhD Student @ Computer Engineering and Networks Laboratory      |
|  ETH Zurich, ETZG76, Gloriastrasse 35, 8092 Zurich, Switzerland  |
|  mail: philipp.miedl@ee.tik.ethz.ch      phone: +41 44 632 70 61 |
--------------------------------------------------------------------

Re: Delayed acpi frequency governor call

[Viresh Kumar]

On 23-01-17, 15:24, Philipp Miedl wrote:
> I would say it only checks whether the elapsed time since the last governor
> call. My question is whether the call to the governor can be delayed for
> such a long time without the CPU going into idle? If it can be guaranteed
> that the governor call cannot be delayed without the CPU going to idle than
> it is true - otherwise not.

You are using a 4.4 kernel, which used the older delayed work stuff at sampling
rate. This has all changed a lot and we hooks paired with the scheduler now to
do this instead. Things have mostly changed since 4.7. Would it be possible for
you to use a later kernel and see if you still see something wrong?

Now coming back to 4.4, the timers (in the delayed works) can't get delayed but
the work can as it is just another process. But your setup has no almost load
and that seems unlikely as well. So, it looks to be a problem at some other
place. Tracers are your friend, only those can confirm something here.

-- 
viresh

Re: Delayed acpi frequency governor call

[Philipp Miedl]

> You are using a 4.4 kernel, which used the older delayed work stuff at sampling
> rate. This has all changed a lot and we hooks paired with the scheduler now to
> do this instead. Things have mostly changed since 4.7. Would it be possible for
> you to use a later kernel and see if you still see something wrong?
I switched to the latest stable kernel available, 4.9.5. Judging from 
the first test I ran, using this kernel I cannot
reproduce this behavior anymore. Looks like it has been fixed by the 
changes introduced.
> Now coming back to 4.4, the timers (in the delayed works) can't get delayed but
> the work can as it is just another process.
Can you please describe how things changed from version 4.4 to 4.9 - I'm 
not sure if I understand correctly.
Before 4.7 the governor was just running as a kernel process and 
triggered by the (free running) timers and
now its hooked with the scheduler. So the scheduler ensures that the 
governor work is not delayed?!
> But your setup has no almost load
> and that seems unlikely as well. So, it looks to be a problem at some other
> place. Tracers are your friend, only those can confirm something here.
Thanks for the tip with the tracers. If time allows I will have a look 
into that - I would like to know whats going
on, just out of curiosity.

-- 
--------------------------------------------------------------------
|  Philipp Miedl                                                   |
|  PhD Student @ Computer Engineering and Networks Laboratory      |
|  ETH Zurich, ETZG76, Gloriastrasse 35, 8092 Zurich, Switzerland  |
|  mail: philipp.miedl@ee.tik.ethz.ch      phone: +41 44 632 70 61 |
--------------------------------------------------------------------

Re: Delayed acpi frequency governor call

[Viresh Kumar]

On 24-01-17, 15:49, Philipp Miedl wrote:
> Can you please describe how things changed from version 4.4 to 4.9 - I'm not
> sure if I understand correctly.
> Before 4.7 the governor was just running as a kernel process and triggered
> by the (free running) timers and
> now its hooked with the scheduler. So the scheduler ensures that the
> governor work is not delayed?!

Now the cpufreq callbacks gets called from within the scheduler every few
milliseconds. And once we have crossed the sampling-rate time, we reevaluate the
frequency. Though ondemand/conservative governors still used workqueue until
very recently for most of the platforms. Recently it is replaced by a RT thread.

-- 
viresh

Re: Delayed acpi frequency governor call

[Philipp Miedl]

On 25.01.2017 04:23, Viresh Kumar wrote:
> On 24-01-17, 15:49, Philipp Miedl wrote:
>> Can you please describe how things changed from version 4.4 to 4.9 - I'm not
>> sure if I understand correctly.
>> Before 4.7 the governor was just running as a kernel process and triggered
>> by the (free running) timers and
>> now its hooked with the scheduler. So the scheduler ensures that the
>> governor work is not delayed?!
> Now the cpufreq callbacks gets called from within the scheduler every few
> milliseconds. And once we have crossed the sampling-rate time, we reevaluate the
> frequency. Though ondemand/conservative governors still used workqueue until
> very recently for most of the platforms. Recently it is replaced by a RT thread.
>

Ok got it now. Thanks a lot for your help!

-- 
--------------------------------------------------------------------
|  Philipp Miedl                                                   |
|  PhD Student @ Computer Engineering and Networks Laboratory      |
|  ETH Zurich, ETZG76, Gloriastrasse 35, 8092 Zurich, Switzerland  |
|  mail: philipp.miedl@ee.tik.ethz.ch      phone: +41 44 632 70 61 |
--------------------------------------------------------------------