[RFC PATCH 0/1]: timers: Timer Migration

[RFC PATCH 0/1]: timers: Timer Migration

[Arun R Bharadwaj]

Hi,

An idle cpu on which device drivers have initialized timers, has to be frequently woken up to service the timers. So, consolidation of timers onto a fewer number of cpus is important. Migration of
timers from idle cpus onto lesser idle cpus is necessary. Currently, timers are migrated during the cpu offline operation. However cpu-hotplug for the sake of idle system power management is too
heavy. So, this patch implements a lightweight timer migration framework.

Also, in a multi-core, multi-package system, it is always desirable to have all the timers firing on the cpus present in the same package. This would enable us to place the idle package in deep sleep
state. So, migration of timers is required for this.

A per_cpu sysfs hook is created at /sys/devices/system/cpu/cpuX/timer_migration. By echo-ing 1 to cpuX/timer_migration, all regular and hrtimers from cpuX are migrated to cpu0. The fact that all
timers are migrated from cpuX to cpu0 is demonstrated by making a note of /proc/interrupts.

$:echo 1 > /sys/devices/system/cpu/cpu3/timer_migration
$:cat /proc/interrupts > file1; sleep 30 ; cat /proc/interrupts > file2
$:diff file1 file2 | grep LOC
< LOC:       5276       2100       2676       1647       2415       1803       2275       2977   Local timer interrupts
> LOC:       6991       2195       2774       1647       2562       1900       2497       3075   Local timer interrupts

As we can observe, first we are enabling timer migration in cpu3. As a result, the number of local timer interrupts on cpu3 over the next 30 seconds are zero. cpu3  has been totally rid of all regular
and hrtimers.

Please refer to the following paper for details:

http://ols.fedoraproject.org/OLS/Reprints-2008/srinivasan1-reprint.pdf


thanks,
Arun

Re: [RFC PATCH 0/1]: timers: Timer Migration

[Peter Zijlstra]

On Tue, 2008-09-16 at 14:42 +0530, Arun R Bharadwaj wrote:
> Hi,

80 char lines please.

> An idle cpu on which device drivers have initialized timers, has to be
> frequently woken up to service the timers. So, consolidation of timers
> onto a fewer number of cpus is important. Migration of
> timers from idle cpus onto lesser idle cpus is necessary. Currently,
> timers are migrated during the cpu offline operation. However
> cpu-hotplug for the sake of idle system power management is too
> heavy. So, this patch implements a lightweight timer migration
> framework.
> 
> Also, in a multi-core, multi-package system, it is always desirable to
> have all the timers firing on the cpus present in the same package.
> This would enable us to place the idle package in deep sleep
> state. So, migration of timers is required for this.
> 
> A per_cpu sysfs hook is created
> at /sys/devices/system/cpu/cpuX/timer_migration. By echo-ing 1 to
> cpuX/timer_migration, all regular and hrtimers from cpuX are migrated
> to cpu0. The fact that all
> timers are migrated from cpuX to cpu0 is demonstrated by making a note
> of /proc/interrupts.

This seems like an exceedingly dumb idea - cpu 0 might be a long long
way from x and by pushing all timers to cpu0 you might actually overload
cpu 0.

Initially you talked about packages an moving timers to busy cpus -
which is something else altogether.

> $:echo 1 > /sys/devices/system/cpu/cpu3/timer_migration
> $:cat /proc/interrupts > file1; sleep 30 ; cat /proc/interrupts >
> file2
> $:diff file1 file2 | grep LOC
> < LOC:       5276       2100       2676       1647       2415
> 1803       2275       2977   Local timer interrupts
> > LOC:       6991       2195       2774       1647       2562
> 1900       2497       3075   Local timer interrupts
> 
> As we can observe, first we are enabling timer migration in cpu3. As a
> result, the number of local timer interrupts on cpu3 over the next 30
> seconds are zero. cpu3  has been totally rid of all regular
> and hrtimers.
> 
> Please refer to the following paper for details:
> 
> http://ols.fedoraproject.org/OLS/Reprints-2008/srinivasan1-reprint.pdf

If there's anything worth reading in there it should have been mentioned
in this email and preferably in some comment in the code and Kconfig
help as well.

Re: [RFC PATCH 0/1]: timers: Timer Migration

[Arun Bharadwaj]

Peter Zijlstra wrote:
> On Tue, 2008-09-16 at 14:42 +0530, Arun R Bharadwaj wrote:
>   
>> Hi,
>>     
>
> 80 char lines please.
>
>   
>> An idle cpu on which device drivers have initialized timers, has to be
>> frequently woken up to service the timers. So, consolidation of timers
>> onto a fewer number of cpus is important. Migration of
>> timers from idle cpus onto lesser idle cpus is necessary. Currently,
>> timers are migrated during the cpu offline operation. However
>> cpu-hotplug for the sake of idle system power management is too
>> heavy. So, this patch implements a lightweight timer migration
>> framework.
>>
>> Also, in a multi-core, multi-package system, it is always desirable to
>> have all the timers firing on the cpus present in the same package.
>> This would enable us to place the idle package in deep sleep
>> state. So, migration of timers is required for this.
>>
>> A per_cpu sysfs hook is created
>> at /sys/devices/system/cpu/cpuX/timer_migration. By echo-ing 1 to
>> cpuX/timer_migration, all regular and hrtimers from cpuX are migrated
>> to cpu0. The fact that all
>> timers are migrated from cpuX to cpu0 is demonstrated by making a note
>> of /proc/interrupts.
>>     
>
> This seems like an exceedingly dumb idea - cpu 0 might be a long long
> way from x and by pushing all timers to cpu0 you might actually overload
> cpu 0.
>
> Initially you talked about packages an moving timers to busy cpus -
> which is something else altogether.
>
>   

Here, I've chosen cpu0 just to demonstrate migration of timers. I'm
working on the algorithm to choose the best cpu or the best set of cpus
to migrate the timers to. This is just the first iteration on which i
intend to build upon. What could be a possible way is that, a package
can be chosen as the destination for the migrated timers and any one or
many cpus in that package can be the target(s). So, this  will prevent
the possible overloading of any particular cpu.

>> $:echo 1 > /sys/devices/system/cpu/cpu3/timer_migration
>> $:cat /proc/interrupts > file1; sleep 30 ; cat /proc/interrupts >
>> file2
>> $:diff file1 file2 | grep LOC
>> < LOC:       5276       2100       2676       1647       2415
>> 1803       2275       2977   Local timer interrupts
>>     
>>> LOC:       6991       2195       2774       1647       2562
>>>       
>> 1900       2497       3075   Local timer interrupts
>>
>> As we can observe, first we are enabling timer migration in cpu3. As a
>> result, the number of local timer interrupts on cpu3 over the next 30
>> seconds are zero. cpu3  has been totally rid of all regular
>> and hrtimers.
>>
>> Please refer to the following paper for details:
>>
>> http://ols.fedoraproject.org/OLS/Reprints-2008/srinivasan1-reprint.pdf
>>     
>
> If there's anything worth reading in there it should have been mentioned
> in this email and preferably in some comment in the code and Kconfig
> help as well.
>
>   
Thanks for the hint. I'll do that next time.

Re: [RFC PATCH 0/1]: timers: Timer Migration

[Andrew Paprocki]

Peter Zijlstra <a.p.zijlstra <at> chello.nl> writes:

> This seems like an exceedingly dumb idea - cpu 0 might be a long long
> way from x and by pushing all timers to cpu0 you might actually overload
> cpu 0.

You have to give credit to Solaris for doing that first :). They implemented
usleep() using nanosleep() which uses the kernel callout queue. That queue is
single threaded and only processed on CPU0 in the clock interrupt handler,
causing CPU0 to spin 100% if enough calls to usleep() are made across the system.

http://bugs.opensolaris.org/view_bug.do?bug_id=6565503

-Andrew

Re: [RFC PATCH 0/1]: timers: Timer Migration

[Andi Kleen]

On Tue, Sep 16, 2008 at 02:42:31PM +0530, Arun R Bharadwaj wrote:
> Hi,
> 
> An idle cpu on which device drivers have initialized timers, has to be frequently woken up to service the timers. So, consolidation of timers onto a fewer number of cpus is important. Migration of
> timers from idle cpus onto lesser idle cpus is necessary. Currently, timers are migrated during the cpu offline operation. However cpu-hotplug for the sake of idle system power management is too
> heavy. So, this patch implements a lightweight timer migration framework.


Who would trigger such migrations?

I'm not sure it's a good idea to make it fully manually triggered. It would be nicer 
if it was bound to the scheduler's power saving heuristic. So if the scheduler consolidates
threads on a single package for power saving the timers should follow too.

> 
> Also, in a multi-core, multi-package system, it is always desirable to have all the timers firing on the cpus present in the same package. This would enable us to place the idle package in deep sleep
> state. So, migration of timers is required for this.
> 
> A per_cpu sysfs hook is created at /sys/devices/system/cpu/cpuX/timer_migration. By echo-ing 1 to cpuX/timer_migration, all regular and hrtimers from cpuX are migrated to cpu0. The fact that all
> timers are migrated from cpuX to cpu0 is demonstrated by making a note of /proc/interrupts.


I suspect moving all to cpu 0 is too inflexible as Peter already mentioned.

(haven't looked at the actual code yet sorry)

-Andi

Re: [RFC PATCH 0/1]: timers: Timer Migration

[Arjan van de Ven]

On Tue, 16 Sep 2008 14:42:31 +0530
Arun R Bharadwaj <arun@linux.vnet.ibm.com> wrote:

> Hi,
> 
> An idle cpu on which device drivers have initialized timers, has to
> be frequently woken up to service the timers. So, consolidation of
> timers onto a fewer number of cpus is important. Migration of timers
> from idle cpus onto lesser idle cpus is necessary. Currently, timers
> are migrated during the cpu offline operation. However cpu-hotplug
> for the sake of idle system power management is too heavy. So, this
> patch implements a lightweight timer migration framework.

while I absolutely like reducing power consumption... I'm not sure this
helps or is the right approach.

First of all, it's of course absolutely better to fix timers and apps
that cause them (and with PowerTOP we fixed basically all the bad
stuff).

Second, in terms of power; a wake up is a wake up, it doesn't really
matter where it happens. 

Now we could do some consolidation (which realistically needs the range
timer feature that's aimed for 2.6.28), but I would much rather do that
in a different way: rather than actively moving stuff, I would instead
suggest sharing the timer queues between logical cpus that share the
same cache. (Now its an admin choice if he wants this on a "shared L1",
"shared L2" or "shared L3" basis). 

Or if you want to forcefully migrate timers, don't move existing ones,
just on queue/requeue put the on not-the-local-cpu. Sure they'll fire
once on the "wrong" cpu, but that's a very short term problem!

-- 
Arjan van de Ven 	Intel Open Source Technology Centre
For development, discussion and tips for power savings, 
visit http://www.lesswatts.org

Re: [RFC PATCH 0/1]: timers: Timer Migration

[Arun R Bharadwaj]

Arjan van de Ven wrote:
> On Tue, 16 Sep 2008 14:42:31 +0530
> Arun R Bharadwaj <arun@linux.vnet.ibm.com> wrote:
>
>   
>> Hi,
>>
>> An idle cpu on which device drivers have initialized timers, has to
>> be frequently woken up to service the timers. So, consolidation of
>> timers onto a fewer number of cpus is important. Migration of timers
>> from idle cpus onto lesser idle cpus is necessary. Currently, timers
>> are migrated during the cpu offline operation. However cpu-hotplug
>> for the sake of idle system power management is too heavy. So, this
>> patch implements a lightweight timer migration framework.
>>     
>
> while I absolutely like reducing power consumption... I'm not sure this
> helps or is the right approach.
>
> First of all, it's of course absolutely better to fix timers and apps
> that cause them (and with PowerTOP we fixed basically all the bad
> stuff).
>
> Second, in terms of power; a wake up is a wake up, it doesn't really
> matter where it happens. 
>   


This is a very valid point. The algorithm which decides which cpu to
migrate the timers to should be intelligent enough in the sense that it
migrates *only* when there is some benefit out of it. That is, the cpu
to which the timers are being migrated, is sufficiently busy and it is
not unnecessarily brought out of its idle state. Else, as you pointed
out, the penalty of wake up is same for either of the cpu.


> Now we could do some consolidation (which realistically needs the range
> timer feature that's aimed for 2.6.28), but I would much rather do that
> in a different way: rather than actively moving stuff, I would instead
> suggest sharing the timer queues between logical cpus that share the
> same cache. (Now its an admin choice if he wants this on a "shared L1",
> "shared L2" or "shared L3" basis). 
>   


While migrating range timers, I would migrate only if the time range of
the source and destination cpus match. Else, there is no point in
migrating. I would rather look at timer migration as an optimization
strategy to increase the cpu idle time and not as a pro-active method to
rid the cpu of timers, no matter what. I would like to migrate timers
only if the wake up penalty of the system as a whole decreases.


> Or if you want to forcefully migrate timers, don't move existing ones,
> just on queue/requeue put the on not-the-local-cpu. Sure they'll fire
> once on the "wrong" cpu, but that's a very short term problem!
>
>