RE: gradual timeofday overhaul

RE: gradual timeofday overhaul

[Perez-Gonzalez, Inaky]

> From: Richard B. Johnson
>
> You need that hardware interrupt for more than time-keeping.
> Without a hardware-interrupt, to force a new time-slice,
> 
>  	for(;;)
>             ;
> 
> ... would allow a user to grab the CPU forever ...

But you can also schedule, before switching to the new task, 
a local interrupt on the running processor to mark the end 
of the timeslice. When you enter the scheduler, you just need 
to remove that; devil is in the details, but it should be possible
to do in a way that doesn't take too much overhead.

Iñaky Pérez-González -- Not speaking for Intel -- all opinions are my own (and my fault)

Re: gradual timeofday overhaul

[George Anzinger]

Perez-Gonzalez, Inaky wrote:
>>From: Richard B. Johnson
>>
>>You need that hardware interrupt for more than time-keeping.
>>Without a hardware-interrupt, to force a new time-slice,
>>
>> 	for(;;)
>>            ;
>>
>>... would allow a user to grab the CPU forever ...
> 
> 
> But you can also schedule, before switching to the new task, 
> a local interrupt on the running processor to mark the end 
> of the timeslice. When you enter the scheduler, you just need 
> to remove that; devil is in the details, but it should be possible
> to do in a way that doesn't take too much overhead.

Well, that is part of the accounting overhead the increases with context switch 
rate.  You also need to include the time it takes to figure out which of the 
time limits is closes (run time limit, profile time, slice time, etc).  Then, 
you also need to remove the timer when switching away.  No, it is not a lot, but 
it is way more than the nothing we do when we can turn it all over to the 
periodic tick.  The choice is load sensitive overhead vs flat overhead.

-- 
George Anzinger   george@mvista.com
High-res-timers:  http://sourceforge.net/projects/high-res-timers/

Re: gradual timeofday overhaul

[Chris Friesen]

George Anzinger wrote:

> Well, that is part of the accounting overhead the increases with context 
> switch rate.  You also need to include the time it takes to figure out 
> which of the time limits is closes (run time limit, profile time, slice 
> time, etc).  Then, you also need to remove the timer when switching 
> away.  No, it is not a lot, but it is way more than the nothing we do 
> when we can turn it all over to the periodic tick.  The choice is load 
> sensitive overhead vs flat overhead.

It should be possible to be clever about this.  Most processes don't use their 
timeslice, so if we have a previous timer running, just keep track of how much 
beyond that timer our timeslice will be.  If we context switch before the timer 
expiry, well and good.  If the timer expires, set it for what's left of our 
timeslice.

Chris

Re: gradual timeofday overhaul

[George Anzinger]

Chris Friesen wrote:
> George Anzinger wrote:
> 
>> Well, that is part of the accounting overhead the increases with 
>> context switch rate.  You also need to include the time it takes to 
>> figure out which of the time limits is closes (run time limit, profile 
>> time, slice time, etc).  Then, you also need to remove the timer when 
>> switching away.  No, it is not a lot, but it is way more than the 
>> nothing we do when we can turn it all over to the periodic tick.  The 
>> choice is load sensitive overhead vs flat overhead.
> 
> 
> It should be possible to be clever about this.  Most processes don't use 
> their timeslice, so if we have a previous timer running, just keep track 
> of how much beyond that timer our timeslice will be.  If we context 
> switch before the timer expiry, well and good.  If the timer expires, 
> set it for what's left of our timeslice.

Me thinks that rather quickly devolves to a periodic tick.

-- 
George Anzinger   george@mvista.com
High-res-timers:  http://sourceforge.net/projects/high-res-timers/

Re: gradual timeofday overhaul

[Chris Friesen]

George Anzinger wrote:
> Chris Friesen wrote:

>> It should be possible to be clever about this.  Most processes don't 
>> use their timeslice, so if we have a previous timer running, just keep 
>> track of how much beyond that timer our timeslice will be.  If we 
>> context switch before the timer expiry, well and good.  If the timer 
>> expires, set it for what's left of our timeslice.
> 
> 
> Me thinks that rather quickly devolves to a periodic tick.

In the busy case, yes.  But on an idle system you get tickless behaviour.

It's still going to be load-sensitive, since you are doing additional work to 
keep track of the timer/timeout values.  But it saves work if reprogramming the 
timer is time-consuming compared to simply reading it.  On something like the 
ppc, it probably doesn't buy you much since the decrementer is cheap to program.

Chris

RE: gradual timeofday overhaul

[Perez-Gonzalez, Inaky]

> From: George Anzinger [mailto:george@mvista.com]
>
> > This is just talking out of my ass, but I guess that for each invocation
> > they will have more or less the same overhead in execution time, let's
> > say T. For the periodic tick, the total overhead (in a second) is T*HZ;
> > with tickless, it'd be T*number_of_context_switches_per_second, right?
> 
> ???  Better look again.  Context switches can and do happen as often as 10 or so
> micro seconds (depends a lot on the cpu speed).  I admit this is with code that
> is just trying to measure the context switch time, but, often the system will
> change it mind just that fast.

As I said, I was talking out of my ass [aka, I didn't know and was just 
guesstimating for the heck of it], so I am happily proven wrong--thanks to
Chris and you--I guess I didn't take into account voluntary yielding of
the CPU by a task; I was more guiding myself for kicked out by a timer
making a task runnable, or a timeslice expiring, etc...which now are 
more or less guided by the tick [and then of course, we have IRQs,
but that's another matter]

> ...
> sourceforge).  On the other hand, where I come from, a system which has
> increasing overhead with load is one that is going to overload.  We are always
> better off if we can figure a way to have fixed overhead.
> 
> As for the idle system ticks, I think the VST stuff we are working on is the
> right answer.

Once my logic is proven wrong, then it makes full sense :]

Thanks for the heads up.

Iñaky Pérez-González -- Not speaking for Intel -- all opinions are my own (and my fault)

RE: gradual timeofday overhaul

[Perez-Gonzalez, Inaky]

> From: George Anzinger [mailto:george@mvista.com]
>
> Perez-Gonzalez, Inaky wrote:
> 
> > But you can also schedule, before switching to the new task,
> > a local interrupt on the running processor to mark the end
> > of the timeslice. When you enter the scheduler, you just need
> > to remove that; devil is in the details, but it should be possible
> > to do in a way that doesn't take too much overhead.
> 
> Well, that is part of the accounting overhead the increases with context switch
> rate.  You also need to include the time it takes to figure out which of the
> time limits is closes (run time limit, profile time, slice time, etc).  Then,

I know these are specific examples, but:

- profile time is a periodic thingie, so if you have it, forget about
  having a tickless system. Periodic interrupt for this guy, get it 
  out of the equation.

- slice time vs runtime limit. I don't remember what is the granularity of
  the runtime limit, but it could be expressed in slice terms. If not,
  we are talking (along with any other times) of min() operations, which
  are just a few cycles each [granted, they add up].

> you also need to remove the timer when switching away.  No, it is not a lot, but
> it is way more than the nothing we do when we can turn it all over to the
> periodic tick.  The choice is load sensitive overhead vs flat overhead.

This is just talking out of my ass, but I guess that for each invocation
they will have more or less the same overhead in execution time, let's
say T. For the periodic tick, the total overhead (in a second) is T*HZ;
with tickless, it'd be T*number_of_context_switches_per_second, right?

Now, the ugly case would be if number_of_context_swiches_per_second > HZ.
In HZ = 100, this could be happening, but in HZ=1000, in a single CPU
...well, that would be TOO weird [of course, a real-time app with a 
1ms period would do that, but it'd require at least an HZ of 10000 to
work more or less ok and we'd be below the watermark].

So in most cases, and given the assumptions, we'd end up winning,
beause number_of_context..., even if variable, is going to be bound
on the upper side by HZ.

Well, you know way more than I do about this, so here is the question:
what is the error in that line of reasoning? 

Iñaky Pérez-González -- Not speaking for Intel -- all opinions are my own (and my fault)

Re: gradual timeofday overhaul

[Chris Friesen]

Perez-Gonzalez, Inaky wrote:

> Now, the ugly case would be if number_of_context_swiches_per_second > HZ.
> In HZ = 100, this could be happening, but in HZ=1000, in a single CPU
> ...well, that would be TOO weird [of course, a real-time app with a 
> 1ms period would do that, but it'd require at least an HZ of 10000 to
> work more or less ok and we'd be below the watermark].

It's easy to have >>1000 context switches per second on a server.  Consider a 
web server that receives a network packet, issues a request to a database, hands 
some work off to a thread so the main app doesn't block, then sends a response. 
  That could be a half dozen context switches per packet.  If you have 20000 
packets/sec coming in....


Chris

Re: gradual timeofday overhaul

[George Anzinger]

Perez-Gonzalez, Inaky wrote:
>>From: George Anzinger [mailto:george@mvista.com]
>>
>>Perez-Gonzalez, Inaky wrote:
>>
>>
>>>But you can also schedule, before switching to the new task,
>>>a local interrupt on the running processor to mark the end
>>>of the timeslice. When you enter the scheduler, you just need
>>>to remove that; devil is in the details, but it should be possible
>>>to do in a way that doesn't take too much overhead.
>>
>>Well, that is part of the accounting overhead the increases with context switch
>>rate.  You also need to include the time it takes to figure out which of the
>>time limits is closes (run time limit, profile time, slice time, etc).  Then,
> 
> 
> I know these are specific examples, but:
> 
> - profile time is a periodic thingie, so if you have it, forget about
>   having a tickless system. Periodic interrupt for this guy, get it 
>   out of the equation.

Not really.  It is only active if the task is running.  At the very least the 
scheduler needs to check to see if it is on and, if so, set up a timer for it.
> 
> - slice time vs runtime limit. I don't remember what is the granularity of
>   the runtime limit, but it could be expressed in slice terms. If not,
>   we are talking (along with any other times) of min() operations, which
>   are just a few cycles each [granted, they add up].

The main issue here is accumulating the run time which is accounting work that 
needs to happen on context switch (out in this case).
> 
> 
>>you also need to remove the timer when switching away.  No, it is not a lot, but
>>it is way more than the nothing we do when we can turn it all over to the
>>periodic tick.  The choice is load sensitive overhead vs flat overhead.
> 
> 
> This is just talking out of my ass, but I guess that for each invocation
> they will have more or less the same overhead in execution time, let's
> say T. For the periodic tick, the total overhead (in a second) is T*HZ;
> with tickless, it'd be T*number_of_context_switches_per_second, right?
> 
> Now, the ugly case would be if number_of_context_swiches_per_second > HZ.
> In HZ = 100, this could be happening, but in HZ=1000, in a single CPU
> ...well, that would be TOO weird [of course, a real-time app with a 
> 1ms period would do that, but it'd require at least an HZ of 10000 to
> work more or less ok and we'd be below the watermark].

???  Better look again.  Context switches can and do happen as often as 10 or so 
micro seconds (depends a lot on the cpu speed).  I admit this is with code that 
is just trying to measure the context switch time, but, often the system will 
change it mind just that fast.
> 
> So in most cases, and given the assumptions, we'd end up winning,
> beause number_of_context..., even if variable, is going to be bound
> on the upper side by HZ.
> 
> Well, you know way more than I do about this, so here is the question:
> what is the error in that line of reasoning? 

The expected number of context switches.  In some real world apps it gets rather 
high.  The cross over of your two curves _might_ be of interest to some (it is 
rather low by my measurements, done with the tickless patch that is still on 
sourceforge).  On the other hand, where I come from, a system which has 
increasing overhead with load is one that is going to overload.  We are always 
better off if we can figure a way to have fixed overhead.

As for the idle system ticks, I think the VST stuff we are working on is the 
right answer.

-- 
George Anzinger   george@mvista.com
High-res-timers:  http://sourceforge.net/projects/high-res-timers/

process start time set wrongly at boot for kernel 2.6.9

[Jerome Borsboom]

Starting with kernel 2.6.9 the process start time is set wrongly for 
processes that get started early in the boot process. Below is a dump from 
my 'ps' command. Note the start time for processes 1-12. After process 12 
the start time is set right.

Jerome


USER       PID %CPU %MEM   VSZ  RSS TTY      STAT START   TIME COMMAND
root         1  0.0  0.1  1372  500 ?        S    20:59   0:00 init [3] 
root         2  0.0  0.0     0    0 ?        SN   20:59   0:00 [ksoftirqd/0]
root         3  0.0  0.0     0    0 ?        S<   20:59   0:00 [events/0]
root         4  0.0  0.0     0    0 ?        S<   20:59   0:00 [khelper]
root         5  0.0  0.0     0    0 ?        S<   20:59   0:00 [kblockd/0]
root         6  0.0  0.0     0    0 ?        S    20:59   0:00 [pdflush]
root         7  0.0  0.0     0    0 ?        S    20:59   0:00 [pdflush]
root         9  0.0  0.0     0    0 ?        S<   20:59   0:00 [aio/0]
root         8  0.0  0.0     0    0 ?        S    20:59   0:00 [kswapd0]
root        10  0.0  0.0     0    0 ?        S    20:59   0:00 [kseriod]
root        11  0.0  0.0     0    0 ?        S    20:59   0:00 [scsi_eh_0]
root        12  0.0  0.0     0    0 ?        S    20:59   0:00 [ahc_dv_0]
root        13  0.0  0.0     0    0 ?        S    19:48   0:00 [scsi_eh_1]
root        14  0.0  0.0     0    0 ?        S    19:48   0:00 [ahc_dv_1]
root        15  0.0  0.0     0    0 ?        S    19:48   0:00 [scsi_eh_2]
root        16  0.0  0.0     0    0 ?        S    19:48   0:00 [ahc_dv_2]
root        17  0.0  0.0     0    0 ?        S    19:49   0:00 [kjournald]
root        43  0.0  0.0     0    0 ?        S    19:49   0:00 [kjournald]
root        44  0.0  0.0     0    0 ?        S    19:49   0:00 [kjournald]
root        45  0.0  0.0     0    0 ?        S    19:49   0:00 [kjournald]
root        46  0.0  0.0     0    0 ?        S    19:49   0:00 [kjournald]
root        47  0.0  0.0     0    0 ?        S    19:49   0:00 [kjournald]
root        48  0.0  0.0     0    0 ?        S    19:49   0:00 [kjournald]
root       122  0.0  0.2  1420  552 ?        Ss   19:49   0:00 /sbin/syslogd -m 0
root       124  0.0  0.1  1376  452 ?        Ss   19:49   0:00 /sbin/klogd
root       131  0.0  0.3  1640  776 ?        Ss   19:49   0:00 /sbin/apcupsd
root       139  0.0  0.9  2444 2444 ?        SLs  19:49   0:00 /usr/bin/ntpd
ldap       148  0.0  1.8 50084 4696 ?        Ssl  19:49   0:00 /usr/sbin/slapd -4 -u ldap -h ldap:/// ldapi:///
nscd       153  0.0  0.6 10208 1640 ?        Ssl  19:49   0:00 /usr/sbin/nscd
root       162  0.0  0.5  3156 1392 ?        Ss   19:49   0:00 /usr/sbin/sshd
root       168  0.0  0.9  5396 2436 ?        Ss   19:49   0:00 sendmail: accepting connections 
smmsp      173  0.0  0.8  5176 2144 ?        Ss   19:49   0:00 sendmail: Queue runner@01:00:00 for /var/spool/clientmqueue
root       177  0.0  0.2  1372  564 ?        Ss   19:49   0:00 /usr/sbin/cron

Re: process start time set wrongly at boot for kernel 2.6.9

[john stultz]

On Tue, 2004-10-19 at 11:21, Jerome Borsboom wrote:
> Starting with kernel 2.6.9 the process start time is set wrongly for 
> processes that get started early in the boot process. Below is a dump from 
> my 'ps' command. Note the start time for processes 1-12. After process 12 
> the start time is set right.

How reproducible is this? Are the correct and incorrect time values
always off by the same amount? 

Are you running NTP? I'm curious if you are changing your system time
during boot. 

thanks
-john

Re: process start time set wrongly at boot for kernel 2.6.9

[Tim Schmielau]

On Tue, 19 Oct 2004, john stultz wrote:

> On Tue, 2004-10-19 at 11:21, Jerome Borsboom wrote:
> > Starting with kernel 2.6.9 the process start time is set wrongly for 
> > processes that get started early in the boot process. Below is a dump from 
> > my 'ps' command. Note the start time for processes 1-12. After process 12 
> > the start time is set right.
> 
> How reproducible is this? Are the correct and incorrect time values
> always off by the same amount? 
> 
> Are you running NTP? I'm curious if you are changing your system time
> during boot. 

I'd bet that some process early in the boot adjusts your system time.
Then this is expected behavior. This is why I would have preferred the 
simple back-out patch for the boot times problem.

I'm sorry I fell of the net for so long and didn't stand up for the 
simpler change in this case. Oh well.

I'll probably supply a back-out patch for -mm then, after wading through
my multi-megabyte email backlog (sorry John, still need to read your time
keeping proposal and all its discussion).

Tim

Re: process start time set wrongly at boot for kernel 2.6.9

[john stultz]

On Tue, 2004-10-19 at 17:42, Tim Schmielau wrote:
> On Tue, 19 Oct 2004, john stultz wrote:
> 
> > On Tue, 2004-10-19 at 11:21, Jerome Borsboom wrote:
> > > Starting with kernel 2.6.9 the process start time is set wrongly for 
> > > processes that get started early in the boot process. Below is a dump from 
> > > my 'ps' command. Note the start time for processes 1-12. After process 12 
> > > the start time is set right.
> > 
> > How reproducible is this? Are the correct and incorrect time values
> > always off by the same amount? 
> > 
> > Are you running NTP? I'm curious if you are changing your system time
> > during boot. 
> 
> I'd bet that some process early in the boot adjusts your system time.

He claims that's not the case (you weren't CC'ed on his reply, but its
on lkml). He believes the time changes before NTP starts up. Might be
something else, but I'm not sure.

> Then this is expected behavior. This is why I would have preferred the 
> simple back-out patch for the boot times problem.
> 
> I'm sorry I fell of the net for so long and didn't stand up for the 
> simpler change in this case. Oh well.
> 
> I'll probably supply a back-out patch for -mm then, after wading through
> my multi-megabyte email backlog (sorry John, still need to read your time
> keeping proposal and all its discussion).

I've begun to agree with you about this issue. It seems that until we
can catch every use of jiffies for time, doing one by one is going to
cause consistency problems.  So I'd support the full backout of the
do_posix_clock_monotonic_gettime changes to the proc interface. 

George, would you protest this?

As for the timeofday overhaul, I've had zero time to work on it
recently. I hate that I dropped code and then went missing for weeks.
I'll have to see if I can get a few cycles at home to sync up my current
tree and send it out. 

thanks
-john

gradual timeofday overhaul

[Tim Schmielau]

On Tue, 19 Oct 2004, john stultz wrote:

> As for the timeofday overhaul, I've had zero time to work on it
> recently. I hate that I dropped code and then went missing for weeks.
> I'll have to see if I can get a few cycles at home to sync up my current
> tree and send it out. 

I still haven't looked at your code and it's discussion. From what I
remember, I liked your proposal very much. It's surely where we want to
end up someday. But from the above mail it strikes me that we just don't
have enough manpower to get there all at once, so we should have a plan 
for the time code to gradually evolve into what we finally want. I think 
we could do it in the following steps:

  1. Sync up jiffies with the monotonic clock, very much like we
     already handle lost ticks. This would immediately remove the
     hassles with incompatible time sources.
     Judging from the jiffies wrap experience, we there probably are
     some drivers which need fixing (mostly because they wait until 
     jiffies==something), but these are bugs already right now
     in the case of lost ticks.

  2. Decouple jiffies from the actual interrupt counter. We could
     then e.g. set HZ to 10000, also increasing the resolution of
     timers, without increasing the interrupt frequency.
     We'd then need to identify the places where this might lead to
     overflows and promote them to use jiffies_64 instead of jiffies
     (where this hasn't been done already).

  3. Increase HZ all the way up to 1e9. jiffies_64 would then be the
     same as your plain 64 bit nanoseconds value.
     This would require an optimization to the timer code to be able
     to increment jiffies in steps larger than 1.

Thoughts?

Re: gradual timeofday overhaul

[Len Brown]

On Tue, 2004-10-19 at 23:05, Tim Schmielau wrote:
> I think we could do it in the following steps:
> 
>   1. Sync up jiffies with the monotonic clock,...
>   2. Decouple jiffies from the actual interrupt counter...
>   3. Increase HZ all the way up to 1e9....

> Thoughts?

Yes, for long periods of idle, I'd like to see the periodic clock tick
disabled entirely.  Clock ticks causes the hardware to exit power-saving
idle states.

The current design with HZ=1000 gives us 1ms = 1000usec between clock
ticks.  But some platforms take nearly that long just to enter/exit low
power states; which means that on Linux the hardware pays a long idle
state exit latency (performance hit) but gets little or no power savings
from the time it resides in that idle state.

thanks,
-Len

Re: gradual timeofday overhaul

[George Anzinger]

Len Brown wrote:
> On Tue, 2004-10-19 at 23:05, Tim Schmielau wrote:
> 
>>I think we could do it in the following steps:
>>
>>  1. Sync up jiffies with the monotonic clock,...
>>  2. Decouple jiffies from the actual interrupt counter...
>>  3. Increase HZ all the way up to 1e9....

Before we do any of the above, I think we need to stop and ponder just what a 
"jiffie" is.  Currently it is, by default (or historically) the "basic tick" of 
the system clock.  On top of this a lot of interpolation code has been "grafted" 
to allow the system to resolve time to finer levels, i.e. to the nanosecond. 
But none of this interpolation code actually changes the tick, i.e. the 
interrupt still happens at the same periodic rate.

As the "basic tick", it is used to do a lot of accounting and scheduling house 
keeping AND as a driver of the system timers.

So, by this definition, it REQUIRES a system interrupt.

I have built a "tick less" system and have evidence from that that such systems 
are over load prone.  The faster the context switch rate, the more accounting 
needs to be done.  On the otherhand, the ticked system has flat accounting 
overhead WRT load.

Regardless of what definitions we settle on, the system needs an interrupt 
source to drive the system timers, and, as I indicate above, the accounting and 
scheduling stuff.  It is a MUST that these interrupts occure at the required 
times or the system timers will be off.  This is why we have a jiffies value 
that is "rather odd" in the x86 today.

George


> 
> 
>>Thoughts?
> 
> 
> Yes, for long periods of idle, I'd like to see the periodic clock tick
> disabled entirely.  Clock ticks causes the hardware to exit power-saving
> idle states.
> 
> The current design with HZ=1000 gives us 1ms = 1000usec between clock
> ticks.  But some platforms take nearly that long just to enter/exit low
> power states; which means that on Linux the hardware pays a long idle
> state exit latency (performance hit) but gets little or no power savings
> from the time it resides in that idle state.
> 
> thanks,
> -Len
> 
> 

-- 
George Anzinger   george@mvista.com
High-res-timers:  http://sourceforge.net/projects/high-res-timers/

Re: gradual timeofday overhaul

[Richard B. Johnson]

On Wed, 20 Oct 2004, George Anzinger wrote:

> Len Brown wrote:
>> On Tue, 2004-10-19 at 23:05, Tim Schmielau wrote:
>> 
>>> I think we could do it in the following steps:
>>> 
>>>  1. Sync up jiffies with the monotonic clock,...
>>>  2. Decouple jiffies from the actual interrupt counter...
>>>  3. Increase HZ all the way up to 1e9....
>
> Before we do any of the above, I think we need to stop and ponder just what a 
> "jiffie" is.  Currently it is, by default (or historically) the "basic tick" 
> of the system clock.  On top of this a lot of interpolation code has been 
> "grafted" to allow the system to resolve time to finer levels, i.e. to the 
> nanosecond. But none of this interpolation code actually changes the tick, 
> i.e. the interrupt still happens at the same periodic rate.
>
> As the "basic tick", it is used to do a lot of accounting and scheduling 
> house keeping AND as a driver of the system timers.
>
> So, by this definition, it REQUIRES a system interrupt.
>
> I have built a "tick less" system and have evidence from that that such 
> systems are over load prone.  The faster the context switch rate, the more 
> accounting needs to be done.  On the otherhand, the ticked system has flat 
> accounting overhead WRT load.
>
> Regardless of what definitions we settle on, the system needs an interrupt 
> source to drive the system timers, and, as I indicate above, the accounting 
> and scheduling stuff.  It is a MUST that these interrupts occure at the 
> required times or the system timers will be off.  This is why we have a 
> jiffies value that is "rather odd" in the x86 today.
>
> George
>
>

You need that hardware interrupt for more than time-keeping.
Without a hardware-interrupt, to force a new time-slice,

 	for(;;)
            ;

... would allow a user to grab the CPU forever ...

So, getting rid of the hardware interrupt can't be done.
Also, much effort has gone into obtaining high resolution
timing without any high resolution hardware to back it
up. This means that user's can get numbers like 987,654
microseconds and the last 654 are as valuable as teats
on a bull. With a HZ timer tick, you get 1/HZ resolution
pure and simple. The rest of the "interpolation" is just
guess-work which leads to lots of problems, especially
when one attempts to read a spinning down-count value
from a hardware device accessed off some ports!

If the ix86 CMOS timer was used you could get better
accuracy than present, but accuracy is something one
can accommodate with automatic adjustment of time,
tracable to some appropriate standard.

The top-level schedule-code could contain some flag that
says; "are we in a power-down mode". If so, it could
execute minimal in-cache code, i.e. :

 		for(;;)
                 {
                    hlt();	// Sleep until next tick
 		   if(mode != power_down)
                        schedule();

                 }

The timer-tick ISR or any other ISR wakes us up from halt.
This keeps the system sleeping, not wasting power grabbing
code/data from RAM and grunching some numbers that are
not going to be used.


>> 
>> 
>>> Thoughts?
>> 
>> 
>> Yes, for long periods of idle, I'd like to see the periodic clock tick
>> disabled entirely.  Clock ticks causes the hardware to exit power-saving
>> idle states.
>> 
>> The current design with HZ=1000 gives us 1ms = 1000usec between clock
>> ticks.  But some platforms take nearly that long just to enter/exit low
>> power states; which means that on Linux the hardware pays a long idle
>> state exit latency (performance hit) but gets little or no power savings
>> from the time it resides in that idle state.
>> 
>> thanks,
>> -Len
>> 
>> 
>
> -- 
> George Anzinger   george@mvista.com
> High-res-timers:  http://sourceforge.net/projects/high-res-timers/
>
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/
>

Cheers,
Dick Johnson
Penguin : Linux version 2.6.9 on an i686 machine (5537.79 GrumpyMips).
                  98.36% of all statistics are fiction.

Re: gradual timeofday overhaul

[George Anzinger]

Len Brown wrote:
> On Tue, 2004-10-19 at 23:05, Tim Schmielau wrote:
> 
>>I think we could do it in the following steps:
>>
>>  1. Sync up jiffies with the monotonic clock,...
>>  2. Decouple jiffies from the actual interrupt counter...
>>  3. Increase HZ all the way up to 1e9....
> 
> 
>>Thoughts?
> 
> 
> Yes, for long periods of idle, I'd like to see the periodic clock tick
> disabled entirely.  Clock ticks causes the hardware to exit power-saving
> idle states.
> 
> The current design with HZ=1000 gives us 1ms = 1000usec between clock
> ticks.  But some platforms take nearly that long just to enter/exit low
> power states; which means that on Linux the hardware pays a long idle
> state exit latency (performance hit) but gets little or no power savings
> from the time it resides in that idle state.


I (and MontaVista) will be expanding on the VST patches.  There are, currently, 
two levels of VST.  VST-I when entering the idle state (task) looks ahead in the 
timer list, finds the next event, and shuts down the "tick" until that time.  An 
interrupts resets things, be it from the end of the time counter or another source.

VST-II adds a call back list to idle entry and exit.  This allows one to add 
code to change (or even remove) timers on idle entry and restore them on exit.

We are doing this work to support deeply embedded applications that often times 
run on small batteries (think cell phone if you like).
-- 
George Anzinger   george@mvista.com
High-res-timers:  http://sourceforge.net/projects/high-res-timers/

Re: gradual timeofday overhaul

[Lee Revell]

On Wed, 2004-10-20 at 03:47, Len Brown wrote:
> The current design with HZ=1000 gives us 1ms = 1000usec between clock
> ticks.  But some platforms take nearly that long just to enter/exit low
> power states; which means that on Linux the hardware pays a long idle
> state exit latency (performance hit) but gets little or no power savings
> from the time it resides in that idle state.

My testing shows that the timer interrupt runs for about 21 usec. 
That's 2.1% of its time just running the timer ISR!  No wonder this
causes PM issues, 2.1% cpu load is not exactly an idle machine.  This is
a 600Mhz C3, so on a slower embedded system this might be 5%.

So, any solution that would allow high res timers with Hz = 100 would be
welcome.

Lee

Re: gradual timeofday overhaul

[Len Brown]

On Wed, 2004-10-20 at 13:09, Lee Revell wrote:
> On Wed, 2004-10-20 at 03:47, Len Brown wrote:
> > The current design with HZ=1000 gives us 1ms = 1000usec between
> > clock ticks.  But some platforms take nearly that long just
> > to enter/exit low power states; which means that on Linux
> > the hardware pays a long idle state exit latency
> > (performance hit) but gets little or no power savings
> > from the time it resides in that idle state.
> 
> My testing shows that the timer interrupt runs for about 21 usec.
> That's 2.1% of its time just running the timer ISR!  No wonder this
> causes PM issues, 2.1% cpu load is not exactly an idle machine.  This
> is a 600Mhz C3, so on a slower embedded system this might be 5%.
> 
> So, any solution that would allow high res timers with Hz = 100 would
> be welcome.

5% residency in the clock tick handler is likely more of a problem when
we're _not_ idle -- a 5% performance hit.  When we're idle we've got
nothing better to do with the processor than run these instructions for
5% of the time and run no instructions 95% of the time -- so tick
handler residency isn't the problem in idle, tick frequency is the
problem.

When an interrupt occurrs, the hardware needs to ramp up its voltages,
resume its clocks and all the stuff it need to do to get out of the
power saving state to run the code that services the interrupt.  This
"exit latency" can take a long time.  On a volume Centrino system today
it is up to 185usec.  On other hardware it is as high as 1000 usec. 
Time spent in this exit latency is a double penalty -- we're not saving
power and we're delaying before the processor starts executing
instructions -- so we want to pay this price only when necessary.

-Len

Re: gradual timeofday overhaul

[john stultz]

On Tue, 2004-10-19 at 20:05, Tim Schmielau wrote:
> On Tue, 19 Oct 2004, john stultz wrote:
> 
> > As for the timeofday overhaul, I've had zero time to work on it
> > recently. I hate that I dropped code and then went missing for weeks.
> > I'll have to see if I can get a few cycles at home to sync up my current
> > tree and send it out. 
> 
> I still haven't looked at your code and it's discussion. From what I
> remember, I liked your proposal very much. It's surely where we want to
> end up someday. But from the above mail it strikes me that we just don't
> have enough manpower to get there all at once, so we should have a plan 
> for the time code to gradually evolve into what we finally want. I think 
> we could do it in the following steps:
> 
>   1. Sync up jiffies with the monotonic clock...
> 
>   2. Decouple jiffies from the actual interrupt counter...
> 
>   3. Increase HZ all the way up to 1e9....
> Thoughts?

They all sound good. I like the notion of basing jiffies off of system
time, rather then interrupt counts. However, I'm a little cautious of
changing the meaning of jiffies too drastically. 

Right now jiffies has two core meanings:
1. Count of the number of timer ticks that have passed.
2. Accurate system uptime, measured in units of 1/HZ
(Let me know if I forgot any others)

The problem being, neither of those meaning are 100% true. 
#1 isn't true because when we loose timer ticks, we try to compensate
for them (i386 specifically). But at the same time #2 isn't true because
the timer interrupts don't necessarily run at exactly HZ (again, i386
specifically).

Basically due to our hardware constraints, we need to break one of these
two assumptions. The problem is which do we choose? 

Do we base jiffies off of monotonic_clock(), guaranteeing #2 and
possibly breaking anyone who is assuming #1? Or do we change all users
of jiffies for time to use monotonic_clock, guaranteeing #1, which will
require quite a bit of work.

And which choice makes it harder for folks to create tickless systems?
Its a tough call.

On top of that, we still have the issue that the current  interpolation
used in the time of day subsystem is broken (in my opinion), and we need
to fix that before we can have a reliable monotonic_clock. 

The joke being of course that I'll need to set my /etc/ntp/ntp.drift
file to 500 to find the time to work on any of this. And really, anyone
who really found that funny needs to go home.

thanks
-john