time jumps forward/backwards

time jumps forward/backwards

[Ben Gamsa]

Our project is currently using 2.6.27.4 with a patch from Paul Mackerras 
("powerpc: Improve resolution of VDSO clock_gettime") running on a 
MPC8555.  We picked up the patch on the hope that it would fix an 
earlier problem we had with time jumping forwards and backwards shortly 
after boot-up.  It seemed to fix that problem, but it seems we have 
another, similar problem.

It appears to be the case that when the time on the system is around the 
epoch (1970), that time will occasionally jump forward and then backward 
by about 17592 seconds.  When it jumps forward, it always jumps back a 
few milliseconds later.  However, it's not always easy to catch these 
occurrences.  The delta is more specifically about 17592186059 usec, 
give or take a few 10s of microseconds (most of the time), despite the 
fact that the user-level program I have that is testing it only checks 
every 10 milliseconds.

The number seems suspiciously close to 2^44 nanoseconds, which suggests 
some sort of overflow error, perhaps caused by the time being set to 
around time 0.

I'm not really sure where to begin looking (I know that the time code is 
one of the more sensitive bits in the system so I definitely don't want 
to go blindly mucking around), so I was hoping to get some suggestions 
from the list.

Thanks for any help.

-- 
Ben Gamsa       ben@somanetworks.com
SOMA Networks   312 Adelaide St. W. Suite 600 Toronto, Ontario, M5V1R2

Re: time jumps forward/backwards

[Paul Mackerras]

Ben Gamsa writes:

> It appears to be the case that when the time on the system is around the 
> epoch (1970), that time will occasionally jump forward and then backward 
> by about 17592 seconds.  When it jumps forward, it always jumps back a 
> few milliseconds later.  However, it's not always easy to catch these 
> occurrences.  The delta is more specifically about 17592186059 usec, 
> give or take a few 10s of microseconds (most of the time), despite the 
> fact that the user-level program I have that is testing it only checks 
> every 10 milliseconds.

I don't think the time code in the kernel is designed to handle
negative values, i.e., times before the epoch.  If you want it to do
that you'll have to check places like arch/powerpc/kernel/time.c,
kernel/time/timekeeping.c, arch/powerpc/include/asm/time.h, etc., and
make sure that it uses signed types where necessary and that the
arithmetic is correct.

Paul.

Re: time jumps forward/backwards

[Benjamin Gamsa]

Paul Mackerras wrote:
> Ben Gamsa writes:
> 
>> It appears to be the case that when the time on the system is around the 
>> epoch (1970), that time will occasionally jump forward and then backward 
>> by about 17592 seconds.  When it jumps forward, it always jumps back a 
>> few milliseconds later.  However, it's not always easy to catch these 
>> occurrences.  The delta is more specifically about 17592186059 usec, 
>> give or take a few 10s of microseconds (most of the time), despite the 
>> fact that the user-level program I have that is testing it only checks 
>> every 10 milliseconds.
> 
> I don't think the time code in the kernel is designed to handle
> negative values, i.e., times before the epoch.  If you want it to do
> that you'll have to check places like arch/powerpc/kernel/time.c,
> kernel/time/timekeeping.c, arch/powerpc/include/asm/time.h, etc., and
> make sure that it uses signed types where necessary and that the
> arithmetic is correct.
> 

The time never goes negative.  It starts off at the epoch and moves 
forward, but sometimes it jumps forward by 17952 seconds, and then 
immediately jumps back.  But it never goes negative (or prior to 1970).

	ben

Re: time jumps forward/backwards

[Benjamin Gamsa]

Benjamin Gamsa wrote:
> Paul Mackerras wrote:
>> Ben Gamsa writes:
>>
>>> It appears to be the case that when the time on the system is around 
>>> the epoch (1970), that time will occasionally jump forward and then 
>>> backward by about 17592 seconds.  When it jumps forward, it always 
>>> jumps back a few milliseconds later.  However, it's not always easy 
>>> to catch these occurrences.  The delta is more specifically about 
>>> 17592186059 usec, give or take a few 10s of microseconds (most of the 
>>> time), despite the fact that the user-level program I have that is 
>>> testing it only checks every 10 milliseconds.
>>
>> I don't think the time code in the kernel is designed to handle
>> negative values, i.e., times before the epoch.  If you want it to do
>> that you'll have to check places like arch/powerpc/kernel/time.c,
>> kernel/time/timekeeping.c, arch/powerpc/include/asm/time.h, etc., and
>> make sure that it uses signed types where necessary and that the
>> arithmetic is correct.
>>
> 
> The time never goes negative.  It starts off at the epoch and moves 
> forward, but sometimes it jumps forward by 17952 seconds, and then 
> immediately jumps back.  But it never goes negative (or prior to 1970).
> 

One important thing I forgot to add is that ntpd is running on this 
system, but the ntp servers are not available.  I suspect the problem 
may be related to ntpd, even though I've seen the time jump even when I 
had ntpd stopped within gdb.  I've not yet been able to confirm if the 
problem still occurs when ntpd is never even started, although I will be 
testing that soon (the tests often require many hours to establish if 
there are no jumps).

	ben

Re: time jumps forward/backwards

[Benjamin Gamsa]

Benjamin Gamsa wrote:
> Benjamin Gamsa wrote:
>> Paul Mackerras wrote:
>>> Ben Gamsa writes:
>>>
>>>> It appears to be the case that when the time on the system is around 
>>>> the epoch (1970), that time will occasionally jump forward and then 
>>>> backward by about 17592 seconds.  When it jumps forward, it always 
>>>> jumps back a few milliseconds later.  However, it's not always easy 
>>>> to catch these occurrences.  The delta is more specifically about 
>>>> 17592186059 usec, give or take a few 10s of microseconds (most of 
>>>> the time), despite the fact that the user-level program I have that 
>>>> is testing it only checks every 10 milliseconds.
>>>
>>> I don't think the time code in the kernel is designed to handle
>>> negative values, i.e., times before the epoch.  If you want it to do
>>> that you'll have to check places like arch/powerpc/kernel/time.c,
>>> kernel/time/timekeeping.c, arch/powerpc/include/asm/time.h, etc., and
>>> make sure that it uses signed types where necessary and that the
>>> arithmetic is correct.
>>>
>>
>> The time never goes negative.  It starts off at the epoch and moves 
>> forward, but sometimes it jumps forward by 17952 seconds, and then 
>> immediately jumps back.  But it never goes negative (or prior to 1970).
>>
> 
> One important thing I forgot to add is that ntpd is running on this 
> system, but the ntp servers are not available.  I suspect the problem 
> may be related to ntpd, even though I've seen the time jump even when I 
> had ntpd stopped within gdb.  I've not yet been able to confirm if the 
> problem still occurs when ntpd is never even started, although I will be 
> testing that soon (the tests often require many hours to establish if 
> there are no jumps).

For what it's worth, the problem occurs even when ntp is not even started.

	ben

Re: time jumps forward/backwards

[Sean MacLennan]

On Mon, 31 Aug 2009 22:20:00 -0400
Benjamin Gamsa <ben@somanetworks.com> wrote:

> For what it's worth, the problem occurs even when ntp is not even
> started.

This is grasping, but could it have anything to do with the jiffies
wrapping near startup?

Cheers,
  Sean

Re: time jumps forward/backwards

[Benjamin Gamsa]

Sean MacLennan wrote:
> On Mon, 31 Aug 2009 22:20:00 -0400
> Benjamin Gamsa <ben@somanetworks.com> wrote:
> 
>> For what it's worth, the problem occurs even when ntp is not even
>> started.
> 
> This is grasping, but could it have anything to do with the jiffies
> wrapping near startup?
> 

I don't know how to test it, but I don't think so, since there are 
multiple of these glitches over an extended period of time.

	ben

Re: time jumps forward/backwards

[Benjamin Herrenschmidt]

On Mon, 2009-08-31 at 23:57 -0400, Benjamin Gamsa wrote:
> Sean MacLennan wrote:
> > On Mon, 31 Aug 2009 22:20:00 -0400
> > Benjamin Gamsa <ben@somanetworks.com> wrote:
> > 
> >> For what it's worth, the problem occurs even when ntp is not even
> >> started.
> > 
> > This is grasping, but could it have anything to do with the jiffies
> > wrapping near startup?
> > 
> 
> I don't know how to test it, but I don't think so, since there are 
> multiple of these glitches over an extended period of time.

I'm not familiar with all the FSL processor variants, but is this
an UP or an SMP platform ? In the later case, are all the core timebases
properly synchronized ?

Cheers,
Ben.

Re: time jumps forward/backwards

[Benjamin Gamsa]

Benjamin Herrenschmidt wrote:
> On Mon, 2009-08-31 at 23:57 -0400, Benjamin Gamsa wrote:
>> Sean MacLennan wrote:
>>> On Mon, 31 Aug 2009 22:20:00 -0400
>>> Benjamin Gamsa <ben@somanetworks.com> wrote:
>>>
>>>> For what it's worth, the problem occurs even when ntp is not even
>>>> started.
>>> This is grasping, but could it have anything to do with the jiffies
>>> wrapping near startup?
>>>
>> I don't know how to test it, but I don't think so, since there are 
>> multiple of these glitches over an extended period of time.
> 
> I'm not familiar with all the FSL processor variants, but is this
> an UP or an SMP platform ? In the later case, are all the core timebases
> properly synchronized ?
> 

This a UP with a single e500 core.

	ben

Re: time jumps forward/backwards

[Ben Gamsa]

Benjamin Gamsa wrote:
> Benjamin Herrenschmidt wrote:
>> On Mon, 2009-08-31 at 23:57 -0400, Benjamin Gamsa wrote:
>>> Sean MacLennan wrote:
>>>> On Mon, 31 Aug 2009 22:20:00 -0400
>>>> Benjamin Gamsa <ben@somanetworks.com> wrote:
>>>>
>>>>> For what it's worth, the problem occurs even when ntp is not even
>>>>> started.
>>>> This is grasping, but could it have anything to do with the jiffies
>>>> wrapping near startup?
>>>>
>>> I don't know how to test it, but I don't think so, since there are 
>>> multiple of these glitches over an extended period of time.
>>
>> I'm not familiar with all the FSL processor variants, but is this
>> an UP or an SMP platform ? In the later case, are all the core timebases
>> properly synchronized ?
>>
> 
> This a UP with a single e500 core.
> 

I take it from the lack of follow-ups that no one has any good ideas as 
to what might be going wrong?

Since the problem seems to be confined to situations where the date is 
around the epoch, I guess I'll just work-around the problem by setting 
the date to a more recent date on startup.

-- 
Ben Gamsa       ben@somanetworks.com
SOMA Networks   312 Adelaide St. W. Suite 600 Toronto, Ontario, M5V1R2