Re: LTP - gettimeofday02 FAIL

Re: LTP - gettimeofday02 FAIL

[Jim Houston]

Hi Everyone,

I just tried gettimeofday02 on an old pentium-pro dual processor, and yes
the time goes backwards with a 2.5.48 kernel.

I believe that this is the result of lost ticks.  It has gotten much
easier to lose a tick since HZ was changed to 1000.  When the timer
interrupt is delayed, the other processors will continue to keep reasonable
time (based on the TSC), but when the timer interrupt eventually happens,
it will add one tick's worth of nanoseconds to xtime.tv_nsec and set
last_tsc_low to the current tsc value.  The other processors now base
their time on this new last_tsc_low and  will see time go backwards.
I accidentally configured in the ACPI power management code and was
disappointed to find that it routinely caused a 9 milli-second interrupt
lock-out (on my 1GHz Athlon).  With the old 100 Hz clock, this delay would
be detected by reading the PIT timer.  With 1000 Hz, the timer would  reload
several times and all we see is a fraction of a tick.

I'm interested in this because I'm working on my "alternative Posix timers
patch".  It gets confused when time backs up.

Jim Houston - Concurrent Computer Corp.

Re: LTP - gettimeofday02 FAIL

[Andi Kleen]

Jim Houston <jim.houston@attbi.com> writes:

> I believe that this is the result of lost ticks.  It has gotten much
> easier to lose a tick since HZ was changed to 1000.  When the timer
> interrupt is delayed, the other processors will continue to keep reasonable
> time (based on the TSC), but when the timer interrupt eventually happens,
> it will add one tick's worth of nanoseconds to xtime.tv_nsec and set
> last_tsc_low to the current tsc value.  The other processors now base
> their time on this new last_tsc_low and  will see time go backwards.

It could be detected by keeping a per cpu last_tsc.

Best would be to use a global timer like HPET, but it's not available
everywhere and much slower than rdtsc too.

-Andi

Re: [LTP] Re: LTP - gettimeofday02 FAIL

[Paul Larson]

[-- Attachment #1: Type: text/plain, Size: 406 bytes --]

On Mon, 2002-11-18 at 19:27, Jim Houston wrote:
> 
> Hi Everyone,
> 
> I just tried gettimeofday02 on an old pentium-pro dual processor, and yes
> the time goes backwards with a 2.5.48 kernel.
This has been noticed, I've posted to lkml about it.  The only person
who replied to me seems to be suggesting it is a hardware issue, but I
can't believe it is impossible to work around.

-Paul Larson

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 240 bytes --]

Re: [LTP] Re: LTP - gettimeofday02 FAIL

[Dave Jones]

On Tue, Nov 19, 2002 at 07:37:23AM -0600, Paul Larson wrote:
 > > I just tried gettimeofday02 on an old pentium-pro dual processor, and yes
 > > the time goes backwards with a 2.5.48 kernel.
 > This has been noticed, I've posted to lkml about it.  The only person
 > who replied to me seems to be suggesting it is a hardware issue, but I
 > can't believe it is impossible to work around.

Especially if earlier kernels got it right..

		Dave

-- 
| Dave Jones.        http://www.codemonkey.org.uk
| SuSE Labs

Re: [LTP] Re: LTP - gettimeofday02 FAIL

[Paul Larson]

[-- Attachment #1: Type: text/plain, Size: 622 bytes --]

On Tue, 2002-11-19 at 08:02, Dave Jones wrote:
> On Tue, Nov 19, 2002 at 07:37:23AM -0600, Paul Larson wrote:
>  > > I just tried gettimeofday02 on an old pentium-pro dual processor, and yes
>  > > the time goes backwards with a 2.5.48 kernel.
>  > This has been noticed, I've posted to lkml about it.  The only person
>  > who replied to me seems to be suggesting it is a hardware issue, but I
>  > can't believe it is impossible to work around.
> 
> Especially if earlier kernels got it right..
This is bug #100 in bugme if anyone wants to track it. 
http://bugme.osdl.org/show_bug.cgi?id=100

-Paul Larson

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 240 bytes --]

Re: [LTP] Re: LTP - gettimeofday02 FAIL

[Andi Kleen]

Paul Larson <plars@linuxtestproject.org> writes:

> This has been noticed, I've posted to lkml about it.  The only person
> who replied to me seems to be suggesting it is a hardware issue, but I
> can't believe it is impossible to work around.

It is very hard to solve properly and efficiently. When you search the
list archives you will find long threads about the problem
(search for "TSC" and gettimeofday and perhaps HPET or cyclone). Last one 
was one or two weeks ago.

The problem has been there always in some way in linux, now it is just
exposed in LTP because it tests for it.

-Andi

Re: [LTP] Re: LTP - gettimeofday02 FAIL

[Alan Cox]

On Tue, 2002-11-19 at 14:24, Andi Kleen wrote:
> It is very hard to solve properly and efficiently. When you search the
> list archives you will find long threads about the problem
> (search for "TSC" and gettimeofday and perhaps HPET or cyclone). Last one 
> was one or two weeks ago.
> 
> The problem has been there always in some way in linux, now it is just
> exposed in LTP because it tests for it.

Dual ppro boxes normally run with a locked synchronous TSC clock. That
suggests the newer code broke stuff. It may also be due to the bug in
the 2.5 timer handling code (missing delays on timer reads, incorrect
assumption that the timer never reads its limit value during the timer 
switch back to zero)

Re: [LTP] Re: LTP - gettimeofday02 FAIL

[Andi Kleen]

On Tue, Nov 19, 2002 at 03:27:31PM +0000, Alan Cox wrote:
> On Tue, 2002-11-19 at 14:24, Andi Kleen wrote:
> > It is very hard to solve properly and efficiently. When you search the
> > list archives you will find long threads about the problem
> > (search for "TSC" and gettimeofday and perhaps HPET or cyclone). Last one 
> > was one or two weeks ago.
> > 
> > The problem has been there always in some way in linux, now it is just
> > exposed in LTP because it tests for it.
> 
> Dual ppro boxes normally run with a locked synchronous TSC clock. That
> suggests the newer code broke stuff. It may also be due to the bug in

It is because of the HZ=1000. See Jim Houston's mail on the same topic,
he analyzed the failure.

Basically the current code cannot handle missing ticks properly on SMP and with
the new 1ms tick it is much more likely that a timer interrupt gets lost.

-Andi

Re: [LTP] Re: LTP - gettimeofday02 FAIL

[Alan Cox]

On Tue, 2002-11-19 at 15:10, Andi Kleen wrote:
> It is because of the HZ=1000. See Jim Houston's mail on the same topic,
> he analyzed the failure.
> 
> Basically the current code cannot handle missing ticks properly on SMP and with
> the new 1ms tick it is much more likely that a timer interrupt gets lost.

Ok so add that to the other existing 2.5 timer handling bugs 8(

Re: [LTP] Re: LTP - gettimeofday02 FAIL

[Andi Kleen]

On Tue, Nov 19, 2002 at 04:12:23PM +0000, Alan Cox wrote:
> On Tue, 2002-11-19 at 15:10, Andi Kleen wrote:
> > It is because of the HZ=1000. See Jim Houston's mail on the same topic,
> > he analyzed the failure.
> > 
> > Basically the current code cannot handle missing ticks properly on SMP and with
> > the new 1ms tick it is much more likely that a timer interrupt gets lost.
> 
> Ok so add that to the other existing 2.5 timer handling bugs 8(

2.4 has the same problem, it is just much less likely to hit because of HZ=100

-Andi