System clock going slow/fast with ntpdate

System clock going slow/fast with ntpdate

[Deepak Gaur]

Hello,

I have board with MIPS 34Kc processor and linux-2.6.29 with CONFIG_HZ_250=y set in kernel configuration (i.e 250 timer
interrupts per 1 real second in /proc/interrupts). When I try to synchronize time using ntpdate command, the time gets
synchronized. This resync is being done every 5 min using cron. The clocksource is set to jiffies and is the only source
available. After some time (1 hr and more) the system clock (kernel/software) sometimes starts slowing down and sometime
goes fast. System time increments very slowly i.e 1 sec on system takes 8-9 real seconds (as per wrist watch) or fast 2
sec in 1 real second.

#date 
Tue Oct 25 15:14:05 IST 2016 
 # date 
 Tue Oct 25 15:14:05 IST 2016 
 # date 
 Tue Oct 25 15:14:05 IST 2016 
 # date 
 Tue Oct 25 15:14:05 IST 2016 
 # date 
 Tue Oct 25 15:14:05 IST 2016 
 # date 
 Tue Oct 25 15:14:05 IST 2016 
 # date 
 Tue Oct 25 15:14:05 IST 2016 
 # date 
 Tue Oct 25 15:14:05 IST 2016 
 # date 
 Tue Oct 25 15:14:05 IST 2016 
 # date 
 Tue Oct 25 15:14:06 IST 2016 
 # date 
 Tue Oct 25 15:14:06 IST 2016

(It took 10 date commands to increment from 14:05 to 15:06)

On further analysing the system we found the number of timer interrupts in /proc/interrupts has actually gone down from
250 to 40 every 1 real second

(1) Normal Operation 10 real sec watch window

cat /proc/interrupts

Start of timer  MIPS timer intr count 3856633 
 End of Timer   MIPS timer intr count 3859268

 Timing approximately 10-11 real sec, the interrupts are 263 per sec.

(2) Clock Slow (Less Timer Interrupts)
After ntpdate run for 1 hr once per 5 min

cat /proc/interrupts

 Start of timer  MIPS timer intr count 985072 
 End of Timer   MIPS timer intr count 985492

 985492 - 985072 = 420 in 10 sec (real) = 42 in 1 sec

(3) Fast Clock with ntpdate (More Timer interrupts)

Start of timer  MIPS timer intr count 4068301 
End of Timer   MIPS timer intr count 4073411 
 985492 - 985072 = 5110 in 10 sec (real) = 511 in 1 real sec

ntpdate uses ntp_adjtime()/adjtimex() GNU libc system call for changing system clock and can change it but by a very
small amount.

But the issue is it is changing system clock so much that other application have started behaving erratically, timers
are not expiring in required time etc.. and once the system clock has slowed down/fast it remains in that state.

# cat /sys/devices/system/clocksource/clocksource0/available_clocksource 
 jiffies 
  
 # cat /sys/devices/system/clocksource/clocksource0/current_clocksource   
 jiffies

We are using gcc version 4.5.2  and gnu libc

Please help me in understanding this behaviour of NTP with MIPS Linux and possible fixes if any. Is it a kernel bug or a
configuration issue?

regards,

Deepak Gaur

System clock going slow/fast with ntpdate

[Deepak Gaur]

Hello,

 I have board with MIPS 34Kc processor and linux-2.6.29 with CONFIG_HZ_250=y set in kernel configuration (i.e 250 timer 
 interrupts per 1 real second in /proc/interrupts). When I try to synchronize time using ntpdate command, the time gets 
 synchronized. This resync is being done every 5 min using cron. The clocksource is set to jiffies and is the only source 
 available. After some time (1 hr and more) the system clock (kernel/software) sometimes starts slowing down and sometime 
 goes fast. System time increments very slowly i.e 1 sec on system takes 8-9 real seconds (as per wrist watch) or fast 2 
 sec in 1 real second.

 #date 
 Tue Oct 25 15:14:05 IST 2016 
 # date 
 Tue Oct 25 15:14:05 IST 2016 
 # date 
 Tue Oct 25 15:14:05 IST 2016 
 # date 
 Tue Oct 25 15:14:05 IST 2016 
 # date 
 Tue Oct 25 15:14:05 IST 2016 
 # date 
 Tue Oct 25 15:14:05 IST 2016 
 # date 
 Tue Oct 25 15:14:05 IST 2016 
 # date 
 Tue Oct 25 15:14:05 IST 2016 
 # date 
 Tue Oct 25 15:14:05 IST 2016 
 # date 
 Tue Oct 25 15:14:06 IST 2016 
 # date 
 Tue Oct 25 15:14:06 IST 2016

 (It took 10 date commands to increment from 14:05 to 15:06)

 On further analysing the system we found the number of timer interrupts in /proc/interrupts has actually gone down from 
 250 to 40 every 1 real second

 (1) Normal Operation 10 real sec watch window

 cat /proc/interrupts

 Start of timer  MIPS timer intr count 3856633 
 End of Timer   MIPS timer intr count 3859268

 Timing approximately 10-11 real sec, the interrupts are 263 per sec.

 (2) Clock Slow (Less Timer Interrupts) 
 After ntpdate run for 1 hr once per 5 min

 cat /proc/interrupts

 Start of timer  MIPS timer intr count 985072 
 End of Timer   MIPS timer intr count 985492

 985492 - 985072 = 420 in 10 sec (real) = 42 in 1 sec

 (3) Fast Clock with ntpdate (More Timer interrupts)

 Start of timer  MIPS timer intr count 4068301 
 End of Timer   MIPS timer intr count 4073411 
 985492 - 985072 = 5110 in 10 sec (real) = 511 in 1 real sec

 ntpdate uses ntp_adjtime()/adjtimex() GNU libc system call for changing system clock and can change it but by a very 
 small amount.

 But the issue is it is changing system clock so much that other application have started behaving erratically, timers 
 are not expiring in required time etc.. and once the system clock has slowed down/fast it remains in that state.

 # cat /sys/devices/system/clocksource/clocksource0/available_clocksource 
 jiffies 
   
 # cat /sys/devices/system/clocksource/clocksource0/current_clocksource   
 jiffies

 We are using gcc version 4.5.2  and gnu libc

 Please help me in understanding this behaviour of NTP with MIPS Linux and possible fixes if any. Is it a kernel bug or a 
 configuration issue?

 regards,

 Deepak Gaur

Re: System clock going slow/fast with ntpdate

[Maciej W. Rozycki]

On Wed, 26 Oct 2016, Deepak Gaur wrote:

> I have board with MIPS 34Kc processor and linux-2.6.29 with 
> CONFIG_HZ_250=y set in kernel configuration (i.e 250 timer interrupts 
> per 1 real second in /proc/interrupts).
[...]
> Please help me in understanding this behaviour of NTP with MIPS Linux 
> and possible fixes if any. Is it a kernel bug or a configuration issue?

 Your Linux kernel is very old, version 2.6.29 was released over 8 years 
ago.  Please try a recent kernel such as 4.8.0 instead so that you don't 
miss all the important fixes and updates.

 If you find yourself still having issues with an updated kernel, then 
please come back and we'll try to help.

  Maciej

Re: System clock going slow/fast with ntpdate

[Maciej W. Rozycki]

On Wed, 26 Oct 2016, Deepak Gaur wrote:

> I have board with MIPS 34Kc processor and linux-2.6.29 with 
> CONFIG_HZ_250=y set in kernel configuration (i.e 250 timer interrupts 
> per 1 real second in /proc/interrupts).
[...]
> Please help me in understanding this behaviour of NTP with MIPS Linux 
> and possible fixes if any. Is it a kernel bug or a configuration issue?

 Your Linux kernel is very old, version 2.6.29 was released over 8 years 
ago.  Please try a recent kernel such as 4.8.0 instead so that you don't 
miss all the important fixes and updates.

 If you find yourself still having issues with an updated kernel, then 
please come back and we'll try to help.

  Maciej

Re: System clock going slow/fast with ntpdate

[Markus Gothe]

[-- Attachment #1: Type: text/plain, Size: 3087 bytes --]

You should calculate your boards time drift and compensate for it.

//Markus - The panama-hat hacker

On 26 Oct 2016, at 10:55 , Deepak Gaur <dgaur@cdot.in> wrote:

> Hello,
> 
> I have board with MIPS 34Kc processor and linux-2.6.29 with CONFIG_HZ_250=y set in kernel configuration (i.e 250 timer
> interrupts per 1 real second in /proc/interrupts). When I try to synchronize time using ntpdate command, the time gets
> synchronized. This resync is being done every 5 min using cron. The clocksource is set to jiffies and is the only source
> available. After some time (1 hr and more) the system clock (kernel/software) sometimes starts slowing down and sometime
> goes fast. System time increments very slowly i.e 1 sec on system takes 8-9 real seconds (as per wrist watch) or fast 2
> sec in 1 real second.
> 
> #date
> Tue Oct 25 15:14:05 IST 2016
> # date
> Tue Oct 25 15:14:05 IST 2016
> # date
> Tue Oct 25 15:14:05 IST 2016
> # date
> Tue Oct 25 15:14:05 IST 2016
> # date
> Tue Oct 25 15:14:05 IST 2016
> # date
> Tue Oct 25 15:14:05 IST 2016
> # date
> Tue Oct 25 15:14:05 IST 2016
> # date
> Tue Oct 25 15:14:05 IST 2016
> # date
> Tue Oct 25 15:14:05 IST 2016
> # date
> Tue Oct 25 15:14:06 IST 2016
> # date
> Tue Oct 25 15:14:06 IST 2016
> 
> (It took 10 date commands to increment from 14:05 to 15:06)
> 
> On further analysing the system we found the number of timer interrupts in /proc/interrupts has actually gone down from
> 250 to 40 every 1 real second
> 
> (1) Normal Operation 10 real sec watch window
> 
> cat /proc/interrupts
> 
> Start of timer  MIPS timer intr count 3856633
> End of Timer   MIPS timer intr count 3859268
> 
> Timing approximately 10-11 real sec, the interrupts are 263 per sec.
> 
> (2) Clock Slow (Less Timer Interrupts)
> After ntpdate run for 1 hr once per 5 min
> 
> cat /proc/interrupts
> 
> Start of timer  MIPS timer intr count 985072
> End of Timer   MIPS timer intr count 985492
> 
> 985492 - 985072 = 420 in 10 sec (real) = 42 in 1 sec
> 
> (3) Fast Clock with ntpdate (More Timer interrupts)
> 
> Start of timer  MIPS timer intr count 4068301
> End of Timer   MIPS timer intr count 4073411
> 985492 - 985072 = 5110 in 10 sec (real) = 511 in 1 real sec
> 
> ntpdate uses ntp_adjtime()/adjtimex() GNU libc system call for changing system clock and can change it but by a very
> small amount.
> 
> But the issue is it is changing system clock so much that other application have started behaving erratically, timers
> are not expiring in required time etc.. and once the system clock has slowed down/fast it remains in that state.
> 
> # cat /sys/devices/system/clocksource/clocksource0/available_clocksource
> jiffies
> 
> # cat /sys/devices/system/clocksource/clocksource0/current_clocksource
> jiffies
> 
> We are using gcc version 4.5.2  and gnu libc
> 
> Please help me in understanding this behaviour of NTP with MIPS Linux and possible fixes if any. Is it a kernel bug or a
> configuration issue?
> 
> regards,
> 
> Deepak Gaur
> 
> 


[-- Attachment #2: Message signed with OpenPGP using GPGMail --]
[-- Type: application/pgp-signature, Size: 186 bytes --]

Re: System clock going slow/fast with ntpdate

[Maciej W. Rozycki]

On Tue, 1 Nov 2016, Markus Gothe wrote:

> You should calculate your boards time drift and compensate for it.

 Well it can't help if the clock rate changes spontaneously, can it?

  Maciej

Re: System clock going slow/fast with ntpdate

[Maciej W. Rozycki]

On Tue, 1 Nov 2016, Markus Gothe wrote:

> You should calculate your boards time drift and compensate for it.

 Well it can't help if the clock rate changes spontaneously, can it?

  Maciej

Re: System clock going slow/fast with ntpdate

[Markus Gothe]

[-- Attachment #1: Type: text/plain, Size: 575 bytes --]

I doubt that it changes spontaneously.

AFAIU this is related to NTP (the issue only shows up when setting time with ntpdate), and
time drift is a know circumstance which must be mitigated as described by Larry Doolittle and his rate.awk-scrip.

On 01 Nov 2016, at 20:04 , Maciej W. Rozycki <macro@imgtec.com> wrote:

> On Tue, 1 Nov 2016, Markus Gothe wrote:
> 
>> You should calculate your boards time drift and compensate for it.
> 
> Well it can't help if the clock rate changes spontaneously, can it?
> 
>  Maciej
> 

//Markus - The panama-hat hacker


[-- Attachment #2: Message signed with OpenPGP using GPGMail --]
[-- Type: application/pgp-signature, Size: 186 bytes --]

Re: System clock going slow/fast with ntpdate

[Maciej W. Rozycki]

On Tue, 1 Nov 2016, Markus Gothe wrote:

> I doubt that it changes spontaneously.
> 
> AFAIU this is related to NTP (the issue only shows up when setting time with ntpdate), and
> time drift is a know circumstance which must be mitigated as described by Larry Doolittle and his rate.awk-scrip.

 You mean you can compensate when suddenly 1 second of system time is only 
elapsed after 8-9 wall clock seconds?  I seriously doubt it.

 What it looks like to me is some sort of an interrupt delivery or 
reprogramming issue with the CP0 Count/Compare timer, which as you may 
recall is a one-shot event and needs rearming.  But it's not worth looking 
into with a kernel which is 8 years old.  It may well have been the time 
around which our CP0 timer suport was converted to the HPT framework and 
new code could have suffered from infancy bugs.

 Switching to another clock source might be a workaround if available with 
the system being used, e.g. the Malta board has an 8254 PIT clone in the 
south bridge wired to an interrupt line, which can be used in its periodic 
mode.  I'm not sure offhand if the PIT clock source is configured though 
in a pristine Malta kernel; it may well be that nobody cared to arrange 
that, given the arcane programming interface and the limited resolution of 
the PIT and the general availability of the CP0 timer.  And then we might 
not be talking about a Malta anyway.

  Maciej

Re: System clock going slow/fast with ntpdate

[Maciej W. Rozycki]

On Tue, 1 Nov 2016, Markus Gothe wrote:

> I doubt that it changes spontaneously.
> 
> AFAIU this is related to NTP (the issue only shows up when setting time with ntpdate), and
> time drift is a know circumstance which must be mitigated as described by Larry Doolittle and his rate.awk-scrip.

 You mean you can compensate when suddenly 1 second of system time is only 
elapsed after 8-9 wall clock seconds?  I seriously doubt it.

 What it looks like to me is some sort of an interrupt delivery or 
reprogramming issue with the CP0 Count/Compare timer, which as you may 
recall is a one-shot event and needs rearming.  But it's not worth looking 
into with a kernel which is 8 years old.  It may well have been the time 
around which our CP0 timer suport was converted to the HPT framework and 
new code could have suffered from infancy bugs.

 Switching to another clock source might be a workaround if available with 
the system being used, e.g. the Malta board has an 8254 PIT clone in the 
south bridge wired to an interrupt line, which can be used in its periodic 
mode.  I'm not sure offhand if the PIT clock source is configured though 
in a pristine Malta kernel; it may well be that nobody cared to arrange 
that, given the arcane programming interface and the limited resolution of 
the PIT and the general availability of the CP0 timer.  And then we might 
not be talking about a Malta anyway.

  Maciej