* System clock going slow/fast with ntpdate
2016-10-26 8:55 ` System clock going slow/fast with ntpdate Deepak Gaur
@ 2016-10-26 9:01 ` Deepak Gaur
2016-11-01 13:49 ` Maciej W. Rozycki
2016-11-01 16:14 ` Markus Gothe
2 siblings, 0 replies; 10+ messages in thread
From: Deepak Gaur @ 2016-10-26 9:01 UTC (permalink / raw)
To: linux-mips
Hello,
I have board with MIPS 34Kc processor and linux-2.6.29 with CONFIG_HZ_250=y set in kernel configuration (i.e 250 timer
interrupts per 1 real second in /proc/interrupts). When I try to synchronize time using ntpdate command, the time gets
synchronized. This resync is being done every 5 min using cron. The clocksource is set to jiffies and is the only source
available. After some time (1 hr and more) the system clock (kernel/software) sometimes starts slowing down and sometime
goes fast. System time increments very slowly i.e 1 sec on system takes 8-9 real seconds (as per wrist watch) or fast 2
sec in 1 real second.
#date
Tue Oct 25 15:14:05 IST 2016
# date
Tue Oct 25 15:14:05 IST 2016
# date
Tue Oct 25 15:14:05 IST 2016
# date
Tue Oct 25 15:14:05 IST 2016
# date
Tue Oct 25 15:14:05 IST 2016
# date
Tue Oct 25 15:14:05 IST 2016
# date
Tue Oct 25 15:14:05 IST 2016
# date
Tue Oct 25 15:14:05 IST 2016
# date
Tue Oct 25 15:14:05 IST 2016
# date
Tue Oct 25 15:14:06 IST 2016
# date
Tue Oct 25 15:14:06 IST 2016
(It took 10 date commands to increment from 14:05 to 15:06)
On further analysing the system we found the number of timer interrupts in /proc/interrupts has actually gone down from
250 to 40 every 1 real second
(1) Normal Operation 10 real sec watch window
cat /proc/interrupts
Start of timer MIPS timer intr count 3856633
End of Timer MIPS timer intr count 3859268
Timing approximately 10-11 real sec, the interrupts are 263 per sec.
(2) Clock Slow (Less Timer Interrupts)
After ntpdate run for 1 hr once per 5 min
cat /proc/interrupts
Start of timer MIPS timer intr count 985072
End of Timer MIPS timer intr count 985492
985492 - 985072 = 420 in 10 sec (real) = 42 in 1 sec
(3) Fast Clock with ntpdate (More Timer interrupts)
Start of timer MIPS timer intr count 4068301
End of Timer MIPS timer intr count 4073411
985492 - 985072 = 5110 in 10 sec (real) = 511 in 1 real sec
ntpdate uses ntp_adjtime()/adjtimex() GNU libc system call for changing system clock and can change it but by a very
small amount.
But the issue is it is changing system clock so much that other application have started behaving erratically, timers
are not expiring in required time etc.. and once the system clock has slowed down/fast it remains in that state.
# cat /sys/devices/system/clocksource/clocksource0/available_clocksource
jiffies
# cat /sys/devices/system/clocksource/clocksource0/current_clocksource
jiffies
We are using gcc version 4.5.2 and gnu libc
Please help me in understanding this behaviour of NTP with MIPS Linux and possible fixes if any. Is it a kernel bug or a
configuration issue?
regards,
Deepak Gaur
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: System clock going slow/fast with ntpdate
@ 2016-11-01 13:49 ` Maciej W. Rozycki
0 siblings, 0 replies; 10+ messages in thread
From: Maciej W. Rozycki @ 2016-11-01 13:49 UTC (permalink / raw)
To: Deepak Gaur; +Cc: linux-mips
On Wed, 26 Oct 2016, Deepak Gaur wrote:
> I have board with MIPS 34Kc processor and linux-2.6.29 with
> CONFIG_HZ_250=y set in kernel configuration (i.e 250 timer interrupts
> per 1 real second in /proc/interrupts).
[...]
> Please help me in understanding this behaviour of NTP with MIPS Linux
> and possible fixes if any. Is it a kernel bug or a configuration issue?
Your Linux kernel is very old, version 2.6.29 was released over 8 years
ago. Please try a recent kernel such as 4.8.0 instead so that you don't
miss all the important fixes and updates.
If you find yourself still having issues with an updated kernel, then
please come back and we'll try to help.
Maciej
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: System clock going slow/fast with ntpdate
@ 2016-11-01 13:49 ` Maciej W. Rozycki
0 siblings, 0 replies; 10+ messages in thread
From: Maciej W. Rozycki @ 2016-11-01 13:49 UTC (permalink / raw)
To: Deepak Gaur; +Cc: linux-mips
On Wed, 26 Oct 2016, Deepak Gaur wrote:
> I have board with MIPS 34Kc processor and linux-2.6.29 with
> CONFIG_HZ_250=y set in kernel configuration (i.e 250 timer interrupts
> per 1 real second in /proc/interrupts).
[...]
> Please help me in understanding this behaviour of NTP with MIPS Linux
> and possible fixes if any. Is it a kernel bug or a configuration issue?
Your Linux kernel is very old, version 2.6.29 was released over 8 years
ago. Please try a recent kernel such as 4.8.0 instead so that you don't
miss all the important fixes and updates.
If you find yourself still having issues with an updated kernel, then
please come back and we'll try to help.
Maciej
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: System clock going slow/fast with ntpdate
2016-10-26 8:55 ` System clock going slow/fast with ntpdate Deepak Gaur
2016-10-26 9:01 ` Deepak Gaur
2016-11-01 13:49 ` Maciej W. Rozycki
@ 2016-11-01 16:14 ` Markus Gothe
2016-11-01 19:04 ` Maciej W. Rozycki
2 siblings, 1 reply; 10+ messages in thread
From: Markus Gothe @ 2016-11-01 16:14 UTC (permalink / raw)
To: Deepak Gaur, linux-mips
[-- Attachment #1: Type: text/plain, Size: 3087 bytes --]
You should calculate your boards time drift and compensate for it.
//Markus - The panama-hat hacker
On 26 Oct 2016, at 10:55 , Deepak Gaur <dgaur@cdot.in> wrote:
> Hello,
>
> I have board with MIPS 34Kc processor and linux-2.6.29 with CONFIG_HZ_250=y set in kernel configuration (i.e 250 timer
> interrupts per 1 real second in /proc/interrupts). When I try to synchronize time using ntpdate command, the time gets
> synchronized. This resync is being done every 5 min using cron. The clocksource is set to jiffies and is the only source
> available. After some time (1 hr and more) the system clock (kernel/software) sometimes starts slowing down and sometime
> goes fast. System time increments very slowly i.e 1 sec on system takes 8-9 real seconds (as per wrist watch) or fast 2
> sec in 1 real second.
>
> #date
> Tue Oct 25 15:14:05 IST 2016
> # date
> Tue Oct 25 15:14:05 IST 2016
> # date
> Tue Oct 25 15:14:05 IST 2016
> # date
> Tue Oct 25 15:14:05 IST 2016
> # date
> Tue Oct 25 15:14:05 IST 2016
> # date
> Tue Oct 25 15:14:05 IST 2016
> # date
> Tue Oct 25 15:14:05 IST 2016
> # date
> Tue Oct 25 15:14:05 IST 2016
> # date
> Tue Oct 25 15:14:05 IST 2016
> # date
> Tue Oct 25 15:14:06 IST 2016
> # date
> Tue Oct 25 15:14:06 IST 2016
>
> (It took 10 date commands to increment from 14:05 to 15:06)
>
> On further analysing the system we found the number of timer interrupts in /proc/interrupts has actually gone down from
> 250 to 40 every 1 real second
>
> (1) Normal Operation 10 real sec watch window
>
> cat /proc/interrupts
>
> Start of timer MIPS timer intr count 3856633
> End of Timer MIPS timer intr count 3859268
>
> Timing approximately 10-11 real sec, the interrupts are 263 per sec.
>
> (2) Clock Slow (Less Timer Interrupts)
> After ntpdate run for 1 hr once per 5 min
>
> cat /proc/interrupts
>
> Start of timer MIPS timer intr count 985072
> End of Timer MIPS timer intr count 985492
>
> 985492 - 985072 = 420 in 10 sec (real) = 42 in 1 sec
>
> (3) Fast Clock with ntpdate (More Timer interrupts)
>
> Start of timer MIPS timer intr count 4068301
> End of Timer MIPS timer intr count 4073411
> 985492 - 985072 = 5110 in 10 sec (real) = 511 in 1 real sec
>
> ntpdate uses ntp_adjtime()/adjtimex() GNU libc system call for changing system clock and can change it but by a very
> small amount.
>
> But the issue is it is changing system clock so much that other application have started behaving erratically, timers
> are not expiring in required time etc.. and once the system clock has slowed down/fast it remains in that state.
>
> # cat /sys/devices/system/clocksource/clocksource0/available_clocksource
> jiffies
>
> # cat /sys/devices/system/clocksource/clocksource0/current_clocksource
> jiffies
>
> We are using gcc version 4.5.2 and gnu libc
>
> Please help me in understanding this behaviour of NTP with MIPS Linux and possible fixes if any. Is it a kernel bug or a
> configuration issue?
>
> regards,
>
> Deepak Gaur
>
>
[-- Attachment #2: Message signed with OpenPGP using GPGMail --]
[-- Type: application/pgp-signature, Size: 186 bytes --]
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: System clock going slow/fast with ntpdate
@ 2016-11-01 19:04 ` Maciej W. Rozycki
0 siblings, 0 replies; 10+ messages in thread
From: Maciej W. Rozycki @ 2016-11-01 19:04 UTC (permalink / raw)
To: Markus Gothe; +Cc: Deepak Gaur, linux-mips
On Tue, 1 Nov 2016, Markus Gothe wrote:
> You should calculate your boards time drift and compensate for it.
Well it can't help if the clock rate changes spontaneously, can it?
Maciej
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: System clock going slow/fast with ntpdate
@ 2016-11-01 19:04 ` Maciej W. Rozycki
0 siblings, 0 replies; 10+ messages in thread
From: Maciej W. Rozycki @ 2016-11-01 19:04 UTC (permalink / raw)
To: Markus Gothe; +Cc: Deepak Gaur, linux-mips
On Tue, 1 Nov 2016, Markus Gothe wrote:
> You should calculate your boards time drift and compensate for it.
Well it can't help if the clock rate changes spontaneously, can it?
Maciej
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: System clock going slow/fast with ntpdate
2016-11-01 19:04 ` Maciej W. Rozycki
(?)
@ 2016-11-01 19:40 ` Markus Gothe
2016-11-01 20:42 ` Maciej W. Rozycki
-1 siblings, 1 reply; 10+ messages in thread
From: Markus Gothe @ 2016-11-01 19:40 UTC (permalink / raw)
To: Maciej W. Rozycki; +Cc: Deepak Gaur, linux-mips
[-- Attachment #1: Type: text/plain, Size: 575 bytes --]
I doubt that it changes spontaneously.
AFAIU this is related to NTP (the issue only shows up when setting time with ntpdate), and
time drift is a know circumstance which must be mitigated as described by Larry Doolittle and his rate.awk-scrip.
On 01 Nov 2016, at 20:04 , Maciej W. Rozycki <macro@imgtec.com> wrote:
> On Tue, 1 Nov 2016, Markus Gothe wrote:
>
>> You should calculate your boards time drift and compensate for it.
>
> Well it can't help if the clock rate changes spontaneously, can it?
>
> Maciej
>
//Markus - The panama-hat hacker
[-- Attachment #2: Message signed with OpenPGP using GPGMail --]
[-- Type: application/pgp-signature, Size: 186 bytes --]
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: System clock going slow/fast with ntpdate
@ 2016-11-01 20:42 ` Maciej W. Rozycki
0 siblings, 0 replies; 10+ messages in thread
From: Maciej W. Rozycki @ 2016-11-01 20:42 UTC (permalink / raw)
To: Markus Gothe; +Cc: Deepak Gaur, linux-mips
On Tue, 1 Nov 2016, Markus Gothe wrote:
> I doubt that it changes spontaneously.
>
> AFAIU this is related to NTP (the issue only shows up when setting time with ntpdate), and
> time drift is a know circumstance which must be mitigated as described by Larry Doolittle and his rate.awk-scrip.
You mean you can compensate when suddenly 1 second of system time is only
elapsed after 8-9 wall clock seconds? I seriously doubt it.
What it looks like to me is some sort of an interrupt delivery or
reprogramming issue with the CP0 Count/Compare timer, which as you may
recall is a one-shot event and needs rearming. But it's not worth looking
into with a kernel which is 8 years old. It may well have been the time
around which our CP0 timer suport was converted to the HPT framework and
new code could have suffered from infancy bugs.
Switching to another clock source might be a workaround if available with
the system being used, e.g. the Malta board has an 8254 PIT clone in the
south bridge wired to an interrupt line, which can be used in its periodic
mode. I'm not sure offhand if the PIT clock source is configured though
in a pristine Malta kernel; it may well be that nobody cared to arrange
that, given the arcane programming interface and the limited resolution of
the PIT and the general availability of the CP0 timer. And then we might
not be talking about a Malta anyway.
Maciej
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: System clock going slow/fast with ntpdate
@ 2016-11-01 20:42 ` Maciej W. Rozycki
0 siblings, 0 replies; 10+ messages in thread
From: Maciej W. Rozycki @ 2016-11-01 20:42 UTC (permalink / raw)
To: Markus Gothe; +Cc: Deepak Gaur, linux-mips
On Tue, 1 Nov 2016, Markus Gothe wrote:
> I doubt that it changes spontaneously.
>
> AFAIU this is related to NTP (the issue only shows up when setting time with ntpdate), and
> time drift is a know circumstance which must be mitigated as described by Larry Doolittle and his rate.awk-scrip.
You mean you can compensate when suddenly 1 second of system time is only
elapsed after 8-9 wall clock seconds? I seriously doubt it.
What it looks like to me is some sort of an interrupt delivery or
reprogramming issue with the CP0 Count/Compare timer, which as you may
recall is a one-shot event and needs rearming. But it's not worth looking
into with a kernel which is 8 years old. It may well have been the time
around which our CP0 timer suport was converted to the HPT framework and
new code could have suffered from infancy bugs.
Switching to another clock source might be a workaround if available with
the system being used, e.g. the Malta board has an 8254 PIT clone in the
south bridge wired to an interrupt line, which can be used in its periodic
mode. I'm not sure offhand if the PIT clock source is configured though
in a pristine Malta kernel; it may well be that nobody cared to arrange
that, given the arcane programming interface and the limited resolution of
the PIT and the general availability of the CP0 timer. And then we might
not be talking about a Malta anyway.
Maciej
^ permalink raw reply [flat|nested] 10+ messages in thread