[Xenomai-help] clock drift

[Xenomai-help] clock drift

[Rob Wheeler]

Hello,

I was recently debugging a problem in our code base and tracked the
issue down to differences between calling gettimeofday() from our
non-realtime code and calling clock_gettime() from our realtime
control loop.  A little poking around with Google led me to clocktest
program in the Xenomai testsuite.  We are routinely seeing differences
of around 4 minutes (~240 seconds) between gettimeofday() and
clock_gettime() immediately after a reboot.  The clocks continue to
drift apart (or together) at the rate of about 50us/s.

My question is, is this large of a disparity between clocks normal?
Are there any tricks or techniques for getting the clocks to be
synchronized?  In our application we have a realtime control loop
reading data from motor control boards at a rate of 1KHz.  The data is
timestamped by the realtime loop.  The non-realtime code can doing
more heavyweight computation on that data.  It uses the timestamp to
correlate the data with other data read from non-realtime sensors
(cameras, laser scanners, etc...).  Some of this data may even be
collected on other computers running on the robot (and connected via
gigabit Ethernet).  What do others do to synchronize time between
realtime, non-realtime, and other network attached computers?

A little more about my setup:
  kernel - 2.6.24.3
  xenomai - v2.4.5
  adeos patch - adeos-ipipe-2.6.24-x86-2.0-07.patch
  processor - 2.5 GHz Core 2 Duo

Thanks for any info you can provide...

-R

Re: [Xenomai-help] clock drift

[Gilles Chanteperdrix]

Rob Wheeler wrote:
> Hello,
> 
> I was recently debugging a problem in our code base and tracked the
> issue down to differences between calling gettimeofday() from our
> non-realtime code and calling clock_gettime() from our realtime
> control loop.  A little poking around with Google led me to clocktest
> program in the Xenomai testsuite.  We are routinely seeing differences
> of around 4 minutes (~240 seconds) between gettimeofday() and
> clock_gettime() immediately after a reboot.  The clocks continue to
> drift apart (or together) at the rate of about 50us/s.
> 
> My question is, is this large of a disparity between clocks normal?
> Are there any tricks or techniques for getting the clocks to be
> synchronized?  In our application we have a realtime control loop
> reading data from motor control boards at a rate of 1KHz.  The data is
> timestamped by the realtime loop.  The non-realtime code can doing
> more heavyweight computation on that data.  It uses the timestamp to
> correlate the data with other data read from non-realtime sensors
> (cameras, laser scanners, etc...).  Some of this data may even be
> collected on other computers running on the robot (and connected via
> gigabit Ethernet).  What do others do to synchronize time between
> realtime, non-realtime, and other network attached computers?
> 
> A little more about my setup:
>   kernel - 2.6.24.3
>   xenomai - v2.4.5
>   adeos patch - adeos-ipipe-2.6.24-x86-2.0-07.patch
>   processor - 2.5 GHz Core 2 Duo
> 
> Thanks for any info you can provide...

Xenomai has no (not yet, this may come) way to correct the clock drift
progressively, like the linux kernel does when using NTP, however, since
version 2.4.x, Xenomai supports setting the clock with clock_settime.
So, as a workaround, you can call clock_settime periodically to
resynchronize Xenomai clock with Linux clock.

Pay attention however: setting the clock has an influence on timers
(like the posix spec asks), so when doing this, you have to carefully
choose whether your timers use CLOCK_REALTIME or CLOCK_MONOTONIC and
whether they are absolute or relative.

http://www.opengroup.org/onlinepubs/000095399/functions/clock_settime.html

-- 
					    Gilles.

Re: [Xenomai-help] clock drift

[Gilles Chanteperdrix]

Rob Wheeler wrote:
> Gilles,

Rob,

please do not drop the list from the CC, the conversation may interest
other people.

> 
> Thanks for the FAST response.  I'm currently calling gettimeofday()
> and clock_settime() one time just prior to creating my realtime
> thread.
> 
> My goal is to have a thread that wakes up every 1ms and performs some
> work and goes back to sleep until the next 1ms interval.  My current
> implementation calls clock_gettime() initially, and then sits in a
> while (1) loop that does some work and then calls
> clock_nanosleep(CLOCK_REALTIME, TIMER_ABSTIME, ...) with a timespec
> exactly 1ms since the last sleep.
> 
> Given my application, is your recommendation that I switch to using
> the CLOCK_MONOTONIC in my realtime loop and then have a non-realtime
> thread which basically does:
> 
> while (1) {
>   gettimeofday()
>   clock_settime(CLOCK_REALTIME)
>   sleep for a bit
> }
> 
> so that I can get close to synchronized time in my realtime code?

Yes.

> 
> Also, just for my education, is 50us/s a typical clock drift value?
> What exactly is the source of this drift and can anything be done to
> reduce the amount of drift?

I have no idea if this is normal. Xenomai uses the tsc as a clock
source. Linux may choose to use something else than the tsc (you should
see messages in the logs if it does), it may correct the tsc clock drift
if you use NTP, and since Xenomai does not correct the drift, you
observe a difference.

-- 
					    Gilles.

Re: [Xenomai-help] clock drift

[Theo Veenker]

Gilles Chanteperdrix wrote:
> Rob Wheeler wrote:
>> Gilles,
> 
> Rob,
> 
> please do not drop the list from the CC, the conversation may interest
> other people.

Yes indeed!

> 
>> Thanks for the FAST response.  I'm currently calling gettimeofday()
>> and clock_settime() one time just prior to creating my realtime
>> thread.
>>
>> My goal is to have a thread that wakes up every 1ms and performs some
>> work and goes back to sleep until the next 1ms interval.  My current
>> implementation calls clock_gettime() initially, and then sits in a
>> while (1) loop that does some work and then calls
>> clock_nanosleep(CLOCK_REALTIME, TIMER_ABSTIME, ...) with a timespec
>> exactly 1ms since the last sleep.
>>
>> Given my application, is your recommendation that I switch to using
>> the CLOCK_MONOTONIC in my realtime loop and then have a non-realtime
>> thread which basically does:
>>
>> while (1) {
>>   gettimeofday()
>>   clock_settime(CLOCK_REALTIME)
>>   sleep for a bit
>> }
>>
>> so that I can get close to synchronized time in my realtime code?
> 
> Yes.
> 
>> Also, just for my education, is 50us/s a typical clock drift value?
>> What exactly is the source of this drift and can anything be done to
>> reduce the amount of drift?
> 
> I have no idea if this is normal. Xenomai uses the tsc as a clock
> source. Linux may choose to use something else than the tsc (you should
> see messages in the logs if it does), it may correct the tsc clock drift
> if you use NTP, and since Xenomai does not correct the drift, you
> observe a difference.

I generally get 152us/s on one system (Asus P4PE/GE) and also 152us/s on
another system (MSI 945P). For the first system I remember (years ago)
seeing the same drift under RTLinux. On a laptop (Fusitsu-Siemens S7110)
the drift somehow converges to zero. I don't use NTP, just ntpdate once
at night.

May ask a (probably silly) question. Does clock_get/settime(clock_id) in
/usr/include/time.h and in .../xenomai/include/posix/time.h refer to exactly
the same clock? Or do these interfaces just refer to similar clocks.

Theo

Re: [Xenomai-help] clock drift

[Gilles Chanteperdrix]

Theo Veenker wrote:
> May ask a (probably silly) question. Does clock_get/settime(clock_id) in
> /usr/include/time.h and in .../xenomai/include/posix/time.h refer to exactly
> the same clock? Or do these interfaces just refer to similar clocks.

The timing subsystems of Xenomai and Linux are different, so they refer
to similar clocks.

Xenomai's implementation of clock_gettime(CLOCK_MONOTONIC) reads the tsc
(which does not need a syscall on most architectures) and converts it to
a struct timespec, whereas the implementation of
clock_gettime(CLOCK_REALTIME) issues a syscall which reads the tsc, adds
an offset, and convert it to a struct timespec.
clock_settime(CLOCK_REALTIME) adjusts the offset used by
clock_gettime(CLOCK_REALTIME) and runs some timers.

-- 
                                                 Gilles.

Re: [Xenomai-help] clock drift

[Gilles Chanteperdrix]

Rob Wheeler wrote:
> Hello,
> 
> I was recently debugging a problem in our code base and tracked the
> issue down to differences between calling gettimeofday() from our
> non-realtime code and calling clock_gettime() from our realtime
> control loop.  A little poking around with Google led me to clocktest
> program in the Xenomai testsuite.  We are routinely seeing differences
> of around 4 minutes (~240 seconds) between gettimeofday() and
> clock_gettime() immediately after a reboot.  The clocks continue to
> drift apart (or together) at the rate of about 50us/s.

Hi Rob,

when chasing unrelated issues, I found what could be the cause of the
original 4 minutes difference. The bug is fixed in the svn repository. I
think it goes in the "very stupid bugs" category.

Well, tooks only 3 monthes...

Regards.

-- 
					    Gilles.