GLIBC IA64 HP_TIMING fixes for SMP systems with unsynchronized ITC

GLIBC IA64 HP_TIMING fixes for SMP systems with unsynchronized ITC

[Christoph Lameter]

The current code in sysdeps/ia64/hp-timing.h expects the ITC to deliver an
monotonic increasing clock value. That is assured only on UP systems and
on SMP systems with synchronized ITC clocks.

Some systems (like the SGI Altix) do not have synchronized ITC clocks. If
a process is moved to another processor and ITC is used to determine time
then time may seem to be running backwards or leaping because ITC values
from different CPUs are compared.

IA64 kernels provide a proc file /proc/sal/itc_drift that contains "1" if
the platform has an unsynchronized ITC.

The following patch checks for drifty ITC clocks based on that file. If
ITC is drifty then it generates timestamps based on the gettimeofday
system call.

This also contains a small fix for elf/rtld-Rules

Index: glibc-2.3.2-200309260658/elf/rtld-Rules
=================================--- glibc-2.3.2-200309260658.orig/elf/rtld-Rules	2003-07-24 20:31:13.000000000 -0700
+++ glibc-2.3.2-200309260658/elf/rtld-Rules	2004-05-28 12:43:27.000000000 -0700
@@ -31,7 +31,7 @@
 ifeq ($(subdir),elf)

 ifndef rtld-subdirs
-error This makefile is a subroutine of elf/Makefile not to be used directly
+$(error This makefile is a subroutine of elf/Makefile not to be used directly)
 endif

 include ../Makeconfig
Index: glibc-2.3.2-200309260658/sysdeps/ia64/Makefile
=================================--- glibc-2.3.2-200309260658.orig/sysdeps/ia64/Makefile	2004-05-25 18:02:27.000000000 -0700
+++ glibc-2.3.2-200309260658/sysdeps/ia64/Makefile	2004-06-01 19:09:43.000000000 -0700
@@ -12,7 +12,7 @@

 ifeq (yes,$(build-shared))
 # Compatibility
-sysdep_routines += libgcc-compat
+sysdep_routines += libgcc-compat
 shared-only-routines += libgcc-compat
 endif
 endif
@@ -21,4 +21,15 @@
 sysdep-dl-routines += dl-symaddr dl-fptr
 sysdep_routines += $(sysdep-dl-routines)
 sysdep-rtld-routines += $(sysdep-dl-routines)
+sysdep-dl-routines += hp-timing
 endif
+
+ifeq ($(subdir),posix)
+sysdep_routines += hp-timing
+endif
+
+ifeq ($(subdir),malloc)
+libmemusage-routines += hp-timing
+endif
+
+
Index: glibc-2.3.2-200309260658/sysdeps/ia64/hp-timing.c
=================================--- glibc-2.3.2-200309260658.orig/sysdeps/ia64/hp-timing.c	2004-05-27 16:57:50.000000000 -0700
+++ glibc-2.3.2-200309260658/sysdeps/ia64/hp-timing.c	2004-06-01 10:08:00.000000000 -0700
@@ -18,7 +18,54 @@
    Software Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA
    02111-1307 USA.  */

+#include <errno.h>
+#include <time.h>
+#include <unistd.h>
+#include <sys/stat.h>
+#include <sys/types.h>
+#include <sys/time.h>
+#include <fcntl.h>
+
 #include <hp-timing.h>

 /* We have to define the variable for the overhead.  */
 hp_timing_t _dl_hp_timing_overhead;
+/* Values
+   itc_drifting =-1 using ITC
+   itc_drifitng =1 using CLOCK_REALTIME
+   itc_drifting =0 itc_drifting not yet determined
+*/
+int itc_drifting=0;
+
+unsigned long int hp_gettimestamp(void)
+{
+	if (itc_drifting>0)
+get_realtimestamp:
+	{
+		struct timeval t;
+		// clock_gettime is not working in the elf loader... use gettimeofday
+		__gettimeofday(&t,NULL);
+		return t.tv_usec+t.tv_sec*1000000UL;
+	}
+	else
+	{
+		int fd = open ("/proc/sal/itc_drift", O_RDONLY);
+		unsigned long x;
+
+		itc_drifting=-1;
+		if (__builtin_expect (fd != -1, 1))
+		{
+			char buf[16];
+			/* We expect the file to contain a single digit followed by
+			   a newline.  If the format changes we better not rely on
+			   the file content.  */
+			if (read (fd, buf, sizeof buf) != 2 || buf[0] != '0' || buf[1] != '\n')
+				itc_drifting=1;
+			close (fd);
+		}
+		if (itc_drifting>=0) goto get_realtimestamp;
+		HP_TIMING_NOW(x);
+		return x;
+	}
+}
+
Index: glibc-2.3.2-200309260658/sysdeps/ia64/hp-timing.h
=================================--- glibc-2.3.2-200309260658.orig/sysdeps/ia64/hp-timing.h	2004-05-27 16:57:50.000000000 -0700
+++ glibc-2.3.2-200309260658/sysdeps/ia64/hp-timing.h	2004-06-01 12:00:26.000000000 -0700
@@ -72,11 +72,13 @@
 /* We always assume having the timestamp register.  */
 #define HP_TIMING_AVAIL		(1)

-/* We indeed have inlined functions.  */
-#define HP_TIMING_INLINE	(1)
+/* We indeed have inlined functions but sometimes we need to make a call.  */
+#define HP_TIMING_INLINE	(0)

 /* We use 64bit values for the times.  */
 typedef unsigned long int hp_timing_t;
+int itc_drifting;
+unsigned long hp_gettimestamp(void);

 /* Set timestamp value to zero.  */
 #define HP_TIMING_ZERO(Var)	(Var) = (0)
@@ -91,10 +93,13 @@
 /* That's quite simple.  Use the `ar.itc' instruction.  */
 #define HP_TIMING_NOW(Var) \
   ({ unsigned long int __itc;						      \
+    if (itc_drifting>=0) Var=hp_gettimestamp();				\
+    else {								\
      do									      \
        asm volatile ("mov %0=ar.itc" : "=r" (__itc) : : "memory");	      \
-     while (REPEAT_READ (__itc));					      \
-     Var = __itc; })
+      while (REPEAT_READ (__itc));					      \
+      Var = __itc; }							\
+   })

 /* Use two 'ar.itc' instructions in a row to find out how long it takes.  */
 #define HP_TIMING_DIFF_INIT() \

Re: GLIBC IA64 HP_TIMING fixes for SMP systems with unsynchronized

[Andreas Schwab]

Christoph Lameter <christoph@lameter.com> writes:

> +ifeq ($(subdir),malloc)
> +libmemusage-routines += hp-timing
> +endif

That needs to be libmemusage-sysdep_routines.

> +		int fd = open ("/proc/sal/itc_drift", O_RDONLY);

This is Linux-specific, isn't it?  Thus it belongs under
sysdeps/.../linux/ia64.

Andreas.

-- 
Andreas Schwab, SuSE Labs, schwab@suse.de
SuSE Linux AG, Maxfeldstraße 5, 90409 Nürnberg, Germany
Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

Re: GLIBC IA64 HP_TIMING fixes for SMP systems with unsynchronized

[Christoph Lameter]

On Thu, 3 Jun 2004, Andreas Schwab wrote:

> That needs to be libmemusage-sysdep_routines.

>
> > +		int fd = open ("/proc/sal/itc_drift", O_RDONLY);
>
> This is Linux-specific, isn't it?  Thus it belongs under
> sysdeps/.../linux/ia64.

Quite a bit of the code is linux specific. Do I leave the generic code
broken?

I could put some hooks in that have to be satisfied by the
operating system specific versions.

That would require the following functins:

int itc_drifting(void)	-> OS dependent way of determining if ITC is drifting.

int __get_freq(void)	-> Determine ITC frequency that has to be simulated.

Are these to be put into a separate file? itc_drift.c that satisfied those
dependencies in an OS specific way?

Re: GLIBC IA64 HP_TIMING fixes for SMP systems with unsynchronized

[Andreas Schwab]

Christoph Lameter <christoph@lameter.com> writes:

> On Thu, 3 Jun 2004, Andreas Schwab wrote:
>
>> That needs to be libmemusage-sysdep_routines.
>
>>
>> > +		int fd = open ("/proc/sal/itc_drift", O_RDONLY);
>>
>> This is Linux-specific, isn't it?  Thus it belongs under
>> sysdeps/.../linux/ia64.
>
> Quite a bit of the code is linux specific. Do I leave the generic code
> broken?

The generic code has to be adjusted for the specific target anyway, if and
when someone ports it to a non-Linux system.

Andreas.

-- 
Andreas Schwab, SuSE Labs, schwab@suse.de
SuSE Linux AG, Maxfeldstraße 5, 90409 Nürnberg, Germany
Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

Re: GLIBC IA64 HP_TIMING fixes for SMP systems with unsynchronized ITC

[David Mosberger]

>>>>> On Thu, 3 Jun 2004 13:28:35 -0700 (PDT), Christoph Lameter <christoph@lameter.com> said:

  Christoph> The following patch checks for drifty ITC clocks based on
  Christoph> that file. If ITC is drifty then it generates timestamps
  Christoph> based on the gettimeofday system call.

Does anyone else find it annoying that gettimeofday() only provides
micro-second resolution?  We can do syscalls significantly faster than
that (especially with a light-weight syscall handler).  Time for a
kernel extension?

	--daivd

Re: GLIBC IA64 HP_TIMING fixes for SMP systems with unsynchronized

[Ulrich Drepper]

David Mosberger wrote:

> Does anyone else find it annoying that gettimeofday() only provides
> micro-second resolution?  We can do syscalls significantly faster than
> that (especially with a light-weight syscall handler).  Time for a
> kernel extension?

What do you think clock_gettime(CLOCK_REALTIME) is for?

-- 
➧ Ulrich Drepper ➧ Red Hat, Inc. ➧ 444 Castro St ➧ Mountain View, CA ❖

Re: GLIBC IA64 HP_TIMING fixes for SMP systems with unsynchronized

[David Mosberger]

>>>>> On Thu, 03 Jun 2004 23:12:49 -0700, Ulrich Drepper <drepper@redhat.com> said:

  Uli> David Mosberger wrote:

  >> Does anyone else find it annoying that gettimeofday() only provides
  >> micro-second resolution?  We can do syscalls significantly faster than
  >> that (especially with a light-weight syscall handler).  Time for a
  >> kernel extension?

  Uli> What do you think clock_gettime(CLOCK_REALTIME) is for?

clock_gettime() is fine.  I think last time I played with it it was
still implemented in user-level.  Probably just a matter of an "old"
libc...

	--david

Re: GLIBC IA64 HP_TIMING fixes for SMP systems with unsynchronized

[Christoph Lameter]

On Thu, 3 Jun 2004, Ulrich Drepper wrote:

> David Mosberger wrote:
>
> > Does anyone else find it annoying that gettimeofday() only provides
> > micro-second resolution?  We can do syscalls significantly faster than
> > that (especially with a light-weight syscall handler).  Time for a
> > kernel extension?
>
> What do you think clock_gettime(CLOCK_REALTIME) is for?

clock_gettime(CLOCK_REALTIME) calls gettimeofday internally and then
multiplies with 1000. I have a patch that supports CLOCK_REALTIME_HR and
CLOCK_MONOTONIC_HR to access the full resolution of the clocks.

Re: GLIBC IA64 HP_TIMING fixes for SMP systems with unsynchronized

[Christoph Lameter]

On Fri, 4 Jun 2004, Ulrich Drepper wrote:

> Christoph Lameter wrote:
>
> > clock_gettime(CLOCK_REALTIME) calls gettimeofday internally and then
> > multiplies with 1000.
>
> Internally where?  clock_gettime is a syscall.  And it can use whatever
> functionality the kernel provides.  There is absolutely no requirement
> to add additional clocks, just make clock_gettime do the right think if
> this isn't happening already.

Internally in the kernel. Excerpts from kernel/posix-timers.c:

static int do_posix_gettime(struct k_clock *clock, struct timespec *tp)
{
        struct timeval tv;

        if (clock->clock_get)
                return clock->clock_get(tp);

        do_gettimeofday(&tv);
        tp->tv_sec = tv.tv_sec;
        tp->tv_nsec = tv.tv_usec * NSEC_PER_USEC;

        return 0;
}

static u64 do_posix_clock_monotonic_gettime_parts(
        struct timespec *tp, struct timespec *mo)
{
        u64 jiff;
        struct timeval tpv;
        unsigned int seq;

        do {
                seq = read_seqbegin(&xtime_lock);
                do_gettimeofday(&tpv);
                *mo = wall_to_monotonic;
                jiff = jiffies_64;

        } while(read_seqretry(&xtime_lock, seq));

        /*
         * Love to get this before it is converted to usec.
         * It would save a div AND a mpy.
         */
        tp->tv_sec = tpv.tv_sec;
        tp->tv_nsec = tpv.tv_usec * NSEC_PER_USEC;

        return jiff;
}

Re: GLIBC IA64 HP_TIMING fixes for SMP systems with unsynchronized ITC

[Peter Chubb]

>>>>> "David" = David Mosberger <davidm@napali.hpl.hp.com> writes:


David> Does anyone else find it annoying that gettimeofday() only
David> provides micro-second resolution?  We can do syscalls
David> significantly faster than that (especially with a light-weight
David> syscall handler).  Time for a kernel extension?

Only problem is that we can get high precision, but unless we can also
get access to the NTP daemon's parameters, we don't know the accuracy
of the result.

Dunno about your systems, but mine drift around the real time by a
couple of *milliseconds* so nanosecond accuracy in gettimeofday()
won't buy much.

For a lot of things, however, all one needs is a precise monotonic
clock; absolute accuracy isn't required.  Making an alternate internal
do_gettimeofday that takes a timespec instead of a timeval, and using
it for the posix monotonic clock might help here.  There's the
standard clock_gettime() function for accessing the values.  Or the
posix time code that George Anzinger wrote (who's CCed on this thread
now, because he understands the issues better than I) could definitely
use a monotonic high precision clock.

--
Dr Peter Chubb  http://www.gelato.unsw.edu.au  peterc AT gelato.unsw.edu.au
The technical we do immediately,  the political takes *forever*

RE: GLIBC IA64 HP_TIMING fixes for SMP systems with unsynchronized ITC

[Luck, Tony]

>Only problem is that we can get high precision, but unless we can also
>get access to the NTP daemon's parameters, we don't know the accuracy
>of the result.
>
>Dunno about your systems, but mine drift around the real time by a
>couple of *milliseconds* so nanosecond accuracy in gettimeofday()
>won't buy much.

It depends what you are trying to do.  If you want to time how long
a short event takes ... then asking for the time of day before and
after and subtracting is the obvious solution.  If the event only
takes 5 micro-seconds, a one micro-second granularity in gettimeofday()
will give a +/- 20% error.  Even if the absolute time you see in the
two calls is off from UTC by a couple of milli-seconds, the rate of
drift is presumably small enough for the subtraction to make sense.

-Tony