Scalability enhancements for gettimeofday

[Christoph Lameter <clameter@sgi.com>]

We noticed that gettimeofday() does not scale very well on large Altix systems with manye CPUs.
We saw the following numbers with a small test program:

 CPUS       WALL  WALL/CPUS 
    1      0.679      0.679
    2      1.079      0.540
    4      5.770      1.442
    8     14.178      1.772
   16     36.630      2.289
   32     53.319      1.666

WALL time is number of seconds to issue 1M calls to gettimeofday().
If gettimeofday() would scale well then all times in the WALL column should be the same.

The Altix systems use a global clock as the time source for gettimeofday and not the ITC. 
In do_gettimeofday() a cmpxchg instruction is used to insure that ITC based time calculations never return an earlier time.
This cmpxchg seems to be the reason for the scalability issues.

After disabling this check we obtained the following results:

 CPUS       WALL  WALL/CPUS 
    1      0.618      0.618
    2      0.614      0.307
    4      0.613      0.153
    8      0.620      0.077
   16      0.627      0.039
   32      0.549      0.017

The Altix systems never use ITC to determine the time so this check and the cmpxchg is not necessary.

Also the fast system call handler for sys_gettimeofday() can only improve scalability if the ITC is the basis for gettimeofday
which is also never the case on Altix SN2 systems.

The following patch skips cmpxchg if the itc interpolator is not used and removes the fast system call for gettimeofday for the Altix systems.

Index: linux-2.6.6/arch/ia64/kernel/fsys.S
=================================--- linux-2.6.6.orig/arch/ia64/kernel/fsys.S	2004-05-20 11:20:50.000000000 -0700
+++ linux-2.6.6/arch/ia64/kernel/fsys.S	2004-05-20 11:36:57.000000000 -0700
@@ -148,6 +148,7 @@
 	FSYS_RETURN
 END(fsys_set_tid_address)
 
+#ifndef CONFIG_IA64_SGI_SN2
 /*
  * Note 1: This routine uses floating-point registers, but only with registers that
  *	   operate on integers.  Because of that, we don't need to set ar.fpsr to the
@@ -329,6 +330,8 @@
 	 *	 can be obtained from System.map), none of this should be security-sensitive
 	 *	 and we should be fine.
 	 */
+END(fsys_gettimeofday)
+#endif
 
 .fail_einval:
 	mov r8=EINVAL			// r8 = EINVAL
@@ -339,7 +342,6 @@
 	mov r8ÔAULT			// r8 = EFAULT
 	mov r10=-1			// r10 = -1
 	FSYS_RETURN
-END(fsys_gettimeofday)
 
 /*
  * long fsys_rt_sigprocmask (int how, sigset_t *set, sigset_t *oset, size_t sigsetsize).
@@ -686,7 +688,14 @@
 	data8 0				// setrlimit
 	data8 0				// getrlimit		// 1085
 	data8 0				// getrusage
+// It is only beneficial to use a fastcall for gettimeofday if ITC can be used
+// to determine time. If a global clock is used then do_gettimeofday must be invoked.
+// SGI SN2 always uses a global clock. There might be other configuration too.
+#ifndef  CONFIG_IA64_SGI_SN2
 	data8 fsys_gettimeofday		// gettimeofday
+#else
+	data8 0				// gettimeofday
+#endif
 	data8 0				// settimeofday
 	data8 0				// select
 	data8 0				// poll			// 1090
Index: linux-2.6.6/arch/ia64/kernel/time.c
=================================--- linux-2.6.6.orig/arch/ia64/kernel/time.c	2004-05-09 19:32:24.000000000 -0700
+++ linux-2.6.6/arch/ia64/kernel/time.c	2004-05-20 11:19:21.000000000 -0700
@@ -146,6 +146,15 @@
 		}
 		if (unlikely(read_seqretry(&xtime_lock, seq)))
 			continue;
+		
+		/* If we are not using the ITC interpolator then the cmpxchg logic is not necessary
+		 * since other clocks provide  monnotonic time that never goes backward.
+		 * There is no need to use last_nsec_offset with a real global clock.
+		 * cmpxchg will generate a scalability problem so skip it.
+		 */
+
+		if (time_interpolator!=&itc_interpolator) break;
+
 		/*
 		 * Ensure that for any pair of causally ordered gettimeofday() calls, time
 		 * never goes backwards (even when ITC on different CPUs are not perfectly
-
To unsubscribe from this list: send the line "unsubscribe linux-ia64" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html