[PATCH] Work around for periodic do_gettimeofday hang

[PATCH] Work around for periodic do_gettimeofday hang

[James Bottomley]

On the voyager systems particularly (but also on some of my slower
parisc boxes) I periodically get hangs where the system just seems to
stop (although it does remain able to execute alt-sysrq).  The traces
always seem to implicate do_gettimeofday in the xtime seqlock.

I've not been able to trace an exact cause for this, but it does seem to
be pretty much eliminated by lowering the clock speed to 100HZ.  I
propose the following patch to do this for the slower intel processors
(basically pentiums and below)

James

--

Signed-off-by: James Bottomley <James.Bottomley@SteelEye.com>

===== include/asm-i386/param.h 1.6 vs edited =====
--- 1.6/include/asm-i386/param.h	2004-06-24 03:55:46 -05:00
+++ edited/include/asm-i386/param.h	2004-11-22 18:36:42 -06:00
@@ -1,8 +1,15 @@
 #ifndef _ASMi386_PARAM_H
 #define _ASMi386_PARAM_H
 
+#include <linux/config.h>
+
 #ifdef __KERNEL__
-# define HZ		1000		/* Internal kernel timer frequency */
+# if defined(CONFIG_M386) || defined(CONFIG_M486) || defined(CONFIG_M586) \
+     || defined(CONFIG_M586TSC) || defined(CONFIG_M586MMX)
+#  define HZ		100
+# else
+#  define HZ		1000		/* Internal kernel timer frequency */
+# endif
 # define USER_HZ	100		/* .. some user interfaces are in "ticks" */
 # define CLOCKS_PER_SEC		(USER_HZ)	/* like times() */
 #endif

Re: [PATCH] Work around for periodic do_gettimeofday hang

[Arjan van de Ven]

> I've not been able to trace an exact cause for this, but it does seem to
> be pretty much eliminated by lowering the clock speed to 100HZ.  I
> propose the following patch to do this for the slower intel processors
> (basically pentiums and below)


while I agree with 100Hz for slower cpus, I rather have a config option for it so people
(and distros) can select it independent of the exact cpu type they want to compile a kernel for

Re: [PATCH] Work around for periodic do_gettimeofday hang

[James Bottomley]

On Wed, 2004-11-24 at 13:13, Arjan van de Ven wrote:
> while I agree with 100Hz for slower cpus, I rather have a config option for it so people
> (and distros) can select it independent of the exact cpu type they want to compile a kernel for

How about this then?

James

===== arch/i386/Kconfig 1.134 vs edited =====
--- 1.134/arch/i386/Kconfig	2004-10-21 20:35:11 -05:00
+++ edited/arch/i386/Kconfig	2004-11-24 20:56:53 -06:00
@@ -330,6 +330,14 @@
 
 endchoice
 
+config X86_HZ
+       int "Clock Tick Rate"
+       default 1000 if !(M386 || M486 || M586 || M586TSC || M586MMX)	
+       default 100 if (M386 || M486 || M586 || M586TSC || M586MMX)	
+       help
+	  Select the kernel clock tick rate in interrupts per second.
+	  Slower processors should choose 100; everything else 1000.
+
 config X86_GENERIC
        bool "Generic x86 support"
        help
===== include/asm-i386/param.h 1.6 vs edited =====
--- 1.6/include/asm-i386/param.h	2004-06-24 03:55:46 -05:00
+++ edited/include/asm-i386/param.h	2004-11-24 20:56:18 -06:00
@@ -1,8 +1,10 @@
 #ifndef _ASMi386_PARAM_H
 #define _ASMi386_PARAM_H
 
+#include <linux/config.h>
+
 #ifdef __KERNEL__
-# define HZ		1000		/* Internal kernel timer frequency */
+# define HZ		(CONFIG_X86_HZ)
 # define USER_HZ	100		/* .. some user interfaces are in "ticks" */
 # define CLOCKS_PER_SEC		(USER_HZ)	/* like times() */
 #endif

Re: [PATCH] Work around for periodic do_gettimeofday hang

[Andrew Morton]

James Bottomley <James.Bottomley@SteelEye.com> wrote:
>
>  +config X86_HZ
>  +       int "Clock Tick Rate"
>  +       default 1000 if !(M386 || M486 || M586 || M586TSC || M586MMX)	
>  +       default 100 if (M386 || M486 || M586 || M586TSC || M586MMX)	
>  +       help
>  +	  Select the kernel clock tick rate in interrupts per second.
>  +	  Slower processors should choose 100; everything else 1000.

I guess we don't need the help, given that it's not a menuisable option. 
(There was a make-HZ-selectable patch once, and Linus spat it out).

Silly question: how come do_gettimeofday() is hanging?

Re: [PATCH] Work around for periodic do_gettimeofday hang

[James Bottomley]

On Wed, 2004-11-24 at 22:33, Andrew Morton wrote:
> Silly question: how come do_gettimeofday() is hanging?

That's what I don't know.  like I said, the xtime seqlock seemed to be
implicated, but I've never managed to trace it down.

I also notice that under heavy load the system starts to lose time (at
least according to ntp).  I suspect there's something wrong with the PIT
timer routine (since that's what voyager has to use) for adding lost
ticks.

James

Re: [PATCH] Work around for periodic do_gettimeofday hang

[Arjan van de Ven]

On Wed, 2004-11-24 at 20:33 -0800, Andrew Morton wrote:
> James Bottomley <James.Bottomley@SteelEye.com> wrote:
> >
> >  +config X86_HZ
> >  +       int "Clock Tick Rate"
> >  +       default 1000 if !(M386 || M486 || M586 || M586TSC || M586MMX)	
> >  +       default 100 if (M386 || M486 || M586 || M586TSC || M586MMX)	
> >  +       help
> >  +	  Select the kernel clock tick rate in interrupts per second.
> >  +	  Slower processors should choose 100; everything else 1000.
> 
> I guess we don't need the help, given that it's not a menuisable option. 
> (There was a make-HZ-selectable patch once, and Linus spat it out).

I'd rather have it as menu-visible option, but I can see how arbitrary
HZ gets spat out. How about a radio-button like thing with "100" and
"1000" in it.

HZ=1000 costs you 1% HPC performance, and for slower machines probably more. Also some hw (with slow smm) really doesn't likeit.

Re: [PATCH] Work around for periodic do_gettimeofday hang

[Roman Zippel]

Hi,

On Thu, 24 Nov 2004, James Bottomley wrote:

> +config X86_HZ
> +       int "Clock Tick Rate"
> +       default 1000 if !(M386 || M486 || M586 || M586TSC || M586MMX)	
> +       default 100 if (M386 || M486 || M586 || M586TSC || M586MMX)	

You don't have to duplicate the conditions, kconfig takes the first match.

bye, Roman

Re: [PATCH] Work around for periodic do_gettimeofday hang

[Jan Engelhardt]

>On Wed, 2004-11-24 at 13:13, Arjan van de Ven wrote:
>> while I agree with 100Hz for slower cpus, I rather have a config option for it so people
>> (and distros) can select it independent of the exact cpu type they want to compile a kernel for
>
>How about this then?

I'd rather make it a number field rather than "choose-me", so the wise user can
choose any Hz he likes.



Jan Engelhardt
-- 
Gesellschaft für Wissenschaftliche Datenverarbeitung
Am Fassberg, 37077 Göttingen, www.gwdg.de

Re: [PATCH] Work around for periodic do_gettimeofday hang

[Alan Cox]

On Mer, 2004-11-24 at 16:49, James Bottomley wrote:
> On the voyager systems particularly (but also on some of my slower
> parisc boxes) I periodically get hangs where the system just seems to
> stop (although it does remain able to execute alt-sysrq).  The traces
> always seem to implicate do_gettimeofday in the xtime seqlock.

What are the other CPUs doing in this case ?

Re: [PATCH] Work around for periodic do_gettimeofday hang

[James Bottomley]

On Thu, 2004-11-25 at 09:47, Alan Cox wrote:
> What are the other CPUs doing in this case ?

Actually, I don't know ... I can't seem to persuade the voyagers to
broadcast an NMI to give up that information.

However, unfortunately, I got a similar hang at HZ=100 this time not
implicating do_gettimeofday.

I'll continue investigating.

James

Re: [PATCH] Work around for periodic do_gettimeofday hang

[Lee Revell]

On Thu, 2004-11-25 at 09:20 +0100, Arjan van de Ven wrote:
> HZ=1000 costs you 1% HPC performance, and for slower machines probably more. Also some hw (with slow smm) really doesn't likeit.

I actually profiled the timer ISR on a 600Mhz C3.  It runs for 22 usec.
So with HZ=1000 the residency of the timer interrupt is 2.2%.

Lee