* Re: Updated Lockup Patches, 2.6.0 Nforce2, apic timer ack delay, ioapic edge for NMI debug
2003-12-21 9:17 Updated Lockup Patches, 2.6.0 Nforce2, apic timer ack delay, ioapic edge for NMI debug Ross Dickson
@ 2003-12-21 16:29 ` Craig Bradney
2003-12-22 18:50 ` Daniel Drake
` (2 subsequent siblings)
3 siblings, 0 replies; 9+ messages in thread
From: Craig Bradney @ 2003-12-21 16:29 UTC (permalink / raw)
To: ross; +Cc: linux-kernel, Ian Kumlien, Maciej W. Rozycki, forming,
Jesse Allen
[-- Attachment #1: Type: text/plain, Size: 13277 bytes --]
Hi Ross
Ok, I have now started testing with apic_tack=2 with the 2.60 kernel
from Gentoo. Before this voluntary reboot I was at 8d 20h+ with old
patches.
I have attached dmesg and /proc/interrupts from a boot to prompt for
both apic_tack=1 and 2, both with nmi 1.
Relevant information would seem to be as follows:
..
..
ENABLING IO-APIC IRQs
init IO_APIC IRQs
IO-APIC (apicid-pin) 2-0, 2-16, 2-17, 2-18, 2-19, 2-20, 2-21, 2-22, 2-23 not connected.
..TIMER: vector=0x31 pin1=2 pin2=-1
..MP-BIOS bug: 8254 timer not connected to IO-APIC INTIN2
..TIMER: Is timer irq0 connected to IO-APIC INTIN0? ...
IOAPIC[0]: Set PCI routing entry (2-0 -> 0x31 -> IRQ 0 Mode:0 Active:0)
activating NMI Watchdog ... done.
testing NMI watchdog ... OK.
..TIMER: works OK on IO-APIC INTIN0 irq0
..
..
..APIC TIMER ack delay, reload:20792, safe:20776
..
..
and CPU disconnect is enabled (as reported by athcool stat)
Thanks for your continuing work!
Craig
ASUS A7N8X Deluxe V2.0 BIOS 1007, Athlon 2600+
On Sun, 2003-12-21 at 10:17, Ross Dickson wrote:
> Firstly I thought a summary overdue so here goes.
>
> My speculations as to why Nforce2 systems lock up are as follows:
>
> a) The Nforce2 DASP speculates and gets it right, pre-fetching the code for the
> local apic timer interrupt, so the interrupt code executes sooner after
> activation than it does with other chipsets for AMD.
>
> b) The AMD cpu may not be over its timing and stability issues when coming
> out of C1 disconnect. Plenty stable soon enough for other chipsets and other
> codepaths in linux which pull the cpu out of C1 disconnect, but not quite soon
> enough for the "cached" short code path to the local apic timer ack. So most
> of the time any latent lockup potential is not realised, but on occasion
> we hit it.
>
> Disclaimer: I can think of so many ways I could be way off so I have a support
> request in with AMD on the topic which I am told is progressing. I do not have
> the resources to be able to confirm or deny the above theory. Patches are
> GPL'ed as per the licenses of the files they go into.
>
>
> Are my patches the only way to workaround lockups?
> NO Firstly I think the manufacturers will fix the problem so I will not have to
> use any workarounds one day. There are early reports that recent Award bios's
> are fixing the lockup issue but not the nmi_watchdog problem.
>
> Others advise you can successfully use the C1 disconnect patch for the kernel,
> also Athcool. Some bios also have options to alter the C1 Disconnect
> state and some early bios are preset with it to always off. If the lockups would
> only occur with disconnect on (yet to be confirmed or denied by manufacturers)
> then this would be one sure way to stop them.
>
> Why don't you just disable the C1 disconnect?
> I don't think it would be wise for me to frob with it for many reasons. Nvidia
> has some seriously patented smarts in their ram controllers, and the ambient
> temperature today, where I am, rose to 30C or about 85F and it gets hotter. I
> also have a system in the north of Western Australia where 40C+ summer is
> normal. The AMD processor data sheet says that the CPU only enters low power
> state if it can disconnect. Web postings indicate 8C to 10C temperature increase
> with disconnect disabled.
>
> Do my patches Fix all of it?
> No they are just workarounds. How well they work around may depend upon your
> system configuration, and how well the delay times chosen suit it.
>
> Any evil side effects?
> Maybe, but I don't know of any yet. Any slowing from the delay is so far
> not noticeable.
>
> Why 2 patches?
> The apic timer ack delay patch is for the hard lockups.
> The io-apic edge patch is for lost interrupts and also gives nmi_watchdog=1
> functionality.
>
> Should I install any or both?
> Depends, you get to decide until the manufacturers fix it.
>
> The io-apic edge seems beneficial to all Nforce2 so far as the bios reports the
> 8254 timer connected to INTIN2. I found it connected directly to INTIN0.
> The patch forces connection to INTIN0 to see if it works only after the existing
> code has tried to connect it to where the bios said it was. The io-apic patch
> compiles in only with acpi on and uniprocessor x86 io-apic on in kern config.
> The acpi config is essential as we use part of it, the uniprocessor x86 io-apic
> is there as a precaution and if the patch is more widely tested it may not be
> required. Remove it if you want the patch in smp mode.
>
> The apic timer ack is not needed if you do not have hard lockups and you can
> solve the hard lockups by other means if you want to. I do not recommend kern
> config of apic without io-apic for nforce2. The two work best together and if
> you use my apic timer ack delay patch then use my io-apic edge patch with it
> for best results.
>
> This version of the apic timer ack delay patch is not on by default. You need
> to use a kernel argument at boot time to invoke it and there are two forms.
>
> apic_tack=1
>
> puts a small delay inline with the code. It is the reliable fallback mode.
>
> apic_tack=2
>
> reads the apic timer count and only delays if needed to prevent a lockup. This
> mode assumes it is always safe to read the apic timer count register but
> unsafe at times to ack it. This assumption could be wrong.
>
> I find apic_tack=1 better at stopping hard lockups on my T/bred XP2200
> (266fsb DDR400 ram) system with my patched 2.4.23 kern but apic_tack=2 ok
> with 2.6.0 on same machine? I also note 2.4.23 gives me 53Mb/s and 2.6.0 only
> 47Mb/s with hdparm -t /dev/hda testing.
>
> Other kernel versions?
> It should work fine with 2.4.22 and 2.4.23 although you will probably have to hand
> patch them for now.
>
> Historical postings on the topic?
> Heaps, many thanks to those who have tested and contributed so far.
> Primary thread on this topic for my patches
> Fixes for nforce2 hard lockup, apic, io-apic, udma133 covered
> and another useful discussion
> Catching NForce2 lockup with NMI watchdog
>
> Debugging output?
> Ioapic patch always reports success or otherwise as part of check_timer if
> it is needed such as:
> ..TIMER: vector=0x31 pin1=2 pin2=-1
> ..MP-BIOS bug: 8254 timer not connected to IO-APIC INTIN2
> ..TIMER: Is timer irq0 connected to IO-APIC INTIN0? ...
> IOAPIC[0]: Set PCI routing entry (2-0 -> 0x31 -> IRQ 0 Mode:0 Active:0)
> ..TIMER: works OK on IO-APIC INTIN0 irq0
>
> None for apic ackdelay apic_tack=0 or apic_tack=1
>
> apic_tack=2 always reports:
> ..APIC TIMER ack delay, reload:16701, safe:16691
>
> but numbers vary with your fsb.
> reload is your existing local timer 1ms reload count.
> safe is the down count at which about 800ns has expired since reload. Patch
> considers it safe to ack after that time.
>
> More can be had if
> #define APIC_DEBUG 0
> is set to 1 in
> /usr/src/linux-2.4.23-rd2/include/asm-i386/apic.h
>
> The io-apic patch will also display the 8259a xtpic interrupt mask.
> I get hex fb and hex ff indicating that the only int enabled on the 8259A xtpic
> which handles irq 0-7 is the cascade irq 2 which is OK because on the other
> 8259A irq 8-15 are masked. This masked xtpic should always be the case.
> We want the 8259A off during our test to see if the 8254 timer is connected
> directly to pin 0 of the ioapic.
>
> The apic ack delay patch will report at boot ten pre delay apic timer times to
> get a feel as to whether the delay had to kick in or not. Note that this is a
> very small sample size but it does give an idea of the timing numbers.
>
> Following are my apic and ioapic patches for 2.6.0.
> bzip also attached.
>
> Regards
> Ross Dickson.
>
> local apic timer ack delay:
>
> --- CUT HERE ---
> --- linux-2.6.0/arch/i386/kernel/apic.c 2003-12-18 12:59:58.000000000 +1000
> +++ linux-2.6.0-rd/arch/i386/kernel/apic.c 2003-12-21 12:39:28.000000000 +1000
> @@ -1070,10 +1070,17 @@ inline void smp_local_timer_interrupt(st
> * we can take more than 100K local irqs per second on a 100 MHz P5.
> */
> }
>
> /*
> + * Athlon nforce2 R.D.
> + * preset timer ack mode if desired
> + * e.g. static int apic_timerack = 2;
> +*/
> +static int apic_timerack;
> +
> +/*
> * Local APIC timer interrupt. This is the most natural way for doing
> * local interrupts, but local timer interrupts can be emulated by
> * broadcast interrupts too. [in case the hw doesn't support APIC timers]
> *
> * [ if a single-CPU system runs an SMP kernel then we call the local
> @@ -1088,10 +1095,54 @@ void smp_apic_timer_interrupt(struct pt_
> * the NMI deadlock-detector uses this.
> */
> irq_stat[cpu].apic_timer_irqs++;
>
> /*
> + * Athlon nforce2 timer ack delay. Ross Dickson.
> + * works around issue of hard lockups in code location
> + * where linux exposes underlying system timing fault?
> + * hopefully manufacturers will fix it soon.
> + * We leave C1 disconnect bit alone as bios/SMM wants?
> + */
> + if(apic_timerack) {
> + if(apic_timerack==1) {
> + /* v1 timer ack delay, inline delay version
> + * on AMDXP & nforce2 chipset we use at least 500ns
> + * try to scale delay time with cpu speed.
> + * safe all cpu cores?
> + */
> + ndelay((cpu_khz >> 12)+200); /* don't ack too soon or hard lockup */
> + } else {
> + static unsigned int passno, safecnt;
> + /* v2 timer ack delay, timeout version, more efficient
> + * on AMDXP & nforce2 chipset we need 800ns?
> + * from timer irq start to apic irq ack, read apic timer,
> + * may be unsafe for thoroughbred cores?
> + */
> + if(!passno) { /* calculate timing */
> + safecnt = apic_read(APIC_TMICT) -
> + ( (800UL * apic_read(APIC_TMICT) ) /
> + (1000000000UL/HZ) );
> + printk("..APIC TIMER ack delay, reload:%lu, safe:%u\n",
> + apic_read(APIC_TMICT), safecnt);
> + passno++;
> + }
> +#if APIC_DEBUG
> + if(passno<12) {
> + unsigned int at1 = apic_read(APIC_TMCCT);
> + if( passno > 1 )
> + Dprintk("..APIC TIMER ack delay, predelay count:%u \n", at1 );
> + passno++;
> + }
> +# endif
> + /* delay only if required */
> + while( apic_read(APIC_TMCCT) > safecnt )
> + ndelay(100);
> + }
> + }
> +
> + /*
> * NOTE! We'd better ACK the irq immediately,
> * because timer handling can be slow.
> */
> ack_APIC_irq();
> /*
> @@ -1157,10 +1208,28 @@ asmlinkage void smp_error_interrupt(void
> smp_processor_id(), v , v1);
> irq_exit();
> }
>
> /*
> +* Athlon nforce2 timer ack delay. R.D.
> +* kernel arg apic_tack=[012]
> +* 0 off, 1 always delay, 2 timeout
> +*/
> +static int __init setup_apic_timerack(char *str)
> +{
> + int tack;
> +
> + get_option(&str, &tack);
> +
> + if ( tack < 0 || tack > 2 )
> + return 0;
> + apic_timerack = tack;
> + return 1;
> +}
> +__setup("apic_tack=", setup_apic_timerack);
> +
> +/*
> * This initializes the IO-APIC and APIC hardware if this is
> * a UP kernel.
> */
> int __init APIC_init_uniprocessor (void)
> {
> --- CUT HERE ---
>
> io-apic edge:
> --- CUT HERE ---
> --- linux-2.6.0/arch/i386/kernel/io_apic.c 2003-12-18 12:58:39.000000000 +1000
> +++ linux-2.6.0-rd/arch/i386/kernel/io_apic.c 2003-12-20 21:41:52.000000000 +1000
> @@ -2123,12 +2123,56 @@ static inline void check_timer(void)
> check_nmi_watchdog();
> }
> return;
> }
> clear_IO_APIC_pin(0, pin1);
> - printk(KERN_ERR "..MP-BIOS bug: 8254 timer not connected to IO-APIC\n");
> + printk(KERN_ERR "..MP-BIOS bug: 8254 timer not connected to IO-APIC INTIN%d\n",pin1);
> + }
> +
> +#if defined(CONFIG_ACPI_BOOT) && defined(CONFIG_X86_UP_IOAPIC)
> + /* for nforce2 try vector 0 on pin0
> + * Note 8259a is already masked, also by default
> + * the io_apic_set_pci_routing call disables the 8259 irq 0
> + * so we must be connected directly to the 8254 timer if this works
> + * Note2: this violates the above comment re Subtle but works!
> + */
> + printk(KERN_INFO "..TIMER: Is timer irq0 connected to IO-APIC INTIN0? ...\n");
> + if (pin1 != -1) {
> + extern spinlock_t i8259A_lock;
> + unsigned long flags;
> + int tok, saved_timer_ack = timer_ack;
> + /*
> + * Ok, does IRQ0 through the IOAPIC work?
> + */
> + io_apic_set_pci_routing ( 0, 0, 0, 0, 0); /* connect pin */
> + unmask_IO_APIC_irq(0);
> + timer_ack = 0;
> +
> + /*
> + * Ok, does IRQ0 through the IOAPIC work?
> + */
> + spin_lock_irqsave(&i8259A_lock, flags);
> + Dprintk("..TIMER 8259A ints disabled?, imr1:%02x, imr2:%02x\n", inb(0x21), inb(0xA1));
> + tok = timer_irq_works();
> + spin_unlock_irqrestore(&i8259A_lock, flags);
> + if (tok) {
> + if (nmi_watchdog == NMI_IO_APIC) {
> + disable_8259A_irq(0);
> + setup_nmi();
> + enable_8259A_irq(0);
> + check_nmi_watchdog();
> + }
> + printk(KERN_INFO "..TIMER: works OK on IO-APIC INTIN0 irq0\n" );
> + return;
> + }
> + /* failed */
> + timer_ack = saved_timer_ack;
> + clear_IO_APIC_pin(0, 0);
> + io_apic_set_pci_routing ( 0, pin1, 0, 0, 0);
> + printk(KERN_ERR "..MP-BIOS: 8254 timer not connected to IO-APIC INTIN0\n");
> }
> +#endif
>
> printk(KERN_INFO "...trying to set up timer (IRQ0) through the 8259A ... ");
> if (pin2 != -1) {
> printk("\n..... (found pin %d) ...", pin2);
> /*
> --- CUT HERE ---
>
>
[-- Attachment #2: 260-gentoo-latestpatches-output.tar.gz --]
[-- Type: application/x-compressed-tar, Size: 5625 bytes --]
^ permalink raw reply [flat|nested] 9+ messages in thread* Re: Updated Lockup Patches, 2.6.0 Nforce2, apic timer ack delay, ioapic edge for NMI debug
2003-12-21 9:17 Updated Lockup Patches, 2.6.0 Nforce2, apic timer ack delay, ioapic edge for NMI debug Ross Dickson
2003-12-21 16:29 ` Craig Bradney
@ 2003-12-22 18:50 ` Daniel Drake
2003-12-22 23:56 ` Ross Dickson
2003-12-24 17:49 ` really bensoo_at_soo_dot_com
2004-02-07 14:55 ` Nforce2 apic timer ack delay Jamie Lokier
3 siblings, 1 reply; 9+ messages in thread
From: Daniel Drake @ 2003-12-22 18:50 UTC (permalink / raw)
To: ross; +Cc: linux-kernel
Hi,
I compiled 2.6.0-test11-mm1, with local APIC and IO-APIC (I hadn't used these
options before) and I encountered a complete system freeze after about 10 mins
of "basic" (mozilla, xmms) usage. Never had any stability problems on this
system at all, before that freeze. I then recompiled the kernel with your
patches and since then it's been fine.
Would your patches (by default) have influenced this? I have yet to play around
with apic_tack - would me testing this be useful?
Hardware: AMD 2600+ throughbred, Abit NF7-S V2.0, DDR333 Corsair RAM.
-Daniel
Ross Dickson wrote:
> Firstly I thought a summary overdue so here goes.
>
> My speculations as to why Nforce2 systems lock up are as follows:
>
> a) The Nforce2 DASP speculates and gets it right, pre-fetching the code for the
> local apic timer interrupt, so the interrupt code executes sooner after
> activation than it does with other chipsets for AMD.
>
> b) The AMD cpu may not be over its timing and stability issues when coming
> out of C1 disconnect. Plenty stable soon enough for other chipsets and other
> codepaths in linux which pull the cpu out of C1 disconnect, but not quite soon
> enough for the "cached" short code path to the local apic timer ack. So most
> of the time any latent lockup potential is not realised, but on occasion
> we hit it.
>
> Disclaimer: I can think of so many ways I could be way off so I have a support
> request in with AMD on the topic which I am told is progressing. I do not have
> the resources to be able to confirm or deny the above theory. Patches are
> GPL'ed as per the licenses of the files they go into.
>
>
> Are my patches the only way to workaround lockups?
> NO Firstly I think the manufacturers will fix the problem so I will not have to
> use any workarounds one day. There are early reports that recent Award bios's
> are fixing the lockup issue but not the nmi_watchdog problem.
>
> Others advise you can successfully use the C1 disconnect patch for the kernel,
> also Athcool. Some bios also have options to alter the C1 Disconnect
> state and some early bios are preset with it to always off. If the lockups would
> only occur with disconnect on (yet to be confirmed or denied by manufacturers)
> then this would be one sure way to stop them.
>
> Why don't you just disable the C1 disconnect?
> I don't think it would be wise for me to frob with it for many reasons. Nvidia
> has some seriously patented smarts in their ram controllers, and the ambient
> temperature today, where I am, rose to 30C or about 85F and it gets hotter. I
> also have a system in the north of Western Australia where 40C+ summer is
> normal. The AMD processor data sheet says that the CPU only enters low power
> state if it can disconnect. Web postings indicate 8C to 10C temperature increase
> with disconnect disabled.
>
> Do my patches Fix all of it?
> No they are just workarounds. How well they work around may depend upon your
> system configuration, and how well the delay times chosen suit it.
>
> Any evil side effects?
> Maybe, but I don't know of any yet. Any slowing from the delay is so far
> not noticeable.
>
> Why 2 patches?
> The apic timer ack delay patch is for the hard lockups.
> The io-apic edge patch is for lost interrupts and also gives nmi_watchdog=1
> functionality.
>
> Should I install any or both?
> Depends, you get to decide until the manufacturers fix it.
>
> The io-apic edge seems beneficial to all Nforce2 so far as the bios reports the
> 8254 timer connected to INTIN2. I found it connected directly to INTIN0.
> The patch forces connection to INTIN0 to see if it works only after the existing
> code has tried to connect it to where the bios said it was. The io-apic patch
> compiles in only with acpi on and uniprocessor x86 io-apic on in kern config.
> The acpi config is essential as we use part of it, the uniprocessor x86 io-apic
> is there as a precaution and if the patch is more widely tested it may not be
> required. Remove it if you want the patch in smp mode.
>
> The apic timer ack is not needed if you do not have hard lockups and you can
> solve the hard lockups by other means if you want to. I do not recommend kern
> config of apic without io-apic for nforce2. The two work best together and if
> you use my apic timer ack delay patch then use my io-apic edge patch with it
> for best results.
>
> This version of the apic timer ack delay patch is not on by default. You need
> to use a kernel argument at boot time to invoke it and there are two forms.
>
> apic_tack=1
>
> puts a small delay inline with the code. It is the reliable fallback mode.
>
> apic_tack=2
>
> reads the apic timer count and only delays if needed to prevent a lockup. This
> mode assumes it is always safe to read the apic timer count register but
> unsafe at times to ack it. This assumption could be wrong.
>
> I find apic_tack=1 better at stopping hard lockups on my T/bred XP2200
> (266fsb DDR400 ram) system with my patched 2.4.23 kern but apic_tack=2 ok
> with 2.6.0 on same machine? I also note 2.4.23 gives me 53Mb/s and 2.6.0 only
> 47Mb/s with hdparm -t /dev/hda testing.
>
> Other kernel versions?
> It should work fine with 2.4.22 and 2.4.23 although you will probably have to hand
> patch them for now.
>
> Historical postings on the topic?
> Heaps, many thanks to those who have tested and contributed so far.
> Primary thread on this topic for my patches
> Fixes for nforce2 hard lockup, apic, io-apic, udma133 covered
> and another useful discussion
> Catching NForce2 lockup with NMI watchdog
>
> Debugging output?
> Ioapic patch always reports success or otherwise as part of check_timer if
> it is needed such as:
> ..TIMER: vector=0x31 pin1=2 pin2=-1
> ..MP-BIOS bug: 8254 timer not connected to IO-APIC INTIN2
> ..TIMER: Is timer irq0 connected to IO-APIC INTIN0? ...
> IOAPIC[0]: Set PCI routing entry (2-0 -> 0x31 -> IRQ 0 Mode:0 Active:0)
> ..TIMER: works OK on IO-APIC INTIN0 irq0
>
> None for apic ackdelay apic_tack=0 or apic_tack=1
>
> apic_tack=2 always reports:
> ..APIC TIMER ack delay, reload:16701, safe:16691
>
> but numbers vary with your fsb.
> reload is your existing local timer 1ms reload count.
> safe is the down count at which about 800ns has expired since reload. Patch
> considers it safe to ack after that time.
>
> More can be had if
> #define APIC_DEBUG 0
> is set to 1 in
> /usr/src/linux-2.4.23-rd2/include/asm-i386/apic.h
>
> The io-apic patch will also display the 8259a xtpic interrupt mask.
> I get hex fb and hex ff indicating that the only int enabled on the 8259A xtpic
> which handles irq 0-7 is the cascade irq 2 which is OK because on the other
> 8259A irq 8-15 are masked. This masked xtpic should always be the case.
> We want the 8259A off during our test to see if the 8254 timer is connected
> directly to pin 0 of the ioapic.
>
> The apic ack delay patch will report at boot ten pre delay apic timer times to
> get a feel as to whether the delay had to kick in or not. Note that this is a
> very small sample size but it does give an idea of the timing numbers.
>
> Following are my apic and ioapic patches for 2.6.0.
> bzip also attached.
>
> Regards
> Ross Dickson.
>
> local apic timer ack delay:
>
> --- CUT HERE ---
> --- linux-2.6.0/arch/i386/kernel/apic.c 2003-12-18 12:59:58.000000000 +1000
> +++ linux-2.6.0-rd/arch/i386/kernel/apic.c 2003-12-21 12:39:28.000000000 +1000
> @@ -1070,10 +1070,17 @@ inline void smp_local_timer_interrupt(st
> * we can take more than 100K local irqs per second on a 100 MHz P5.
> */
> }
>
> /*
> + * Athlon nforce2 R.D.
> + * preset timer ack mode if desired
> + * e.g. static int apic_timerack = 2;
> +*/
> +static int apic_timerack;
> +
> +/*
> * Local APIC timer interrupt. This is the most natural way for doing
> * local interrupts, but local timer interrupts can be emulated by
> * broadcast interrupts too. [in case the hw doesn't support APIC timers]
> *
> * [ if a single-CPU system runs an SMP kernel then we call the local
> @@ -1088,10 +1095,54 @@ void smp_apic_timer_interrupt(struct pt_
> * the NMI deadlock-detector uses this.
> */
> irq_stat[cpu].apic_timer_irqs++;
>
> /*
> + * Athlon nforce2 timer ack delay. Ross Dickson.
> + * works around issue of hard lockups in code location
> + * where linux exposes underlying system timing fault?
> + * hopefully manufacturers will fix it soon.
> + * We leave C1 disconnect bit alone as bios/SMM wants?
> + */
> + if(apic_timerack) {
> + if(apic_timerack==1) {
> + /* v1 timer ack delay, inline delay version
> + * on AMDXP & nforce2 chipset we use at least 500ns
> + * try to scale delay time with cpu speed.
> + * safe all cpu cores?
> + */
> + ndelay((cpu_khz >> 12)+200); /* don't ack too soon or hard lockup */
> + } else {
> + static unsigned int passno, safecnt;
> + /* v2 timer ack delay, timeout version, more efficient
> + * on AMDXP & nforce2 chipset we need 800ns?
> + * from timer irq start to apic irq ack, read apic timer,
> + * may be unsafe for thoroughbred cores?
> + */
> + if(!passno) { /* calculate timing */
> + safecnt = apic_read(APIC_TMICT) -
> + ( (800UL * apic_read(APIC_TMICT) ) /
> + (1000000000UL/HZ) );
> + printk("..APIC TIMER ack delay, reload:%lu, safe:%u\n",
> + apic_read(APIC_TMICT), safecnt);
> + passno++;
> + }
> +#if APIC_DEBUG
> + if(passno<12) {
> + unsigned int at1 = apic_read(APIC_TMCCT);
> + if( passno > 1 )
> + Dprintk("..APIC TIMER ack delay, predelay count:%u \n", at1 );
> + passno++;
> + }
> +# endif
> + /* delay only if required */
> + while( apic_read(APIC_TMCCT) > safecnt )
> + ndelay(100);
> + }
> + }
> +
> + /*
> * NOTE! We'd better ACK the irq immediately,
> * because timer handling can be slow.
> */
> ack_APIC_irq();
> /*
> @@ -1157,10 +1208,28 @@ asmlinkage void smp_error_interrupt(void
> smp_processor_id(), v , v1);
> irq_exit();
> }
>
> /*
> +* Athlon nforce2 timer ack delay. R.D.
> +* kernel arg apic_tack=[012]
> +* 0 off, 1 always delay, 2 timeout
> +*/
> +static int __init setup_apic_timerack(char *str)
> +{
> + int tack;
> +
> + get_option(&str, &tack);
> +
> + if ( tack < 0 || tack > 2 )
> + return 0;
> + apic_timerack = tack;
> + return 1;
> +}
> +__setup("apic_tack=", setup_apic_timerack);
> +
> +/*
> * This initializes the IO-APIC and APIC hardware if this is
> * a UP kernel.
> */
> int __init APIC_init_uniprocessor (void)
> {
> --- CUT HERE ---
>
> io-apic edge:
> --- CUT HERE ---
> --- linux-2.6.0/arch/i386/kernel/io_apic.c 2003-12-18 12:58:39.000000000 +1000
> +++ linux-2.6.0-rd/arch/i386/kernel/io_apic.c 2003-12-20 21:41:52.000000000 +1000
> @@ -2123,12 +2123,56 @@ static inline void check_timer(void)
> check_nmi_watchdog();
> }
> return;
> }
> clear_IO_APIC_pin(0, pin1);
> - printk(KERN_ERR "..MP-BIOS bug: 8254 timer not connected to IO-APIC\n");
> + printk(KERN_ERR "..MP-BIOS bug: 8254 timer not connected to IO-APIC INTIN%d\n",pin1);
> + }
> +
> +#if defined(CONFIG_ACPI_BOOT) && defined(CONFIG_X86_UP_IOAPIC)
> + /* for nforce2 try vector 0 on pin0
> + * Note 8259a is already masked, also by default
> + * the io_apic_set_pci_routing call disables the 8259 irq 0
> + * so we must be connected directly to the 8254 timer if this works
> + * Note2: this violates the above comment re Subtle but works!
> + */
> + printk(KERN_INFO "..TIMER: Is timer irq0 connected to IO-APIC INTIN0? ...\n");
> + if (pin1 != -1) {
> + extern spinlock_t i8259A_lock;
> + unsigned long flags;
> + int tok, saved_timer_ack = timer_ack;
> + /*
> + * Ok, does IRQ0 through the IOAPIC work?
> + */
> + io_apic_set_pci_routing ( 0, 0, 0, 0, 0); /* connect pin */
> + unmask_IO_APIC_irq(0);
> + timer_ack = 0;
> +
> + /*
> + * Ok, does IRQ0 through the IOAPIC work?
> + */
> + spin_lock_irqsave(&i8259A_lock, flags);
> + Dprintk("..TIMER 8259A ints disabled?, imr1:%02x, imr2:%02x\n", inb(0x21), inb(0xA1));
> + tok = timer_irq_works();
> + spin_unlock_irqrestore(&i8259A_lock, flags);
> + if (tok) {
> + if (nmi_watchdog == NMI_IO_APIC) {
> + disable_8259A_irq(0);
> + setup_nmi();
> + enable_8259A_irq(0);
> + check_nmi_watchdog();
> + }
> + printk(KERN_INFO "..TIMER: works OK on IO-APIC INTIN0 irq0\n" );
> + return;
> + }
> + /* failed */
> + timer_ack = saved_timer_ack;
> + clear_IO_APIC_pin(0, 0);
> + io_apic_set_pci_routing ( 0, pin1, 0, 0, 0);
> + printk(KERN_ERR "..MP-BIOS: 8254 timer not connected to IO-APIC INTIN0\n");
> }
> +#endif
>
> printk(KERN_INFO "...trying to set up timer (IRQ0) through the 8259A ... ");
> if (pin2 != -1) {
> printk("\n..... (found pin %d) ...", pin2);
> /*
> --- CUT HERE ---
>
>
^ permalink raw reply [flat|nested] 9+ messages in thread* Re: Updated Lockup Patches, 2.6.0 Nforce2, apic timer ack delay, ioapic edge for NMI debug
2003-12-22 18:50 ` Daniel Drake
@ 2003-12-22 23:56 ` Ross Dickson
2003-12-24 15:41 ` Daniel Drake
0 siblings, 1 reply; 9+ messages in thread
From: Ross Dickson @ 2003-12-22 23:56 UTC (permalink / raw)
To: Daniel Drake; +Cc: linux-kernel
On Tuesday 23 December 2003 04:50, Daniel Drake wrote:
> Hi,
>
> I compiled 2.6.0-test11-mm1, with local APIC and IO-APIC (I hadn't used these
> options before) and I encountered a complete system freeze after about 10 mins
> of "basic" (mozilla, xmms) usage. Never had any stability problems on this
> system at all, before that freeze. I then recompiled the kernel with your
> patches and since then it's been fine.
>
The freezes really show up with the apic on. The old XTPIC has slower access
times and different code paths which appear to make it more stable.
2.6.0-test11-mm1 picked up the earlier two patches which were the topic
of "Catching NForce2 lockup with NMI watchdog"
Others had reported that those patches achieved stability?
Those patches are known in the test11-mm1 changelog as
+nforce2-disconnect-quirk.patch
+nforce2-apic.patch
The nforce2-disconnect-quirk.patch disables the C1 disconnect bit.
It roughly equates in function to my apic ack delay patch.
The nforce2-apic.patch tries to correct the 8254 timer route but gets it wrong.
It roughly equates in function to my ioapic edge patch.
> Would your patches (by default) have influenced this? I have yet to play around
> with apic_tack - would me testing this be useful?
YES by default you have corrected the 8254 timer route with my io-apic patch.
My apic ack delay patch is off in your system as evidenced by not using the
apic_tack arg.
The disconnect quirk patch in the test11-mm1 is achieving much the same goal
as the apic_tack arg does but your cpu temperature is probably 8C to 10C hotter
than if the disconnect quirk was removed and you tried apic_tack=1.
You can try apic_tack=2 but its performance in stopping lockups is more system and
linux version dependent.
I say probably on the temperature as it depends if your bios has C1 disconnect
on or off at boot.
You can check it the C1 disconnect bit at runtime and manipulate it if desired
with athcool.
On is cool
Off is hot.
Regards
Ross.
>
> Hardware: AMD 2600+ throughbred, Abit NF7-S V2.0, DDR333 Corsair RAM.
>
> -Daniel
>
> Ross Dickson wrote:
> > Firstly I thought a summary overdue so here goes.
> >
> > My speculations as to why Nforce2 systems lock up are as follows:
> >
> > a) The Nforce2 DASP speculates and gets it right, pre-fetching the code for the
> > local apic timer interrupt, so the interrupt code executes sooner after
> > activation than it does with other chipsets for AMD.
> >
> > b) The AMD cpu may not be over its timing and stability issues when coming
> > out of C1 disconnect. Plenty stable soon enough for other chipsets and other
> > codepaths in linux which pull the cpu out of C1 disconnect, but not quite soon
> > enough for the "cached" short code path to the local apic timer ack. So most
> > of the time any latent lockup potential is not realised, but on occasion
> > we hit it.
> >
> > Disclaimer: I can think of so many ways I could be way off so I have a support
> > request in with AMD on the topic which I am told is progressing. I do not have
> > the resources to be able to confirm or deny the above theory. Patches are
> > GPL'ed as per the licenses of the files they go into.
> >
> >
> > Are my patches the only way to workaround lockups?
> > NO Firstly I think the manufacturers will fix the problem so I will not have to
> > use any workarounds one day. There are early reports that recent Award bios's
> > are fixing the lockup issue but not the nmi_watchdog problem.
> >
> > Others advise you can successfully use the C1 disconnect patch for the kernel,
> > also Athcool. Some bios also have options to alter the C1 Disconnect
> > state and some early bios are preset with it to always off. If the lockups would
> > only occur with disconnect on (yet to be confirmed or denied by manufacturers)
> > then this would be one sure way to stop them.
> >
> > Why don't you just disable the C1 disconnect?
> > I don't think it would be wise for me to frob with it for many reasons. Nvidia
> > has some seriously patented smarts in their ram controllers, and the ambient
> > temperature today, where I am, rose to 30C or about 85F and it gets hotter. I
> > also have a system in the north of Western Australia where 40C+ summer is
> > normal. The AMD processor data sheet says that the CPU only enters low power
> > state if it can disconnect. Web postings indicate 8C to 10C temperature increase
> > with disconnect disabled.
> >
> > Do my patches Fix all of it?
> > No they are just workarounds. How well they work around may depend upon your
> > system configuration, and how well the delay times chosen suit it.
> >
> > Any evil side effects?
> > Maybe, but I don't know of any yet. Any slowing from the delay is so far
> > not noticeable.
> >
> > Why 2 patches?
> > The apic timer ack delay patch is for the hard lockups.
> > The io-apic edge patch is for lost interrupts and also gives nmi_watchdog=1
> > functionality.
> >
> > Should I install any or both?
> > Depends, you get to decide until the manufacturers fix it.
> >
> > The io-apic edge seems beneficial to all Nforce2 so far as the bios reports the
> > 8254 timer connected to INTIN2. I found it connected directly to INTIN0.
> > The patch forces connection to INTIN0 to see if it works only after the existing
> > code has tried to connect it to where the bios said it was. The io-apic patch
> > compiles in only with acpi on and uniprocessor x86 io-apic on in kern config.
> > The acpi config is essential as we use part of it, the uniprocessor x86 io-apic
> > is there as a precaution and if the patch is more widely tested it may not be
> > required. Remove it if you want the patch in smp mode.
> >
> > The apic timer ack is not needed if you do not have hard lockups and you can
> > solve the hard lockups by other means if you want to. I do not recommend kern
> > config of apic without io-apic for nforce2. The two work best together and if
> > you use my apic timer ack delay patch then use my io-apic edge patch with it
> > for best results.
> >
> > This version of the apic timer ack delay patch is not on by default. You need
> > to use a kernel argument at boot time to invoke it and there are two forms.
> >
> > apic_tack=1
> >
> > puts a small delay inline with the code. It is the reliable fallback mode.
> >
> > apic_tack=2
> >
> > reads the apic timer count and only delays if needed to prevent a lockup. This
> > mode assumes it is always safe to read the apic timer count register but
> > unsafe at times to ack it. This assumption could be wrong.
> >
> > I find apic_tack=1 better at stopping hard lockups on my T/bred XP2200
> > (266fsb DDR400 ram) system with my patched 2.4.23 kern but apic_tack=2 ok
> > with 2.6.0 on same machine? I also note 2.4.23 gives me 53Mb/s and 2.6.0 only
> > 47Mb/s with hdparm -t /dev/hda testing.
> >
> > Other kernel versions?
> > It should work fine with 2.4.22 and 2.4.23 although you will probably have to hand
> > patch them for now.
> >
> > Historical postings on the topic?
> > Heaps, many thanks to those who have tested and contributed so far.
> > Primary thread on this topic for my patches
> > Fixes for nforce2 hard lockup, apic, io-apic, udma133 covered
> > and another useful discussion
> > Catching NForce2 lockup with NMI watchdog
> >
> > Debugging output?
> > Ioapic patch always reports success or otherwise as part of check_timer if
> > it is needed such as:
> > ..TIMER: vector=0x31 pin1=2 pin2=-1
> > ..MP-BIOS bug: 8254 timer not connected to IO-APIC INTIN2
> > ..TIMER: Is timer irq0 connected to IO-APIC INTIN0? ...
> > IOAPIC[0]: Set PCI routing entry (2-0 -> 0x31 -> IRQ 0 Mode:0 Active:0)
> > ..TIMER: works OK on IO-APIC INTIN0 irq0
> >
> > None for apic ackdelay apic_tack=0 or apic_tack=1
> >
> > apic_tack=2 always reports:
> > ..APIC TIMER ack delay, reload:16701, safe:16691
> >
> > but numbers vary with your fsb.
> > reload is your existing local timer 1ms reload count.
> > safe is the down count at which about 800ns has expired since reload. Patch
> > considers it safe to ack after that time.
> >
> > More can be had if
> > #define APIC_DEBUG 0
> > is set to 1 in
> > /usr/src/linux-2.4.23-rd2/include/asm-i386/apic.h
> >
> > The io-apic patch will also display the 8259a xtpic interrupt mask.
> > I get hex fb and hex ff indicating that the only int enabled on the 8259A xtpic
> > which handles irq 0-7 is the cascade irq 2 which is OK because on the other
> > 8259A irq 8-15 are masked. This masked xtpic should always be the case.
> > We want the 8259A off during our test to see if the 8254 timer is connected
> > directly to pin 0 of the ioapic.
> >
> > The apic ack delay patch will report at boot ten pre delay apic timer times to
> > get a feel as to whether the delay had to kick in or not. Note that this is a
> > very small sample size but it does give an idea of the timing numbers.
> >
> > Following are my apic and ioapic patches for 2.6.0.
> > bzip also attached.
> >
> > Regards
> > Ross Dickson.
> >
<snip>
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: Updated Lockup Patches, 2.6.0 Nforce2, apic timer ack delay, ioapic edge for NMI debug
2003-12-22 23:56 ` Ross Dickson
@ 2003-12-24 15:41 ` Daniel Drake
0 siblings, 0 replies; 9+ messages in thread
From: Daniel Drake @ 2003-12-24 15:41 UTC (permalink / raw)
To: linux-kernel
(also sent to Ross, forgot to CC LKML)...
Ross Dickson wrote:
> On Tuesday 23 December 2003 04:50, Daniel Drake wrote:
>
>> I then recompiled the kernel with your patches and since then it's been fine.
I spoke to soon. After your two newest patches (APIC & IOAPIC from thread
starting mail) the system did freeze again (twice) after several hours of
usage each time. I have been running with apic_tack=2 since then and have not
experienced any lockups.
> 2.6.0-test11-mm1 picked up the earlier two patches which were the topic
> of "Catching NForce2 lockup with NMI watchdog"
>
> Others had reported that those patches achieved stability?
Not for me.
Thanks
-Daniel
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: Updated Lockup Patches, 2.6.0 Nforce2, apic timer ack delay, ioapic edge for NMI debug
2003-12-21 9:17 Updated Lockup Patches, 2.6.0 Nforce2, apic timer ack delay, ioapic edge for NMI debug Ross Dickson
2003-12-21 16:29 ` Craig Bradney
2003-12-22 18:50 ` Daniel Drake
@ 2003-12-24 17:49 ` really bensoo_at_soo_dot_com
2004-02-07 14:55 ` Nforce2 apic timer ack delay Jamie Lokier
3 siblings, 0 replies; 9+ messages in thread
From: really bensoo_at_soo_dot_com @ 2003-12-24 17:49 UTC (permalink / raw)
To: Ross Dickson, linux-kernel
i have an MSI K7N2, the original NFORCE2
chipset that doesn't do over 200MHz FSB,
running an overclocked Barton 2500 at
195MHz FSB, on 1Gig RAM.
These patches have worked for me. At
apic_tack=2 been up for close to two days
running the kind of load that before would
lock up the box in less than an hour.
So nice to have a reliable box again.
Season's Greetings; many thanks,
b
On Sun, Dec 21, 2003 at 07:17:05PM +1000, Ross Dickson wrote:
> Do my patches Fix all of it?
> No they are just workarounds. How well they work around may depend upon your
> system configuration, and how well the delay times chosen suit it.
>
> Any evil side effects?
> Maybe, but I don't know of any yet. Any slowing from the delay is so far
> not noticeable.
>
> Why 2 patches?
> The apic timer ack delay patch is for the hard lockups.
> The io-apic edge patch is for lost interrupts and also gives nmi_watchdog=1
> functionality.
>
> Should I install any or both?
> Depends, you get to decide until the manufacturers fix it.
>
^ permalink raw reply [flat|nested] 9+ messages in thread
* Nforce2 apic timer ack delay
2003-12-21 9:17 Updated Lockup Patches, 2.6.0 Nforce2, apic timer ack delay, ioapic edge for NMI debug Ross Dickson
` (2 preceding siblings ...)
2003-12-24 17:49 ` really bensoo_at_soo_dot_com
@ 2004-02-07 14:55 ` Jamie Lokier
2004-02-08 11:41 ` Ross Dickson
3 siblings, 1 reply; 9+ messages in thread
From: Jamie Lokier @ 2004-02-07 14:55 UTC (permalink / raw)
To: Ross Dickson; +Cc: linux-kernel
Ross Dickson wrote:
> a) The Nforce2 DASP speculates and gets it right, pre-fetching the
> code for the local apic timer interrupt, so the interrupt code
> executes sooner after activation than it does with other chipsets
> for AMD.
>
> b) The AMD cpu may not be over its timing and stability issues when
> coming out of C1 disconnect. Plenty stable soon enough for other
> chipsets and other codepaths in linux which pull the cpu out of C1
> disconnect, but not quite soon enough for the "cached" short code
> path to the local apic timer ack. So most of the time any latent
> lockup potential is not realised, but on occasion we hit it.
Ross,
Is the AMD C1 Disconnect state only entered when the CPU is idle, as in a
"hlt" instruction?
If it is, we could set a flag just before the "hlt" instruction in the
idle task and clear it afterwards. If the flag is set, the interrupt
path could clear the flag and do the delay thing. Then you could use
a longer, safe delay, and have it in the generic intrrupt path not
just the local apic timer. A delay coming out of the idle state is no
big deal.
-- Jamie
^ permalink raw reply [flat|nested] 9+ messages in thread* Re: Nforce2 apic timer ack delay
2004-02-07 14:55 ` Nforce2 apic timer ack delay Jamie Lokier
@ 2004-02-08 11:41 ` Ross Dickson
2004-02-09 12:33 ` Jamie Lokier
0 siblings, 1 reply; 9+ messages in thread
From: Ross Dickson @ 2004-02-08 11:41 UTC (permalink / raw)
To: Jamie Lokier; +Cc: linux-kernel, Ian Kumlien, Jesse Allen
On Sunday 08 February 2004 00:55, Jamie Lokier wrote:
> Ross Dickson wrote:
> > a) The Nforce2 DASP speculates and gets it right, pre-fetching the
> > code for the local apic timer interrupt, so the interrupt code
> > executes sooner after activation than it does with other chipsets
> > for AMD.
> >
> > b) The AMD cpu may not be over its timing and stability issues when
> > coming out of C1 disconnect. Plenty stable soon enough for other
> > chipsets and other codepaths in linux which pull the cpu out of C1
> > disconnect, but not quite soon enough for the "cached" short code
> > path to the local apic timer ack. So most of the time any latent
> > lockup potential is not realised, but on occasion we hit it.
>
> Ross,
>
> Is the AMD C1 Disconnect state only entered when the CPU is idle, as in a
> "hlt" instruction?
Yes that is how I read the Athlon data sheet. It also needs the disconnect bit
set in the Northbridge.
>
> If it is, we could set a flag just before the "hlt" instruction in the
> idle task and clear it afterwards. If the flag is set, the interrupt
> path could clear the flag and do the delay thing. Then you could use
> a longer, safe delay, and have it in the generic intrrupt path not
> just the local apic timer. A delay coming out of the idle state is no
> big deal.
>
> -- Jamie
>
>
>
Great Idea and I think well worth trying but...
I just tried booting with "idle=poll" and without my apic_tack arg
which to my understanding should avoid using the hlt instruction. I know
my system has the disconnect bit set by the bios.
I was just writing you to tell you how well it prevented lockups when my system
experienced a hard lockup in a slightly different form. Firstly mplayer bailed
complaining about misuse of cpu ram etc, which is an abnormal failure for
this system. I then swapped desktops back to continue my email and then
no mouse or anything else just like one of the original hard lockups.
A bit early to draw conclusions but could it be that the bios also executes
a hlt instruction in a System Management Interrupt?? Perhaps on detecting a
hot cpu or something??
Other info I have uncovered which I think disproves my above theory:
I earlier decided to try some more things with my apic ack delay patch and found
that it was still able to prevent the lockups on my system when moved to the end
of the smp_apic_timer_interrupt()????, way after the apic ack????
At least this is so on my preempt lowlat vhz64 1000hz 2.4.24 kernel.
What this tells me is that on an nforce2 system if the smp_apic_timer_interrupt()
routine completes execution in under about 1us time since the hardware
interrupt occurred then the system can (will?) hard lock-up.
Why?
I also altered the CLK_Ctl MSR bits along the lines of changes since model 4
Athlon and found nothing useful - they appear to me to be already optimised
on my model 8 cpu.
Regards
Ross.
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: Nforce2 apic timer ack delay
2004-02-08 11:41 ` Ross Dickson
@ 2004-02-09 12:33 ` Jamie Lokier
0 siblings, 0 replies; 9+ messages in thread
From: Jamie Lokier @ 2004-02-09 12:33 UTC (permalink / raw)
To: Ross Dickson; +Cc: linux-kernel, Ian Kumlien, Jesse Allen
Ross Dickson wrote:
> I was just writing you to tell you how well it prevented lockups
> when my system experienced a hard lockup in a slightly different
> form. Firstly mplayer bailed complaining about misuse of cpu ram
> etc, which is an abnormal failure for this system. I then swapped
> desktops back to continue my email and then no mouse or anything
> else just like one of the original hard lockups.
Well, that's a different form of lockup so it might be a different
culprit. What was the mplayer error?
> A bit early to draw conclusions but could it be that the bios also
> executes a hlt instruction in a System Management Interrupt??
> Perhaps on detecting a hot cpu or something??
Perhaps. Are you using APM or ACPI? I think they override
"idle=poll" and call into the BIOS for idling.
-- Jamie
^ permalink raw reply [flat|nested] 9+ messages in thread