Patch - Sun4d SMP now boots

Patch - Sun4d SMP now boots

[Chris Newport]

I have now applied the patch posted to the list by Pasi Pirhonen
on 19Nov2003. This needs to be applied AFTER the patches from Thomas.

More work is needed to get rid of the previously reported compiler
warnings in the same function before this is fit to be submitted, but
I will have to leave that to those who understand what they are doing
<B-).

This now boots and runs, but is horribly unstable.

==== BEGIN INCLUDE

root@phobos:~# cd /usr/src/linux/arch/sparc/kernel
root@phobos:/usr/src/linux/arch/sparc/kernel# diff -ru sun4d_smp.csaved sun4d_s
mp.c

--- sun4d_smp.csaved	2004-08-19 13:12:24.000000000 +0100
+++ sun4d_smp.c	2004-08-19 13:46:07.000000000 +0100
@@ -340,7 +340,8 @@
 		register int high = smp_highest_cpu;
 		unsigned long flags;

-		spin_lock_irqsave(&cross_call_lock, flags);
+		//spin_lock_irqsave(&cross_call_lock, flags);
+		spin_lock_bh(&cross_call_lock);

 		{
 			/* If you make changes here, make sure gcc generates proper code... */
@@ -373,6 +374,8 @@
 				}
 			}
 		}
+		/* first run local copy */
+		func(arg1, arg2, arg3, arg4, arg5);

 		{
 			register int i;
@@ -390,8 +393,10 @@
 			} while(++i <= high);
 		}

-		spin_unlock_irqrestore(&cross_call_lock, flags);
-	}
+		//spin_unlock_irqrestore(&cross_call_lock, flags);
+		spin_unlock_bh(&cross_call_lock);
+	} else
+		func(arg1, arg2, arg3, arg4, arg5); /* just need to run local copy */
 }

 /* Running cross calls. */

===== END INCLUDE

Re: Patch - Sun4d SMP now boots

[William Lee Irwin III]

On Thu, Aug 19, 2004 at 02:32:47PM +0100, Chris Newport wrote:
> I have now applied the patch posted to the list by Pasi Pirhonen
> on 19Nov2003. This needs to be applied AFTER the patches from Thomas.
> More work is needed to get rid of the previously reported compiler
> warnings in the same function before this is fit to be submitted, but
> I will have to leave that to those who understand what they are doing
> <B-).
> This now boots and runs, but is horribly unstable.

Okay, this is great news. I know the spin_lock_bh() is wrong, but it
gets us to a point where we can actually do something and we actually
have enough to reproduce all this.

One major question is *WHY* spin_lock_bh() vs. spin_lock_irqsave()
works. Only trouble there is that we can't see what happens after
spin_lock_irqsave() very easily. One rather large possibility is that
since disabling bh's doesn't disable interrupts, some smp_call_function()
or similar that would deadlock if interrupts weren't disabled suddenly
appears to work and/or works for some limited cases...

This is in my queue also, though preferably some explanation of what's
going on with this (e.g. the real issue, maybe a better fix) will
happen instead.


-- wli

Re: Patch - Sun4d SMP now boots

[David S. Miller]

On Thu, 19 Aug 2004 07:12:13 -0700
William Lee Irwin III <wli@holomorphy.com> wrote:

> One major question is *WHY* spin_lock_bh() vs. spin_lock_irqsave()
> works. Only trouble there is that we can't see what happens after
> spin_lock_irqsave() very easily. One rather large possibility is that
> since disabling bh's doesn't disable interrupts, some smp_call_function()
> or similar that would deadlock if interrupts weren't disabled suddenly
> appears to work and/or works for some limited cases...
> 
> This is in my queue also, though preferably some explanation of what's
> going on with this (e.g. the real issue, maybe a better fix) will
> happen instead.

Another thing to keep in mind on sparc32 is that %psr writes have
a 3 instruction latency, which is why we usually put 3 nops after it.
Also, writing garbage into the %psr can easily wedge the chip or
cause it to take a fatal trap.

Re: Patch - Sun4d SMP now boots

[C.Newport]

On Thursday 19 August 2004 3:12 pm, William Lee Irwin III wrote:
> On Thu, Aug 19, 2004 at 02:32:47PM +0100, Chris Newport wrote:

> One major question is *WHY* spin_lock_bh() vs. spin_lock_irqsave()
> works. Only trouble there is that we can't see what happens after
> spin_lock_irqsave() very easily. One rather large possibility is that
> since disabling bh's doesn't disable interrupts, some smp_call_function()
> or similar that would deadlock if interrupts weren't disabled suddenly
> appears to work and/or works for some limited cases...

Two possible clues :-

Other architectures have 2 sets of functions here, one for use before it is
safe to allow SMP with IRQs.

sun4d_irq.c is almost certainly involved, and does not appear to deal with
SMP in a sane manner. Compare with sun4m_irq.c which is much newer.
This is consistent with the observed failure, the system locks as soon as
a write to disk happens. Pasi also noted failures as soon as interrupts 
happen.

Re: Patch - Sun4d SMP now boots

[William Lee Irwin III]

On Thursday 19 August 2004 3:12 pm, William Lee Irwin III wrote:
>> One major question is *WHY* spin_lock_bh() vs. spin_lock_irqsave()
>> works. Only trouble there is that we can't see what happens after
>> spin_lock_irqsave() very easily. One rather large possibility is that
>> since disabling bh's doesn't disable interrupts, some smp_call_function()
>> or similar that would deadlock if interrupts weren't disabled suddenly
>> appears to work and/or works for some limited cases...

On Thu, Aug 19, 2004 at 03:49:18PM +0100, C.Newport wrote:
> Two possible clues :-
> Other architectures have 2 sets of functions here, one for use before it is
> safe to allow SMP with IRQs.
> sun4d_irq.c is almost certainly involved, and does not appear to deal with
> SMP in a sane manner. Compare with sun4m_irq.c which is much newer.
> This is consistent with the observed failure, the system locks as soon as
> a write to disk happens. Pasi also noted failures as soon as interrupts 
> happen.

Sounds like the analogous rewrite must happen for sun4d, then. I'll look
into that... some kind of sun4d hacking-level access may need to happen,
or someone may have to look into this on my behalf.

-- wli

Re: Patch - Sun4d SMP now boots

[C.Newport]

On Thursday 19 August 2004 4:00 pm, William Lee Irwin III wrote:

>
> Sounds like the analogous rewrite must happen for sun4d, then. I'll look
> into that... some kind of sun4d hacking-level access may need to happen,
> or someone may have to look into this on my behalf.

How long before you get your machine ?.

You can have both telnet and serial console access to my SS1000E 
but we will probably need to keep an IRC channel open so that I can
pull the plug every time you wedge it to the point where <break> will
not help.

Re: Patch - Sun4d SMP now boots

[William Lee Irwin III]

On Thursday 19 August 2004 4:00 pm, William Lee Irwin III wrote:
>> Sounds like the analogous rewrite must happen for sun4d, then. I'll look
>> into that... some kind of sun4d hacking-level access may need to happen,
>> or someone may have to look into this on my behalf.

On Thu, Aug 19, 2004 at 04:22:51PM +0100, C.Newport wrote:
> How long before you get your machine ?.
> You can have both telnet and serial console access to my SS1000E 
> but we will probably need to keep an IRC channel open so that I can
> pull the plug every time you wedge it to the point where <break> will
> not help.

It looks months out. I've been reluctant to cash in on this as this
sort of arrangement is rather awkward, but I probably should cash in
on it soon.


-- wli