4way smp on ppc32

4way smp on ppc32

[ERIC NORTHUP]

Hi all,

I have been trying to get my 4-processor Daystar GenesisMP system to run Linux in smp mode, without success.  The system works reliably with a uniprocessor kernel.  But it oopses on boot with an smp kernel.  I have attached a boot log from a serial console for a 2.6 kernel, but I get similar messages  with other kernels I've tried.  Although the log is from 2.6.4, I've tried 2.6.5-pre3 which included some ppc32 fixes, though they didn't fix this problem.  I've also tried different gcc versions.

I don't understand why only 2 of the 4 cpus would respond to the smp_call_function().  Any suggestions for where to start looking?  I am not subscribed to linuxppc-dev, so please CC me on replies.  Thanks,

--Eric
Total memory = 576MB; using 2048kB for hash table (at c0400000)
Linux version 2.6.4up (eric@debian-genesismp) (gcc version 2.95.4 20011002 (Debian prerelease)) #5 SMP Mon Mar 29 13:55:55 EST 2004
Found a Grand Central mac-io controller, rev: 2, mapped at 0xfddd4000
PowerMac motherboard: PowerMac 9500/9600
Cache coherency enabled for bandit/PSX
Found Bandit PCI host bridge at 0xf2000000. Firmware bus number: 0->0
Cache coherency enabled for bandit/PSX
Found Bandit PCI host bridge at 0xf4000000. Firmware bus number: 1->1
nvram: OF partition at 0x1800
nvram: XP partition at 0x1300
nvram: NR partition at 0x1400
On node 0 totalpages: 147456
  DMA zone: 147456 pages, LIFO batch:16
  Normal zone: 0 pages, LIFO batch:1
  HighMem zone: 0 pages, LIFO batch:1
Built 1 zonelists
Kernel command line: root=/dev/sda5 console=ttyS0 init=/bin/bash
System has 32 possible interrupts
PID hash table entries: 4096 (order 12: 32768 bytes)
GMT Delta read from XPRAM: 0 minutes, DST: off
via_calibrate_decr: ticks per jiffy = 11249 (674990 ticks)
Console: colour dummy device 80x25
Memory: 577920k available (1980k kernel code, 1080k data, 348k init, 0k highmem)
Calibrating delay loop... 358.40 BogoMIPS
Dentry cache hash table entries: 131072 (order: 7, 524288 bytes)
Inode-cache hash table entries: 65536 (order: 6, 262144 bytes)
Mount-cache hash table entries: 512 (order: 0, 4096 bytes)
POSIX conformance testing by UNIFIX
CPU 1 done callin...
CPU 1 done setup...
CPU 1 done timebase take...
Processor 1 found.
CPU 2 done callin...
CPU 2 done setup...
CPU 2 done timebase take...
Processor 2 found.
CPU 3 done callin...
CPU 3 done setup...
CPU 3 done timebase take...
Processor 3 found.
Brought up 4 CPUs
NET: Registered protocol family 16
PCI: Probing PCI hardware
smp_call_function on cpu 0: other cpus not responding (2)
kernel BUG in smp_call_function_all_cpus at mm/slab.c:1364!
Oops: Exception in kernel mode, sig: 5 [#1]
NIP: C00405E0 LR: C00405D8 SP: C0BABED0 REGS: c0babe20 TRAP: 0700    Not tainted
MSR: 00029032 EE: 1 PR: 0 FP: 0 ME: 1 IR/DR: 11
TASK = c0badb40[1] 'swapper' Last syscall: 120  CPU: 0
GPR00: C00405D8 C0BABED0 C0BADB40 FFFFFFFF 000007C3 00000000 C0290000 C0255AA4
GPR08: 000007C3 00000000 C0300000 C0255A9C 42002024 003530A4 ABB20028 46200028
GPR16: 480002BD 1AC2ABCD 73637269 00000000 C0290000 C0300000 00000000 E3FFA348
GPR24: 00000008 00000010 00000020 C0BABEFC C0BABEF8 C0042760 E3FFA300 00000000
Call trace:
 [c0042858] do_tune_cpucache+0xbc/0x290
 [c0042ae0] enable_cpucache+0xb4/0xe4
 [c004027c] kmem_cache_create+0x3e4/0x6e0
 [c02af6e8] init_bio+0x2c/0xd0
 [c029a85c] do_initcalls+0x90/0x10c
 [c029a8f8] do_basic_setup+0x20/0x30
 [c0003968] init+0x4c/0x114
 [c000afc4] kernel_thread+0x44/0x60
Kernel panic: Attempted to kill init!
 smp_call_function on cpu 0: other cpus not responding (2)


** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Re: 4way smp on ppc32

[Benjamin Herrenschmidt]

On Wed, 2004-03-31 at 11:57, ERIC NORTHUP wrote:
> Hi all,
>
> I have been trying to get my 4-processor Daystar GenesisMP system to run Linux in smp mode,
> without success.  The system works reliably with a uniprocessor kernel.
> But it oopses on boot with an smp kernel.  I have attached a boot log from a
> serial console for a 2.6 kernel, but I get similar messages  with other
> kernels I've tried.  Although the log is from 2.6.4, I've tried 2.6.5-pre3 which
> included some ppc32 fixes, though they didn't fix this problem.
> I've also tried different gcc versions.
>
> I don't understand why only 2 of the 4 cpus would respond to the smp_call_function().
> Any suggestions for where to start looking?  I am not subscribed to linuxppc-dev,
> so please CC me on replies.  Thanks,

Interesting... Apparently, all the CPUs are coming in, but interrupts
don't get properly delivered to the "other" CPUs...

I'll have a look, that setup should work. Did you try a 2.4 kernel ?

Ben.


** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Re: 4way smp on ppc32

[Benjamin Herrenschmidt]

Actually, there is at least one problem, can you try this patch:

===== arch/ppc/platforms/pmac_smp.c 1.30 vs edited =====
--- 1.30/arch/ppc/platforms/pmac_smp.c	Tue Mar 23 12:46:59 2004
+++ edited/arch/ppc/platforms/pmac_smp.c	Wed Mar 31 18:16:54 2004
@@ -316,6 +316,8 @@

 	psurge_type = psurge_quad_probe();
 	if (psurge_type != PSURGE_DUAL) {
+		psurge_smp_ops.take_timebase = smp_generic_take_timebase;
+		psurge_smp_ops.give_timebase = smp_generic_give_timebase;
 		psurge_quad_init();
 		/* All released cards using this HW design have 4 CPUs */
 		ncpus = 4;


** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Re: 4way smp on ppc32

[Mark Guertin]

I'll try and find some time to test this stuff on my daystar quad as
well.  I know that right up to latest 2.4.x all 4 of my CPU's are still
working fine.

Gerk

On 31-Mar-04, at 3:17 AM, Benjamin Herrenschmidt wrote:

>
> Actually, there is at least one problem, can you try this patch:
>
> ===== arch/ppc/platforms/pmac_smp.c 1.30 vs edited =====
> --- 1.30/arch/ppc/platforms/pmac_smp.c	Tue Mar 23 12:46:59 2004
> +++ edited/arch/ppc/platforms/pmac_smp.c	Wed Mar 31 18:16:54 2004
> @@ -316,6 +316,8 @@
>
>  	psurge_type = psurge_quad_probe();
>  	if (psurge_type != PSURGE_DUAL) {
> +		psurge_smp_ops.take_timebase = smp_generic_take_timebase;
> +		psurge_smp_ops.give_timebase = smp_generic_give_timebase;
>  		psurge_quad_init();
>  		/* All released cards using this HW design have 4 CPUs */
>  		ncpus = 4;
>
>
>
>


** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Re: 4way smp on ppc32

[ERIC NORTHUP]

On Wednesday, March 31, 2004 12:56 pm, Mark Guertin wrote:
> I'll try and find some time to test this stuff on my daystar quad
> as
> well.  I know that right up to latest 2.4.x all 4 of my CPU's are
> still
> working fine.

OK, I have just gotten a 2.4 smp kernel booted, and it does find all 4 cpus.  However, the machine is far from stable.  GCC dies on a fatal signal or reports an "internal compiler error", but when re-inovked on the same file it works.  Other times, it takes the ssh connection down with it.  I was trying to build LTP or some other code to narrow down the problem areas, but gcc is too flaky for this to work.

> On 31-Mar-04, at 3:17 AM, Benjamin Herrenschmidt wrote:
>
> >
> > Actually, there is at least one problem, can you try this patch:
> >
> > ===== arch/ppc/platforms/pmac_smp.c 1.30 vs edited =====
> > --- 1.30/arch/ppc/platforms/pmac_smp.c	Tue Mar 23 12:46:59 2004
> > +++ edited/arch/ppc/platforms/pmac_smp.c	Wed Mar 31 18:16:54 2004
> > @@ -316,6 +316,8 @@
> >
> >  	psurge_type = psurge_quad_probe();
> >  	if (psurge_type != PSURGE_DUAL) {
> > +        	psurge_smp_ops.take_timebase = smp_generic_take_timebase;
> > +        	psurge_smp_ops.give_timebase = smp_generic_give_timebase;
> >          	psurge_quad_init();
> >          	/* All released cards using this HW design have 4 CPUs */
> >          	ncpus = 4;

I applied this patch with the addition of a forward declaration

struct smp_ops_t psurge_smp_ops;

The system did not boot (it crashed earler than without the patchbut i did not get an oops on the serial console)

--Eric


** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Re: 4way smp on ppc32

[Mark Guertin]

I've not had any of those problems with my machine, it runs very solid
and I've in fact compiled a _lot_ of code on it, including hundreds
(maybe thousands) of kernel builds for ppckernel.org .. it was the main
build machine for over a year.  It also did a lot of rebuilding for
testing of various packages when I was doing Gentoo PPC work.

What distro are you using?  What versions of GCC, etc?

Mark

On 3-Apr-04, at 7:59 PM, ERIC NORTHUP wrote:

> On Wednesday, March 31, 2004 12:56 pm, Mark Guertin wrote:
>> I'll try and find some time to test this stuff on my daystar quad
>> as
>> well.  I know that right up to latest 2.4.x all 4 of my CPU's are
>> still
>> working fine.
>
> OK, I have just gotten a 2.4 smp kernel booted, and it does find all 4
> cpus.  However, the machine is far from stable.  GCC dies on a fatal
> signal or reports an "internal compiler error", but when re-inovked on
> the same file it works.  Other times, it takes the ssh connection down
> with it.  I was trying to build LTP or some other code to narrow down
> the problem areas, but gcc is too flaky for this to work.


** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Re: 4way smp on ppc32

[ERIC NORTHUP]

> I've not had any of those problems with my machine, it runs very
> solid
> and I've in fact compiled a _lot_ of code on it, including hundreds
> (maybe thousands) of kernel builds for ppckernel.org .. it was the
> main
> build machine for over a year.  It also did a lot of rebuilding for
> testing of various packages when I was doing Gentoo PPC work.
>
> What distro are you using?  What versions of GCC, etc?

I'm mostly cross-compiling the kernels with gcc 3.3.2.  The userland is an old Debian, but I'll upgrade that as soon as I get a kernel booting.

Do you have recommendations of gcc/kernel combos that should work?  I'm also starting to wonder if my h/w is bad (it did come from eBay. . .)

--Eric


** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Re: 4way smp on ppc32

[Benjamin Herrenschmidt]

> I'm mostly cross-compiling the kernels with gcc 3.3.2.  The userland is an old Debian, but I'll upgrade that as soon as I get a kernel booting.
>
> Do you have recommendations of gcc/kernel combos that should work?  I'm also starting to wonder if my h/w is bad (it did come from eBay. . .)

Could well be bad RAM or bad cache

Ben.


** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Re: 4way smp on ppc32

[Mark Guertin]

On 6-Apr-04, at 2:15 AM, Benjamin Herrenschmidt wrote:

>
>> I'm mostly cross-compiling the kernels with gcc 3.3.2.  The userland
>> is an old Debian, but I'll upgrade that as soon as I get a kernel
>> booting.
>>
>> Do you have recommendations of gcc/kernel combos that should work?
>> I'm also starting to wonder if my h/w is bad (it did come from eBay.
>> . .)
>
> Could well be bad RAM or bad cache
>
> Ben.
>

I was going to mention the same thing, to run some hardware testing if
you could.  The only time I've seen any real ICE problems was with gcc
3.1.x and projects that weren't really adjusted for gcc 3.x yet, other
than that things have been fine for my setup.

Mark


** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/