SMP in 32-bit arch/powerpc

SMP in 32-bit arch/powerpc

[Adrian Cox]

Is anybody else having problems with 32-bit SMP support in arch/powerpc?
I'm using 2.6.17 as my current base, because I've not yet merged the
latest mpic changes.

I'm currently bringing up a dual-7448 board, and when I build the kernel
with CONFIG_SMP, the bootmem allocator corrupts the device tree. The
strange thing is, this still happens when I don't start the second CPU.
Kernels built without CONFIG_SMP run flawlessly on the same hardware.

-- 
Adrian Cox <adrian@humboldt.co.uk>

Re: SMP in 32-bit arch/powerpc

[Michael Ellerman]

[-- Attachment #1: Type: text/plain, Size: 1605 bytes --]

On Wed, 2006-08-02 at 09:25 +0100, Adrian Cox wrote:
> Is anybody else having problems with 32-bit SMP support in arch/powerpc?
> I'm using 2.6.17 as my current base, because I've not yet merged the
> latest mpic changes.
> 
> I'm currently bringing up a dual-7448 board, and when I build the kernel
> with CONFIG_SMP, the bootmem allocator corrupts the device tree. The
> strange thing is, this still happens when I don't start the second CPU.
> Kernels built without CONFIG_SMP run flawlessly on the same hardware.

Seems to be working OK here:

fuego:~# uname -a
Linux fuego 2.6.17 #1 SMP Mon Jul 3 12:10:43 EST 2006 ppc GNU/Linux
fuego:~# cat /proc/cpuinfo
processor       : 0
cpu             : 604r
clock           : 332.000000MHz
revision        : 49.2 (pvr 0009 3102)
bogomips        : 82.68

processor       : 1
cpu             : 604r
clock           : 332.000000MHz
revision        : 49.2 (pvr 0009 3102)
bogomips        : 82.68

processor       : 2
cpu             : 604r
clock           : 332.000000MHz
revision        : 49.2 (pvr 0009 3102)
bogomips        : 82.68

processor       : 3
cpu             : 604r
clock           : 332.000000MHz
revision        : 49.2 (pvr 0009 3102)
bogomips        : 82.68

total bogomips  : 330.75
timebase        : 41500582
platform        : CHRP
machine         : CHRP IBM,7025-F50

cheers

-- 
Michael Ellerman
IBM OzLabs

wwweb: http://michael.ellerman.id.au
phone: +61 2 6212 1183 (tie line 70 21183)

We do not inherit the earth from our ancestors,
we borrow it from our children. - S.M.A.R.T Person

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 191 bytes --]

RE: SMP in 32-bit arch/powerpc

[Zhang Wei-r63237]

Do you have more detailed descriptions? Our MPC8641D(dual-7448 core CPU)
board works well.

Best Regards,
Zhang Wei=20

> -----Original Message-----
> From: linuxppc-dev-bounces+wei.zhang=3Dfreescale.com@ozlabs.org=20
> [mailto:linuxppc-dev-bounces+wei.zhang=3Dfreescale.com@ozlabs.or
> g] On Behalf Of Adrian Cox
> Sent: Wednesday, August 02, 2006 4:26 PM
> To: linuxppc-dev@ozlabs.org
> Subject: SMP in 32-bit arch/powerpc
>=20
> Is anybody else having problems with 32-bit SMP support in=20
> arch/powerpc?
> I'm using 2.6.17 as my current base, because I've not yet merged the
> latest mpic changes.
>=20
> I'm currently bringing up a dual-7448 board, and when I build=20
> the kernel
> with CONFIG_SMP, the bootmem allocator corrupts the device tree. The
> strange thing is, this still happens when I don't start the=20
> second CPU.
> Kernels built without CONFIG_SMP run flawlessly on the same hardware.
>=20
> --=20
> Adrian Cox <adrian@humboldt.co.uk>
>=20
> _______________________________________________
> Linuxppc-dev mailing list
> Linuxppc-dev@ozlabs.org
> https://ozlabs.org/mailman/listinfo/linuxppc-dev
>=20

Re: SMP in 32-bit arch/powerpc

[Kumar Gala]

On Aug 2, 2006, at 3:25 AM, Adrian Cox wrote:

> Is anybody else having problems with 32-bit SMP support in arch/ 
> powerpc?
> I'm using 2.6.17 as my current base, because I've not yet merged the
> latest mpic changes.
>
> I'm currently bringing up a dual-7448 board, and when I build the  
> kernel
> with CONFIG_SMP, the bootmem allocator corrupts the device tree. The
> strange thing is, this still happens when I don't start the second  
> CPU.
> Kernels built without CONFIG_SMP run flawlessly on the same hardware.

There are a few patches to prom.c that may help.  I know I've run  
into this on other 32-bit systems booting from u-boot.  These patches  
resolved my issue.

* [PATCH] powerpc: Auto reserve of device tree blob
* [POWERPC] Prevent duplicate lmb reservations for Device...

- kumar

Re: SMP in 32-bit arch/powerpc

[Jon Loeliger]

So, like, the other day Adrian Cox mumbled:
> Is anybody else having problems with 32-bit SMP support in arch/powerpc?
> I'm using 2.6.17 as my current base, because I've not yet merged the
> latest mpic changes.
> 
> I'm currently bringing up a dual-7448 board, and when I build the kernel
> with CONFIG_SMP, the bootmem allocator corrupts the device tree. The
> strange thing is, this still happens when I don't start the second CPU.
> Kernels built without CONFIG_SMP run flawlessly on the same hardware.

As a point of reference, the 32-bit 8641 HPCN seems to be working fine
with both CONFIG_SMP and the device tree mechanism in place.  It was working
both before and after the IRQ changes.

Can you nail down any more specifics on how it is failing or where
the corruption happens?

jdl

Re: SMP in 32-bit arch/powerpc

[Adrian Cox]

On Wed, 2006-08-02 at 08:32 -0500, Kumar Gala wrote:
> On Aug 2, 2006, at 3:25 AM, Adrian Cox wrote:
> > Is anybody else having problems with 32-bit SMP support in arch/ 
> > powerpc?
> > I'm using 2.6.17 as my current base, because I've not yet merged the
> > latest mpic changes.

> There are a few patches to prom.c that may help.  I know I've run  
> into this on other 32-bit systems booting from u-boot.  These patches  
> resolved my issue.
> * [PATCH] powerpc: Auto reserve of device tree blob
> * [POWERPC] Prevent duplicate lmb reservations for Device...

That was the problem. It now boots correctly, thanks.

-- 
Adrian Cox <adrian@humboldt.co.uk>

Re: SMP in 32-bit arch/powerpc

[Benjamin Herrenschmidt]

On Wed, 2006-08-02 at 09:42 -0500, Jon Loeliger wrote:
> So, like, the other day Adrian Cox mumbled:
> > Is anybody else having problems with 32-bit SMP support in arch/powerpc?
> > I'm using 2.6.17 as my current base, because I've not yet merged the
> > latest mpic changes.
> > 
> > I'm currently bringing up a dual-7448 board, and when I build the kernel
> > with CONFIG_SMP, the bootmem allocator corrupts the device tree. The
> > strange thing is, this still happens when I don't start the second CPU.
> > Kernels built without CONFIG_SMP run flawlessly on the same hardware.
> 
> As a point of reference, the 32-bit 8641 HPCN seems to be working fine
> with both CONFIG_SMP and the device tree mechanism in place.  It was working
> both before and after the IRQ changes.
> 
> Can you nail down any more specifics on how it is failing or where
> the corruption happens?

For completeness, there is a known bug with 32 bits and SMP regarding
icache coherency.... If you have random SIGILL/SEGV under load, that's
probably what you are hitting. The problem is due to the way we do the
coherency and isn't trivial to fix unfortunately, though it's also
fairly rare.

Ben.

Re: SMP in 32-bit arch/powerpc

[Adrian Cox]

On Thu, 2006-08-17 at 01:12 +0200, Benjamin Herrenschmidt wrote:
> For completeness, there is a known bug with 32 bits and SMP regarding
> icache coherency.... If you have random SIGILL/SEGV under load, that's
> probably what you are hitting. The problem is due to the way we do the
> coherency and isn't trivial to fix unfortunately, though it's also
> fairly rare.

Could you give a few more details?  I'd not heard of the problem before,
and I couldn't find any references with a few quick searches. I've not
seen the problem myself, but I've not run any heavy page fault loads,
only Altivec load.

-- 
Adrian Cox <adrian@humboldt.co.uk>

Re: SMP in 32-bit arch/powerpc

[Benjamin Herrenschmidt]

On Thu, 2006-08-17 at 08:19 +0100, Adrian Cox wrote:
> On Thu, 2006-08-17 at 01:12 +0200, Benjamin Herrenschmidt wrote:
> > For completeness, there is a known bug with 32 bits and SMP regarding
> > icache coherency.... If you have random SIGILL/SEGV under load, that's
> > probably what you are hitting. The problem is due to the way we do the
> > coherency and isn't trivial to fix unfortunately, though it's also
> > fairly rare.
> 
> Could you give a few more details?  I'd not heard of the problem before,
> and I couldn't find any references with a few quick searches. I've not
> seen the problem myself, but I've not run any heavy page fault loads,
> only Altivec load.

Currently, we do the sync in update_mmu_cache() which is called some
time after we instert a new PTE. Thus, there is the scenario where CPU A
gets in set_pte() to insert a new PTE for some code, and before it
reaches update_mmu_cache(), CPU B will hash in that PTE and try to
execute the code (typical case is swapping in code).

The "fix" would be to do the flush in hash_page but it's a bit
complicated since 32 bits hash_page is completely assembly, and going to
struct page PG_arch1 from there will require a few contorsions. We also
want to avoid 2 CPUs flushing at the same time. One thing I thought
about was to have set_pte "mirror" PG_arch1 into a PTE bit and have
hash_page fail with a minor fault whenever that bit isn't set,
effectively delaying until update_mmu_cache is called (you gotta hope
it's always called sometime after we do set_pte ... I'm not even sure we
can rely on that nowadays, I have to check)

Ben.