How many pentium-3 processors does SMP support?

How many pentium-3 processors does SMP support?

[Xinwei Xue]

Hi, 

My research group is planning to buy a multi-processor linux machine
for parallel computing. Does anyone know how many pentium processors the
linux SMP support? Does it support 8(pentium-three) processors? What
companies produce such reliable multi-processor (>4) machines? Thanks a
lot! 

Xinwei

Re: How many pentium-3 processors does SMP support?

[H. Peter Anvin]

Followup to:  <Pine.GSO.4.21.0107092315140.493-100000@faith.cs.utah.edu>
By author:    Xinwei Xue <xwxue@cs.utah.edu>
In newsgroup: linux.dev.kernel
> 
> Hi, 
> 
> My research group is planning to buy a multi-processor linux machine
> for parallel computing. Does anyone know how many pentium processors the
> linux SMP support? Does it support 8(pentium-three) processors? What
> companies produce such reliable multi-processor (>4) machines? Thanks a
> lot! 
> 

It supports up to 32, if you can find a machine that has that many.

	-hpa
-- 
<hpa@transmeta.com> at work, <hpa@zytor.com> in private!
"Unix gives you enough rope to shoot yourself in the foot."
http://www.zytor.com/~hpa/puzzle.txt

Re: How many pentium-3 processors does SMP support?

[Chris Wedgwood]

On Mon, Jul 09, 2001 at 10:34:24PM -0700, H. Peter Anvin wrote:

    It supports up to 32, if you can find a machine that has that
    many.

I think 8-way is about as high as anything common goes to, maybe
16. The cpu array is declared 32 long, maybe this should be changed to
8 by default?



  --cw

Re: How many pentium-3 processors does SMP support?

[Christoph Hellwig]

In article <20010711015128.E31799@weta.f00f.org> you wrote:
> On Mon, Jul 09, 2001 at 10:34:24PM -0700, H. Peter Anvin wrote:
>
>     It supports up to 32, if you can find a machine that has that
>     many.
>
> I think 8-way is about as high as anything common goes to, maybe
> 16. The cpu array is declared 32 long, maybe this should be changed to
> 8 by default?

Why limit the user?  There are more than enough Linux system that have
more than 32 CPUs (SGI, DEC, Sun).

Making it a per-architecture value or even a config option make a lot
more sense.

	Christoph

-- 
Of course it doesn't work. We've performed a software upgrade.

Re: How many pentium-3 processors does SMP support?

[Chris Wedgwood]

On Tue, Jul 10, 2001 at 04:12:00PM +0200, Christoph Hellwig wrote:

    Why limit the user?  There are more than enough Linux system that
    have more than 32 CPUs (SGI, DEC, Sun).

The limit is architecture dependant, for ia32/i386 is it 32.

    Making it a per-architecture value or even a config option make a
    lot more sense.

It turns out I was full of it, you can buy off the shelf 32-processor
systems.

In anyone from Compaq is reading this, you should send me a 32-way
Xeon ASAP just to prove they really work :)



  --cw

Re: How many pentium-3 processors does SMP support?

[Christoph Hellwig]

On Wed, Jul 11, 2001 at 02:16:39AM +1200, Chris Wedgwood wrote:
> On Tue, Jul 10, 2001 at 04:12:00PM +0200, Christoph Hellwig wrote:
> 
>     Why limit the user?  There are more than enough Linux system that
>     have more than 32 CPUs (SGI, DEC, Sun).
> 
> The limit is architecture dependant, for ia32/i386 is it 32.

The number of CPUs is currently globally limited to 32 by NR_CPUS in
include/linux/threads.h.

>     Making it a per-architecture value or even a config option make a
>     lot more sense.
> 
> It turns out I was full of it, you can buy off the shelf 32-processor
> systems.

You can.  But you cannot buy 32-processor PII (-Xeon) systems that are
supported by Linux.

> In anyone from Compaq is reading this, you should send me a 32-way
> Xeon ASAP just to prove they really work :)

It doesn't.

	Christoph

-- 
Of course it doesn't work. We've performed a software upgrade.

Re: How many pentium-3 processors does SMP support?

[Chris Wedgwood]

On Tue, Jul 10, 2001 at 04:19:43PM +0200, Christoph Hellwig wrote:

    The number of CPUs is currently globally limited to 32 by NR_CPUS in
    include/linux/threads.h.

Really?

<pause>

Ah, so it is... yes, making this architecture dependant might be a
good idea. Large PPC and MIPS boxen need to adjust this already. Also,
someone did a starfire port, I think that had 64 processors, not sure.

    You can.  But you cannot buy 32-processor PII (-Xeon) systems that are
    supported by Linux.

What is the limit here? The 8/16 way SE chipsets?

    > In anyone from Compaq is reading this, you should send me a 32-way
    > Xeon ASAP just to prove they really work :)
    
    It doesn't.

Oh, then they definately need to send me one.

Are these not MP1.4 based? Something different?



  --cw

Re: How many pentium-3 processors does SMP support?

[Christoph Hellwig]

On Wed, Jul 11, 2001 at 02:25:09AM +1200, Chris Wedgwood wrote:
> What is the limit here? The 8/16 way SE chipsets?

The largest Chipset I know about are the 8 Way ones.
(What is SE?).

>     > In anyone from Compaq is reading this, you should send me a 32-way
>     > Xeon ASAP just to prove they really work :)
>     
>     It doesn't.
> 
> Oh, then they definately need to send me one.

Heh :)

> Are these not MP1.4 based? Something different?

They must be the Unisys OEM machines.  They are based on some
crossbar-architecture called CMP that allows logical partioning, etc..

I have talked to Unisys engineers on last Cebit who said that the NT
(now W2k) port required a huge amount of work.
Also I noticed that UnixWare^H^H^H^H^HOpenUnix needed work to run on it.

	Christoph

-- 
Whip me.  Beat me.  Make me maintain AIX.

Re: How many pentium-3 processors does SMP support?

[Chris Wedgwood]

On Tue, Jul 10, 2001 at 04:32:43PM +0200, Christoph Hellwig wrote:

    The largest Chipset I know about are the 8 Way ones.
    (What is SE?).

A moronic attempt as using this funny qwerty layout thing.  It should
read SW (ServerWorks).

    They must be the Unisys OEM machines.  They are based on some
    crossbar-architecture called CMP that allows logical partioning,
    etc..

Yup, turns out they are.

    I have talked to Unisys engineers on last Cebit who said that the
    NT (now W2k) port required a huge amount of work.  Also I noticed
    that UnixWare^H^H^H^H^HOpenUnix needed work to run on it.

Well, I guess NetBSD is a good candidate, it has a nice portable MM
system, linux might not be so bad either, but without knowledge of how
this works, I am just guessing and probably still talking shit.



  --cw

Re: How many pentium-3 processors does SMP support?

[Randy.Dunlap]

Chris Wedgwood wrote:
> 
> On Tue, Jul 10, 2001 at 04:19:43PM +0200, Christoph Hellwig wrote:
> 
>     The number of CPUs is currently globally limited to 32 by NR_CPUS in
>     include/linux/threads.h.
> 
> Really?
> 
> <pause>
> 
> Ah, so it is... yes, making this architecture dependant might be a
> good idea. Large PPC and MIPS boxen need to adjust this already. Also,
> someone did a starfire port, I think that had 64 processors, not sure.
> 
>     You can.  But you cannot buy 32-processor PII (-Xeon) systems that are
>     supported by Linux.
> 
> What is the limit here? The 8/16 way SE chipsets?
> 
>     > In anyone from Compaq is reading this, you should send me a 32-way
>     > Xeon ASAP just to prove they really work :)
> 
>     It doesn't.
> 
> Oh, then they definately need to send me one.
> 
> Are these not MP1.4 based? Something different?
> 
>   --cw

For IA32/i386/x86:

The Linus-kernel only supports a maximum of 15 Pentium IIIs
due to APIC addressing (4 bits, with 0xf meaning "broadcast").

Pentium 4 uses 0xff for broadcast, so lots more of them can
be supported (when you can find a P-4 MP server).

I have heard of some IBM/Sequent patches that modify the
logical vs. physical APIC addressing scheme to make 16-way
systems work.

-- 
~Randy

Re: How many pentium-3 processors does SMP support?

[Chris Wedgwood]

On Tue, Jul 10, 2001 at 08:26:09AM -0700, Randy.Dunlap wrote:

    I have heard of some IBM/Sequent patches that modify the
    logical vs. physical APIC addressing scheme to make 16-way
    systems work.

The Unisys machine is actually four quad-CPU machines with shared
memory, dynamically configurable and other neat stuff.



  --cw

Re: How many pentium-3 processors does SMP support?

[Christoph Hellwig]

On Wed, Jul 11, 2001 at 03:50:38AM +1200, Chris Wedgwood wrote:
> On Tue, Jul 10, 2001 at 08:26:09AM -0700, Randy.Dunlap wrote:
> 
>     I have heard of some IBM/Sequent patches that modify the
>     logical vs. physical APIC addressing scheme to make 16-way
>     systems work.
> 
> The Unisys machine is actually four quad-CPU machines with shared
> memory, dynamically configurable and other neat stuff.

If it is the Unisys design I mean it is done rather different.
The design I saw has two central crossbars and attached either CPU
modules with a GTL+ bus and two CPUs after a L3 cache or a IO
Module with a few (IIRC 3) PCI busses.

-- 
Of course it doesn't work. We've performed a software upgrade.

Re: How many pentium-3 processors does SMP support?

[Ralf Baechle]

On Wed, Jul 11, 2001 at 02:25:09AM +1200, Chris Wedgwood wrote:

>     The number of CPUs is currently globally limited to 32 by NR_CPUS in
>     include/linux/threads.h.
> 
> Really?
> 
> <pause>

A define which can easily be changed.

> Ah, so it is... yes, making this architecture dependant might be a
> good idea. Large PPC and MIPS boxen need to adjust this already. Also,
> someone did a starfire port, I think that had 64 processors, not sure.

The next limit are bitfields for processors in a number of places.  They're
stored in unsigned long variables, so this limits the kernel to 32
processors on 32-bit machines and 64 on 64-bit machines.  Kanoj once
fixed that and we had Linux booting on a 128p Origin but I'm not sure
if those fixes went back to Linus.  Once this is fixed there is a number
of arrays with NR_CPUS elements in struct task_struct.  Once you reach
an estimated number of 200-300 processors (for 16kb kernel stacks) those
make task_struct that large that the kernel stack may overflow.  I
haven't really researched all the gotchas that may cause problems with
the kernel when going to machines of more than 128 processors.

  Ralf

Re: How many pentium-3 processors does SMP support?

[Jesper Juhl]

Chris Wedgwood wrote:
> 
> On Mon, Jul 09, 2001 at 10:34:24PM -0700, H. Peter Anvin wrote:
> 
>     It supports up to 32, if you can find a machine that has that
>     many.
> 
> I think 8-way is about as high as anything common goes to, maybe
> 16. The cpu array is declared 32 long, maybe this should be changed to
> 8 by default?
> 

There are some machines (like the Compaq Proliant ML770 - 
http://www.compaq.com/products/quickspecs/10698_div/10698_div.html) that
are actually sold as 32 way systems based on Pentium III Xeon CPU's, so
why not let the cpu array be able to handle that many CPU's by default
(maybe make a config option?)?


- Jesper Juhl
  juhl@eisenstein.dk

Re: How many pentium-3 processors does SMP support?

[Chris Wedgwood]

On Tue, Jul 10, 2001 at 04:10:32PM +0000, Jesper Juhl wrote:

    There are some machines (like the Compaq Proliant ML770 -
    http://www.compaq.com/products/quickspecs/10698_div/10698_div.html)
    that are actually sold as 32 way systems based on Pentium III Xeon
    CPU's, so why not let the cpu array be able to handle that many
    CPU's by default (maybe make a config option?)?

Wow... what a beats.  Its CMP based which isn't supported.  I wonder
if more details on that are available somewhere?



  --cw

Re: How many pentium-3 processors does SMP support?

[Jesper Juhl]

Chris Wedgwood wrote:
> 
> On Tue, Jul 10, 2001 at 04:10:32PM +0000, Jesper Juhl wrote:
> 
>     There are some machines (like the Compaq Proliant ML770 -
[snip]
> 
> Wow... what a beats.  Its CMP based which isn't supported.  I wonder
> if more details on that are available somewhere?
> 

More info on the Proliant ML770 is available here:
http://www.compaq.com/products/servers/proliantml770/specs.html

There is a White Paper on CMP available from unisys on this page:
http://www.unisys.com/hw/servers/enterprise/libr-cm-architecture.asp


- Jesper Juhl

Re: How many pentium-3 processors does SMP support?

[Martin J. Bligh]

> For IA32/i386/x86:
>
> The Linus-kernel only supports a maximum of 15 Pentium IIIs
> due to APIC addressing (4 bits, with 0xf meaning "broadcast").
>
> Pentium 4 uses 0xff for broadcast, so lots more of them can
> be supported (when you can find a P-4 MP server).
>
> I have heard of some IBM/Sequent patches that modify the
> logical vs. physical APIC addressing scheme to make 16-way
> systems work.

I'll be posting the patches for this in about a week's time (just
going through a code review first). They should make up to a
32 way (8 quad) system work, though I've only actually tested
it up to 16 procs.

The main changes are to use "Logical clustered APIC addressing"
mode rather than flat logical, and to bootstrap via NMIs rather than
the usual INIT, INIT, STARTUP IPI sequence.

I can't keep up with the traffic on this list, but if you're interested,
bug me by email (mbligh@sequent.com / mbligh@us.ibm.com).

Martin.

PS. Takes 32 secs to compile the kernel, and that's on an older,
slower machine ;-)