Re: What about...

Re: What about...

[Alex deVries]

On Thu, 16 Jul 1998, Igor Loncarevic wrote:
> What about Linux for Origin2000 SGI?
> Is there any possibility to run Linux on Origin?

I would say there's almost no chance of running Linux on an Origin 2000
anytime this century.

- Alex

Re: What about...

[Greg Chesson]

Alex is right.
Linux on Origin 2000 would be a huge project - not just a port.

We have much better chances getting Linux up on SGIs yet-to-be-revealed
Intel-based platforms.

g

-- 
Greg Chesson

Re: What about...

[William J. Earl]

Greg Chesson writes:
 > Alex is right.
 > Linux on Origin 2000 would be a huge project - not just a port.
...

     Note that the reason it would be a big project is not so much the
hardware itself, but rather the scale of the system.  IRIX needed a lot
of infrastructure work to be useful on very large system with a very large
number of I/O buses and devices.  That is, a toy port would not be all
that giant a project, but it would not be particularly useful.  If there
were a good linux on some other very large ccNUMA machine, then an Origin
port would be much simpler.  By "good", I mean a linux which scale well
for large (greater than 32) processor and I/O count (many I/O buses
and thousands of disk).  It expect such a linux will happen eventually,
but not yet.

Re: What about...

[Alan Cox]

> that giant a project, but it would not be particularly useful.  If there
> were a good linux on some other very large ccNUMA machine, then an Origin
> port would be much simpler.  By "good", I mean a linux which scale well
> for large (greater than 32) processor and I/O count (many I/O buses
> and thousands of disk).  It expect such a linux will happen eventually,
> but not yet.

The obvious starting point would probably be the older (386/486) sequent
boxes. I almost got somewhere with this but one bit of sequent wasnt 
willing to be helpful and counted its bus arbitrators as staying NDA.

That was 2 years ago so I ought to chase them again I guess

Re: What about...

[Alan Cox]

> that giant a project, but it would not be particularly useful.  If there
> were a good linux on some other very large ccNUMA machine, then an Origin
> port would be much simpler.  By "good", I mean a linux which scale well
> for large (greater than 32) processor and I/O count (many I/O buses
> and thousands of disk).  It expect such a linux will happen eventually,
> but not yet.

The obvious starting point would probably be the older (386/486) sequent
boxes. I almost got somewhere with this but one bit of sequent wasnt 
willing to be helpful and counted its bus arbitrators as staying NDA.

That was 2 years ago so I ought to chase them again I guess

Re: What about...

[ralf]

On Fri, Jul 17, 1998 at 07:11:42AM -0700, William J. Earl wrote:

> Greg Chesson writes:
>  > Alex is right.
>  > Linux on Origin 2000 would be a huge project - not just a port.
> ...
> 
>      Note that the reason it would be a big project is not so much the
> hardware itself, but rather the scale of the system.  IRIX needed a lot
> of infrastructure work to be useful on very large system with a very large
> number of I/O buses and devices.  That is, a toy port would not be all
> that giant a project, but it would not be particularly useful.  If there
> were a good linux on some other very large ccNUMA machine, then an Origin
> port would be much simpler.  By "good", I mean a linux which scale well
> for large (greater than 32) processor and I/O count (many I/O buses
> and thousands of disk).  It expect such a linux will happen eventually,
> but not yet.

Actually some people begin to be interested in Linux for huge installations.
For example a bank recently offer us some machine time on a maximum
configuration (64 CPU, 64GB RAM) E10000 for porting Linux.  This didn't
happen (yet?) for other reasons, but it shows that there is interest.

For the time being most large Linux installations are based on the Beowulf
clustering software - and I think place 318 on the list on the top 500 list
of supercomputer installations is quite remarkable for something which's
OS software is being shipped for USD 29.95 by RedHat.

Anyway, as far as our MIPS port goes we first need to implement SMP at
all before we should think about scaling.  More about Linux in general -
most people are currentl using Linux/SMP on dual processors, few on
quad CPU systems and even fewer on larger systems.  This is about where
the market of Linux and it's limits are.  But the tradition of Linux
shows that technology for which a demand exists will be developed sooner
or later.  Where Linux technology exists, usually a market will develop.
So far I see a demand for older SGI SMP systems, Origin 200 systems (quite
some inquiries) but not yet for Origin 2000 systems.

  Ralf

Re: What about...

[ralf]

On Fri, Jul 17, 1998 at 04:07:04PM +0100, Alan Cox wrote:

> > that giant a project, but it would not be particularly useful.  If there
> > were a good linux on some other very large ccNUMA machine, then an Origin
> > port would be much simpler.  By "good", I mean a linux which scale well
> > for large (greater than 32) processor and I/O count (many I/O buses
> > and thousands of disk).  It expect such a linux will happen eventually,
> > but not yet.
> 
> The obvious starting point would probably be the older (386/486) sequent
> boxes. I almost got somewhere with this but one bit of sequent wasnt 
> willing to be helpful and counted its bus arbitrators as staying NDA.
> 
> That was 2 years ago so I ought to chase them again I guess

Do you have hardware?  I know an installation which might want to get rid
of their 486 based 32 processor Sequent system if they didn't already.

  Ralf

Re: What about...

[Greg Chesson]

Bill is, of course, quite correct.
In addition to the observations about the scale of the system,
realize also that a ccNUMA machine has a memory system for each
cpu node in the system.  The physical base addresses of these blocks
of memory are aligned on multi-gigabyte boundaries.  The high-order
bits of the address designate the cpu node, the rest address the physical
memory, etc etc.  What this means is that physical memory space has
many "holes"...  The idea of a simple buddy-system allocator as is
ingrained in the Linux kernel falls apart completely in the face of
this kind of architecture.   I suppose you could run a copy of Linux
on every node, but I consider that an excuse rather than a solution.

g



-- 
Greg Chesson

Re: What about...

[Alan Cox]

> Do you have hardware?  I know an installation which might want to get rid

I dont. I had an offer at the time, and I did some digging on the older boxes
also recently on the new ones - did you know Sequent Numa-Q appears to use non
standard microcode ?

> of their 486 based 32 processor Sequent system if they didn't already.

I can just imagine the shipping cost ;)

Re: What about...

[Greg Chesson]

Ralf,

Beowolf machines don't have adequate IO for applications other than
things that resemble Linpack.  The teraflop machine at Sandia would like
to generate output every 5 or 10 minutes, but it takes an hour to get
the computation result out of the machine. Beowolf machines tend to not
have adequate bisection bandwidth more many kinds of codes.  Anything that
requires a cornerturn (transpose) is a good example.  Also, Beowolf machines
tend to run a single code for many hours: they don't multiplex well,
or sometimes at all.  But multiplexing and adapting to a dynamic load
is expected of many large scale machines and that requires some OS support
not found in the single-cpu or small-cpu-count OS or existing middleware.

 We cannot afford to build
machines that are good only for Linpack.... it is better
to build a Beowolf cluster if that is the best fit for the job.
It is rather ignorant to think that because something like Beowolf can do some
some types of applications well, that some software work can make it do
all jobs well.  That's like saying a Volkswagon could go 300 kph
if you drive carefully.

Linux has proven to be worthwhile as a node controller in an MPP architecture -
that's what a Beowolf is.  But that does not make it ready for SMP nodes
that scale to large numbers.  It seems wasteful to program a large scale
ccNUMA machine the same way as a Beowolf cluster: you'd be throwing away
most of the capabilities of the hardware.  That's why I don't
think it is interesting or particularly useful... unless a massive amount
of work went into rewriting the io and memory management subsystems,
not to mention scheduling, administration, etc.

g


-- 
Greg Chesson

Re: What about...

[Alan Cox]

> many "holes"...  The idea of a simple buddy-system allocator as is
> ingrained in the Linux kernel falls apart completely in the face of
> this kind of architecture.   I suppose you could run a copy of Linux
> on every node, but I consider that an excuse rather than a solution.

Actually the Linux buddy stuff is quite happy with holes. Its still
completely inappropriate. From the above I deduce we'd have to do
mips64 before we even considerd it anyway

Re: What about...

[Alan Cox]

> many "holes"...  The idea of a simple buddy-system allocator as is
> ingrained in the Linux kernel falls apart completely in the face of
> this kind of architecture.   I suppose you could run a copy of Linux
> on every node, but I consider that an excuse rather than a solution.

Actually the Linux buddy stuff is quite happy with holes. Its still
completely inappropriate. From the above I deduce we'd have to do
mips64 before we even considerd it anyway

Re: What about...

[William J. Earl]

Greg Chesson writes:
...
 > Linux has proven to be worthwhile as a node controller in an MPP architecture -
 > that's what a Beowolf is.  But that does not make it ready for SMP nodes
 > that scale to large numbers.  It seems wasteful to program a large scale
 > ccNUMA machine the same way as a Beowolf cluster: you'd be throwing away
 > most of the capabilities of the hardware.  That's why I don't
 > think it is interesting or particularly useful... unless a massive amount
 > of work went into rewriting the io and memory management subsystems,
 > not to mention scheduling, administration, etc.
...

     I expect that much of the work will gradually happen in linux, once
the remaining small-CPU-count issues are resolved.  Right now, there
is no shortage of interesting problems to attack.  :-)

     One possible way to approach the large-CPU-count space with linux
is to indeed run multiple linux kernels, one per node in a ccNUMA
machine, and add a distributed OS layer at a fairly high level.  If it
is not underway already somewhere, I would expect someone to take up
the project as a graduate school project.  Some systems of this sort
have been built or attempted, with varying degrees of success.  Given
the linux bias toward small and simple, a linux-based distributed OS
might actually work.

     One important ingredient in such a system, which would be
valuable immediately for clusters, would be a efficient distributed
volume manager and file system.  

     In any case, trying to port linux straight to a large ccNUMA 
(or even a large SMP) system would be a lot of effort for limited
return at present.

Re: What about...

[William J. Earl]

Alan Cox writes:
 > > many "holes"...  The idea of a simple buddy-system allocator as is
 > > ingrained in the Linux kernel falls apart completely in the face of
 > > this kind of architecture.   I suppose you could run a copy of Linux
 > > on every node, but I consider that an excuse rather than a solution.
 > 
 > Actually the Linux buddy stuff is quite happy with holes. Its still
 > completely inappropriate. From the above I deduce we'd have to do
 > mips64 before we even considerd it anyway

     Yes, the address space is very large, even in a single rack.

Re: What about...

[William J. Earl]

Alan Cox writes:
 > > many "holes"...  The idea of a simple buddy-system allocator as is
 > > ingrained in the Linux kernel falls apart completely in the face of
 > > this kind of architecture.   I suppose you could run a copy of Linux
 > > on every node, but I consider that an excuse rather than a solution.
 > 
 > Actually the Linux buddy stuff is quite happy with holes. Its still
 > completely inappropriate. From the above I deduce we'd have to do
 > mips64 before we even considerd it anyway

     Yes, the address space is very large, even in a single rack.

Re: What about...

[Jeffrey Watts]

On Fri, 17 Jul 1998, Alan Cox wrote:

> > that giant a project, but it would not be particularly useful.  If there
> > were a good linux on some other very large ccNUMA machine, then an Origin
> > port would be much simpler.  By "good", I mean a linux which scale well
> > for large (greater than 32) processor and I/O count (many I/O buses
> > and thousands of disk).  It expect such a linux will happen eventually,
> > but not yet.

Boy would it be fun to work on that port.  Origin2000s are seriously cool. 
I am getting 9 new Origin2000 systems in about 6 weeks.  I wonder if
Sprint would mind if I used one for Linux porting...  :-) 

J.

o-----------------------------------o
| Jeffrey Watts                     |
| watts@sunflower.com           o-------------------------------------o
| Systems Analyst               | "Proprietary system advocates       |
| Sprint - Systems Management   |  aren't evil or stupid.  They are   |
o-------------------------------|  the victims.  They have a disease  |
                                |  and they need help."               |
                                |  -- Donald B. Marti Jr.             |
                                o-------------------------------------o

Re: What about...

[ralf]

On Fri, Jul 17, 1998 at 07:14:04PM +0100, Alan Cox wrote:

> > many "holes"...  The idea of a simple buddy-system allocator as is
> > ingrained in the Linux kernel falls apart completely in the face of
> > this kind of architecture.   I suppose you could run a copy of Linux
> > on every node, but I consider that an excuse rather than a solution.
> 
> Actually the Linux buddy stuff is quite happy with holes. Its still
> completely inappropriate. From the above I deduce we'd have to do
> mips64 before we even considerd it anyway

At least in Vger CVS we alredy have code to efficiently deal with
non-dense memory architectures with buddy.

I think the memory allocator design as presented by Rik van Riel looks as
if it is a good base to deal much more efficiently with such an
memory architecture.  And then there is Sct with his big iron experience
working on something better than the buddy system.

I completly agree that we first have to go to MIPS64 before we really
can attack big iron problems.  The current Linux/MIPS kernel design limits
the memory to the size of KSEG0 which is 512mb.  Putting Linux into a
64bit universe we'd extend that design limit to the size of XKSEG0, or
1TB for current silicon.

  Ralf

Re: What about...

[ralf]

On Fri, Jul 17, 1998 at 11:19:54AM -0700, William J. Earl wrote:

>      I expect that much of the work will gradually happen in linux, once
> the remaining small-CPU-count issues are resolved.  Right now, there
> is no shortage of interesting problems to attack.  :-)

Actually I'd hate it if Linux'd be ``finished'' ...

>      One possible way to approach the large-CPU-count space with linux
> is to indeed run multiple linux kernels, one per node in a ccNUMA
> machine, and add a distributed OS layer at a fairly high level.  If it
> is not underway already somewhere, I would expect someone to take up
> the project as a graduate school project.  Some systems of this sort
> have been built or attempted, with varying degrees of success.  Given
> the linux bias toward small and simple, a linux-based distributed OS
> might actually work.

I don't know the exact technical details but something similar has already
been done back in '95 (?) when some guys in Australia ported Linux to
Fujitsu's AP1000.  That's of course a different architecture and a
multikernel approach makes much more sense there.

>      One important ingredient in such a system, which would be
> valuable immediately for clusters, would be a efficient distributed
> volume manager and file system.  

Hans Reiser is currently working on a new filesystem with alot of fresh
ideas.  In some aspects what he is aiming at is similar to XFS, in some
aspects not.  He basically started on a white sheet of paper with his design,
so his team's code isn't contaminated by old ideas.  His work looks pretty
promising.  Among his plans is also the implementation of a distributed
filesystem.  Current benchmarks are looking pretty good, in fact in
some cases extremly good.  The URL to checkout for interested people is
http://idiom.com/~beverly/reiserfs.html.

>      In any case, trying to port linux straight to a large ccNUMA 
> (or even a large SMP) system would be a lot of effort for limited
> return at present.

Explecitly not talking about the MIPS port - I think it makes sense in
working on porting to machines beyond what we currently scale to.  But
I agree about the ``large'' thing.

  Ralf

Re: What about...

[Greg Chesson]

I think everyone is suggesting the same general path, and I'm gratified
to know that the buddy system code anticipates non-contiguous physical
memory.

I think you'd want to extend the design limit (XKSEG0) beyond 1 TB
to handle the next rev of silicon.  I'd suggest 64 TB (48 bits) as
the appropriate goal.

g

-- 
Greg Chesson

Re: What about...

[Greg Chesson]

On Jul 18,  3:57am, ralf@uni-koblenz.de wrote:
>
> Actually I'd hate it if Linux'd be ``finished'' ...
>

Ancient Chinese proverb:  "House finish; man die."

g

-- 
Greg Chesson

Re: What about...

[ralf]

On Fri, Jul 17, 1998 at 06:58:07PM -0700, Greg Chesson wrote:

> I think everyone is suggesting the same general path, and I'm gratified
> to know that the buddy system code anticipates non-contiguous physical
> memory.
> 
> I think you'd want to extend the design limit (XKSEG0) beyond 1 TB
> to handle the next rev of silicon.  I'd suggest 64 TB (48 bits) as
> the appropriate goal.

As far as the maximum amount of RAM goes, no problem.  The only assumption
which I make is that the entire available memory is visible in XKSEG0,
whatever it's size is.  That means the actual design limit is the maximum
possible size of XKSEG0 for the MIPS 64bit architecture which is 2^62 bytes.

I assume this also means the size limit of the XKUSEG has been extended
to 48 bits.  There we actually a somewhat more limiting design limit since
we only have three level page tables for 64 bit architectures which limits
us to a maximum mappable size of

  PAGE_SIZE * (PAGE_SIZE / sizeof(pte_t)) ^ 3

which for PAGE_SIZE = 4 kbytes and sizeof(pte_t) = 8 bytes is 512gb.

(And actually things are somewhat more complex.)

  Ralf

Re: What about...

[ralf]

On Sat, Jul 18, 1998 at 04:44:03AM +0200, ralf@uni-koblenz.de wrote:

> On Fri, Jul 17, 1998 at 06:58:07PM -0700, Greg Chesson wrote:

> > I think you'd want to extend the design limit (XKSEG0) beyond 1 TB
> > to handle the next rev of silicon.  I'd suggest 64 TB (48 bits) as
> > the appropriate goal.
> 
> As far as the maximum amount of RAM goes, no problem.  The only assumption
> which I make is that the entire available memory is visible in XKSEG0,
> whatever it's size is.  That means the actual design limit is the maximum
> possible size of XKSEG0 for the MIPS 64bit architecture which is 2^62 bytes.

Ooops, the real limit is of course how much memory can be presented in XKPHYS
which is a bit less, just 2^58 bytes or 4 exabytes.  XKSEG or XKSSEG would
be used for kernel virtual memory which usually isn't used very much under
Linux - it's virtual, but not swappable.

  Ralf