Can linux support ccNUMA machine now?

Can linux support ccNUMA machine now?

[Barry Wu]

Hi, all,

Can linux support ccNUMA machine now? 
How mcuh memory linux support? 64GB or more?
If someone knows, please help me. TIA.

Barry

__________________________________________________
Do You Yahoo!?
Send FREE video emails in Yahoo! Mail!
http://promo.yahoo.com/videomail/

Re: Can linux support ccNUMA machine now?

[Martin J. Bligh]

> Can linux support ccNUMA machine now? 

Yes.

> How mcuh memory linux support? 64GB or more?

Theoritically 64Gb on 32 bit machines, in practice
significantly less than that (IIRC, something like
25-30Gb). It's not terribly efficient in using it though ;-)

Martin.

Re: Can linux support ccNUMA machine now?

[William Lee Irwin III]

At some point in the past, someone wrote:
>> How mcuh memory linux support? 64GB or more?

On Wed, Jan 23, 2002 at 02:45:24AM -0800, Martin J. Bligh wrote:
> Theoritically 64Gb on 32 bit machines, in practice
> significantly less than that (IIRC, something like
> 25-30Gb). It's not terribly efficient in using it though ;-)

With some simple calculations for various things, I predict ZONE_NORMAL
will get filled by large boot-time allocations on x86 with PAE and 64GB
of RAM. I'm not entirely sure what other sorts of pathologies arise
while these beasts still function; but without enough ZONE_NORMAL to
satisfy all the combined boot-time allocation requests, the kernel
will surely panic.

The sizes of these boot-time allocations are not entirely constant
across kernel versions, but there should be some threshhold of
memory size at which any given Linux version to date drops dead on
large memory x86 machines.

On 64-bit machines, there is obviously no such behavior, and Linux
will be able to use the full memory capacity of the system.


Cheers,
Bill

P.S.: Blame it on struct page.

Re: Can linux support ccNUMA machine now?

[Denis Vlasenko]

> With some simple calculations for various things, I predict ZONE_NORMAL
> will get filled by large boot-time allocations on x86 with PAE and 64GB
> of RAM. I'm not entirely sure what other sorts of pathologies arise
> while these beasts still function; but without enough ZONE_NORMAL to
> satisfy all the combined boot-time allocation requests, the kernel
> will surely panic.
...
> P.S.: Blame it on struct page.

Looks like running x86 with more than 16GB RAM is not a good idea.
If you need it, you need 64bit arch.

This limit can be raised substantially by reducing low 4GB memory 
requirements, but don't you feel it's like running 16-bit DOS
on 686 class CPU? HIMEM.SYS, EMM, horde of DOS extenders - sounds familiar?

However, CPU vendors war over common 64-bit arch is still ahead...
--
vda

Re: Can linux support ccNUMA machine now?

[William Lee Irwin III]

At some point in the past, I wrote:
>> P.S.: Blame it on struct page.

On Thu, Jan 24, 2002 at 02:15:30PM -0200, Denis Vlasenko wrote:
> Looks like running x86 with more than 16GB RAM is not a good idea.
> If you need it, you need 64bit arch.

There are more polite ways of refusing to boot in such situations,
ranging from just dropping the RAM that can't be used on the floor to
some effort to at make console messages explaining the panic visible.

On Thu, Jan 24, 2002 at 02:15:30PM -0200, Denis Vlasenko wrote:
> This limit can be raised substantially by reducing low 4GB memory 
> requirements, but don't you feel it's like running 16-bit DOS
> on 686 class CPU? HIMEM.SYS, EMM, horde of DOS extenders - sounds familiar?

Not familiar to me.

Reducing overhead helps all boxen everywhere all the time. Turning the
kernel upside-down for the corner case of 64GB isn't worth it, but
finding more graceful ways to fail than not booting with no visible
error messages, and perhaps extending the range of configurations where
the kernel actually functions (within reason) by reducing space
overhead is worthwhile.

I haven't seen many people posting questions to the list saying "Linux
won't boot, my machine is umpteen-way i386 SMP with 64GB of RAM". So I
largely think of it as a sort of reminder of the negative consequences
of overhead more than an actual goal, and the largest concern is that
the failure mode is not graceful.

On Thu, Jan 24, 2002 at 02:15:30PM -0200, Denis Vlasenko wrote:
> However, CPU vendors war over common 64-bit arch is still ahead...

64-bit machines have been what most CPU vendors have been selling for a
few years now. Unfortunately, "most vendors" does not translate to "most
CPU's in use", for non-technical reasons beyond the scope of this forum.


Cheers,
Bill

Re: Can linux support ccNUMA machine now?

[Andreas Dilger]

On Jan 24, 2002  10:50 -0800, William Lee Irwin III wrote:
> On Thu, Jan 24, 2002 at 02:15:30PM -0200, Denis Vlasenko wrote:
> > Looks like running x86 with more than 16GB RAM is not a good idea.
> > If you need it, you need 64bit arch.

Actually, Andrea made a patch to move the page tables into HIMEM on
such machines.  I believe it is in the latest -aa patch.

> Reducing overhead helps all boxen everywhere all the time. Turning the
> kernel upside-down for the corner case of 64GB isn't worth it, but
> finding more graceful ways to fail than not booting with no visible
> error messages, and perhaps extending the range of configurations where
> the kernel actually functions (within reason) by reducing space
> overhead is worthwhile.

Yes, there also have been several patches floating around to reduce
the size of struct page.  I don't think any are in the kernel yet.

Cheers, Andreas
--
Andreas Dilger
http://sourceforge.net/projects/ext2resize/
http://www-mddsp.enel.ucalgary.ca/People/adilger/

Re: Can linux support ccNUMA machine now?

[William Lee Irwin III]

On Thu, Jan 24, 2002 at 02:15:30PM -0200, Denis Vlasenko wrote:
>>> Looks like running x86 with more than 16GB RAM is not a good idea.
>>> If you need it, you need 64bit arch.

On Thu, Jan 24, 2002 at 12:19:52PM -0700, Andreas Dilger wrote:
> Actually, Andrea made a patch to move the page tables into HIMEM on
> such machines.  I believe it is in the latest -aa patch.

That doesn't help boot-time allocations that go into ->zone_mem_map
and various other places, but it certainly helps runtime space
overhead on machines that have enough ZONE_NORMAL left to boot.
... and I don't believe it's in the kernel yet.

On Jan 24, 2002  10:50 -0800, William Lee Irwin III wrote:
>> Reducing overhead helps all boxen everywhere all the time. Turning the
>> kernel upside-down for the corner case of 64GB isn't worth it, but
>> finding more graceful ways to fail than not booting with no visible
>> error messages, and perhaps extending the range of configurations where
>> the kernel actually functions (within reason) by reducing space
>> overhead is worthwhile.

On Thu, Jan 24, 2002 at 12:19:52PM -0700, Andreas Dilger wrote:
> Yes, there also have been several patches floating around to reduce
> the size of struct page.  I don't think any are in the kernel yet.

Actually, I made a patch to eliminate ->virtual, another to eliminate
->zone, and another to eliminate ->wait. I believe they can be found in
revised forms in the latest -rmap patch, and the reduction in space
overhead is available for all machines, not just i386 machines where
there's so much physical memory the kernel can't use direct-mapped
virtual-to-physical translations for the kernel virtual address space.

I will take this as further prompting to push these things to mainline
and at long last kill the multiplicative hash in the waitqueue table.
Pagetables in highmem sound good for mainline too.

Cheers,
Bill

P.S.: ->zone came back as an unsigned char because putting it into the
->flags field looked ugly, but it still beats the heck out of a full word.