128 GiB memory question-- kernel only sees 64GiB?

128 GiB memory question-- kernel only sees 64GiB?

[Justin Piszcz]

Hello,

To be able to utilize 128 GiB of memory on a host, does one need to enable 
NUMA support? Currently only 64 GiB is showing up.

As far as machine name/motherboard type, I do not have this information 
currently but I am told there are one or two of these boxes are running 
CentOS 5.x and/or RHEL5/AS.

Is there something special you need to do to utilize all 128 GiB of 
memory?

Thanks,

Justin.

Re: 128 GiB memory question-- kernel only sees 64GiB?

[Peter Zijlstra]

On Tue, 2008-07-22 at 10:21 -0400, Justin Piszcz wrote:
> Hello,
> 
> To be able to utilize 128 GiB of memory on a host, does one need to enable 
> NUMA support? Currently only 64 GiB is showing up.
> 
> As far as machine name/motherboard type, I do not have this information 
> currently but I am told there are one or two of these boxes are running 
> CentOS 5.x and/or RHEL5/AS.

Sounds like you need to bug RH with that ;-)

> Is there something special you need to do to utilize all 128 GiB of 
> memory?

Does a recent kernel work for you? Also, you could at the very least
have mentioned your machine architecture; is it x86_64, ia64 or ppc64?

Re: 128 GiB memory question-- kernel only sees 64GiB?

[Rik van Riel]

On Tue, 22 Jul 2008 10:21:44 -0400 (EDT)
Justin Piszcz <jpiszcz@lucidpixels.com> wrote:

> To be able to utilize 128 GiB of memory on a host, does one need to enable 
> NUMA support? Currently only 64 GiB is showing up.

How much memory does the BIOS say the system has?

Can you get the e820 memory map (from early boot up) of the systems?
 
> As far as machine name/motherboard type, I do not have this information 
> currently but I am told there are one or two of these boxes are running 
> CentOS 5.x and/or RHEL5/AS.

Hopefully not with 64GB each :))
 
> Is there something special you need to do to utilize all 128 GiB of 
> memory?

You need to use a 64 bit kernel, for one.

-- 
All rights reversed.

Re: 128 GiB memory question-- kernel only sees 64GiB?

[Justin Piszcz]

On Tue, 22 Jul 2008, Peter Zijlstra wrote:

> On Tue, 2008-07-22 at 10:21 -0400, Justin Piszcz wrote:
>> Hello,
>>
>> To be able to utilize 128 GiB of memory on a host, does one need to enable
>> NUMA support? Currently only 64 GiB is showing up.
>>
>> As far as machine name/motherboard type, I do not have this information
>> currently but I am told there are one or two of these boxes are running
>> CentOS 5.x and/or RHEL5/AS.
>
> Sounds like you need to bug RH with that ;-)
>
>> Is there something special you need to do to utilize all 128 GiB of
>> memory?
>
> Does a recent kernel work for you? Also, you could at the very least
> have mentioned your machine architecture; is it x86_64, ia64 or ppc64?
>

Sun X4600, will find out if they are running the right kernel and also 
check the BIOS, thanks.

Justin.

Re: 128 GiB memory question-- kernel only sees 64GiB?

[Justin Piszcz]

On Tue, 22 Jul 2008, Rik van Riel wrote:

> On Tue, 22 Jul 2008 10:21:44 -0400 (EDT)
> Justin Piszcz <jpiszcz@lucidpixels.com> wrote:
>
>> To be able to utilize 128 GiB of memory on a host, does one need to enable
>> NUMA support? Currently only 64 GiB is showing up.
>
> How much memory does the BIOS say the system has?
Asked them.

>
> Can you get the e820 memory map (from early boot up) of the systems?
Probably can't get this.

>
>> As far as machine name/motherboard type, I do not have this information
>> currently but I am told there are one or two of these boxes are running
>> CentOS 5.x and/or RHEL5/AS.
>
> Hopefully not with 64GB each :))
Nope :P

>
>> Is there something special you need to do to utilize all 128 GiB of
>> memory?
>
> You need to use a 64 bit kernel, for one.
Will verify what they are running, thanks.

Re: 128 GiB memory question-- kernel only sees 64GiB?

[Rik van Riel]

On Tue, 22 Jul 2008 16:55:32 -0400 (EDT)
Justin Piszcz <jpiszcz@lucidpixels.com> wrote:
> On Tue, 22 Jul 2008, Rik van Riel wrote:
> 
> > On Tue, 22 Jul 2008 10:21:44 -0400 (EDT)
> > Justin Piszcz <jpiszcz@lucidpixels.com> wrote:
> >
> >> To be able to utilize 128 GiB of memory on a host, does one need to enable
> >> NUMA support? Currently only 64 GiB is showing up.
> >
> > How much memory does the BIOS say the system has?
> Asked them.
> 
> >
> > Can you get the e820 memory map (from early boot up) of the systems?
> Probably can't get this.

It looks roughly like this:

BIOS-provided physical RAM map:
 BIOS-e820: 0000000000000000 - 00000000000a0000 (usable)
 BIOS-e820: 0000000000100000 - 00000000bfb50000 (usable)
 BIOS-e820: 00000000bfb50000 - 00000000bfb66000 (reserved)
 BIOS-e820: 00000000bfb66000 - 00000000bfb85c00 (ACPI data)
 BIOS-e820: 00000000bfb85c00 - 00000000c0000000 (reserved)
 BIOS-e820: 00000000e0000000 - 00000000f0000000 (reserved)
 BIOS-e820: 00000000fe000000 - 0000000100000000 (reserved)
 BIOS-e820: 0000000100000000 - 0000000440000000 (usable)

Getting this info will be very useful.
 
-- 
All rights reversed.

Re: 128 GiB memory question-- kernel only sees 64GiB?

[Yinghai Lu]

On Tue, Jul 22, 2008 at 1:55 PM, Justin Piszcz <jpiszcz@lucidpixels.com> wrote:
>
>
> On Tue, 22 Jul 2008, Peter Zijlstra wrote:
>
>> On Tue, 2008-07-22 at 10:21 -0400, Justin Piszcz wrote:
>>>
>>> Hello,
>>>
>>> To be able to utilize 128 GiB of memory on a host, does one need to
>>> enable
>>> NUMA support? Currently only 64 GiB is showing up.
>>>
>>> As far as machine name/motherboard type, I do not have this information
>>> currently but I am told there are one or two of these boxes are running
>>> CentOS 5.x and/or RHEL5/AS.
>>
>> Sounds like you need to bug RH with that ;-)
>>
>>> Is there something special you need to do to utilize all 128 GiB of
>>> memory?
>>
>> Does a recent kernel work for you? Also, you could at the very least
>> have mentioned your machine architecture; is it x86_64, ia64 or ppc64?
>>
>
> Sun X4600, will find out if they are running the right kernel and also check
> the BIOS, thanks.

so you are using 32 bit kernel?

uname -a

could tell arch

YH