HIGHMEM

HIGHMEM

[Nagy Tibor]

Hi,

We have two Dell Poweredge servers, an older one (PowerEdge 6300) and a 
newer one (PowerEdge 6400). Both servers have 4GB RAM, but the Linux 
kernel uses about 500MB less memory in the newer machine.

See /var/log/boot.msg on the old one:

<4>Linux version 2.4.20 (root@dell632) (gcc version 2.95.3 20010315
(SuSE)) #4 SMP Fri Jan 10 12:07:00 CET 2003
<6>BIOS-provided physical RAM map:
<4> BIOS-e820: 0000000000000000 - 00000000000a0000 (usable)
<4> BIOS-e820: 0000000000100000 - 00000000fbffe000 (usable)
<4> BIOS-e820: 00000000fbffe000 - 00000000fc000000 (reserved)
<4> BIOS-e820: 00000000fec00000 - 00000000fec10000 (reserved)
<4> BIOS-e820: 00000000fee00000 - 00000000fee10000 (reserved)
<4> BIOS-e820: 00000000fff80000 - 0000000100000000 (reserved)
<5>3135MB HIGHMEM available.
<5>896MB LOWMEM available.

and on the new one:

<4>Linux version 2.4.20 (root@alfa) (gcc version 2.95.3 20010315 (SuSE))
#10 SMP Fri Mar 28 15:40:45 CET 2003
<6>BIOS-provided physical RAM map:
<4> BIOS-e820: 0000000000000000 - 00000000000a0000 (usable)
<4> BIOS-e820: 0000000000100000 - 00000000dfff0000 (usable)
<4> BIOS-e820: 00000000dfff0000 - 00000000dfffec00 (ACPI data)
<4> BIOS-e820: 00000000dfffec00 - 00000000dffff000 (reserved)
<4> BIOS-e820: 00000000fec00000 - 00000000fec10000 (reserved)
<4> BIOS-e820: 00000000fee00000 - 00000000fee10000 (reserved)
<4> BIOS-e820: 00000000fff80000 - 0000000100000000 (reserved)
<5>2687MB HIGHMEM available.
<5>896MB LOWMEM available.


There is a big hole between 00000000dffff000 and 00000000fec00000, which
is not used on the new machine. What can I do?

Thanks for your help.

Tibor


------------------------------------------------------------------------
Tibor Nagy
E-mail: nagyt@otpbank.hu
------------------------------------------------------------------------

Re: HIGHMEM

[William Lee Irwin III]

On Tue, Apr 15, 2003 at 04:39:30PM +0200, Nagy Tibor wrote:
> <6>BIOS-provided physical RAM map:
> <4> BIOS-e820: 0000000000000000 - 00000000000a0000 (usable)
> <4> BIOS-e820: 0000000000100000 - 00000000fbffe000 (usable)
> <4> BIOS-e820: 00000000fbffe000 - 00000000fc000000 (reserved)
> <4> BIOS-e820: 00000000fec00000 - 00000000fec10000 (reserved)
> <4> BIOS-e820: 00000000fee00000 - 00000000fee10000 (reserved)
> <4> BIOS-e820: 00000000fff80000 - 0000000100000000 (reserved)
> <5>3135MB HIGHMEM available.
> <5>896MB LOWMEM available.
[...]
> <6>BIOS-provided physical RAM map:
> <4> BIOS-e820: 0000000000000000 - 00000000000a0000 (usable)
> <4> BIOS-e820: 0000000000100000 - 00000000dfff0000 (usable)
> <4> BIOS-e820: 00000000dfff0000 - 00000000dfffec00 (ACPI data)
> <4> BIOS-e820: 00000000dfffec00 - 00000000dffff000 (reserved)
> <4> BIOS-e820: 00000000fec00000 - 00000000fec10000 (reserved)
> <4> BIOS-e820: 00000000fee00000 - 00000000fee10000 (reserved)
> <4> BIOS-e820: 00000000fff80000 - 0000000100000000 (reserved)
> <5>2687MB HIGHMEM available.
> <5>896MB LOWMEM available.
> There is a big hole between 00000000dffff000 and 00000000fec00000, which
> is not used on the new machine. What can I do?

Presumably that was lost to ACPI. The hole between 0xdffff000 and
0xfec00000 appears to not be covered by the e820.

Try turning ACPI off in your .config since there's something that
looks relevant to it different between 2.4 and 2.5. You also might
want to follow up with .config's just in case. I'll look at 2.5's e820
stuff, but no promises.


-- wli

Re: HIGHMEM

[Samuel Flory]

Nagy Tibor wrote:

>
>
> There is a big hole between 00000000dffff000 and 00000000fec00000, which
> is not used on the new machine. What can I do?
>

  You might also try making sure you have the latest bios, and try 
turning off acpi in the bios.

-- 
There is no such thing as obsolete hardware.
Merely hardware that other people don't want.
(The Second Rule of Hardware Acquisition)
Sam Flory  <sflory@rackable.com>

HIGHMEM

[Nagy Tibor]

Hi,

I am sorry, I have found your e-mail address in 
./arch/i386/kernel/setup.c. I have the problem below since a year, and 
there is no solution yet. I guess, the problem is about e820. The 
problem exists in 2.4.x and also in 2.6.1.

We have two Dell Poweredge servers, an older one (PowerEdge 6300) and a
newer one (PowerEdge 6400). Both servers have 4GB RAM, but the Linux
kernel uses about 500MB less memory in the newer machine.

----------------------- 2.4.20 ---------------------------------

See /var/log/boot.msg on the old one:

<4>Linux version 2.4.20 (root@dell632) (gcc version 2.95.3 20010315
(SuSE)) #4 SMP Fri Jan 10 12:07:00 CET 2003
<6>BIOS-provided physical RAM map:
<4> BIOS-e820: 0000000000000000 - 00000000000a0000 (usable)
<4> BIOS-e820: 0000000000100000 - 00000000fbffe000 (usable)
<4> BIOS-e820: 00000000fbffe000 - 00000000fc000000 (reserved)
<4> BIOS-e820: 00000000fec00000 - 00000000fec10000 (reserved)
<4> BIOS-e820: 00000000fee00000 - 00000000fee10000 (reserved)
<4> BIOS-e820: 00000000fff80000 - 0000000100000000 (reserved)
<5>3135MB HIGHMEM available.
<5>896MB LOWMEM available.

and on the new one:

<4>Linux version 2.4.20 (root@alfa) (gcc version 2.95.3 20010315 (SuSE))
#10 SMP Fri Mar 28 15:40:45 CET 2003
<6>BIOS-provided physical RAM map:
<4> BIOS-e820: 0000000000000000 - 00000000000a0000 (usable)
<4> BIOS-e820: 0000000000100000 - 00000000dfff0000 (usable)
<4> BIOS-e820: 00000000dfff0000 - 00000000dfffec00 (ACPI data)
<4> BIOS-e820: 00000000dfffec00 - 00000000dffff000 (reserved)
<4> BIOS-e820: 00000000fec00000 - 00000000fec10000 (reserved)
<4> BIOS-e820: 00000000fee00000 - 00000000fee10000 (reserved)
<4> BIOS-e820: 00000000fff80000 - 0000000100000000 (reserved)
<5>2687MB HIGHMEM available.
<5>896MB LOWMEM available.

There is a big hole between 00000000dffff000 and 00000000fec00000, which
is not used on the new machine. What can I do?

------------------------------ 2.6.1 ------------------------------
I do not see in boot.msg such a detailed memory map, but the next line 
in boot.msg indicates, that the situation is the same:

<6>Memory: 3627908k/3669952k available (2918k kernel code, 40908k 
reserved, 1049k data, 180k init, 2752448k highmem)

--------------------------------------------------------------------

ACPI can not be disabled in BIOS. If ACPI is disbled in the kernel, the 
memory mapping remains the same, but CPU hyperthreading does not work also.


Thanks for your help.

Tibor


------------------------------------------------------------------------
Tibor Nagy
E-mail: nagyt@otpbank.hu
------------------------------------------------------------------------

Re: HIGHMEM

[Sean Neakums]

Nagy Tibor <nagyt@otpbank.hu> writes:

> Hi,
>
> I am sorry, I have found your e-mail address in
> ./arch/i386/kernel/setup.c. I have the problem below since a year, and
> there is no solution yet. I guess, the problem is about e820. The
> problem exists in 2.4.x and also in 2.6.1.
>
> We have two Dell Poweredge servers, an older one (PowerEdge 6300) and a
> newer one (PowerEdge 6400). Both servers have 4GB RAM, but the Linux
> kernel uses about 500MB less memory in the newer machine.

I may be talking through my hat, but I think that in this case you
need to select the option for support of 64G highmem.  If I recall,
"4G highmem" refers not to the total amount to the memory, but to the
highest physical address that can be accessed.

Re: HIGHMEM

[Martin J. Bligh]

>> I am sorry, I have found your e-mail address in
>> ./arch/i386/kernel/setup.c. I have the problem below since a year, and
>> there is no solution yet. I guess, the problem is about e820. The
>> problem exists in 2.4.x and also in 2.6.1.
>> 
>> We have two Dell Poweredge servers, an older one (PowerEdge 6300) and a
>> newer one (PowerEdge 6400). Both servers have 4GB RAM, but the Linux
>> kernel uses about 500MB less memory in the newer machine.
> 
> I may be talking through my hat, but I think that in this case you
> need to select the option for support of 64G highmem.  If I recall,
> "4G highmem" refers not to the total amount to the memory, but to the
> highest physical address that can be accessed.

That's exactly correct. Whether the gain of 500MB of RAM is worth the
overhead of PAE is another question ... but that's how to do it ;-)

M.

Re: HIGHMEM

[Matt Domsch]

On Fri, Feb 13, 2004 at 08:05:14AM -0800, Martin J. Bligh wrote:
> >> We have two Dell Poweredge servers, an older one (PowerEdge 6300) and a
> >> newer one (PowerEdge 6400). Both servers have 4GB RAM, but the Linux
> >> kernel uses about 500MB less memory in the newer machine.
> > 
> > I may be talking through my hat, but I think that in this case you
> > need to select the option for support of 64G highmem.  If I recall,
> > "4G highmem" refers not to the total amount to the memory, but to the
> > highest physical address that can be accessed.
> 
> That's exactly correct. Whether the gain of 500MB of RAM is worth the
> overhead of PAE is another question ... but that's how to do it ;-)

If the chipset and BIOS can't remap the physical RAM out of the
address space needed by the PCI devices and into PAE space, then PAE
doesn't buy you anything.  You need chipset support for RAM remapping,
which doesn't exist on the servers mentioned.

Thanks,
Matt

-- 
Matt Domsch
Sr. Software Engineer, Lead Engineer
Dell Linux Solutions linux.dell.com & www.dell.com/linux
Linux on Dell mailing lists @ http://lists.us.dell.com

Re: HIGHMEM

[Martin J. Bligh]

> On Fri, Feb 13, 2004 at 08:05:14AM -0800, Martin J. Bligh wrote:
>> >> We have two Dell Poweredge servers, an older one (PowerEdge 6300) and a
>> >> newer one (PowerEdge 6400). Both servers have 4GB RAM, but the Linux
>> >> kernel uses about 500MB less memory in the newer machine.
>> > 
>> > I may be talking through my hat, but I think that in this case you
>> > need to select the option for support of 64G highmem.  If I recall,
>> > "4G highmem" refers not to the total amount to the memory, but to the
>> > highest physical address that can be accessed.
>> 
>> That's exactly correct. Whether the gain of 500MB of RAM is worth the
>> overhead of PAE is another question ... but that's how to do it ;-)
> 
> If the chipset and BIOS can't remap the physical RAM out of the
> address space needed by the PCI devices and into PAE space, then PAE
> doesn't buy you anything.  You need chipset support for RAM remapping,
> which doesn't exist on the servers mentioned.

Ah, didn't realise you had a hardware problem as well ;-)

M.

Re: HIGHMEM

[david parsons]

On Fri, Feb 13, 2004 at 01:12:48PM +0000, Sean Neakums wrote:
> Nagy Tibor <nagyt@otpbank.hu> writes:
> 
> > Hi,
> >
> > I am sorry, I have found your e-mail address in
> > ./arch/i386/kernel/setup.c. I have the problem below since a year, and
> > there is no solution yet. I guess, the problem is about e820. The
> > problem exists in 2.4.x and also in 2.6.1.
> >
> > We have two Dell Poweredge servers, an older one (PowerEdge 6300) and a
> > newer one (PowerEdge 6400). Both servers have 4GB RAM, but the Linux
> > kernel uses about 500MB less memory in the newer machine.
> 
> I may be talking through my hat, but I think that in this case you
> need to select the option for support of 64G highmem.  If I recall,
> "4G highmem" refers not to the total amount to the memory, but to the
> highest physical address that can be accessed.


   Its been several years since I did anything with the 820 code,
   but it all uses long longs so it should be fine with up to 4gb
   and a casual look at the now much hacked code doesn't seem to
   show anything that would leap out and start laughing at you.


                 ____
   david parsons \bi/ ``Sanitize the BIOS e820 map''?  
                  \/

Re: HIGHMEM

[Matt Domsch]

On Fri, Feb 13, 2004 at 01:20:36PM +0100, Nagy Tibor wrote:
> We have two Dell Poweredge servers, an older one (PowerEdge 6300) and a
> newer one (PowerEdge 6400). Both servers have 4GB RAM, but the Linux
> kernel uses about 500MB less memory in the newer machine.

This is a FAQ on the Linux-PowerEdge mailing list, and is captured here.
http://lists.us.dell.com/fom-serve/cache/68.html

I have 4GB (or more) RAM in my system. How come Linux sees less than
that?

BIOS must reserve some address space below 4GB for PCI devices such as
RAID controllers, SCSI controllers, NICs, etc. RAID controllers in
particular may request and be given 256MB each. This is address space
that would normally be occupied by RAM, but instead is used by PCI
devices.

RAM addresses start at 0 and grow up. PCI device addresses start at
~4GB and grow down. As long as there is no overlap, the OS will see
all available RAM and make use of it. If there is overlap, the PCI
devices win, and that RAM is not made available to the OS.

This is working as designed per PCI, BIOS, and system chipset
specifications. 


You may wish to subscribe to the Linux-PowerEdge@dell.com mailing list
at http://lists.us.dell.com/.

Thanks,
Matt

-- 
Matt Domsch
Sr. Software Engineer, Lead Engineer
Dell Linux Solutions linux.dell.com & www.dell.com/linux
Linux on Dell mailing lists @ http://lists.us.dell.com