Re: Opteron box and 4Gb memory

["J.A. Magallón" <jamagallon@ono.com>]

On Thu, 25 Oct 2007 14:58:10 -0700, "H. Peter Anvin" <hpa@zytor.com> wrote:

> J.A. Magallon wrote:
> > Hi...
> > 
> > I have some Quad-Opteron boxes with 4Gb memory and two of them are
> > running two different Linux distros.
> > 
> > Box one sees 4Gb of memory, but box two just sees 3.
> > Their mtrr setups are different:
> > 
> > Why ? Is it a bios setup problem ? A kernel problem ?
> > grep HIGHMEN in configs for both kernels does not give anything, so
> > I still understand less this thing...
> > 
> 
> It would depend on how the BIOS programmed the memory controllers.  For 
> 32-bit (and lots of device) compatibility, a memory hole is required 
> below 4 GB.  Not all memory controllers can remap memory in the 3-4 GB 
> range above the 4 GB memory; I'm not sure if that varies with the 
> different Opteron processors.

I have collected several pieces of info around the internet...

- Some people uses this options in the BIOS:
	Node interleave: off
	Bank interleave: auto
	SW memory hole: disable
	HW memory hole: enable
	MTRR: Continuous

- Node Memory Interleaving DISABLES NUMA and generally is a bad thing

- MTRR setting -should be set to "discrete" for Linux, and probably for Windows too.

- This is what SuperMicro's tech support said about 2.96GB vs. 4GB.

"This is as expected, as soon as you set "software memory hole" to disabled,
you also disable option ROM remapping functionality, this option normally
remaps used option rom (option rom= raid bios, lan pxe ; usb legacy, bioses
on add-on cards, etc) in the 4GB region, so no basis memory is lost, while
this feature is now disabled the option rom space occupies the space between
3 and 4 GB which results in lower main memory availability.
There is no solution or work around for this phenomenon"

so software memory hole enabled might be needed to get all 4GB to show up

>From mobo manual:

Software Memory Hole
When "Enabled", allows software memory remapping around the memory
hole. Options are Enabled and Disabled.

Hardware Memory Hole
When "Enabled", allows software memory remapping around the memory
hole. Options are Enabled and Disabled. Note: this is only supported by
Rev E0 processors and above.
( I have two Opteron 275 processors, no idea about revision)

So _my_ conclussion is:

	Node interleave: off    (numa mode)
	Bank interleave: auto
	SW memory hole: disable |
	HW memory hole: enable  | allow remapping
	MTRR: Discrete          |

But then, do I need to enable NUMA options in the kernel ?

> 
> Also, if you run a 32-bit distribution, you need to have HIGHMEM_64G 
> enabled in the kernel.
> 

I run a 64 bit one, then I don't need anything, isn't it ? That's why I
don't see any _HIGHMEM in the kernel configs...

Some day I will understand this crappy BIOS thing (or burn a photo of its
inventor...).
Why can't we have OpenFirmware PC's, like my MacPro  and Sparcs ?

--
J.A. Magallon <jamagallon()ono!com>     \               Software is like sex:
                                         \         It's better when it's free
Mandriva Linux release 2008.1 (Cooker) for i586
Linux 2.6.23-jam01 (gcc 4.2.2 20070909 (4.2.2-0.RC.1mdv2008.0)) SMP PREEMPT
09 F9 11 02 9D 74 E3 5B D8 41 56 C5 63 56 88 C0