[x86_32] With 4GB installed, in which cases low mem total is less than 896MB?

[x86_32] With 4GB installed, in which cases low mem total is less than 896MB?

[Fabio De Francesco]

Hi all,

I've got a Linux box, Intel Core 2 T7700 with 4GB RAM installed, with vanilla kernel 2.6.24 + TuxOnIce patch-set compiled for x86_32bit (CONFIG_X86_32=y), whose /proc/meminfo shows "LowTotal 794MB".

I've always known that, in this specific platform, Linux permanently maps 896MB as "Low Memory" into its linear address space (as this last value is the one /proc/meminfo shows in the other two x86_32bit computers I own). 

What am I missing?

A further issue is about the "MemTotal" value that only shows more or less 3550MB with HIGHMEM4G selected, and it shows a  more satisfying value of 4050MB with HIGHMEM64G + X86_PAE + RESOURCES_64BIT selected. In both cases LowTotal is always less than 800MB.

Why is it needed to activate the PAE mechanism in order to see and use the whole 4GB memory? Is it what it is expected to be done?

Thanks in advance to anyone who will reply.

Regards,

fabio de francesco
metanix.org

Re: [x86_32] With 4GB installed, in which cases low mem total is less than 896MB?

[Arjan van de Ven]

On Wed, 30 Apr 2008 17:02:02 +0200
Fabio De Francesco <fabiomdf@alice.it> wrote:

> Hi all,
> 
> I've got a Linux box, Intel Core 2 T7700 with 4GB RAM installed, with
> vanilla kernel 2.6.24 + TuxOnIce patch-set compiled for x86_32bit
> (CONFIG_X86_32=y), whose /proc/meminfo shows "LowTotal 794MB".

I'm not an expert in this, but one thing is that the more memory the system has, the more overhead there is.
(roughly 64 bytes per 4Kb of memory).

> 
> A further issue is about the "MemTotal" value that only shows more or
> less 3550MB with HIGHMEM4G selected, and it shows a  more satisfying
> value of 4050MB with HIGHMEM64G + X86_PAE + RESOURCES_64BIT selected.
> In both cases LowTotal is always less than 800MB.
> 
> Why is it needed to activate the PAE mechanism in order to see and
> use the whole 4GB memory? Is it what it is expected to be done?

yes this is expected.
Let me explain: even though you have 4Gb of memory, it's spread out like this:

0     - 3.5Gb	Memory
3.5Gb - 4Gb  	Gap
4Gb   - 4.5Gb   Memory

This "gap" is needed for PCI. PCI devices have "memory" on them (technical term is "MMIO", memory mapped IO) and
this needs to be in the first 4Gb of memory (traditionally). So the system leaves a 0.5Gb gap on your system
for this purpose.
Some other systems have 0.25Gb or 1Gb gaps; some systems even have a BIOS option for the size of this gap.
But a gap.. all machines with >= 4Gb have that.

Just enable PAE and be done with it. (And you get the non-executable stack security feature as well then)

Re: [x86_32] With 4GB installed, in which cases low mem total is less than 896MB?

[Fabio De Francesco]

On Tuesday 29 April 2008 19:18:52 Arjan van de Ven wrote:
>On Wed, 30 Apr 2008 17:02:02 +0200
>Fabio De Francesco <fabiomdf@alice.it> wrote:
>
>> Hi all,
>> 
>> I've got a Linux box, Intel Core 2 T7700 with 4GB RAM installed, with
>> vanilla kernel 2.6.24 + TuxOnIce patch-set compiled for x86_32bit
>> (CONFIG_X86_32=y), whose /proc/meminfo shows "LowTotal 794MB".
>
>I'm not an expert in this, but one thing is that the more memory the system has, the more overhead there is.
>(roughly 64 bytes per 4Kb of memory).
>

Arjan,

Unfortunetally my poor knowledge of Linux kernel mechanisms (and code!) prevents me from understanding what could be the causes for that overhead. Do you mean it is a hardware issue? Or is it a kernel one? Anyway another machine of mine, that is equipped with 2GB RAM, doesn't show any proportional reduction of low memory at all. So this makes me think (oh, dear!).

Anyway I started a little research (eh?) in Linux code in order to understand how the low memory mapping is done, just in case... Please be patient if what I've done shows nothing relevant for this subject.  

I've just read arch/x86/mm/init_32.c where, if I understand well(?) the mapping happens. I started from pagetable_init() following the flow to kernel_physical_mapping_init() where the hard work is done. Comments before this last function say that "maps the physical memory to kernel virtual address space, a total of max_low_pfn pages, by creating page tables starting from address PAGE_OFFSET".

Ok, then... max_low_pfn is set upon what the BIOS had provided in arch/x86/kernel/setup_32.c, by function find_max_low_pfn().

After having inserted a couple of printk() showing the value of max_low_pfn and pfn (a local variable of kernel_physical_mapping_init() that is incremented in a for() at each iteration, and which counts how many pages have really been mapped) in the relevant(?) sections/functions, this is the first part of dmesg:

[    0.000000] Linux version 2.6.24-tuxonice-r6-080430 (root@host11) (gcc version 4.2.3 (Gentoo 4.2.3 p1.0)) #5 SMP PREEMPT Thu May 1 22:24:26 CEST 2008
[    0.000000] BIOS-provided physical RAM map:
[    0.000000]  BIOS-e820: 0000000000000000 - 000000000009f000 (usable)
[    0.000000]  BIOS-e820: 000000000009f000 - 00000000000a0000 (reserved)
[    0.000000]  BIOS-e820: 0000000000100000 - 00000000dfe72000 (usable)
[    0.000000]  BIOS-e820: 00000000dfe72000 - 00000000e0000000 (reserved)
[    0.000000]  BIOS-e820: 00000000f4000000 - 00000000f8000000 (reserved)
[    0.000000]  BIOS-e820: 00000000fec00000 - 00000000fec10000 (reserved)
[    0.000000]  BIOS-e820: 00000000fed18000 - 00000000fed1c000 (reserved)
[    0.000000]  BIOS-e820: 00000000fed20000 - 00000000fed90000 (reserved)
[    0.000000]  BIOS-e820: 00000000feda0000 - 00000000feda6000 (reserved)
[    0.000000]  BIOS-e820: 00000000fee00000 - 00000000fee10000 (reserved)
[    0.000000]  BIOS-e820: 00000000ffe00000 - 0000000100000000 (reserved)
[    0.000000]  BIOS-e820: 0000000100000000 - 0000000120000000 (usable)
[    0.000000] ---> find_max_lox_pfn() returns max_low_pfn value of 229376 <---
[    0.000000] 3712MB HIGHMEM available.
[    0.000000] 896MB LOWMEM available.
[    0.000000] NX (Execute Disable) protection: active
[    0.000000] ---> In kernel_physical_mapping_init(), max_low_pfn value is 229376 <---
[    0.000000] ---> In kernel_physical_mapping_init(), after mapping, max_low_pfn and pfn values are 229376 and 229376 <---
[    0.000000] Entering add_active_range(0, 0, 1179648) 0 entries of 256 used
[    0.000000] Zone PFN ranges:
[    0.000000]   DMA             0 ->     4096
[    0.000000]   Normal       4096 ->   229376
[    0.000000]   HighMem    229376 ->  1179648
[    0.000000] Movable zone start PFN for each node
[    0.000000] early_node_map[1] active PFN ranges
[    0.000000]     0:        0 ->  1179648
[    0.000000] On node 0 totalpages: 1179648
[    0.000000]   DMA zone: 32 pages used for memmap
[    0.000000]   DMA zone: 0 pages reserved
[    0.000000]   DMA zone: 4064 pages, LIFO batch:0
[    0.000000]   Normal zone: 1760 pages used for memmap
[    0.000000]   Normal zone: 223520 pages, LIFO batch:31
[    0.000000]   HighMem zone: 7424 pages used for memmap
[    0.000000]   HighMem zone: 942848 pages, LIFO batch:31
[    0.000000]   Movable zone: 0 pages used for memmap
...skip...
Memory: 4068032k/4718592k available (4030k kernel code, 123504k reserved, 1642k data, 264k init, 3275208k highmem)
[   10.657852] virtual kernel memory layout:
[   10.657852]     fixmap  : 0xfff9b000 - 0xfffff000   ( 400 kB)
[   10.657853]     pkmap   : 0xffc00000 - 0xffe00000   (2048 kB)
[   10.657854]     vmalloc : 0xf8800000 - 0xffbfe000   ( 115 MB)
[   10.657855]     lowmem  : 0xc0000000 - 0xf8000000   ( 896 MB)
[   10.657856]       .init : 0xc0691000 - 0xc06d3000   ( 264 kB)
[   10.657856]       .data : 0xc04ef916 - 0xc068a154   (1642 kB)
[   10.657857]       .text : 0xc0100000 - 0xc04ef916   (4030 kB)

I think it only remains to me to understand which policy /proc/meminfo uses in calculation of total low memory.

As for it is concerned with my other question, I understand yours and Alan replies on the PCI hole.

Thank you.

-- 
Best Regards,

fabio de francesco
metanix.org

Re: [x86_32] With 4GB installed, in which cases low mem total is less than 896MB?

[Andi Kleen]

Fabio De Francesco <fabiomdf@alice.it> writes:
>
> Unfortunetally my poor knowledge of Linux kernel mechanisms (and
> code!) prevents me from understanding what could be the causes for
> that overhead. Do you mean it is a hardware issue? Or is it a kernel
> one? Anyway another machine of mine, that is equipped with 2GB RAM,
> doesn't show any proportional reduction of low memory at all. So
> this makes me think (oh, dear!).

You might want to read http://halobates.de/memorywaste.pdf, in
particular 3.5.2. The numbers arjan quoted and in the paper
are for 64bit kernels, on 32bit the overhead is ~32bytes/page. This
mem_map overhead happens for both highmem and lowmem, but is always
allocated in lowmem.

-Andi

Re: [x86_32] With 4GB installed, in which cases low mem total is less than 896MB?

[Alan Cox]

> Why is it needed to activate the PAE mechanism in order to see and use the whole 4GB memory? Is it what it is expected to be done?
> 
> Thanks in advance to anyone who will reply.

The PCI hole has to be below 4GB so the Intel chipset maps some RAM above
4GB where PAE is needed to address it.

Re: [x86_32] With 4GB installed, in which cases low mem total is less than 896MB?

[Rene Herman]

On 30-04-08 17:02, Fabio De Francesco wrote:

> I've got a Linux box, Intel Core 2 T7700 with 4GB RAM installed, with
> vanilla kernel 2.6.24 + TuxOnIce patch-set compiled for x86_32bit
> (CONFIG_X86_32=y), whose /proc/meminfo shows "LowTotal 794MB".
> 
> I've always known that, in this specific platform, Linux permanently maps
> 896MB as "Low Memory" into its linear address space (as this last value
> is the one /proc/meminfo shows in the other two x86_32bit computers I
> own).

Others have already answered as well so this just as by the way and just in 
case... why are you running a 32-bit kernel on it?

Rene.

Re: [x86_32] With 4GB installed, in which cases low mem total is less than 896MB?

[Fabio De Francesco]

On Sunday 04 May 2008 16:37:08 you wrote:
>On 30-04-08 17:02, Fabio De Francesco wrote:
>
>> I've got a Linux box, Intel Core 2 T7700 with 4GB RAM installed, with
>> vanilla kernel 2.6.24 + TuxOnIce patch-set compiled for x86_32bit
>> (CONFIG_X86_32=y), whose /proc/meminfo shows "LowTotal 794MB".
>> 
>> I've always known that, in this specific platform, Linux permanently maps
>> 896MB as "Low Memory" into its linear address space (as this last value
>> is the one /proc/meminfo shows in the other two x86_32bit computers I
>> own).
>
>Others have already answered as well so this just as by the way and just in 
>case... why are you running a 32-bit kernel on it?
>
>Rene.
>

Hello Rene,

It's only because when I got the box I had to make it run as soon as possible, sooner than I'd had time to investigate which possible benefits I would have had from running a 64bit distribution.

Further, since I had to install a Gentoo GNU/Linux distribution by "emerging" from a chroot(-ed) environment inside an Ubuntu 32bit one (it takes too long to explain why), I supposed I couldn't do that for installing a 64bit Gentoo. Still today I don't know whether or not I assumed that wrongly.

Today I would switch to a 64bit Gentoo but, first I don't know how to do that in a secure and fast way, second I still don't know if it is worth the job... 

Could you please summarize which benefits I would gain from that?

-- 

Best Regards,

fabio de francesco
metanix.org

Re: [x86_32] With 4GB installed, in which cases low mem total is less than 896MB?

[Rene Herman]

On 08-05-08 09:48, Fabio De Francesco wrote:

>>> I've got a Linux box, Intel Core 2 T7700 with 4GB RAM installed, with

>> Others have already answered as well so this just as by the way and just in 
>> case... why are you running a 32-bit kernel on it?

> It's only because when I got the box I had to make it run as soon as
> possible, sooner than I'd had time to investigate which possible
> benefits I would have had from running a 64bit distribution.
> 
> Further, since I had to install a Gentoo GNU/Linux distribution by
> "emerging" from a chroot(-ed) environment inside an Ubuntu 32bit one
> (it takes too long to explain why), I supposed I couldn't do that for
> installing a 64bit Gentoo. Still today I don't know whether or not I
> assumed that wrongly.
> 
> Today I would switch to a 64bit Gentoo but, first I don't know how to
> do that in a secure and fast way, second I still don't know if it is
> worth the job...
> 
> Could you please summarize which benefits I would gain from that?

Not directly as it's quite dependent on your use of the box but generally
I'd quite definitely use 64-bit on that. Running a 32-bit OS on a 64-bit
capable 4G machine today is sort of like running MS-DOS on a 64 MB PC a
decade ago (or like putting a Fiat 500 engine in an Alfa Romeo if you
prefer car analogies...).

32-bit architectures have a 32-bit address space but this does not
translate directly into being able to cope with 4G of memory, and
certainly not well. You already remarked on the 896 MB lowmem; note
that it might've been called 'sanemem' just as well really. If you
want to address more than that 896M you have to resort to highmem or
adjust the traditional UNIX 3/1 address space split down. Worse still
in your situation -- not just more than 896M but 4G -- the system can
not even address all of your memory _at all_ without hardware hacks
such as PAE due to it also needing a bit of the physical address
space for things such as hardware mappings. In any case, not any
single application can make use of all of it on 32-bit.

Basically, 32-bit stopped being a good idea once physical memory
below 1G (or 768M) stopped being a good idea. If you'd experience
much practical downside depends on use. Highmem, lots of it, is
slower but you might not care if you don't use all that memory
intensively, nor about the application addressing limit, and with 4G
you'd not yet be running into situations that the amount of lowmem
available gets to be too little to support the amount of highmem ...

... but running a 32-bit on that machine while it's capable of 64-bit
is still Just Wrong(TM). It works, but it's stapling kludges onto
kludges.

Rene.