Does Linux have plan to support memory hole remapping?

Does Linux have plan to support memory hole remapping?

[Zhao Forrest]

Hi experts,

I ask this because I run kernel 2.6.25-rc8 on a x64 system with 32GB
physical memory, and kernel only use (32GB-512MB) physical memory. See
below related information:
E820 table:
BIOS-provided physical RAM map:
 BIOS-e820: 0000000000000000 - 0000000000098c00 (usable)
 BIOS-e820: 0000000000098c00 - 00000000000a0000 (reserved)
 BIOS-e820: 00000000000e6000 - 0000000000100000 (reserved)
 BIOS-e820: 0000000000100000 - 00000000dffa0000 (usable)
 BIOS-e820: 00000000dffae000 - 00000000dffb0000 type 9
 BIOS-e820: 00000000dffb0000 - 00000000dffbe000 (ACPI data)
 BIOS-e820: 00000000dffbe000 - 00000000dfff0000 (ACPI NVS)
 BIOS-e820: 00000000dfff0000 - 00000000dfffe000 (reserved)
 BIOS-e820: 00000000e0000000 - 00000000f0000000 (reserved)
 BIOS-e820: 00000000fec00000 - 00000000fec01000 (reserved)
 BIOS-e820: 00000000fee00000 - 00000000fee01000 (reserved)
 BIOS-e820: 00000000ff700000 - 0000000100000000 (reserved)
 BIOS-e820: 0000000100000000 - 0000000820000000 (usable)

/proc/mtrr:
reg00: base=0x00000000 (   0MB), size=32768MB: write-back, count=1
reg01: base=0x800000000 (32768MB), size= 512MB: write-back, count=1
reg02: base=0xe0000000 (3584MB), size= 512MB: uncachable, count=1

/proc/meminfo:
MemTotal:     33010240 kB
MemFree:      32715924 kB
Buffers:          1624 kB
Cached:          31412 kB
SwapCached:          0 kB
Active:          22348 kB
Inactive:        19728 kB
SwapTotal:           0 kB
SwapFree:            0 kB
Dirty:               0 kB
Writeback:           0 kB
AnonPages:        9120 kB
Mapped:           6408 kB
Slab:            24828 kB
SReclaimable:     8004 kB
SUnreclaim:      16824 kB
PageTables:       1296 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  16505120 kB
Committed_AS:    16556 kB
VmallocTotal: 34359738367 kB
VmallocUsed:     73524 kB
VmallocChunk: 34359664507 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
HugePages_Surp:      0
Hugepagesize:     2048 kB

As we can see from above information that the physical memory in system
is 32768MB(32GB). However OS is only using about
(32768-512)MB(MemTotal:     33010240 kB). Does this mean that this
linux kernel can't use the physical memory remapped
from (4G-512M, 4G) to (32G, 32G+512M)?

Thanks,
Forrest

Re: Does Linux have plan to support memory hole remapping?

[Andi Kleen]

"Zhao Forrest" <forrest.zhao@gmail.com> writes:
>
> As we can see from above information that the physical memory in system
> is 32768MB(32GB). However OS is only using about
> (32768-512)MB(MemTotal:     33010240 kB). Does this mean that this
> linux kernel can't use the physical memory remapped
> from (4G-512M, 4G) to (32G, 32G+512M)?

The linux kernel can only use the memory passed to it by the BIOS.
Sometimes they need special BIOS setup options to enable remapping. If
there are no such options and you can't upgrade it you're out of luck

I would suggest you contact your system vendor.

-Andi

Re: Does Linux have plan to support memory hole remapping?

[Matti Aarnio]

On Wed, Apr 09, 2008 at 10:01:59AM +0200, Andi Kleen wrote:
> "Zhao Forrest" <forrest.zhao@gmail.com> writes:
> >
> > As we can see from above information that the physical memory in system
> > is 32768MB(32GB). However OS is only using about
> > (32768-512)MB(MemTotal:     33010240 kB). Does this mean that this
> > linux kernel can't use the physical memory remapped
> > from (4G-512M, 4G) to (32G, 32G+512M)?
> 
> The linux kernel can only use the memory passed to it by the BIOS.
> Sometimes they need special BIOS setup options to enable remapping. If
> there are no such options and you can't upgrade it you're out of luck
> 
> I would suggest you contact your system vendor.

I second that.   On my home machine I have early ASUS  AMD x64 board from
a few years ago.  For it I did buy CPU with this memory mapping hardware
inside, but the BIOS did not support it correctly until a year or two latter,
thus I was not able to use all of 4 GB of memory I had installed there.
There was support in BIOS from start, but it did things wrong.

Looks like BIOS-writers don't really have test environments for extreme
far edges of system configurations - "yes, we support --> sales $$$".
And once product ships, rare extreme users rarely report anything back.


Long time ago (very long ?) when 386 machines got more than 1 MB memory,
there was (and probably still is) a hole in low 1 MB address space, because
of VGA display memory space and BIOS ROM.  With 1-32 MB of memory in system,
the hardware makers created inventive ways to map the masked 384 kB to
the top of the address space.

I make a prediction that the same way as that old VGA/BIOS mapping, also
the address space masked by PCI window will eventually simply be ignored.
.. or be used in automatic memory chip internal "faulty block mapper".
That will start to happen in about 10 years.  (Perhaps also PCIe starts
to disappear around that time, like PCI does now.)

> -Andi

/Matti Aarnio

Re: Does Linux have plan to support memory hole remapping?

[Andi Kleen]

> I second that.   On my home machine I have early ASUS  AMD x64 board from
> a few years ago.  For it I did buy CPU with this memory mapping hardware
> inside, but the BIOS did not support it correctly until a year or two latter,
> thus I was not able to use all of 4 GB of memory I had installed there.

To be fair the low-end boards often state in their documentation that they only 
got tested with upto 2GB. So strictly you were out of spec if you plug in 4GB.

> Looks like BIOS-writers don't really have test environments for extreme
> far edges of system configurations - "yes, we support --> sales $$$".
> And once product ships, rare extreme users rarely report anything back.

It depends on how much you pay. Higher end boards tend to get tested
in higher end configurations. Low end boards not.

-Andi

Re: Does Linux have plan to support memory hole remapping?

[Ingo Molnar]

* Matti Aarnio <matti.aarnio@zmailer.org> wrote:

> On Wed, Apr 09, 2008 at 10:01:59AM +0200, Andi Kleen wrote:
> > "Zhao Forrest" <forrest.zhao@gmail.com> writes:
> > >
> > > As we can see from above information that the physical memory in system
> > > is 32768MB(32GB). However OS is only using about
> > > (32768-512)MB(MemTotal:     33010240 kB). Does this mean that this
> > > linux kernel can't use the physical memory remapped
> > > from (4G-512M, 4G) to (32G, 32G+512M)?
> > 
> > The linux kernel can only use the memory passed to it by the BIOS.
> > Sometimes they need special BIOS setup options to enable remapping. If
> > there are no such options and you can't upgrade it you're out of luck
> > 
> > I would suggest you contact your system vendor.
> 
> I second that.  On my home machine I have early ASUS AMD x64 board 
> from a few years ago.  For it I did buy CPU with this memory mapping 
> hardware inside, but the BIOS did not support it correctly until a 
> year or two latter, thus I was not able to use all of 4 GB of memory I 
> had installed there. There was support in BIOS from start, but it did 
> things wrong.

having said that - but we'll still consider sane looking patches that 
allow the remapping of otherwise inactive RAM. [ We might not even have 
to touch the system chipset beyond what Linux already knows about it: in 
theory someone might want to try a hack to GART remap such RAM areas and 
use a sparse mem map entry to map it to Linux - if the identity of that 
RAM is crystal-clear and there's enough GART space available. But even 
then it's probably still at best a dangerous and fragile hack. ]

	Ingo

Re: Does Linux have plan to support memory hole remapping?

[Andi Kleen]

> having said that - but we'll still consider sane looking patches that 
> allow the remapping of otherwise inactive RAM. 

I doubt there is any way to do this sanely in the kernel. Usually you 
would need to move either the PCI hole (which is hard or impossible) or all 
RAM beyond it (potentially breaking SMM assumptions and causing other problems). 

Normally the memory controllers don't have a way to just remap pieces because
that would need much hardware in critical paths. 
 
The only half way sane way to attack this without vendor support 
would be LinuxBIOS imho.

-Andi

Re: Does Linux have plan to support memory hole remapping?

[Kasper Sandberg]

On Wed, 2008-04-09 at 10:54 +0200, Ingo Molnar wrote:
[..]
> 
> having said that - but we'll still consider sane looking patches that 
> allow the remapping of otherwise inactive RAM. [ We might not even have 
> to touch the system chipset beyond what Linux already knows about it: in 
A somewhat related question is.. How come linux does not(as i understand
it) support talking to the chipsets to do these things? wouldnt this
also give other benefits, such as being able to more precisely control
interrupts and stuff?

> theory someone might want to try a hack to GART remap such RAM areas and 
> use a sparse mem map entry to map it to Linux - if the identity of that 
> RAM is crystal-clear and there's enough GART space available. But even 
> then it's probably still at best a dangerous and fragile hack. ]
> 
> 	Ingo
> --
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/

Re: Does Linux have plan to support memory hole remapping?

[H. Peter Anvin]

Kasper Sandberg wrote:
> A somewhat related question is.. How come linux does not(as i understand
> it) support talking to the chipsets to do these things? wouldnt this
> also give other benefits, such as being able to more precisely control
> interrupts and stuff?

Sure.  Plus, it's virtually guaranteed to crash or corrupt your system 
if you try to do anything which requires ACPI or SMM.

	-hpa

Re: Does Linux have plan to support memory hole remapping?

[Kasper Sandberg]

On Wed, 2008-04-09 at 19:09 -0700, H. Peter Anvin wrote:
> Kasper Sandberg wrote:
> > A somewhat related question is.. How come linux does not(as i understand
> > it) support talking to the chipsets to do these things? wouldnt this
> > also give other benefits, such as being able to more precisely control
> > interrupts and stuff?
> 
> Sure.  Plus, it's virtually guaranteed to crash or corrupt your system 
> if you try to do anything which requires ACPI or SMM.
Ah, i just thought that a specific other OS actually did this (which,
now that you mention this, makes a whole lot of sense :-))
> 
> 	-hpa
> --
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/

Re: Does Linux have plan to support memory hole remapping?

[Andi Kleen]

Kasper Sandberg <lkml@metanurb.dk> writes:

> A somewhat related question is.. How come linux does not(as i understand
> it) support talking to the chipsets to do these things? wouldnt this
> also give other benefits, such as being able to more precisely control
> interrupts and stuff?

Linux is an portable operating system and not a hardware specific BIOS. So it
prefers to not know about some things.

-Andi

Re: Does Linux have plan to support memory hole remapping?

[Zhao Forrest]

On 4/9/08, Andi Kleen <andi@firstfloor.org> wrote:
> "Zhao Forrest" <forrest.zhao@gmail.com> writes:
> >
> > As we can see from above information that the physical memory in system
> > is 32768MB(32GB). However OS is only using about
> > (32768-512)MB(MemTotal:     33010240 kB). Does this mean that this
> > linux kernel can't use the physical memory remapped
> > from (4G-512M, 4G) to (32G, 32G+512M)?
>
> The linux kernel can only use the memory passed to it by the BIOS.
> Sometimes they need special BIOS setup options to enable remapping. If
> there are no such options and you can't upgrade it you're out of luck

Yes, I have enabled mem hole remapping in BIOS SETUP.
After looking into dmesg, I found that all 32GB physical memory has
been allocated to 8 nodes, but why MemTotal shows a less amount of
memory than 32GB? Interesting......
Does MemTotal means the amount of physical memory allocated to all
nodes? Or it has different meanings?

Thanks,
Forrest

Re: Does Linux have plan to support memory hole remapping?

[Andi Kleen]

On Wed, Apr 09, 2008 at 05:04:24PM +0800, Zhao Forrest wrote:
> On 4/9/08, Andi Kleen <andi@firstfloor.org> wrote:
> > "Zhao Forrest" <forrest.zhao@gmail.com> writes:
> > >
> > > As we can see from above information that the physical memory in system
> > > is 32768MB(32GB). However OS is only using about
> > > (32768-512)MB(MemTotal:     33010240 kB). Does this mean that this
> > > linux kernel can't use the physical memory remapped
> > > from (4G-512M, 4G) to (32G, 32G+512M)?
> >
> > The linux kernel can only use the memory passed to it by the BIOS.
> > Sometimes they need special BIOS setup options to enable remapping. If
> > there are no such options and you can't upgrade it you're out of luck
> 
> Yes, I have enabled mem hole remapping in BIOS SETUP.
> After looking into dmesg, I found that all 32GB physical memory has
> been allocated to 8 nodes, but why MemTotal shows a less amount of
> memory than 32GB? Interesting......

There is some loss of memory before MemTotal, both from the BIOS
(SMM, Frame Buffer for integrated graphics etc.) and from the kernel
(mem_map which costs a few percent and some other data structures
which are allocated early) 

I covered this in detail in http://halobates.de/memorywaste.pdf

> Does MemTotal means the amount of physical memory allocated to all
> nodes? Or it has different meanings?

Amount of memory left over after early boot on all nodes and visible
to the kernel (so e.g. minus reservations for kdump kernels) 

-Andi

Re: Does Linux have plan to support memory hole remapping?

[Zhao Forrest]

> There is some loss of memory before MemTotal, both from the BIOS
> (SMM, Frame Buffer for integrated graphics etc.) and from the kernel
> (mem_map which costs a few percent and some other data structures
> which are allocated early)
>
> I covered this in detail in http://halobates.de/memorywaste.pdf
>

I read your paper, which is very helpful for us to understand the
memory usage in Linux kernel.
And I have a question about output of slabtop:
 Active / Total Objects (% used)    : 730986 / 750823 (97.4%)
 Active / Total Slabs (% used)      : 46124 / 46125 (100.0%)
 Active / Total Caches (% used)     : 99 / 149 (66.4%)
 Active / Total Size (% used)       : 169383.63K / 172467.11K (98.2%)

Could you please elaborate on Active / Total Caches? The meaning of
other lines are obvious to me.

Thanks,
Forrest

Re: Does Linux have plan to support memory hole remapping?

[Andi Kleen]

> Could you please elaborate on Active / Total Caches? The meaning of
> other lines are obvious to me.

slab keeps some objects ready on a free list for quick access. Total
includes those. active are only the objects which are truly allocated.

-Andi

Re: Does Linux have plan to support memory hole remapping?

[Arne Georg Gleditsch]

Andi Kleen <andi@firstfloor.org> writes:

> "Zhao Forrest" <forrest.zhao@gmail.com> writes:
>>
>> As we can see from above information that the physical memory in system
>> is 32768MB(32GB). However OS is only using about
>> (32768-512)MB(MemTotal:     33010240 kB). Does this mean that this
>> linux kernel can't use the physical memory remapped
>> from (4G-512M, 4G) to (32G, 32G+512M)?
>
> The linux kernel can only use the memory passed to it by the BIOS.
> Sometimes they need special BIOS setup options to enable remapping. If
> there are no such options and you can't upgrade it you're out of luck

Hm, wouldn't the given e820 map and mtrr listing indicate that 512M were
actually remapped to 32G+ in this case?

-- 
								Arne.

Re: Does Linux have plan to support memory hole remapping?

[Andi Kleen]

On Wed, Apr 09, 2008 at 11:50:00AM +0200, Arne Georg Gleditsch wrote:
> Andi Kleen <andi@firstfloor.org> writes:
> 
> > "Zhao Forrest" <forrest.zhao@gmail.com> writes:
> >>
> >> As we can see from above information that the physical memory in system
> >> is 32768MB(32GB). However OS is only using about
> >> (32768-512)MB(MemTotal:     33010240 kB). Does this mean that this
> >> linux kernel can't use the physical memory remapped
> >> from (4G-512M, 4G) to (32G, 32G+512M)?
> >
> > The linux kernel can only use the memory passed to it by the BIOS.
> > Sometimes they need special BIOS setup options to enable remapping. If
> > there are no such options and you can't upgrade it you're out of luck
> 
> Hm, wouldn't the given e820 map and mtrr listing indicate that 512M were
> actually remapped to 32G+ in this case?

The way it usually works (if it is implemented correctly in the BIOS) 
is that all memory starting at the hole moves up together 
(often subject to DIMM boundaries etc.),
not that the area below the hole is remapped individually.

BTW it is not actually 512MB that is lost. MemTotal does not 
include mem_map and that alone is ~512MB (64 bytes for each 4K page)

So as far as I can see there is no missing memory remapping in Zhao's case, 
he's just confused by the MemTotal semantics.

-Andi

Re: Does Linux have plan to support memory hole remapping?

[Arne Georg Gleditsch]

Andi Kleen <andi@firstfloor.org> writes:
> On Wed, Apr 09, 2008 at 11:50:00AM +0200, Arne Georg Gleditsch wrote:
>> Hm, wouldn't the given e820 map and mtrr listing indicate that 512M were
>> actually remapped to 32G+ in this case?
>
> The way it usually works (if it is implemented correctly in the BIOS) 
> is that all memory starting at the hole moves up together 
> (often subject to DIMM boundaries etc.),
> not that the area below the hole is remapped individually.

Yes, bad choice of words on my part.

> BTW it is not actually 512MB that is lost. MemTotal does not 
> include mem_map and that alone is ~512MB (64 bytes for each 4K page)
>
> So as far as I can see there is no missing memory remapping in Zhao's case, 
> he's just confused by the MemTotal semantics.

...meaning that upgrading his BIOS isn't going to change anything for
him, which is what I suspected from the e820 readouts.

-- 
								Arne.

Re: Does Linux have plan to support memory hole remapping?

[Yinghai Lu]

On Wed, Apr 9, 2008 at 3:03 AM, Andi Kleen <andi@firstfloor.org> wrote:
>
> On Wed, Apr 09, 2008 at 11:50:00AM +0200, Arne Georg Gleditsch wrote:
>  > Andi Kleen <andi@firstfloor.org> writes:
>  >
>  > > "Zhao Forrest" <forrest.zhao@gmail.com> writes:
>  > >>
>  > >> As we can see from above information that the physical memory in system
>  > >> is 32768MB(32GB). However OS is only using about
>  > >> (32768-512)MB(MemTotal:     33010240 kB). Does this mean that this
>  > >> linux kernel can't use the physical memory remapped
>  > >> from (4G-512M, 4G) to (32G, 32G+512M)?
>  > >
>  > > The linux kernel can only use the memory passed to it by the BIOS.
>  > > Sometimes they need special BIOS setup options to enable remapping. If
>  > > there are no such options and you can't upgrade it you're out of luck
>  >
>  > Hm, wouldn't the given e820 map and mtrr listing indicate that 512M were
>  > actually remapped to 32G+ in this case?
>
>  The way it usually works (if it is implemented correctly in the BIOS)
>  is that all memory starting at the hole moves up together
>  (often subject to DIMM boundaries etc.),
>  not that the area below the hole is remapped individually.
>
>  BTW it is not actually 512MB that is lost. MemTotal does not
>  include mem_map and that alone is ~512MB (64 bytes for each 4K page)
>
>  So as far as I can see there is no missing memory remapping in Zhao's case,
>  he's just confused by the MemTotal semantics.

agreed. the HW/BIOS already enable HW memory hole.

YH

Re: Does Linux have plan to support memory hole remapping?

[Lennart Sorensen]

On Wed, Apr 09, 2008 at 11:50:00AM +0200, Arne Georg Gleditsch wrote:
> Hm, wouldn't the given e820 map and mtrr listing indicate that 512M were
> actually remapped to 32G+ in this case?

The mtrr does list the 512MB as being remapped.  It sais there was 32GB
starting at 0, and 512MB starting at 32GB, and then that there was an
uncacheable 512MB hole at 3.5GB.  So the mtrr does say everything looks
right.

The e820 map also shows it:
BIOS-e820: 0000000000000000 - 0000000000098c00 (usable)
       625,664 bytes
BIOS-e820: 0000000000100000 - 00000000dffa0000 (usable)
 3,756,654,592 bytes
BIOS-e820: 0000000100000000 - 0000000820000000 (usable)
30,601,641,984 bytes

Total of 34,358,922,240 bytes or 32767.221 MiB.  Seems rather close to
the full 32768 MiB in the system.  So looks to me like the mtrr and e820
covers all the ram.  Maybe the kernel looses some of it somewhere for
other purposes.

Isn't there usually a line in dmesg similar to this:
Memory: 1023120k/1048256k available (1927k kernel code, 24748k reserved, 868k data, 176k init)

Maybe it can show how much the kernel reserved for itself in this case.

-- 
Len Sorensen