x86_64 virtual memory map

x86_64 virtual memory map

[hayfeng Lee]

Hi,folks.
Recently I'm studying some things of x86_64 on Linux. And the virsion
is 2.6.18.8. From the document of Documentation/x86_64/mm.txt,I found
the mapping method for x86_64 virtual memory map. I want to know ,why
use this method for virtual memory mapping?

-----------------------------------------------------------------------------------------------
  1
  2 <previous description obsolete, deleted>
  3
  4 Virtual memory map with 4 level page tables:
  5
  6 0000000000000000 - 00007fffffffffff (=47bits) user space, different per mm
  7 hole caused by [48:63] sign extension
  8 ffff800000000000 - ffff80ffffffffff (=40bits) guard hole
  9 ffff810000000000 - ffffc0ffffffffff (=46bits) direct mapping of
all phys. memory
 10 ffffc10000000000 - ffffc1ffffffffff (=40bits) hole
 11 ffffc20000000000 - ffffe1ffffffffff (=45bits) vmalloc/ioremap space
 12 ... unused hole ...
 13 ffffffff80000000 - ffffffff82800000 (=40MB)   kernel text mapping,
from phys 0
 14 ... unused hole ...
 15 ffffffff88000000 - fffffffffff00000 (=1919MB) module mapping space
 16
 17 The direct mapping covers all memory in the system upto the highest
 18 memory address (this means in some cases it can also include PCI memory
 19 holes)
 20
 21 vmalloc space is lazily synchronized into the different PML4 pages of
 22 the processes using the page fault handler, with init_level4_pgt as
 23 reference.
 24
 25 Current X86-64 implementations only support 40 bit of address space,
 26 but we support upto 46bits. This expands into MBZ space in the page tables.
 27
 28 -Andi Kleen, Jul 2004

I urgently want to know the answer.

Re: x86_64 virtual memory map

[Eric W. Biederman]

hayfeng Lee <omycle@gmail.com> writes:

> Hi,folks.
> Recently I'm studying some things of x86_64 on Linux. And the virsion
> is 2.6.18.8. From the document of Documentation/x86_64/mm.txt,I found
> the mapping method for x86_64 virtual memory map. I want to know ,why
> use this method for virtual memory mapping?
>
> -----------------------------------------------------------------------------------------------
>   1
>   2 <previous description obsolete, deleted>
>   3
>   4 Virtual memory map with 4 level page tables:
>   5
>   6 0000000000000000 - 00007fffffffffff (=47bits) user space, different per mm
>   7 hole caused by [48:63] sign extension
>   8 ffff800000000000 - ffff80ffffffffff (=40bits) guard hole
>   9 ffff810000000000 - ffffc0ffffffffff (=46bits) direct mapping of
> all phys. memory
>  10 ffffc10000000000 - ffffc1ffffffffff (=40bits) hole
>  11 ffffc20000000000 - ffffe1ffffffffff (=45bits) vmalloc/ioremap space
>  12 ... unused hole ...
>  13 ffffffff80000000 - ffffffff82800000 (=40MB)   kernel text mapping,
> from phys 0
>  14 ... unused hole ...
>  15 ffffffff88000000 - fffffffffff00000 (=1919MB) module mapping space
>  16
>  17 The direct mapping covers all memory in the system upto the highest
>  18 memory address (this means in some cases it can also include PCI memory
>  19 holes)
>  20
>  21 vmalloc space is lazily synchronized into the different PML4 pages of
>  22 the processes using the page fault handler, with init_level4_pgt as
>  23 reference.
>  24
>  25 Current X86-64 implementations only support 40 bit of address space,
>  26 but we support upto 46bits. This expands into MBZ space in the page tables.
>  27
>  28 -Andi Kleen, Jul 2004
>
> I urgently want to know the answer.

We can't give you the answer unless you give us the question, and enough
context that the question makes sense.  I recommend looking up the AMD and
possibly the intel architecture documents on x86_64.  They very completely
cover what the processors can do and are freely available online.

Eric

Re: x86_64 virtual memory map

[Bob Zhang]

thanks in advanced!

I searched this question , so I directly reply to this mail.

1, for x86_64 machine ,for example ,HP's proliant machine.
from the kernel point of view , what is the max physical ram size ?
if I install 2TB physical memory(I mean physical RAM) , can linux
kernel recognize 2TB physical memory ?  from mm.txt , it seems that
physical ram size should be 64TB.

2,
from mm.txt :(http://lxr.linux.no/linux+v2.6.39/Documentation/x86/x86_64/mm.txt
from latest 2.6.39.

I am a little confused .
above said that :
ffff880000000000 - ffffc7ffffffffff (=64 TB) direct mapping of all phys. memory
10ffffc80000000000 - ffffc8ffffffffff (=40 bits) hole
11ffffc90000000000 - ffffe8ffffffffff (=45 bits) vmalloc/ioremap space  :32TB
ffffea0000000000 - ffffeaffffffffff (=40 bits) virtual memory map (1TB)
from this , we can know ,we can used max 64TB physical memory(physical
RAM).  vmalloc/ioremap space :32TB . but why tell us 1TB virtual
memory map ?  if the address that  vmalloc allocated are the address
that we can use , why say like this ? I am puzzled.
for example ,for 32bit kernel .vmalloc() range

and in the end of mm.txt , it says :
>>Current X86-64 implementations only support 40 bits of address space,
 >> 27but we support up to 46 bits. This expands into MBZ space in the
>>page tables.
so this 40bit(1TB) is responding to virtual memory map(1TB) ?

3,  if I installed 64TB physical memory (RAM) , but linux kernel only
support 1TB address space , what should I do next ?

4 , it seems that MBZ can support up to 256TB ,but who can tell how to
expand address space by MBZ ?
thanks very much!

-Bob

Re: x86_64 virtual memory map

[Bob Zhang]

I have HP's latest Proliant machine , The test involves using
configurations of 1 TB of Ram then 2TB of RAM. The system ROM is
reporting the proper amount of installed memory, however, Linux is
only reporting 992GB with both 1TB and 2 TB of memory in the system.
The customer has made the changes in the RMSU from 40 to 44 bit
addressing when using the 2TB of memory but this does not change
anything.
I am using SLES 11 SP1 (linux kernel is 2.6.32)
Does anyone have any idea how to get Linux to see the full amount of memory?

Re: x86_64 virtual memory map

[Eric W. Biederman]

Bob Zhang <zhanglinbao@gmail.com> writes:

> I have HP's latest Proliant machine , The test involves using
> configurations of 1 TB of Ram then 2TB of RAM. The system ROM is
> reporting the proper amount of installed memory, however, Linux is
> only reporting 992GB with both 1TB and 2 TB of memory in the system.
> The customer has made the changes in the RMSU from 40 to 44 bit
> addressing when using the 2TB of memory but this does not change
> anything.
> I am using SLES 11 SP1 (linux kernel is 2.6.32)
> Does anyone have any idea how to get Linux to see the full amount of
> memory?

What does the e820 map printed by the kernel at the start of time
report?  In particular is all of memory visible there?

Eric

Re: x86_64 virtual memory map

[Bob Zhang]

who can answer my first email of questions ?thanks .

Good idea , yes ,yes ,BIOS probes the real physical RAM size ,and
linux kernel will get this by e820 BIOS call(INT 0x15) .we can check
e820 output to judge whether firmware 's problem or not.

 I will grab e820 output information from dmesg .now ,I can't access
that machine , so wait .thanks .

Re: x86_64 virtual memory map

[Valdis.Kletnieks]

[-- Attachment #1: Type: text/plain, Size: 319 bytes --]

On Mon, 25 Jul 2011 18:13:16 +0800, Bob Zhang said:

> 3,  if I installed 64TB physical memory (RAM) , but linux kernel only
> support 1TB address space , what should I do next ?

Prepare to explain to your boss why you blew the project budget by buying 63T
too much RAM without checking first if it was supported? :)


[-- Attachment #2: Type: application/pgp-signature, Size: 227 bytes --]

Re: x86_64 virtual memory map

[Andi Kleen]

> and in the end of mm.txt , it says :
> >>Current X86-64 implementations only support 40 bits of address space,
>  >> 27but we support up to 46 bits. This expands into MBZ space in the
> >>page tables.
> so this 40bit(1TB) is responding to virtual memory map(1TB) ?

The 1TB limit was dropped a long time ago.
Your googling found an old file.

> 3,  if I installed 64TB physical memory (RAM) , but linux kernel only

What system is that out of curiosity? Are you sure it's not
a cluster?

> support 1TB address space , what should I do next ?

There's no known 1TB limit currently.

However at least for Intel Xeons there is no CPU which supports
more than 16TB physical address space.

Are you sure you have your units correct?

The theoretical x86 kernel limits are higher and it should upto support
64TB. However since that's untested territory and there 
have been bugs in large memory support in the past
you may run into problems.

Other architectures like IA64 will support more (but you
may be also in untested territory)

I believe 4+TB x86 systems have been tested.

-Andi

Re: x86_64 virtual memory map

[Bob Zhang]

2011/7/27 Andi Kleen <ak@linux.intel.com>:
>> and in the end of mm.txt , it says :
>> >>Current X86-64 implementations only support 40 bits of address space,
>>  >> 27but we support up to 46 bits. This expands into MBZ space in the
>> >>page tables.
>> so this 40bit(1TB) is responding to virtual memory map(1TB) ?
>
> The 1TB limit was dropped a long time ago.
> Your googling found an old file.

I am not googling this , I get this file from lxr . I get that file
from kernel v3.0 again .
http://lxr.linux.no/linux+v3.0/Documentation/x86/x86_64/mm.txt
it still said 1TB limit, if this number is really obsolete , it seems
that related people should update it to correct number.thanks.

>
>> 3,  if I installed 64TB physical memory (RAM) , but linux kernel only
>
> What system is that out of curiosity? Are you sure it's not
> a cluster?
This system is HP's latest Proliant machine , DL980G7 .
I have verified with firmware guys , they have tested sles11sp1 on
DL980G7 ,2TB ,4TB ,they tested successfully. not BIOS problem. it
seems that customer's configuration may has some issues. we don't
think this is linux kernel issue again. thanks for help.

>
>> support 1TB address space , what should I do next ?
> There's no known 1TB limit currently.
so as you said below ,it should be 64TB, but no one test with 64TB really.

> However at least for Intel Xeons there is no CPU which supports
> more than 16TB physical address space.
This information is very important, I will check System spec for details.
which spec can get this information ? x86_64 ,which spec ?



>
> Are you sure you have your units correct?
>
hoho , I am not richer , I don't have 64TB installed, I just supposed that.
in fact, my DL980G7 just have 2 TB installed , but linux kernel
recognized only  992GB. as I said above , firmware guys said they have
tested 4TB on DL980G7 with sles11sp1 , so we are investigating why our
environment will happen this issue . wait for future reply.


another question:
For BIOS ,BIOS has absolutely supported 4TB+ ?
Does BIOS theoretically support 64TB ?

so I basicly understand what you mean , you mean intel Xeon just
support 16TB , linux kernel should have no issue to support this 16TB
?


thanks very much! If I have some update info , I will reply in time.

-Bob

Re: x86_64 virtual memory map

[Andi Kleen]

On Wed, Jul 27, 2011 at 03:13:19PM +0800, Bob Zhang wrote:
> 2011/7/27 Andi Kleen <ak@linux.intel.com>:
> >> and in the end of mm.txt , it says :
> >> >>Current X86-64 implementations only support 40 bits of address space,
> >>  >> 27but we support up to 46 bits. This expands into MBZ space in the
> >> >>page tables.
> >> so this 40bit(1TB) is responding to virtual memory map(1TB) ?
> >
> > The 1TB limit was dropped a long time ago.
> > Your googling found an old file.
> 
> I am not googling this , I get this file from lxr . I get that file
> from kernel v3.0 again .
> http://lxr.linux.no/linux+v3.0/Documentation/x86/x86_64/mm.txt
> it still said 1TB limit, if this number is really obsolete , it seems
> that related people should update it to correct number.thanks.

Ah that virtual memory map just revers to a specific kernel data structure,
it's not a limit you need to be concerned about.

The max user mapping is 128TB, the direct mapping (physical limit)
64TB at the moment.


> 
> >
> >> 3,  if I installed 64TB physical memory (RAM) , but linux kernel only
> >
> > What system is that out of curiosity? Are you sure it's not
> > a cluster?
> This system is HP's latest Proliant machine , DL980G7 .
> I have verified with firmware guys , they have tested sles11sp1 on
> DL980G7 ,2TB ,4TB ,they tested successfully. not BIOS problem. it
> seems that customer's configuration may has some issues. we don't
> think this is linux kernel issue again. thanks for help.

I don't think 64TB are really supported by this machine.

Xeon E7-7xxx has a limit of 16TB physical address space.

The HP web site says it's capable of 4TB.

> 
> >
> >> support 1TB address space , what should I do next ?
> > There's no known 1TB limit currently.
> so as you said below ,it should be 64TB, but no one test with 64TB really.
> 
> > However at least for Intel Xeons there is no CPU which supports
> > more than 16TB physical address space.
> This information is very important, I will check System spec for details.
> which spec can get this information ? x86_64 ,which spec ?

It's probably somewhere in the datasheet, or when you boot the system
grep "address sizes" /proc/cpuinfo 
reports the limits reported by the CPU.


> 
> 
> 
> >
> > Are you sure you have your units correct?
> >
> hoho , I am not richer , I don't have 64TB installed, I just supposed that.
> in fact, my DL980G7 just have 2 TB installed , but linux kernel
> recognized only  992GB. as I said above , firmware guys said they have
> tested 4TB on DL980G7 with sles11sp1 , so we are investigating why our
> environment will happen this issue . wait for future reply.


As someone earlier suggested check the beginning of the kernel log
for the e820 map.  This tells how much memory the BIOS reported
to the OS.


> so I basicly understand what you mean , you mean intel Xeon just
> support 16TB , linux kernel should have no issue to support this 16TB
> ?

I don't know if especially 16TB has been tested. At least there's 
no theoretical limit that should prevent it.

-Andi
-- 
ak@linux.intel.com -- Speaking for myself only

Re: x86_64 virtual memory map

[Bob Zhang]

in fact , there are two factors to affect how much DRAM final user can use .

for example , for HP's this kind of machine , BIOS can support 40-bit
size and 44 -bit size.
if in BIOS ,user select 44-bit size , then every CPU(socket) can
support 512GB , if user install 8 CPU(socket) , it totally support
4TB.
if in BIOS ,user select 40-bit size , then every CPU(socket) only can
support 128GB , so if user install 8s(socket) , it totally only can
support 1TB, if 4s, only support totally 512GB.

certainly ,this is also decided by OS support 44-bit size or not. if
OS,i.e. rhel5.x  , only support 40bit ,though BIOS 44-bit selected ,
only 40-bit size support every CPU(socket).