kernel page table mapping for >1GB <3 GB for x86 arch without PAE

kernel page table mapping for >1GB <3 GB for x86 arch without PAE

[joe Shmoe]

Assuming I have 3GB RAM, why does not kernel create page tables in a way that all of this 3GB physical memory can be addressed . (for x86 arch 32 bit with PAE disabled )

I understand the 3GB/1GB split for user/kernel address space.Isn't  this just a matter of setting up the page table entries in such a way that so all the available RAM upto 3GB can mapped in the kernel page tables. In this way CPU's MMU can take care of mapping any virtual address within first 4GB to physical address using the page table entries regardless of a process is in kernel/user mode. 

Why can't some of first 768 page directory entries ( + 1024 page table 
entries for each PDE) be used to map btw 1GB and 3GB. What is stopping the kernel from doing this?

Instead why does kernel uses high memory mapping with zone model for 
addressing physical memory above 1GB?

Can anyone please shed some light into this,


Thanks,


      

Re: kernel page table mapping for >1GB <3 GB for x86 arch without PAE

[Roland Dreier]

 > Assuming I have 3GB RAM, why does not kernel create page tables in a way that all of this 3GB physical memory can be addressed . (for x86 arch 32 bit with PAE disabled )
 > 
 > I understand the 3GB/1GB split for user/kernel address space.Isn't  this just a matter of setting up the page table entries in such a way that so all the available RAM upto 3GB can mapped in the kernel page tables. In this way CPU's MMU can take care of mapping any virtual address within first 4GB to physical address using the page table entries regardless of a process is in kernel/user mode. 

On a 32-bit architecture, there is only 4GB of address space available.
If the kernel uses 3GB of address space for a direct mapping of physical
memory, then only 1GB is available for everything else (userspace,
vmalloc, etc).  BTW, this is available by enabling CONFIG_VMSPLIT_1G.

 - R.

Re: kernel page table mapping for >1GB <3 GB for x86 arch without PAE

[joe Shmoe]

Thanks for the reply,
But my question is with regard to setting up the page tables during the kernel initialization in such a way so that entire RAM upto 4GB can be mapped. i.e virtual to physical address mapping. My question is strictly with reg. to physical address mapping only.

Why does kernel stop at 1GB (896 MB to be exact) while mapping from linear to physical address. 

Since a given process runs in both user/kernel mode how does it matter whether page table mapping  is for entire 4GB or 1 GB. Just because kernel setup the page tables does not mean that allocated dynamic memory cannot be accessed by given process while running in user mode. 


For the sake of discussion, let us say I have 3.5 GB of RAM of x86 with PAE disabled and I have only one process running other than kernel.
Why can't kernel simply setup the page table mapping in a way that entire 3.5GB is mapped from linear to physical during the initialization. Assuming this single process is given access upto 3 GB, while running in user mode this single process can access any memory location in the allocated 2 or 3 GB virtual space.






--- On Fri, 8/28/09, Roland Dreier <rdreier@cisco.com> wrote:

> From: Roland Dreier <rdreier@cisco.com>
> Subject: Re: kernel page table mapping for >1GB <3 GB for x86 arch without PAE
> To: "joe Shmoe" <jsmoe3@yahoo.com>
> Cc: linux-kernel@vger.kernel.org
> Date: Friday, August 28, 2009, 2:12 PM
> 
>  > Assuming I have 3GB RAM, why does not kernel create
> page tables in a way that all of this 3GB physical memory
> can be addressed . (for x86 arch 32 bit with PAE disabled )
>  > 
>  > I understand the 3GB/1GB split for user/kernel
> address space.Isn't  this just a matter of setting up the
> page table entries in such a way that so all the available
> RAM upto 3GB can mapped in the kernel page tables. In this
> way CPU's MMU can take care of mapping any virtual address
> within first 4GB to physical address using the page table
> entries regardless of a process is in kernel/user mode. 
> 
> On a 32-bit architecture, there is only 4GB of address
> space available.
> If the kernel uses 3GB of address space for a direct
> mapping of physical
> memory, then only 1GB is available for everything else
> (userspace,
> vmalloc, etc).  BTW, this is available by enabling
> CONFIG_VMSPLIT_1G.
> 
>  - R.
> 


      

Re: kernel page table mapping for >1GB <3 GB for x86 arch without PAE

[Valdis.Kletnieks]

[-- Attachment #1: Type: text/plain, Size: 444 bytes --]

On Fri, 28 Aug 2009 11:40:56 PDT, joe Shmoe said:

> For the sake of discussion, let us say I have 3.5 GB of RAM of x86 with PAE
> disabled and I have only one process running other than kernel.

Oh, so an embedded environment. Sure, you could play games with that.

But most Linux systems have more processes running than just 'init'. And
in that case, what do you do with your memory map?  It gets a bit more
complicated then....


[-- Attachment #2: Type: application/pgp-signature, Size: 226 bytes --]

Re: kernel page table mapping for >1GB <3 GB for x86 arch without PAE

[joe Shmoe]

No I am not talking about embedded systems. It is just an example.
My question is what is preventing the kernel from setting up the page tables so that entire RAM can be mapped upto 4 GB.

page tables have to be setup for every process anyway with first 768 PDE entries being unique and remaining 256 entries being same for all the processes.


--- On Fri, 8/28/09, Valdis.Kletnieks@vt.edu <Valdis.Kletnieks@vt.edu> wrote:

> From: Valdis.Kletnieks@vt.edu <Valdis.Kletnieks@vt.edu>
> Subject: Re: kernel page table mapping for >1GB <3 GB for x86 arch without PAE
> To: "joe Shmoe" <jsmoe3@yahoo.com>
> Cc: "Roland Dreier" <rdreier@cisco.com>, linux-kernel@vger.kernel.org
> Date: Friday, August 28, 2009, 3:13 PM
> On Fri, 28 Aug 2009 11:40:56 PDT, joe
> Shmoe said:
> 
> > For the sake of discussion, let us say I have 3.5 GB
> of RAM of x86 with PAE
> > disabled and I have only one process running other
> than kernel.
> 
> Oh, so an embedded environment. Sure, you could play games
> with that.
> 
> But most Linux systems have more processes running than
> just 'init'. And
> in that case, what do you do with your memory map?  It
> gets a bit more
> complicated then....
> 
> 


      

Re: kernel page table mapping for >1GB <3 GB for x86 arch without PAE

[Alan Cox]

On Fri, 28 Aug 2009 12:44:25 -0700 (PDT)
joe Shmoe <jsmoe3@yahoo.com> wrote:

> No I am not talking about embedded systems. It is just an example.
> My question is what is preventing the kernel from setting up the page tables so that entire RAM can be mapped upto 4 GB.

The design of the x86 processor

You only have 4GB of virtual address space at any one time so you have to
fit the following within your 4GB virtual space

- Virtual addresses of the user space
- Mappings for physical memory
- Kernel virtual spaces (vmalloc etc)
- Various miscellaneous kernel mappings
- I/O MMIO space

Now you could in theory switch the page tables on kernel entry to get a
very big user space mapping and then instead of relying on the user space
being mapped you could map specific user pages or access them via their
physical address lookup.

That however means reloading %cr3 (page table base) which is very
expensive, and some very hard to implement locking on user space access.

Some other processor families have the notion of "spaces" or MMU
bypassing. The Sparc64 for example can access MMIO space without an MMU
mapping or virtual address being needed for it. x86 does not - it has
segments but the segments map into the 4GB space so they don't help here
hence the current implementation and 3G/1G split.

Re: kernel page table mapping for >1GB <3 GB for x86 arch without PAE

[joe Shmoe]

>That however means reloading %cr3 (page table base) which is very
>expensive, and some very hard to implement locking on user space access

I understand the implications of reloading CR3. But once the page tables are  setup to map all the available physical RAM to virtual (linear) address it could be for eg. 1, 2, 3 or 4 GB how does it matter.

Now If a process wants to access a memory location which is above 1GB , since kernel already setup the page tables CPU MMU will do the translation since it consults this page table setup by kernel. Of course it also makes sure the page table has appropriate privilege level for access.

Am I missing something here?

 



--- On Fri, 8/28/09, Alan Cox <alan@lxorguk.ukuu.org.uk> wrote:

> From: Alan Cox <alan@lxorguk.ukuu.org.uk>
> Subject: Re: kernel page table mapping for >1GB <3 GB for x86 arch without PAE
> To: "joe Shmoe" <jsmoe3@yahoo.com>
> Cc: Valdis.Kletnieks@vt.edu, linux-kernel@vger.kernel.org
> Date: Friday, August 28, 2009, 3:53 PM
> On Fri, 28 Aug 2009 12:44:25 -0700
> (PDT)
> joe Shmoe <jsmoe3@yahoo.com>
> wrote:
> 
> > No I am not talking about embedded systems. It is just
> an example.
> > My question is what is preventing the kernel from
> setting up the page tables so that entire RAM can be mapped
> upto 4 GB.
> 
> The design of the x86 processor
> 
> You only have 4GB of virtual address space at any one time
> so you have to
> fit the following within your 4GB virtual space
> 
> - Virtual addresses of the user space
> - Mappings for physical memory
> - Kernel virtual spaces (vmalloc etc)
> - Various miscellaneous kernel mappings
> - I/O MMIO space
> 
> Now you could in theory switch the page tables on kernel
> entry to get a
> very big user space mapping and then instead of relying on
> the user space
> being mapped you could map specific user pages or access
> them via their
> physical address lookup.
> 
> That however means reloading %cr3 (page table base) which
> is very
> expensive, and some very hard to implement locking on user
> space access.
> 
> Some other processor families have the notion of "spaces"
> or MMU
> bypassing. The Sparc64 for example can access MMIO space
> without an MMU
> mapping or virtual address being needed for it. x86 does
> not - it has
> segments but the segments map into the 4GB space so they
> don't help here
> hence the current implementation and 3G/1G split.
> 


      

Re: kernel page table mapping for >1GB <3 GB for x86 arch without PAE

[Alan Cox]

O> I understand the implications of reloading CR3. But once the page tables are  setup to map all the available physical RAM to virtual (linear) address it could be for eg. 1, 2, 3 or 4 GB how does it matter.

Where are you putting the user virtual addresses. User addresses don't
map direct to physical addresses so you need both sets of translations at
once

Right now you have

0-3GB		MMU translations to arbitary basically random
physical addresses (with many pages shared and some absent)

3GB+		Physical mapping only accessible in kernel mode

If user applications ran with a 1:1 mapping of application space to
physical addresses you would be fine - but they don't and it would be
rather hard to run like that because you want page sharing, lazy unshare,
vfork etc to all work.

Re: kernel page table mapping for >1GB <3 GB for x86 arch without PAE

[joe Shmoe]

Thanks for the reply Alan,
What I am looking for is this: Let us keep aside the virtual addresses for a process aside for a second. My question does not relate to that at all.

why does kernel map only 896MB of RAM to linear addresses in kernel page tables at the startup even if RAM size is more than that say 3.5GB of RAM
Why does it use dynamic memory mapping technique using zones for mapping the remaining 2.5 GB of RAM( 3.5GB - 1GB = 2.5GB)

Why not it does the following?
kernel sets up the page tables during initialization phase such that it maps all the available physical RAM i.e 3.5GB to linear addresses.
what is wrong with this approach. 




--- On Fri, 8/28/09, Alan Cox <alan@lxorguk.ukuu.org.uk> wrote:

> From: Alan Cox <alan@lxorguk.ukuu.org.uk>
> Subject: Re: kernel page table mapping for >1GB <3 GB for x86 arch without PAE
> To: "joe Shmoe" <jsmoe3@yahoo.com>
> Cc: linux-kernel@vger.kernel.org
> Date: Friday, August 28, 2009, 6:16 PM
> O> I understand the implications
> of reloading CR3. But once the page tables are  setup
> to map all the available physical RAM to virtual (linear)
> address it could be for eg. 1, 2, 3 or 4 GB how does it
> matter.
> 
> Where are you putting the user virtual addresses. User
> addresses don't
> map direct to physical addresses so you need both sets of
> translations at
> once
> 
> Right now you have
> 
> 0-3GB        MMU translations
> to arbitary basically random
> physical addresses (with many pages shared and some
> absent)
> 
> 3GB+        Physical mapping
> only accessible in kernel mode
> 
> If user applications ran with a 1:1 mapping of application
> space to
> physical addresses you would be fine - but they don't and
> it would be
> rather hard to run like that because you want page sharing,
> lazy unshare,
> vfork etc to all work.
> 


      

Re: kernel page table mapping for >1GB <3 GB for x86 arch without PAE

[Alan Cox]

On Fri, 28 Aug 2009 17:20:31 -0700 (PDT)
joe Shmoe <jsmoe3@yahoo.com> wrote:

> Thanks for the reply Alan,
> What I am looking for is this: Let us keep aside the virtual addresses for a process aside for a second. My question does not relate to that at all.

When you understand why that first statement is wrong you will understand
the answer.

Alan

Re: kernel page table mapping for >1GB <3 GB for x86 arch without PAE

[joe Shmoe]

Hey Alan,
May be I confused you. Sorry

> 0-3GB        MMU translations
> to arbitary basically random
> physical addresses (with many pages shared and some
> absent)

Can those "arbitary basically random physical addresses" fall outside 
1GB-3GB is my question. Why does kernel stop at 896MB during page table setup. Why not go ahead and map the remaining available RAM (where RAM size is > 1GB < 4GB)
But kernel uses different technique for addressing the high memory > 896MB

Not sure this is being done so that it will be easier for kernel to maintain page table assignments/swaping to various processes.
 
> 3GB+        Physical mapping
> only accessible in kernel mode

Yes. This part I understand obviously these addresses are issued only when a process is running in kernel mode.


--- On Fri, 8/28/09, Alan Cox <alan@lxorguk.ukuu.org.uk> wrote:

> From: Alan Cox <alan@lxorguk.ukuu.org.uk>
> Subject: Re: kernel page table mapping for >1GB <3 GB for x86 arch without PAE
> To: "joe Shmoe" <jsmoe3@yahoo.com>
> Cc: linux-kernel@vger.kernel.org
> Date: Friday, August 28, 2009, 6:16 PM
> O> I understand the implications
> of reloading CR3. But once the page tables are  setup
> to map all the available physical RAM to virtual (linear)
> address it could be for eg. 1, 2, 3 or 4 GB how does it
> matter.
> 
> Where are you putting the user virtual addresses. User
> addresses don't
> map direct to physical addresses so you need both sets of
> translations at
> once
> 
> Right now you have
> 
> 0-3GB        MMU translations
> to arbitary basically random
> physical addresses (with many pages shared and some
> absent)
> 
> 3GB+        Physical mapping
> only accessible in kernel mode
> 
> If user applications ran with a 1:1 mapping of application
> space to
> physical addresses you would be fine - but they don't and
> it would be
> rather hard to run like that because you want page sharing,
> lazy unshare,
> vfork etc to all work.
> 


      

Re: kernel page table mapping for >1GB <3 GB for x86 arch without PAE

[joe Shmoe]

doesn't CR3 gets reloaded for process switch? If so , how does it matter if you setup page table mapping for all the available RAM instead of limiting to just 1GB

--- On Fri, 8/28/09, Alan Cox <alan@lxorguk.ukuu.org.uk> wrote:

> From: Alan Cox <alan@lxorguk.ukuu.org.uk>
> Subject: Re: kernel page table mapping for >1GB <3 GB for x86 arch without PAE
> To: "joe Shmoe" <jsmoe3@yahoo.com>
> Cc: Valdis.Kletnieks@vt.edu, linux-kernel@vger.kernel.org
> Date: Friday, August 28, 2009, 3:53 PM
> On Fri, 28 Aug 2009 12:44:25 -0700
> (PDT)
> joe Shmoe <jsmoe3@yahoo.com>
> wrote:
> 
> > No I am not talking about embedded systems. It is just
> an example.
> > My question is what is preventing the kernel from
> setting up the page tables so that entire RAM can be mapped
> upto 4 GB.
> 
> The design of the x86 processor
> 
> You only have 4GB of virtual address space at any one time
> so you have to
> fit the following within your 4GB virtual space
> 
> - Virtual addresses of the user space
> - Mappings for physical memory
> - Kernel virtual spaces (vmalloc etc)
> - Various miscellaneous kernel mappings
> - I/O MMIO space
> 
> Now you could in theory switch the page tables on kernel
> entry to get a
> very big user space mapping and then instead of relying on
> the user space
> being mapped you could map specific user pages or access
> them via their
> physical address lookup.
> 
> That however means reloading %cr3 (page table base) which
> is very
> expensive, and some very hard to implement locking on user
> space access.
> 
> Some other processor families have the notion of "spaces"
> or MMU
> bypassing. The Sparc64 for example can access MMIO space
> without an MMU
> mapping or virtual address being needed for it. x86 does
> not - it has
> segments but the segments map into the 4GB space so they
> don't help here
> hence the current implementation and 3G/1G split.
> 


      

Re: kernel page table mapping for >1GB <3 GB for x86 arch without PAE

[Alan Cox]

On Fri, 28 Aug 2009 14:07:01 -0700 (PDT)
joe Shmoe <jsmoe3@yahoo.com> wrote:

> doesn't CR3 gets reloaded for process switch? If so , how does it matter if you setup page table mapping for all the available RAM instead of limiting to just 1GB

Because

- You don't switch process every syscall
- You don't switch process every interrupt

The latter is the real killer.

Re: kernel page table mapping for >1GB <3 GB for x86 arch without PAE

[Roland Dreier]

 > But my question is with regard to setting up the page tables during the kernel initialization in such a way so that entire RAM upto 4GB can be mapped. i.e virtual to physical address mapping. My question is strictly with reg. to physical address mapping only.
 > 
 > Why does kernel stop at 1GB (896 MB to be exact) while mapping from linear to physical address. 

Because it wants to be able to give userspace 3GB of address space.  And
userspace mapping cannot be a linear mapping to physical memory -- it
must be real virtual memory that has proper access permissions and can
handle mmap'ed files, shared libraries, etc etc.  So userspace cannot
share the kernel's linear mapping.