How to use memory over 4GB

How to use memory over 4GB

[Roberto Fichera]

Hi All,

I've a dual Xeon 3.2GHz HT with 8GB of memory running kernel 2.6.11.
I whould like to know the way how to use all the memory in a single
process, the application is a big simulation which needs big memory chuncks.
I have readed about hugetlbfs, shmfs and tmpfs, but don't understand how I 
can access
the whole memory. Ok! I can create a big file on tmpfs using shm_open() and
than map it by using mmap() or mmap2() but how can I access over 4GB using
standard pointers (if I had to use it)?

So, please send me some url, documents etc and/or piece
of source code that shows how to do it in user space.

Thanks _very much_ in advance,

Roberto Fichera. 

Re: How to use memory over 4GB

[Michael Tokarev]

Roberto Fichera wrote:
> Hi All,
> 
> I've a dual Xeon 3.2GHz HT with 8GB of memory running kernel 2.6.11.
> I whould like to know the way how to use all the memory in a single
> process, the application is a big simulation which needs big memory
> chuncks.
> I have readed about hugetlbfs, shmfs and tmpfs, but don't understand how
> I can access
> the whole memory. Ok! I can create a big file on tmpfs using shm_open() and
> than map it by using mmap() or mmap2() but how can I access over 4GB using
> standard pointers (if I had to use it)?

There's no "standard" and simple way to utilize more than 4Gb memory on
i386 hardware, especially in a userspace.  That is, the size of a pointer
is 32bits, which is 4GB addresspace maximum.  i386 architecture just can't
have a pointer of greather size.

All "extra" (>4GB) space can be used like a file in a filesystem, not like
a plain memory.  Think of read()/write() (or pread()/pwrite() for that matter),
but much faster ones compared to disk-based storage -- in tmpfs.  You can
also mmap() *parts* of such a file, but will be still limited to 4GB at
once -- in order to have more, you will have to unmap() something.

All the "large applications" (most notable large database systems such as
Oracle) can't use more than 4GB memory directly, but can utilize it for
database cache.  In directly-addressible space there's a "table of content"
of cached buffers is keept, and when a buffer is needed, it is mmap()'ed
into the application's address space, and unmapped right away when it isn't
needed anymore (but it is still in memory).  Ofcourse you can't have
usual pointers into that memory, but you can use something like
(block-number,offset) instead of a pointer (pagetables).

/mjt

Re: How to use memory over 4GB

[Roberto Fichera]

At 14.56 16/05/2005, Michael Tokarev wrote:

>Roberto Fichera wrote:
> > Hi All,
> >
> > I've a dual Xeon 3.2GHz HT with 8GB of memory running kernel 2.6.11.
> > I whould like to know the way how to use all the memory in a single
> > process, the application is a big simulation which needs big memory
> > chuncks.
> > I have readed about hugetlbfs, shmfs and tmpfs, but don't understand how
> > I can access
> > the whole memory. Ok! I can create a big file on tmpfs using shm_open() and
> > than map it by using mmap() or mmap2() but how can I access over 4GB using
> > standard pointers (if I had to use it)?
>
>There's no "standard" and simple way to utilize more than 4Gb memory on
>i386 hardware, especially in a userspace.  That is, the size of a pointer
>is 32bits, which is 4GB addresspace maximum.  i386 architecture just can't
>have a pointer of greather size.
>
>All "extra" (>4GB) space can be used like a file in a filesystem, not like
>a plain memory.  Think of read()/write() (or pread()/pwrite() for that 
>matter),
>but much faster ones compared to disk-based storage -- in tmpfs.  You can
>also mmap() *parts* of such a file, but will be still limited to 4GB at
>once -- in order to have more, you will have to unmap() something.
>
>All the "large applications" (most notable large database systems such as
>Oracle) can't use more than 4GB memory directly, but can utilize it for
>database cache.  In directly-addressible space there's a "table of content"
>of cached buffers is keept, and when a buffer is needed, it is mmap()'ed
>into the application's address space, and unmapped right away when it isn't
>needed anymore (but it is still in memory).

I was thinking to create as many object I need by the usual shm_open(), 
than mmap()
it on a process until I get ENOMEM. So, when I get a ENOMEM I start to
munmap() objects in order to free some user space memory and create the needed
space to complete mmap() for the requested object.

>Ofcourse you can't have
>usual pointers into that memory, but you can use something like
>(block-number,offset) instead of a pointer (pagetables).

Could you explain better this approch?


>/mjt
>-
>To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
>the body of a message to majordomo@vger.kernel.org
>More majordomo info at  http://vger.kernel.org/majordomo-info.html
>Please read the FAQ at  http://www.tux.org/lkml/

Roberto Fichera. 

Re: How to use memory over 4GB

[William Lee Irwin III]

On Mon, May 16, 2005 at 03:16:28PM +0200, Roberto Fichera wrote:
> I was thinking to create as many object I need by the usual shm_open(), 
> than mmap() it on a process until I get ENOMEM. So, when I get a
> ENOMEM I start to munmap() objects in order to free some user space
> memory and create the needed space to complete mmap() for the
> requested object.

This approach has already been used in production by various major
applications and is even obsolete, now replaced by remap_file_pages()
(in Linux), where it and its counterparts in other operating systems
have been in use in production by various major applications for some time.

remap_file_pages() allows virtual pages in an mmap() area to correspond
in an unrestricted fashion to the pages of the underlying file, and to
alter this correspondence at will.

In particular, Oracle's "vlm" option does this.


-- wli

Re: How to use memory over 4GB

[Roberto Fichera]

At 17.10 16/05/2005, William Lee Irwin III wrote:

>On Mon, May 16, 2005 at 03:16:28PM +0200, Roberto Fichera wrote:
> > I was thinking to create as many object I need by the usual shm_open(),
> > than mmap() it on a process until I get ENOMEM. So, when I get a
> > ENOMEM I start to munmap() objects in order to free some user space
> > memory and create the needed space to complete mmap() for the
> > requested object.
>
>This approach has already been used in production by various major
>applications and is even obsolete, now replaced by remap_file_pages()
>(in Linux), where it and its counterparts in other operating systems
>have been in use in production by various major applications for some time.
>
>remap_file_pages() allows virtual pages in an mmap() area to correspond
>in an unrestricted fashion to the pages of the underlying file, and to
>alter this correspondence at will.
>
>In particular, Oracle's "vlm" option does this.

So, you are suggesting to create one big tmpfs area, 6GB for example, than 
mmap() it
to the user process and use the remap_file_pages() for all the objects I 
want make
"addressable" on the user process taking care the return value of -1 which 
implies
to munmap() something to free vm space?



>-- wli

Roberto Fichera. 

Re: How to use memory over 4GB

[William Lee Irwin III]

At 17.10 16/05/2005, William Lee Irwin III wrote:
>> This approach has already been used in production by various major
>> applications and is even obsolete, now replaced by remap_file_pages()
>> (in Linux), where it and its counterparts in other operating systems
>> have been in use in production by various major applications for some time.
>> remap_file_pages() allows virtual pages in an mmap() area to correspond
>> in an unrestricted fashion to the pages of the underlying file, and to
>> alter this correspondence at will.
>> In particular, Oracle's "vlm" option does this.

On Mon, May 16, 2005 at 05:47:31PM +0200, Roberto Fichera wrote:
> So, you are suggesting to create one big tmpfs area, 6GB for example, than 
> mmap() it to the user process and use the remap_file_pages() for all
> the objects I want make "addressable" on the user process taking care
> the return value of -1 which implies to munmap() something to free vm space?

I don't have any specific suggestion regarding layout or usage patterns
besides pointing to remap_file_pages() being significantly lighter-weight
than mmap() for the purposes of virtual windowing. The other aspects of
all this (and even the use of remap_file_pages() at all) are, of course,
at your own discretion. It is, however, notable that Oracle has had some
success with a tactic similar to what you describe, where few objects are
used and the application instead manages space within the objects
dedicated to various purposes.

In general, I'd recommend experimenting with different strategies to see
which works best for you. This is all rather vague, and the mechanics of
getting all this working with your application are sure to have enough
alternative implementations to merit some decision-making and the like.


-- wli

Re: How to use memory over 4GB

[Roberto Fichera]

At 17.54 16/05/2005, William Lee Irwin III wrote:
>At 17.10 16/05/2005, William Lee Irwin III wrote:
> >> This approach has already been used in production by various major
> >> applications and is even obsolete, now replaced by remap_file_pages()
> >> (in Linux), where it and its counterparts in other operating systems
> >> have been in use in production by various major applications for some 
> time.
> >> remap_file_pages() allows virtual pages in an mmap() area to correspond
> >> in an unrestricted fashion to the pages of the underlying file, and to
> >> alter this correspondence at will.
> >> In particular, Oracle's "vlm" option does this.
>
>On Mon, May 16, 2005 at 05:47:31PM +0200, Roberto Fichera wrote:
> > So, you are suggesting to create one big tmpfs area, 6GB for example, than
> > mmap() it to the user process and use the remap_file_pages() for all
> > the objects I want make "addressable" on the user process taking care
> > the return value of -1 which implies to munmap() something to free vm 
> space?
>
>I don't have any specific suggestion regarding layout or usage patterns
>besides pointing to remap_file_pages() being significantly lighter-weight
>than mmap() for the purposes of virtual windowing. The other aspects of
>all this (and even the use of remap_file_pages() at all) are, of course,
>at your own discretion. It is, however, notable that Oracle has had some
>success with a tactic similar to what you describe, where few objects are
>used and the application instead manages space within the objects
>dedicated to various purposes.
>
>In general, I'd recommend experimenting with different strategies to see
>which works best for you. This is all rather vague, and the mechanics of
>getting all this working with your application are sure to have enough
>alternative implementations to merit some decision-making and the like.

Thanks to pointing me to another solution, I will start to experimenting 
very soon :-)!



>-- wli
>-
>To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
>the body of a message to majordomo@vger.kernel.org
>More majordomo info at  http://vger.kernel.org/majordomo-info.html
>Please read the FAQ at  http://www.tux.org/lkml/

Roberto Fichera. 

Re: How to use memory over 4GB

[Bill Davidsen]

Michael Tokarev wrote:
> Roberto Fichera wrote:
> 
>>Hi All,
>>
>>I've a dual Xeon 3.2GHz HT with 8GB of memory running kernel 2.6.11.
>>I whould like to know the way how to use all the memory in a single
>>process, the application is a big simulation which needs big memory
>>chuncks.
>>I have readed about hugetlbfs, shmfs and tmpfs, but don't understand how
>>I can access
>>the whole memory. Ok! I can create a big file on tmpfs using shm_open() and
>>than map it by using mmap() or mmap2() but how can I access over 4GB using
>>standard pointers (if I had to use it)?
> 
> 
> There's no "standard" and simple way to utilize more than 4Gb memory on
> i386 hardware, especially in a userspace.  That is, the size of a pointer
> is 32bits, which is 4GB addresspace maximum.  i386 architecture just can't
> have a pointer of greather size.

And the original i286 couldn't address more than 64k, either. But of 
course it could and did, using the large model which used segment 
registers to address additional memory. In terms of hardware this 
certainly could be done using 4GB segments on newer processors. In terms 
of available compiler and O/S support, you're stuck with using tricks, 
as various people have noted.

In terms of generating correct code, with all the bad things people have 
said about the cost of doing segmentation, it was vastly more likely to 
be correct code if you could just pretend that you have linear address 
space and let the compiler/lib/os play let's pretend for you.

If you really need to address more than 4GB perhaps a 64 bit hardware is 
the better solution.
> 
> All "extra" (>4GB) space can be used like a file in a filesystem, not like
> a plain memory.  Think of read()/write() (or pread()/pwrite() for that matter),
> but much faster ones compared to disk-based storage -- in tmpfs.  You can
> also mmap() *parts* of such a file, but will be still limited to 4GB at
> once -- in order to have more, you will have to unmap() something.
> 
> All the "large applications" (most notable large database systems such as
> Oracle) can't use more than 4GB memory directly, but can utilize it for
> database cache.  In directly-addressible space there's a "table of content"
> of cached buffers is keept, and when a buffer is needed, it is mmap()'ed
> into the application's address space, and unmapped right away when it isn't
> needed anymore (but it is still in memory).  Ofcourse you can't have
> usual pointers into that memory, but you can use something like
> (block-number,offset) instead of a pointer (pagetables).
> 
> /mjt

Re: How to use memory over 4GB

[Eric Dumazet]

Roberto Fichera a écrit :
> Hi All,
> 
> I've a dual Xeon 3.2GHz HT with 8GB of memory running kernel 2.6.11.
> I whould like to know the way how to use all the memory in a single
> process, the application is a big simulation which needs big memory 
> chuncks.

AFAIK the best you can have with a 32bits processor, is 4GB for one process.

But still you need a 4GB/4GB user/kernel split, because the standard is 3GB/1GB.

Eric

Re: How to use memory over 4GB

[Roberto Fichera]

At 14.57 16/05/2005, Eric Dumazet wrote:

>Roberto Fichera a écrit :
>>Hi All,
>>I've a dual Xeon 3.2GHz HT with 8GB of memory running kernel 2.6.11.
>>I whould like to know the way how to use all the memory in a single
>>process, the application is a big simulation which needs big memory chuncks.
>
>AFAIK the best you can have with a 32bits processor, is 4GB for one process.

Yes! I know ;-)!


>But still you need a 4GB/4GB user/kernel split, because the standard is 
>3GB/1GB.

Why I need 4GB/4GB split? What are the beneficts?


>Eric
>-
>To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
>the body of a message to majordomo@vger.kernel.org
>More majordomo info at  http://vger.kernel.org/majordomo-info.html
>Please read the FAQ at  http://www.tux.org/lkml/

Roberto Fichera. 

Re: How to use memory over 4GB

[Eric Dumazet]

Roberto Fichera a écrit :

> 
>> But still you need a 4GB/4GB user/kernel split, because the standard 
>> is 3GB/1GB.
> 
> 
> Why I need 4GB/4GB split? What are the beneficts?

Well... 4GB for your process is better than 3GB, that's 33% more space...

If your process is cpu bounded (and not issuing too many system calls),
then 4GB/4GB split let it address more ram, reducing the need to shift windows in
mmaped files for example.

Eric

Re: How to use memory over 4GB

[Roberto Fichera]

At 16.17 16/05/2005, Eric Dumazet wrote:

>Roberto Fichera a écrit :
>
>>
>>>But still you need a 4GB/4GB user/kernel split, because the standard is 
>>>3GB/1GB.
>>
>>Why I need 4GB/4GB split? What are the beneficts?
>
>Well... 4GB for your process is better than 3GB, that's 33% more space...

Right! So, the 4GB/4GB split and tmpfs pair should be the best solution to
address as much memory possible on a single user process.

>If your process is cpu bounded (and not issuing too many system calls),
>then 4GB/4GB split let it address more ram, reducing the need to shift 
>windows in
>mmaped files for example.

... any source code that explain better what you say ;-)!


>Eric
>

Roberto Fichera. 

Re: How to use memory over 4GB

[Nix]

On 16 May 2005, Roberto Fichera whispered secretively:
> At 16.17 16/05/2005, Eric Dumazet wrote:
>>If your process is cpu bounded (and not issuing too many system calls),
>> then 4GB/4GB split let it address more ram, reducing the need to shift windows in
>>mmaped files for example.
> 
> ... any source code that explain better what you say ;-)!

<http://lkml.org/lkml/2003/7/8/246> perhaps?

(In a nutshell: it gives processes an extra 1Gb of virtual memory, at
the cost of making system calls --- and everything else that must
transition to kernel space --- *much* slower.)

Re: How to use memory over 4GB

[Roberto Fichera]

At 23.34 16/05/2005, Nix wrote:
>On 16 May 2005, Roberto Fichera whispered secretively:
> > At 16.17 16/05/2005, Eric Dumazet wrote:
> >>If your process is cpu bounded (and not issuing too many system calls),
> >> then 4GB/4GB split let it address more ram, reducing the need to shift 
> windows in
> >>mmaped files for example.
> >
> > ... any source code that explain better what you say ;-)!
>
><http://lkml.org/lkml/2003/7/8/246> perhaps?

Yes! I already know this patch ... I was asking for some user space code,
just to show up the thinks ;-)!


>(In a nutshell: it gives processes an extra 1Gb of virtual memory, at
>the cost of making system calls --- and everything else that must
>transition to kernel space --- *much* slower.)
>-
>To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
>the body of a message to majordomo@vger.kernel.org
>More majordomo info at  http://vger.kernel.org/majordomo-info.html
>Please read the FAQ at  http://www.tux.org/lkml/

Roberto Fichera. 

Re: How to use memory over 4GB

[William Lee Irwin III]

Roberto Fichera a ?crit :
>> Why I need 4GB/4GB split? What are the beneficts?

On Mon, May 16, 2005 at 04:17:14PM +0200, Eric Dumazet wrote:
> Well... 4GB for your process is better than 3GB, that's 33% more space...
> If your process is cpu bounded (and not issuing too many system calls),
> then 4GB/4GB split let it address more ram, reducing the need to shift 
> windows in mmaped files for example.

Most of the benefit of this can be recovered by organizing the user
address space for compactness without incurring the context switch
overhead of TLB reloads or TLB footprint of a separate kernel address
space (or other issues of a third-party patch). Linking with low
starting text addresses, stack below text, and manual placement of
mmap()'s all help to conserve user virtualspace. It's also useful
in tandem with such in order to extract the maximum benefit from
the additional virtualspace (as the costs above warrant recovery).


-- wli

Re: How to use memory over 4GB

[li nux]

I guess u r on 32-bit platform.
Then, u need to enable PAE ( if not enabled alredy)
and rebuild the kernel.
This will be transparent to the user space.
Ur word size will still be 32-bit, only thing is PAE
will change the way u translate the address by adding
one more level of indexing to the page table.

-lnxluv.

--- Roberto Fichera <kernel@tekno-soft.it> wrote:
> Hi All,
> 
> I've a dual Xeon 3.2GHz HT with 8GB of memory
> running kernel 2.6.11.
> I whould like to know the way how to use all the
> memory in a single
> process, the application is a big simulation which
> needs big memory chuncks.
> I have readed about hugetlbfs, shmfs and tmpfs, but
> don't understand how I 
> can access
> the whole memory. Ok! I can create a big file on
> tmpfs using shm_open() and
> than map it by using mmap() or mmap2() but how can I
> access over 4GB using
> standard pointers (if I had to use it)?
> 
> So, please send me some url, documents etc and/or
> piece
> of source code that shows how to do it in user
> space.
> 
> Thanks _very much_ in advance,
> 
> Roberto Fichera. 
> 
> -
> To unsubscribe from this list: send the line
> "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at 
> http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/
> 


		
__________________________________ 
Yahoo! Mail Mobile 
Take Yahoo! Mail with you! Check email on your mobile phone. 
http://mobile.yahoo.com/learn/mail 

Re: How to use memory over 4GB

[Roberto Fichera]

At 14.55 16/05/2005, li nux wrote:

>I guess u r on 32-bit platform.
>Then, u need to enable PAE ( if not enabled alredy)
>and rebuild the kernel.
>This will be transparent to the user space.
>Ur word size will still be 32-bit, only thing is PAE
>will change the way u translate the address by adding
>one more level of indexing to the page table.

Yes! It's already PAE enabled.


>-lnxluv.
>
>--- Roberto Fichera <kernel@tekno-soft.it> wrote:
> > Hi All,
> >
> > I've a dual Xeon 3.2GHz HT with 8GB of memory
> > running kernel 2.6.11.
> > I whould like to know the way how to use all the
> > memory in a single
> > process, the application is a big simulation which
> > needs big memory chuncks.
> > I have readed about hugetlbfs, shmfs and tmpfs, but
> > don't understand how I
> > can access
> > the whole memory. Ok! I can create a big file on
> > tmpfs using shm_open() and
> > than map it by using mmap() or mmap2() but how can I
> > access over 4GB using
> > standard pointers (if I had to use it)?
> >
> > So, please send me some url, documents etc and/or
> > piece
> > of source code that shows how to do it in user
> > space.
> >
> > Thanks _very much_ in advance,
> >
> > Roberto Fichera.
> >
> > -
> > To unsubscribe from this list: send the line
> > "unsubscribe linux-kernel" in
> > the body of a message to majordomo@vger.kernel.org
> > More majordomo info at
> > http://vger.kernel.org/majordomo-info.html
> > Please read the FAQ at  http://www.tux.org/lkml/
> >
>
>
>
>__________________________________
>Yahoo! Mail Mobile
>Take Yahoo! Mail with you! Check email on your mobile phone.
>http://mobile.yahoo.com/learn/mail
>-
>To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
>the body of a message to majordomo@vger.kernel.org
>More majordomo info at  http://vger.kernel.org/majordomo-info.html
>Please read the FAQ at  http://www.tux.org/lkml/

Roberto Fichera.