mmap'ing a large file

mmap'ing a large file

[Mike Black]

Is there a logical reason why a process can't mmap more than a 2G file?

I seem to get stuck at 2142208000 with
mmap: Cannot allocate memory

#include <stdio.h>
#include <string.h>
#include <stdlib.h>
#include <unistd.h>
#include <sys/mman.h>
#include <sys/types.h>
#include <sys/stat.h>
#include <fcntl.h>

#define FILESIZE 2500000000

int
main ()
{
    unsigned long long offset = 0;
    unsigned long maplength = getpagesize () * 1000;
    int i;
    unsigned char *p;
    char mynull = 0;
    int fd = open ("test.map", O_RDWR | O_CREAT, S_IRUSR | S_IWUSR);
    if (fd < 0) {
        perror ("test.map");
        exit (-1);
    }
    lseek (fd, FILESIZE - 1, SEEK_SET);
    write (fd, &mynull, 1);
    for (offset = 0; offset < FILESIZE - maplength; offset += maplength) {
        p = mmap (p, maplength, PROT_READ | PROT_WRITE, MAP_SHARED, fd,offset);
        printf ("%lld %p\n", offset, p);
        fflush (stdout);
        if (p == (unsigned char *) -1) {
            perror ("mmap");
            exit (-1);
        }
        memset (p, 1, maplength);
#if 0
        munmap (p, maplength);  /* this of course let's things go on */
#endif
    }
    return 0;
}


Michael D. Black mblack@csi-inc.com
http://www.csi-inc.com/
http://www.csi-inc.com/~mike
321-676-2923, x203
Melbourne FL

Re: mmap'ing a large file

[Gianni Tedesco]

[-- Attachment #1: Type: text/plain, Size: 932 bytes --]

On Wed, 2002-08-14 at 16:42, Mike Black wrote:
> Is there a logical reason why a process can't mmap more than a 2G file?
> 
> I seem to get stuck at 2142208000 with
> mmap: Cannot allocate memory

Perhaps this should be an FAQ item.

Intel is a 32bit architecture, that is to say the address space is 2^32
bytes (4GB), of this address space the kernel takes the top 2GB and
userspace the bottom 2GB.

There are patches that allow userspace to have 3GB or even 3.5GB
floating around. Obviously the kernel then only has 1GB/512MB - I'm not
sure what affect that will have.

The workaround to this is to only map in the portion(s) of the file you
actually need dynamically but this isn't always simple depending on the
application.

I suppose the short answer is 'its a hardware problem'.

HTH

-- 
// Gianni Tedesco (gianni at ecsc dot co dot uk)
8646BE7D: 6D9F 2287 870E A2C9 8F60 3A3C 91B5 7669 8646 BE7D

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 232 bytes --]

Re: mmap'ing a large file

[Jamie Lokier]

Gianni Tedesco wrote:
> On Wed, 2002-08-14 at 16:42, Mike Black wrote:
> > Is there a logical reason why a process can't mmap more than a 2G file?
> > 
> > I seem to get stuck at 2142208000 with
> > mmap: Cannot allocate memory
> 
> Perhaps this should be an FAQ item.
> 
> Intel is a 32bit architecture, that is to say the address space is 2^32
> bytes (4GB), of this address space the kernel takes the top 2GB and
> userspace the bottom 2GB.

No, firstly those numbers are incorrect and secondly, that's not the
reason why Mike's program stops at 2142208000.

The standard kernel's address space provides exactly 3GB for userspace.
The range of user addresses is 0x00000000 to 0xbfffffff.  So the
absolute maximum that can be mmaped at a time is a little under 3GB.

The reason why Mike's program won't mmap() more than about 2GB of the
file is for two reasons:

  1. mmap() will search for a free address starting at 0x40000000, up to
     0xbffff000.  It won't search lower addresses, unless you give it a
     hint, and the hinted address is actually unmapped.  I think this is
     to reserve about 1GB for brk().

  2. The executable and shared libraries take some room, too.

Btw Mike, the original program uses `p' uninitialised.  If you weren't
so lucky (say if the stack is initialised differently) you could get
much less than 2GB.

-- Jamie

Re: mmap'ing a large file

[H. Peter Anvin]

Followup to:  <1029342745.8255.6.camel@lemsip>
By author:    Gianni Tedesco <gianni@ecsc.co.uk>
In newsgroup: linux.dev.kernel
> 
> Intel is a 32bit architecture, that is to say the address space is 2^32
> bytes (4GB), of this address space the kernel takes the top 2GB and
> userspace the bottom 2GB.
> 

NAK.  3 GB userspace, 1 GB kernel space is the default.

The 3 GB is subdivided into 1 GB for the program, and 2 GB for libraries
and other mmaps (grows up) and the stack (grows down).

	-hpa
-- 
<hpa@transmeta.com> at work, <hpa@zytor.com> in private!
"Unix gives you enough rope to shoot yourself in the foot."
http://www.zytor.com/~hpa/puzzle.txt	<amsp@zytor.com>

Re: mmap'ing a large file

[DervishD]

    Hi Mike :)

>Is there a logical reason why a process can't mmap more than a 2G file?

    Seems to be the value of TASK_MAX if you don't have HIGHMEM.

    Moreover, if you try to mmap a size between TASK_MAX and
ULONG_MAX, you hit a corner case that I fixed in the -ac series, the
2.5.x series and other trees, *except* the 2.4.x 'official', because
Marcelo doesn't apply the patch (three lines... trivial I think).

    Raúl