mmap vs. real memory

mmap vs. real memory

[Brian McGrew]

Good afternoon all,

Right now, I'm in what appears [to me] to be a very tricky situation.  We
have several programs that all need access to the same memory.  Currently,
we use mmap and map to a file on disk like this:

const u_long    kCommsegSize            = 1024L * 1024L * 1024L;
const u_long    kCommsegMappedAddress   = 0xb0000000;

And then the code is:

    char *cseg_addr = reinterpret_cast<char *>(mmap(
            reinterpret_cast<char *>(kCommsegMappedAddress),
            size,
            PROT_READ  | PROT_WRITE,
            MAP_SHARED | MAP_FIXED,
            fd,
            0));
    gShmemOffset = kCommsegMappedAddress - kCommseg;
    return(cseg_addr);

And, all of this has worked great ever since we went from a physical shared
memory board on a vme bus to emulating shared memory on the computer.  In
the beginning, kCommsegSize was only 4MB (4L * 1024L * 1024L).

Now that we are needing more and more shared memory, we're seeing random
performance issues.  With 16MB, 64MB and 256MB (in recent past) all was
good.  But now, trying to allocate 1GB of shared memory, we see it taking
anywhere from .6 to 9 SEONDS to access the file.  No good!

What I'm wondering and needed to do is map a chunk of mymoery (1GB today,
maybe 2GB later on) so that all my apps can access it.  Short of creating a
ramdisk and moving the mmap'ed file to ramdisk, what is the best way to do
this???

Thanks,

-brian

Re: mmap vs. real memory

[Chris Friesen]

On 09/10/2009 04:38 PM, Brian McGrew wrote:

> Now that we are needing more and more shared memory, we're seeing random
> performance issues.  With 16MB, 64MB and 256MB (in recent past) all was
> good.  But now, trying to allocate 1GB of shared memory, we see it taking
> anywhere from .6 to 9 SEONDS to access the file.  No good!
> 
> What I'm wondering and needed to do is map a chunk of mymoery (1GB today,
> maybe 2GB later on) so that all my apps can access it.  Short of creating a
> ramdisk and moving the mmap'ed file to ramdisk, what is the best way to do
> this???

Maybe make a ramfs partition, mount it somewhere, and create your file
there?  This is more efficient and simpler than creating a ramdisk.

Alternately, have you considered using hugetlbfs and mapping the shared
memory with large pages?

Chris

Re: mmap vs. real memory

[Luciano Rocha]

[-- Attachment #1: Type: text/plain, Size: 1848 bytes --]

On Thu, Sep 10, 2009 at 03:38:11PM -0700, Brian McGrew wrote:
> Good afternoon all,
> 
> Right now, I'm in what appears [to me] to be a very tricky situation.  We
> have several programs that all need access to the same memory.  Currently,
> we use mmap and map to a file on disk like this:
> 
> const u_long    kCommsegSize            = 1024L * 1024L * 1024L;
> const u_long    kCommsegMappedAddress   = 0xb0000000;
> 
> And then the code is:
> 
>     char *cseg_addr = reinterpret_cast<char *>(mmap(
>             reinterpret_cast<char *>(kCommsegMappedAddress),
>             size,
>             PROT_READ  | PROT_WRITE,
>             MAP_SHARED | MAP_FIXED,
>             fd,
>             0));
>     gShmemOffset = kCommsegMappedAddress - kCommseg;
>     return(cseg_addr);
> 
> And, all of this has worked great ever since we went from a physical shared
> memory board on a vme bus to emulating shared memory on the computer.  In
> the beginning, kCommsegSize was only 4MB (4L * 1024L * 1024L).
> 
> Now that we are needing more and more shared memory, we're seeing random
> performance issues.  With 16MB, 64MB and 256MB (in recent past) all was
> good.  But now, trying to allocate 1GB of shared memory, we see it taking
> anywhere from .6 to 9 SEONDS to access the file.  No good!
> 
> What I'm wondering and needed to do is map a chunk of mymoery (1GB today,
> maybe 2GB later on) so that all my apps can access it.  Short of creating a
> ramdisk and moving the mmap'ed file to ramdisk, what is the best way to do
> this???

Were are you storing the file? For memory-only, you should use /dev/shm.
Another more portable solution is to use shm_open(3), that will open the
file were the system wants it.

-- 
Luciano Rocha <luciano@eurotux.com>
Eurotux Informática, S.A. <http://www.eurotux.com/>

[-- Attachment #2: Type: application/pgp-signature, Size: 197 bytes --]