Virtual memory leaking through IA32 emulation layer for mmap and

Virtual memory leaking through IA32 emulation layer for mmap and

[Shaun]

Hi,

I'm not sure if this is a known issue or not, but we have been having some
trouble running some of our programs under the IA32 emulation layer of
linux-ia64.

What is happening in our case is that our program is indirectly causing
lots of mmaps of anonymous memory for 4096 bytes. Shortly thereafter these
blocks are munmaped. The munmap is effectively ignored causing the program
to leak a page (16K in our kernel) of virtual memory.

The situation can be illustrated with the following code when compiled on
an IA32 system then run on a IA64 system:

for (int i = 0; i < 10000; i++)
{
   pvAddr = mmap(NULL, getpagesize(), PROT_READ|PROT_WRITE,
                 MAP_PRIVATE|MAP_ANONYMOUS, -1, 0);
   printf("%p\n", pvAddr);
   munmap(pvAddr, getpagesize());
}

Given that getpagesize() is hardcoded in glibc it returns 4096, the mmap
succeeds and the compatibility layers actually allocate a full page (the
kernel in question has PAGE_SHIFT set to 14 for a page size of 16K).
However the munmap hits the following bit of code in sys_ia32.c:

asmlinkage long
sys32_munmap (unsigned int start, unsigned int len)
{
   unsigned int end = start + len;
   long ret;
...
   start = PAGE_ALIGN(start);
   end = PAGE_START(end);

   if (start >= end)
      return 0;

Because end is always less than a page size away from start, start and end
are equal causing munmap to return 0 without doing anything (leaking
memory).

The most obvious solution to this in userland is using the correct page
size, which is what we have done inside our product (we determine it at
run time). Unfortunately, the memory we're leaking now is being leaked by
the glibc standard io routines. They hardcode the page size as 4k and use
mmap to allocate a buffer big enough for the first fgets and fputs on a
stream. Since we use a lot of streams, we leak quickly.

Unfortunately, even avoiding streams won't really save us from this
problem since any other routine we use could well perform mmaps of this
type under the covers now or in the future.

Does anyone have any suggestions for how this could/should be fixed kernel
wise?  We're happy to implement what we can and contribute the changes
back.

Thanks,
Shaun

Re: Virtual memory leaking through IA32 emulation layer for mmap and munmap

[H. J. Lu]

On Tue, Mar 09, 2004 at 10:43:45AM +1100, Shaun wrote:
> 
> Hi,
> 
> I'm not sure if this is a known issue or not, but we have been having some
> trouble running some of our programs under the IA32 emulation layer of
> linux-ia64.
> 
> What is happening in our case is that our program is indirectly causing
> lots of mmaps of anonymous memory for 4096 bytes. Shortly thereafter these
> blocks are munmaped. The munmap is effectively ignored causing the program
> to leak a page (16K in our kernel) of virtual memory.
> 
> The situation can be illustrated with the following code when compiled on
> an IA32 system then run on a IA64 system:
> 
> for (int i = 0; i < 10000; i++)
> {
>    pvAddr = mmap(NULL, getpagesize(), PROT_READ|PROT_WRITE,
>                  MAP_PRIVATE|MAP_ANONYMOUS, -1, 0);
>    printf("%p\n", pvAddr);
>    munmap(pvAddr, getpagesize());
> }
> 
> Given that getpagesize() is hardcoded in glibc it returns 4096, the mmap

getpagesize () isn't hardcoded to 4K in glibc. You can change it with
AT_PAGESZ. The problem is the x86 executable is aligned at 4K page
size, which is specified by the i386 ABI. The dynamic linker in glibc
won't load the normal x86 executables with a bigger page size. I have
a glibc patch which allows >4K page size. It may work for you.


H.J.

Re: Virtual memory leaking through IA32 emulation layer for mmap and munmap

[Peter Chubb]

>>>>> "Shaun" = Shaun  <delius@progsoc.uts.edu.au> writes:

Shaun> Hi,

Shaun> Given that getpagesize() is hardcoded in glibc it returns 4096,
Shaun> the mmap succeeds and the compatibility layers actually
Shaun> allocate a full page (the kernel in question has PAGE_SHIFT set
Shaun> to 14 for a page size of 16K).  However the munmap hits the
Shaun> following bit of code in sys_ia32.c:

Shaun> asmlinkage long sys32_munmap (unsigned int start, unsigned int
Shaun> len) { unsigned int end = start + len; long ret; ...  start Shaun> PAGE_ALIGN(start); end = PAGE_START(end);

Shaun>    if (start >= end) return 0;

 I think thius may be a bug.  From the man page for munmap:

   The  address  start must be a multiple of the page size. All pages con-
   taining a part of the indicated range are unmapped, and subsequent ref-
   erences to these pages will generate SIGSEGV. It is not an error if the
   indicated range does not contain any mapped pages.

This follows the SUS, which means that end should be rounded up not
down.

--
Dr Peter Chubb  http://www.gelato.unsw.edu.au  peterc AT gelato.unsw.edu.au
The technical we do immediately,  the political takes *forever*

Re: Virtual memory leaking through IA32 emulation layer for mmap and munmap

[David Mosberger]

>>>>> On Tue, 9 Mar 2004 11:11:04 +1100, Peter Chubb <peter@chubb.wattle.id.au> said:

>>>>> "Shaun" = Shaun  <delius@progsoc.uts.edu.au> writes:
  Shaun> Hi,

  Shaun> Given that getpagesize() is hardcoded in glibc it returns
  Shaun> 4096, the mmap succeeds and the compatibility layers actually
  Shaun> allocate a full page (the kernel in question has PAGE_SHIFT
  Shaun> set to 14 for a page size of 16K).  However the munmap hits
  Shaun> the following bit of code in sys_ia32.c:

  Shaun> asmlinkage long sys32_munmap (unsigned int start, unsigned
  Shaun> int len) { unsigned int end = start + len; long ret; ...
  Shaun> start = PAGE_ALIGN(start); end = PAGE_START(end);

  Shaun> if (start >= end) return 0;

  Peter>  I think thius may be a bug.  From the man page for munmap:

  Peter>    The address start must be a multiple of the page size. All
  Peter> pages con- taining a part of the indicated range are
  Peter> unmapped, and subsequent ref- erences to these pages will
  Peter> generate SIGSEGV. It is not an error if the indicated range
  Peter> does not contain any mapped pages.

  Peter> This follows the SUS, which means that end should be rounded
  Peter> up not down.

Rounded up to the _page-size_ which is 4KB for x86.  In general, you
can't just round up to 16KB for munmap() or all hell will break lose.

	--david

Re: Virtual memory leaking through IA32 emulation layer for mmap

[Shaun]

On Mon, 8 Mar 2004, H. J. Lu wrote:

> On Tue, Mar 09, 2004 at 10:43:45AM +1100, Shaun wrote:
> >
> > Given that getpagesize() is hardcoded in glibc it returns 4096, the mmap
>
> getpagesize () isn't hardcoded to 4K in glibc.

Indeed, it isn't quite hardcoded, but it may as well be, it certainly
doesn't do what the man page states it does:

  The function getpagesize() returns the number of bytes in a page,  where
  a "page" is the thing used where it says in the description of mmap(2)
  that files are mapped in page-sized units.

And without a doubt the page size is hardcoded in the libio
(stdio) routines.

> The problem is the x86 executable is
> aligned at 4K page size, which is specified by the i386 ABI. The dynamic
> linker in glibc won't load the normal x86 executables with a bigger page
> size. I have a glibc patch which allows >4K page size. It may work for
> you.

I guess that makes sense, the dynamic linker would need special code to
handle this case.

However, I can't see how this relates to my problem unless your patch
fixes the libio routines too?

Cheers,
Shaun

Re: Virtual memory leaking through IA32 emulation layer for mmap and munmap

[Peter Chubb]

>>>>> "David" = David Mosberger <davidm@napali.hpl.hp.com> writes:


David> Rounded up to the _page-size_ which is 4KB for x86.  In
David> general, you can't just round up to 16KB for munmap() or all
David> hell will break lose.

As munmap()'s behaviour is undefined on memory that wasn't allocated
with mmap(), rounding to the underlaying true page size should work
for well-behaved programs.

     x = mmap(... 4k ....)
maps a 16k chunk

     munmap(x, 4k)
deallocates a 16k chunk, if page_size is 16k

Of course at present, mmap32 tries to be clever and pretend PAGE_SIZE
is 4k, so it doesn't work...

--
Dr Peter Chubb  http://www.gelato.unsw.edu.au  peterc AT gelato.unsw.edu.au
The technical we do immediately,  the political takes *forever*

Re: Virtual memory leaking through IA32 emulation layer for mmap and munmap

[David Mosberger]

>>>>> On Tue, 9 Mar 2004 11:57:47 +1100, Peter Chubb <peter@chubb.wattle.id.au> said:

>>>>> "David" = David Mosberger <davidm@napali.hpl.hp.com> writes:
  David> Rounded up to the _page-size_ which is 4KB for x86.  In
  David> general, you can't just round up to 16KB for munmap() or all
  David> hell will break lose.

  Peter> As munmap()'s behaviour is undefined on memory that wasn't
  Peter> allocated with mmap(), rounding to the underlaying true page
  Peter> size should work for well-behaved programs.

  Peter>      x = mmap(... 4k ....)  maps a 16k chunk

  Peter>      munmap(x, 4k) deallocates a 16k chunk, if page_size is
  Peter> 16k

  Peter> Of course at present, mmap32 tries to be clever and pretend
  Peter> PAGE_SIZE is 4k, so it doesn't work...

All of this is old news and has come up at least twice in the past.
Don Dugger started to work on a patch to track partial page
allocations but I believe the patch was never finished.  Please check
the mail archives for details.

	--david

Re: Virtual memory leaking through IA32 emulation layer for mmap

[Arun Sharma]

Peter Chubb wrote:
> 
> As munmap()'s behaviour is undefined on memory that wasn't allocated
> with mmap(), rounding to the underlaying true page size should work
> for well-behaved programs.
> 
>      x = mmap(... 4k ....)
> maps a 16k chunk
> 
>      munmap(x, 4k)
> deallocates a 16k chunk, if page_size is 16k

As David said, this has been discussed before. Basically if there were 
other 4k mappings with a different permission in the same 16k chunk, you 
can't deallocate the 16k chunk safely. Tracking the sub pages without 
changes to machine independent code (struct page) is possible, but may 
be expensive.

	-Arun

Re: Virtual memory leaking through IA32 emulation layer for mmap

[Arun Sharma]

H. J. Lu wrote:
> 
> getpagesize () isn't hardcoded to 4K in glibc. You can change it with
> AT_PAGESZ. The problem is the x86 executable is aligned at 4K page
> size, which is specified by the i386 ABI. The dynamic linker in glibc
> won't load the normal x86 executables with a bigger page size. I have
> a glibc patch which allows >4K page size. It may work for you.

I did some very light testing with HJ's glibc patch with this kernel 
change a few months ago and didn't find any obvious breakages. YMMV.

	-Arun

--- 1.21/fs/binfmt_elf.c        Sat May  4 10:05:24 2002
+++ edited/fs/binfmt_elf.c      Fri Jun 13 12:25:10 2003
@@ -184,7 +184,7 @@
         }
         sp -= DLINFO_ITEMS*2;
         NEW_AUX_ENT( 0, AT_HWCAP, hwcap);
-       NEW_AUX_ENT( 1, AT_PAGESZ, ELF_EXEC_PAGESIZE);
+       NEW_AUX_ENT( 1, AT_PAGESZ, PAGE_SIZE);
         NEW_AUX_ENT( 2, AT_CLKTCK, CLOCKS_PER_SEC);
         NEW_AUX_ENT( 3, AT_PHDR, load_addr + exec->e_phoff);
         NEW_AUX_ENT( 4, AT_PHENT, sizeof (struct elf_phdr));

Re: Virtual memory leaking through IA32 emulation layer for mmap and munmap

[n0ano]

What Dave said.  I only see 2 solutions to this problem:

1)  Find the preliminary patch I wrote (it's in the email archives,
let me know if you can't find it) and finish it.  It should be
fairly functional but it needs to be made optional, either as a
config option or something at run time.  Most `well behaved' IA32
programs don't need the overhead required to implement that patch.

2)  The easiest solution is to just configure your kernel to use
4K pages.  This might cause some performance degradation for IA64
programs but it should be minor and will definitely solve this
problem.

PS: You can read through the email archives to follow the detailed
discussion of this issue but, trust me, there is no easy solution.
Trying to modify the IA32 `mmap' calls to align all requests onto
the kerenl's page size, the obvious solution, will break the way
shared libraries are loaded and then no IA32 programs will run.

On Mon, Mar 08, 2004 at 05:04:47PM -0800, David Mosberger wrote:
> >>>>> On Tue, 9 Mar 2004 11:57:47 +1100, Peter Chubb <peter@chubb.wattle.id.au> said:
> 
> >>>>> "David" = David Mosberger <davidm@napali.hpl.hp.com> writes:
>   David> Rounded up to the _page-size_ which is 4KB for x86.  In
>   David> general, you can't just round up to 16KB for munmap() or all
>   David> hell will break lose.
> 
>   Peter> As munmap()'s behaviour is undefined on memory that wasn't
>   Peter> allocated with mmap(), rounding to the underlaying true page
>   Peter> size should work for well-behaved programs.
> 
>   Peter>      x = mmap(... 4k ....)  maps a 16k chunk
> 
>   Peter>      munmap(x, 4k) deallocates a 16k chunk, if page_size is
>   Peter> 16k
> 
>   Peter> Of course at present, mmap32 tries to be clever and pretend
>   Peter> PAGE_SIZE is 4k, so it doesn't work...
> 
> All of this is old news and has come up at least twice in the past.
> Don Dugger started to work on a patch to track partial page
> allocations but I believe the patch was never finished.  Please check
> the mail archives for details.
> 
> 	--david
> -
> To unsubscribe from this list: send the line "unsubscribe linux-ia64" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

-- 
Don Dugger
"Censeo Toto nos in Kansa esse decisse." - D. Gale
n0ano@n0ano.com
Ph: 303/447-2201

Re: Virtual memory leaking through IA32 emulation layer for mmap

[Shaun]

On Tue, 9 Mar 2004 n0ano@n0ano.com wrote:

> What Dave said.  I only see 2 solutions to this problem:
>
> 1)  Find the preliminary patch I wrote (it's in the email archives,
> let me know if you can't find it) and finish it.  It should be
> fairly functional but it needs to be made optional, either as a
> config option or something at run time.  Most `well behaved' IA32
> programs don't need the overhead required to implement that patch.

Well, I can't see how this has anything to do with being well behaved? It
seems to me that any well behaved program could (and does) easily cause
this problem. Is there any chance of the patch being merged if I do the
remaining work to make it optional?

Thanks,
Shaun

Re: Virtual memory leaking through IA32 emulation layer for mmap

[David Mosberger]

>>>>> On Wed, 10 Mar 2004 16:16:22 +1100 (EST), Shaun <delius@progsoc.uts.edu.au> said:

  Shaun> Well, I can't see how this has anything to do with being well
  Shaun> behaved? It seems to me that any well behaved program could
  Shaun> (and does) easily cause this problem. Is there any chance of
  Shaun> the patch being merged if I do the remaining work to make it
  Shaun> optional?

Assuming the patch is clean and the performance impact is reasonably
small (I think it should be), I'm sure the patch would be welcome
(that's just my opinion; Arun is the ia32 subsystem maintainer so he's
the final decision maker as far as this is concerned).

	--david

Re: Virtual memory leaking through IA32 emulation layer for mmap

[Arun Sharma]

David Mosberger wrote:
> 
> Assuming the patch is clean and the performance impact is reasonably
> small (I think it should be), I'm sure the patch would be welcome
> (that's just my opinion; Arun is the ia32 subsystem maintainer so he's
> the final decision maker as far as this is concerned).

Yes, I'd really like to see this problem fixed. I think, the patch needs to be as memory efficient as possible - we're using non-pageable kernel memory. So some numbers on the two level bitmap scheme vs keeping lists of head/tail pages for common ia32 workloads would be good.

    -Arun 

Re: Virtual memory leaking through IA32 emulation layer for mmap

[David Mosberger]

>>>>> On Wed, 10 Mar 2004 11:31:50 -0800, Arun Sharma <arun.sharma@intel.com> said:

  Arun> Yes, I'd really like to see this problem fixed. I think, the
  Arun> patch needs to be as memory efficient as possible - we're
  Arun> using non-pageable kernel memory. So some numbers on the two
  Arun> level bitmap scheme vs keeping lists of head/tail pages for
  Arun> common ia32 workloads would be good.

Agreed.  The good news is that the track is only needed for parially
allocated pages, so the overhead should, in the worst-case, be
proportional to the number of vmareas, not to the amount of memory
mapped.

	--david