Re: [PATCH 0/3] mm/mmap.c: don't unmap the overlapping VMA(s)

[Piotr Kwapulinski <kwapulinski.piotr@gmail.com>]

On Mon, Apr 04, 2016 at 05:26:43PM +0200, Vlastimil Babka wrote:
> On 04/04/2016 09:31 AM, Michal Hocko wrote:
> >On Sat 02-04-16 21:17:31, Piotr Kwapulinski wrote:
> >>Currently the mmap(MAP_FIXED) discards the overlapping part of the
> >>existing VMA(s).
> >>Introduce the new MAP_DONTUNMAP flag which forces the mmap to fail
> >>with ENOMEM whenever the overlapping occurs and MAP_FIXED is set.
> >>No existing mapping(s) is discarded.
> >
> >You forgot to tell us what is the use case for this new flag.
> 
> Exactly. Also, returning ENOMEM is strange, EINVAL might be a better match,
> otherwise how would you distinguish a "geunine" ENOMEM from passing a wrong
> address?
> 
> 

Thanks to all for suggestions. I'll fix them.

The example use case:
#include <stdio.h>
#include <string.h>
#include <sys/mman.h>

void main(void)
{
  void* addr = (void*)0x1000000;
  size_t size = 0x600000;
  void* start = 0;
  start = mmap(addr,
               size,
               PROT_WRITE,
               MAP_PRIVATE | MAP_ANONYMOUS | MAP_FIXED,
               -1, 0);

  strcpy(start, "PPPP");
  printf("%s\n", start);        // == PPPP

  addr = (void*)0x1000000;
  size = 0x9000;
  start = mmap(addr,
               size,
               PROT_READ | PROT_WRITE,
               MAP_PRIVATE | MAP_ANONYMOUS | MAP_FIXED,
               -1, 0);
  
  printf("%s\n", start);        // != PPPP
}

Another use case, this time with huge pages in action.
The limit configured in proc's nr_hugepages is exceeded.
mmap unmaps the area and fails. No new mapping is created.
The program segfaults.

echo 0 > /sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages

#include <stdio.h>
#include <string.h>
#include <sys/mman.h>
#include <unistd.h>

void main(void)
{
  void* addr = (void*)0x1000000;
  size_t size = 0x600000;
  void* start = 0;
  start = mmap(addr,
               size,
               PROT_WRITE,
               MAP_PRIVATE | MAP_ANONYMOUS | MAP_FIXED,
               -1, 0);

  strcpy(start, "PPPP");
  printf("%s\n", start);        // == PPPP

  addr = (void*)0x1000000;
  size = 0x400000;
  start = mmap(addr,
               size,
               PROT_READ | PROT_WRITE,
               MAP_PRIVATE | MAP_ANONYMOUS | MAP_FIXED | MAP_HUGETLB,
               -1, 0);         // mmap fails but unmaps the area

  printf("%s\n", addr);       // segfault
}

Piotr