[bug report?] unintuitive behavior when mapping over hugepage-backed PROT_NONE regions

[Uday Shankar <ushankar@purestorage.com>]

I was debugging an issue with a malloc implementation when I noticed
some unintuitive behavior that happens when someone attempts to
overwrite part of a hugepage-backed PROT_NONE mapping with another
mapping. I've isolated the issue and reproduced it with the following
program:

[root@localhost ~]# cat test.c
#include <assert.h>
#include <errno.h>
#include <stdio.h>
#include <stdlib.h>
#include <sys/mman.h>
#include <unistd.h>

#define MMAP_FLAGS_COMMON (MAP_PRIVATE | MAP_ANONYMOUS | MAP_HUGETLB)

int main()
{
        size_t len = 2ULL << 30;
        void *a = mmap(
                (void *)0x7c8000000000, len, PROT_NONE,
                MMAP_FLAGS_COMMON | MAP_FIXED_NOREPLACE | MAP_NORESERVE, -1, 0);
        printf("a=%p errno %d %m\n", a, errno);
        errno = 0;

        char buf[128];
        sprintf(buf, "cp /proc/%d/smaps smaps1", getpid());
        assert(system(buf) == 0);

        len = 4096;
        void *b = mmap(
                a, len, PROT_READ | PROT_WRITE,
                MMAP_FLAGS_COMMON | MAP_POPULATE | MAP_FIXED, -1, 0);
        printf("b=%p errno %d %m\n", b, errno);
        errno = 0;

        sprintf(buf, "cp /proc/%d/smaps smaps2", getpid());
        assert(system(buf) == 0);

        return 0;
}
[root@localhost ~]# gcc -o test test.c && ./test
a=0x7c8000000000 errno 0 Success
b=0xffffffffffffffff errno 12 Cannot allocate memory
[root@localhost ~]# diff smaps1 smaps2
157,158c157,158
< 7c8000000000-7c8080000000 ---p 00000000 00:10 7332                       /anon_hugepage (deleted)
< Size:            2097152 kB
---
> 7c8000200000-7c8080000000 ---p 00200000 00:10 7332                       /anon_hugepage (deleted)
> Size:            2095104 kB

First, we map a 2G PROT_NONE region using hugepages. This succeeds. Then
we try to map a 4096-length PROT_READ | PROT_WRITE region at the
beginning of the PROT_NONE region, still using hugepages. This fails, as
expected, because 4096 is much smaller than the hugepage size configured
on the system (this is x86 with a default hugepage size of 2M). The
surprising thing is the difference in /proc/pid/smaps before and after
the failed mmap. Even though the mmap failed, the value in
/proc/pid/smaps changed, with a 2M-sized bite being taken out the front
of the mapping. This feels unintuitive to me, as I'd expect a failed
mmap to have no effect on the virtual memory mappings of the calling
process whatsoever.

I initially saw this on an ancient redhat kernel, but I was able to
reproduce it on 6.13 as well. So I assume this behavior still exists and
has been around forever.