[PATCH] x86/mm: lower MAP_32BIT begin to reduce heap collisions

[PATCH] x86/mm: lower MAP_32BIT begin to reduce heap collisions

[Simon Liebold]

Commit 03475167fda5 ("x86: Increase brk randomness entropy for 64-bit
systems") increased the brk randomness from 32 MiB to 1 GiB. MAP_32BIT
looks between 1 GiB and 2 GiB for an unmapped area. Depending on the
randomization, a heap starting high enough and being big enough can use
up all the area that MAP_32BIT looks at, leading to allocation failures.

For example, if the heap starts at 800 MiB and is 1.2 GiB large,
allocations with MAP_32BIT will always fail despite unused addresses
below 800 MiB.

Lower the begin of the address space which is available to MAP32_BIT
from 0x40000000 to 0x10000000 to give mmap more room if the randomly
allocated brk address turns out to be unfavourable high. This allows
mmap to allocate up to 75% more space.

Signed-off-by: Simon Liebold <simonlie@amazon.de>
---

Notes:
    Background: LuaJIT v2.0 uses MAP32_BIT for allocating memory. Because of
    the restriction of MAP32_BIT to limit all allocation of mmap to the
    address space from 1 GiB to 2 GiB, LuaJIT v2.0 can fail to work,
    depending on the random location of brk.
    
    I tested this change using the following reproducer:
    
    int main() {
        uintptr_t mmap_end = 0x80000000;
        uintptr_t heap_start = (uintptr_t)sbrk(0);
        printf("heap start: %p\n", heap_start);
        uintptr_t alloc_size = mmap_end - heap_start;
        uintptr_t heap_end = (uintptr_t)sbrk(alloc_size);
        printf("heap allocated until: %p\n", heap_end);
        void* addr = mmap(NULL,
            8,
            PROT_READ | PROT_WRITE,
            MAP_PRIVATE | MAP_ANONYMOUS | MAP_32BIT,
            -1,
            0);
    
        if(addr == MAP_FAILED)
            printf("mmap allocation failed\n");
        else
            printf("mmap allocation at %p\n", addr);
    
        return 0;
    }
    
    Before the change, the allocation failed:
        [root@localhost ~]# ./repro
        heap start: 0x24bce000
        heap allocated until: 0x24bef000
        mmap allocation failed
    
    After the change, it succeeded:
        [root@localhost ~]# ./repro
        heap start: 0x38f24000
        heap allocated until: 0x38f45000
        mmap allocation at 0x11962000
    
    Note that this does not guarantee to succeed. If the randomized heap
    start is below 0x10000000, it still fails.

 arch/x86/kernel/sys_x86_64.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/arch/x86/kernel/sys_x86_64.c b/arch/x86/kernel/sys_x86_64.c
index 776ae6fa7f2d..29c6277aa31e 100644
--- a/arch/x86/kernel/sys_x86_64.c
+++ b/arch/x86/kernel/sys_x86_64.c
@@ -100,7 +100,7 @@ static void find_start_end(unsigned long addr, unsigned long flags,
 		   conflicts with the heap, but we assume that glibc
 		   malloc knows how to fall back to mmap. Give it 1GB
 		   of playground for now. -AK */
-		*begin = 0x40000000;
+		*begin = 0x10000000;
 		*end = 0x80000000;
 		if (current->flags & PF_RANDOMIZE) {
 			*begin = randomize_page(*begin, 0x02000000);

base-commit: 260f6f4fda93c8485c8037865c941b42b9cba5d2
-- 
2.47.3




Amazon Web Services Development Center Germany GmbH
Tamara-Danz-Str. 13
10243 Berlin
Geschaeftsfuehrung: Christian Schlaeger, Jonathan Weiss
Eingetragen am Amtsgericht Charlottenburg unter HRB 257764 B
Sitz: Berlin
Ust-ID: DE 365 538 597

Re: [PATCH] x86/mm: lower MAP_32BIT begin to reduce heap collisions

[Dave Hansen]

On 8/25/25 03:48, Simon Liebold wrote:
> Commit 03475167fda5 ("x86: Increase brk randomness entropy for 64-bit
> systems") increased the brk randomness from 32 MiB to 1 GiB. MAP_32BIT
> looks between 1 GiB and 2 GiB for an unmapped area. Depending on the
> randomization, a heap starting high enough and being big enough can use
> up all the area that MAP_32BIT looks at, leading to allocation failures.

Isn't that still a really unreasonably gigantic heap?

Would you mind posting some actual /proc/$pid/maps output from one of
the failure cases?

Re: [PATCH] x86/mm: lower MAP_32BIT begin to reduce heap collisions

[Simon Liebold]

Dave Hansen <dave.hansen@intel.com> writes:

> On 8/25/25 03:48, Simon Liebold wrote:
>> Commit 03475167fda5 ("x86: Increase brk randomness entropy for 64-bit
>> systems") increased the brk randomness from 32 MiB to 1 GiB. MAP_32BIT
>> looks between 1 GiB and 2 GiB for an unmapped area. Depending on the
>> randomization, a heap starting high enough and being big enough can use
>> up all the area that MAP_32BIT looks at, leading to allocation failures.
>
> Isn't that still a really unreasonably gigantic heap?
>
> Would you mind posting some actual /proc/$pid/maps output from one of
> the failure cases?

Hello Dave,

we are seeing heaps this large on real workloads. This is a Lua
application using LuaJit v2.0.

This is an excerpt from the output of one of the failure cases:

    00400000-00566000 r-xp 00000000 103:01 4476567               [...]/bin/httpd.orig
    00765000-0076b000 r--p 00165000 103:01 4476567               [...]/bin/httpd.orig
    0076b000-00772000 rw-p 0016b000 103:01 4476567               [...]/bin/httpd.orig
    00772000-00778000 rw-p 00000000 00:00 0
    34a21000-35021000 rw-p 00000000 00:00 0                      [heap]
    35021000-82ea7000 rw-p 00000000 00:00 0                      [heap]
    7fee7c0ba000-7fee7c11f000 r-xp 00000000 103:01 3836609       [...]/lib/libluajit-5.1.so
    7fee7c11f000-7fee7c31f000 ---p 00065000 103:01 3836609       [...]/lib/libluajit-5.1.so
    7fee7c31f000-7fee7c321000 r--p 00065000 103:01 3836609       [...]/lib/libluajit-5.1.so
    7fee7c321000-7fee7c322000 rw-p 00067000 103:01 3836609       [...]/lib/libluajit-5.1.so
    [Other maps at high addresses...]

Regards,

Simon Liebold



Amazon Web Services Development Center Germany GmbH
Tamara-Danz-Str. 13
10243 Berlin
Geschaeftsfuehrung: Christian Schlaeger, Jonathan Weiss
Eingetragen am Amtsgericht Charlottenburg unter HRB 257764 B
Sitz: Berlin
Ust-ID: DE 365 538 597