[PATCH] x86/mm: don't apply va_align to hugetlb mappings on AMD F15h

[PATCH] x86/mm: don't apply va_align to hugetlb mappings on AMD F15h

[Laurent Wandrebeck]

Commit 1317a5e7f7b1 ("arch/x86: teach arch_get_unmapped_area_vmflags to
handle hugetlb mappings") taught get_align_mask() to return
huge_page_mask_align() for a hugetlbfs file, and skipped the pgoff-derived
align_offset for one. It missed the other write to align_offset:

	if (filp) {
		info.align_mask = get_align_mask(filp);
		info.align_offset += get_align_bits();
	}

get_align_bits() calls get_align_mask(NULL), so a hugetlbfs file still gets
the F15h I$ anti-aliasing randomization that its own align_mask already
excludes it from. vm_unmapped_area() therefore returns an address
deliberately offset from the huge page boundary, the hugetlb VMA's vm_start
is only PAGE_SIZE aligned, and tearing it down trips
BUG_ON(start & ~huge_page_mask(h)) in __unmap_hugepage_range():

  kernel BUG at mm/hugetlb.c:5161!
  RIP: 0010:__unmap_hugepage_range+0x64f/0x660
  RAX: 000000003fffffff  RDX: 00007e9280003000
  Call Trace:
   __zap_vma_range+0x523/0x680
   unmap_vmas+0xa5/0x1a0
   exit_mmap+0x13b/0x3f0
   do_exit+0x1e4/0x470

That is a 1 GiB mapping on an A10-8770E (family 0x15, model 0x65) running
7.2.0, 0x3000 below a 1 GiB boundary, RAX being ~huge_page_mask(h). Both
hstates crash, and so do both on an FX-8370E (family 0x15, model 0x02)
running 7.1.8, there 0x5000 low. The offset is va_align.bits, drawn once
per boot: identical across hstates within a boot, different between boots
and machines, and a boot that draws zero does not reproduce at any size -
hence the apparent intermittency. The crash is in the teardown path, so the
reservation leaks as well, HugePages_Rsvd owned by nobody until reboot.

Reproduced by mmap()ing MAP_HUGETLB and returning. A Ryzen 5 2500U (family
0x17) on the same 7.2.0 does not reproduce it, as expected since va_align
is only set up for family 0x15. With the patch both hstates return aligned
addresses, and PostgreSQL has mapped a 4 GB hugetlbfs segment for 18.8 h on
2 MiB and 4+ h on 1 GiB pages with no BUG and no leaked reservations.

Fixes: 1317a5e7f7b1 ("arch/x86: teach arch_get_unmapped_area_vmflags to handle hugetlb mappings")
Cc: stable@vger.kernel.org # 6.13+
Signed-off-by: Laurent Wandrebeck <l.wandrebeck@quelquesmots.fr>
---
 arch/x86/kernel/sys_x86_64.c | 6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)

diff --git a/arch/x86/kernel/sys_x86_64.c b/arch/x86/kernel/sys_x86_64.c
index 776ae6fa7f2d..6b2be065304f 100644
--- a/arch/x86/kernel/sys_x86_64.c
+++ b/arch/x86/kernel/sys_x86_64.c
@@ -157,7 +157,8 @@ arch_get_unmapped_area(struct file *filp, unsigned long addr, unsigned long len,
 	}
 	if (filp) {
 		info.align_mask = get_align_mask(filp);
-		info.align_offset += get_align_bits();
+		if (!is_file_hugepages(filp))
+			info.align_offset += get_align_bits();
 	}
 
 	return vm_unmapped_area(&info);
@@ -222,7 +223,8 @@ arch_get_unmapped_area_topdown(struct file *filp, unsigned long addr0,
 
 	if (filp) {
 		info.align_mask = get_align_mask(filp);
-		info.align_offset += get_align_bits();
+		if (!is_file_hugepages(filp))
+			info.align_offset += get_align_bits();
 	}
 	addr = vm_unmapped_area(&info);
 	if (!(addr & ~PAGE_MASK))
-- 
2.34.1

Re: [PATCH] x86/mm: don't apply va_align to hugetlb mappings on AMD F15h

[Dave Hansen]

Hey Boris,

These AMD F15h CPUs are, what, 15 years old now? How bad of a
performance hit *was* this issue? Is this optimization worth keeping around?

On 8/28/26 06:57, Laurent Wandrebeck wrote:
> Commit 1317a5e7f7b1 ("arch/x86: teach arch_get_unmapped_area_vmflags to
> handle hugetlb mappings") taught get_align_mask() to return
> huge_page_mask_align() for a hugetlbfs file, and skipped the pgoff-derived
> align_offset for one. It missed the other write to align_offset:

I'm not crazy about this changelog.

> diff --git a/arch/x86/kernel/sys_x86_64.c b/arch/x86/kernel/sys_x86_64.c
> index 776ae6fa7f2d..6b2be065304f 100644
> --- a/arch/x86/kernel/sys_x86_64.c
> +++ b/arch/x86/kernel/sys_x86_64.c
> @@ -157,7 +157,8 @@ arch_get_unmapped_area(struct file *filp, unsigned long addr, unsigned long len,
>  	}
>  	if (filp) {
>  		info.align_mask = get_align_mask(filp);
> -		info.align_offset += get_align_bits();
> +		if (!is_file_hugepages(filp))
> +			info.align_offset += get_align_bits();
>  	}

Rather than special-casing hugetlb in two more places, could we just do:

	info.align_mask = get_align_mask(filp);
	info.align_offset += get_align_bits(filp);

and then pass filp through get_align_bits() to get_align_mask()? I think
that would end up masking the troublesome bits out of va_align.bits.