[PATCH] mm/hugetlb: account for allowed nodes when gathering surplus pages

[PATCH] mm/hugetlb: account for allowed nodes when gathering surplus pages

[Huaisheng Ye]

Hugetlb reservations are accounted globally, but hugetlb_acct_memory()
also verifies that the current cpuset and MPOL_BIND policy contain enough
free huge pages to add a new reservation.

gather_surplus_pages() calculates its allocation shortfall from the global
free and reserved counters. If the global pool has enough free pages, but
those pages reside outside the nodes allowed by the task, it allocates no
surplus pages. The subsequent allowed_mems_nr() check then rejects the
reservation and mmap() fails with ENOMEM, even when
nr_overcommit_hugepages permits allocating surplus pages on the allowed
nodes.

Calculate both the global shortfall and the shortfall within the allowed
nodes, and allocate the larger of the two. Include surplus pages allocated
outside hugetlb_lock in both calculations when rechecking after reacquiring
the lock. These pages are constrained by alloc_nodemask, so they satisfy
both shortages.

Easy way to reproduce this issue with 2+ NUMA nodes system:

  # echo 0 > /sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepage
  # echo 3 > /sys/devices/system/node/node0/hugepages/hugepages-2048kB/nr_hugepages
  # echo 1 > /sys/kernel/mm/hugepages/hugepages-2048kB/nr_overcommit_hugepages
  # cd tools/testing/selftests/mm
  # numactl --membind=1 ./hugetlb-mmap 2 21
  TAP version 13
  # [INFO] detected hugetlb page size: 2048 KiB
  # [INFO] detected hugetlb page size: 1048576 KiB
  # 2048 kB hugepages
  1..2
  # Mapping 2 Mbytes
  Bail out! mmap: Cannot allocate memory (12)
  # Planned tests != run tests (2 != 0)
  # Totals: pass:0 fail:0 xfail:0 xpass:0 skip:0 error:0

This fixes hugetlb mappings when, for example, a task runs with
MPOL_BIND on Node 1 while the existing free huge pages are on Node 0.

Similar issue also could be found in ltp if the free pages of global pool
reside outside the nodes allowed by the application.
  # cd ltp/testcases/kernel/mem/hugetlb/hugemmap/
  # numactl --cpunodebind=0 --membind=1 ./hugemmap10

Signed-off-by: Huaisheng Ye <yehuaisheng@open-hieco.net>
---
 mm/hugetlb.c | 21 +++++++++++++++++----
 1 file changed, 17 insertions(+), 4 deletions(-)

diff --git a/mm/hugetlb.c b/mm/hugetlb.c
index 7857728457952..b078953099fe5 100644
--- a/mm/hugetlb.c
+++ b/mm/hugetlb.c
@@ -116,6 +116,7 @@ struct mutex *hugetlb_fault_mutex_table __ro_after_init;
 
 /* Forward declaration */
 static int hugetlb_acct_memory(struct hstate *h, long delta);
+static unsigned int allowed_mems_nr(struct hstate *h);
 static void hugetlb_vma_lock_free(struct vm_area_struct *vma);
 static void hugetlb_vma_lock_alloc(struct vm_area_struct *vma);
 static void __hugetlb_vma_unlock_write_free(struct vm_area_struct *vma);
@@ -2224,6 +2225,19 @@ static nodemask_t *policy_mbind_nodemask(gfp_t gfp)
 	return NULL;
 }
 
+/*
+ * Reservations are globally accounted, but they must also be backed by free
+ * pages on nodes allowed by the current cpuset and MPOL_BIND policy.
+ */
+static long surplus_pages_needed(struct hstate *h, long delta, long allocated)
+{
+	long global_free = (long)h->free_huge_pages + allocated;
+	long allowed_free = (long)allowed_mems_nr(h) + allocated;
+
+	return max((long)h->resv_huge_pages + delta - global_free,
+		   delta - allowed_free);
+}
+
 /*
  * Increase the hugetlb pool such that it can accommodate a reservation
  * of size 'delta'.
@@ -2246,7 +2260,7 @@ static int gather_surplus_pages(struct hstate *h, long delta)
 		alloc_nodemask = cpuset_current_mems_allowed;
 
 	lockdep_assert_held(&hugetlb_lock);
-	needed = (h->resv_huge_pages + delta) - h->free_huge_pages;
+	needed = surplus_pages_needed(h, delta, 0);
 	if (needed <= 0) {
 		h->resv_huge_pages += delta;
 		return 0;
@@ -2277,11 +2291,10 @@ static int gather_surplus_pages(struct hstate *h, long delta)
 
 	/*
 	 * After retaking hugetlb_lock, we need to recalculate 'needed'
-	 * because either resv_huge_pages or free_huge_pages may have changed.
+	 * because either resv_huge_pages or the free page counts may have changed.
 	 */
 	spin_lock_irq(&hugetlb_lock);
-	needed = (h->resv_huge_pages + delta) -
-			(h->free_huge_pages + allocated);
+	needed = surplus_pages_needed(h, delta, allocated);
 	if (needed > 0) {
 		if (alloc_ok)
 			goto retry;
-- 
2.52.0

Re: [PATCH] mm/hugetlb: account for allowed nodes when gathering surplus pages

[Muchun Song]

> On Sep 9, 2026, at 15:46, Huaisheng Ye <yehuaisheng@open-hieco.net> wrote:
> 
> Hugetlb reservations are accounted globally, but hugetlb_acct_memory()
> also verifies that the current cpuset and MPOL_BIND policy contain enough
> free huge pages to add a new reservation.
> 
> gather_surplus_pages() calculates its allocation shortfall from the global
> free and reserved counters. If the global pool has enough free pages, but
> those pages reside outside the nodes allowed by the task, it allocates no
> surplus pages. The subsequent allowed_mems_nr() check then rejects the
> reservation and mmap() fails with ENOMEM, even when
> nr_overcommit_hugepages permits allocating surplus pages on the allowed
> nodes.
> 
> Calculate both the global shortfall and the shortfall within the allowed
> nodes, and allocate the larger of the two. Include surplus pages allocated
> outside hugetlb_lock in both calculations when rechecking after reacquiring
> the lock. These pages are constrained by alloc_nodemask, so they satisfy
> both shortages.
> 
> Easy way to reproduce this issue with 2+ NUMA nodes system:
> 
>  # echo 0 > /sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepage
>  # echo 3 > /sys/devices/system/node/node0/hugepages/hugepages-2048kB/nr_hugepages
>  # echo 1 > /sys/kernel/mm/hugepages/hugepages-2048kB/nr_overcommit_hugepages
>  # cd tools/testing/selftests/mm
>  # numactl --membind=1 ./hugetlb-mmap 2 21
>  TAP version 13
>  # [INFO] detected hugetlb page size: 2048 KiB
>  # [INFO] detected hugetlb page size: 1048576 KiB
>  # 2048 kB hugepages
>  1..2
>  # Mapping 2 Mbytes
>  Bail out! mmap: Cannot allocate memory (12)
>  # Planned tests != run tests (2 != 0)
>  # Totals: pass:0 fail:0 xfail:0 xpass:0 skip:0 error:0
> 
> This fixes hugetlb mappings when, for example, a task runs with
> MPOL_BIND on Node 1 while the existing free huge pages are on Node 0.
> 
> Similar issue also could be found in ltp if the free pages of global pool
> reside outside the nodes allowed by the application.
>  # cd ltp/testcases/kernel/mem/hugetlb/hugemmap/
>  # numactl --cpunodebind=0 --membind=1 ./hugemmap10
> 
> Signed-off-by: Huaisheng Ye <yehuaisheng@open-hieco.net>

Acked-by: Muchun Song <muchun.song@linux.dev>

Thanks.

Re: [PATCH] mm/hugetlb: account for allowed nodes when gathering surplus pages

[Andrew Morton]

On Thu, 10 Sep 2026 16:48:56 +0800 Muchun Song <muchun.song@linux.dev> wrote:

> > This fixes hugetlb mappings when, for example, a task runs with
> > MPOL_BIND on Node 1 while the existing free huge pages are on Node 0.
> > 
> > Similar issue also could be found in ltp if the free pages of global pool
> > reside outside the nodes allowed by the application.
> >  # cd ltp/testcases/kernel/mem/hugetlb/hugemmap/
> >  # numactl --cpunodebind=0 --membind=1 ./hugemmap10
> > 
> > Signed-off-by: Huaisheng Ye <yehuaisheng@open-hieco.net>
> 
> Acked-by: Muchun Song <muchun.song@linux.dev>

Should we backport this?

If so, a Fixes: target would be helpful.

Re: [PATCH] mm/hugetlb: account for allowed nodes when gathering surplus pages

[Muchun Song]

> On Sep 11, 2026, at 06:20, Andrew Morton <akpm@linux-foundation.org> wrote:
> 
> On Thu, 10 Sep 2026 16:48:56 +0800 Muchun Song <muchun.song@linux.dev> wrote:
> 
>>> This fixes hugetlb mappings when, for example, a task runs with
>>> MPOL_BIND on Node 1 while the existing free huge pages are on Node 0.
>>> 
>>> Similar issue also could be found in ltp if the free pages of global pool
>>> reside outside the nodes allowed by the application.
>>> # cd ltp/testcases/kernel/mem/hugetlb/hugemmap/
>>> # numactl --cpunodebind=0 --membind=1 ./hugemmap10
>>> 
>>> Signed-off-by: Huaisheng Ye <yehuaisheng@open-hieco.net>
>> 
>> Acked-by: Muchun Song <muchun.song@linux.dev>
> 
> Should we backport this?
> 
> If so, a Fixes: target would be helpful.

I believe the fix should be e4e574b767ba ("hugetlb: Try to grow hugetlb
pool for MAP_SHARED mappings"). This commit was introduced back in 2007.
Regarding the impact of this issue, I consulted an AI:

Severity: Medium
Impact: valid HugeTLB mappings can be rejected with ENOMEM
Scope: NUMA + cpuset/MPOL_BIND + nr_overcommit_hugepages
Risk: availability/configuration failure, no data corruption


The fact that it has only been triggered after such a long time also
precisely shows how uncommon its configuration is. I expect that
backporting may encounter some conflicts. In the absence of clear user
feedback, I personally suggest not doing a backport.

Of course, as the author of this patch, you could also provide more
information, for example, whether it was discovered as a real issue
in a production environment, so as to decide whether to proceed with
a backport.

Thanks.

Re: [PATCH] mm/hugetlb: account for allowed nodes when gathering surplus pages

[Huaisheng Ye]

On 9/11/2026 10:41, Muchun Song wrote:
>
>> On Sep 11, 2026, at 06:20, Andrew Morton <akpm@linux-foundation.org> wrote:
>>
>> On Thu, 10 Sep 2026 16:48:56 +0800 Muchun Song <muchun.song@linux.dev> wrote:
>>
>>>> This fixes hugetlb mappings when, for example, a task runs with
>>>> MPOL_BIND on Node 1 while the existing free huge pages are on Node 0.
>>>>
>>>> Similar issue also could be found in ltp if the free pages of global pool
>>>> reside outside the nodes allowed by the application.
>>>> # cd ltp/testcases/kernel/mem/hugetlb/hugemmap/
>>>> # numactl --cpunodebind=0 --membind=1 ./hugemmap10
>>>>
>>>> Signed-off-by: Huaisheng Ye <yehuaisheng@open-hieco.net>
>>> Acked-by: Muchun Song <muchun.song@linux.dev>
>> Should we backport this?
>>
>> If so, a Fixes: target would be helpful.
> I believe the fix should be e4e574b767ba ("hugetlb: Try to grow hugetlb
> pool for MAP_SHARED mappings"). This commit was introduced back in 2007.
> Regarding the impact of this issue, I consulted an AI:
>
> Severity: Medium
> Impact: valid HugeTLB mappings can be rejected with ENOMEM
> Scope: NUMA + cpuset/MPOL_BIND + nr_overcommit_hugepages
> Risk: availability/configuration failure, no data corruption
>
>
> The fact that it has only been triggered after such a long time also
> precisely shows how uncommon its configuration is. I expect that
> backporting may encounter some conflicts. In the absence of clear user
> feedback, I personally suggest not doing a backport.
>
> Of course, as the author of this patch, you could also provide more
> information, for example, whether it was discovered as a real issue
> in a production environment, so as to decide whether to proceed with
> a backport.
We found this defect when doing the hugemmap10 test of LTP, it appears
with a certain probability (5%) in scenarios involving multiple NUMA nodes.
After debugging, I believe its occurrence is related to the location of
the regular huge page pool and the MPOL_BIND strategy used for allocating 
surplus pages.
So I moved to hugetlb-mmap with some simple setup steps to make it reproduce stably.

A more realistic scenario I have in mind is this, user uses huge pages in memory
pooling solutions like CXL memory expander or others, if there are already some 
regular
huge pages locates on the local node, but user wants to restrict application to use
surplus pages from remote nodes — and that's when the failure occurs.

CC linux-cxl.

BRs,
Huaisheng Ye