[PATCH v2 0/5] Fix bugs on HugeTLB folio allocation failure paths

[PATCH v2 0/5] Fix bugs on HugeTLB folio allocation failure paths

[Ackerley Tng via B4 Relay]

When mem_cgroup_charge_hugetlb() fails (e.g. when hitting memcg limits),
the HugeTLB allocation path currently handles the errors incorrectly.

This series fixes 5 issues related to HugeTLB allocation failure paths.

Some of these issues were pointed out by Sashiko while I was working on
[1].

0. The fix for (1.) below was in v1 of this series was incomplete, because it
   fixed the case where used_hpages was not restored, but introduced another
   issue, where subpool reservations are always restored even when a subpool
   reservation was not used. I looked into this more, I believe there's a deeper
   conceptual conflict in the way hugepage_subpool_put_pages() is used. I added
   a new patch to this series, which addresses the issue.

With the additional change, (1.) now does the correct thing. It always calls
hugepage_subpool_put_pages(), which restores used_hpages, and is able to
correctly skip updating reservations as well. The other patches fall in place
after this additional patch.

1. Fix subpool usage leak on allocation failure: (Sashiko pointed this out
   in [2])

   When alloc_hugetlb_folio() fails early (e.g. buddy allocation failure or
   hugetlb cgroup charging failure) and gbl_chg == 1, the error path skips
   restoring the page to the subpool, leaking the subpool's used_hpages
   counter. Fix this by calling hugepage_subpool_put_pages()
   unconditionally if map_chg is true.

(1.) above could technically be a separate patch series, but I'm relying on that
for fix (4.) below.

2. Fix folio refcount mismatch on memcg charge failure: (Sashiko pointed this
   out in [3])

   The error path in alloc_hugetlb_folio() calls free_huge_folio() directly
   on a folio with a refcount of 1 (set via folio_ref_unfreeze() earlier).
   This triggers VM_BUG_ON_FOLIO(folio_ref_count(folio), folio) if
   CONFIG_DEBUG_VM is enabled, and can corrupt allocator state otherwise.
   Fix this by using folio_put() instead of free_huge_folio() to correctly
   drop the refcount before freeing.

While fixing the above, the reproducer (I'll send these separately) was causing
an infinite loop during allocation, unveiling issue (3.).

3. Return -ENOSPC on memcg charge failure to prevent infinite loop:

   alloc_hugetlb_folio() propagates -ENOMEM on charge failure, which maps
   to VM_FAULT_OOM. Because HugeTLB physical allocations are high-order and
   use __GFP_RETRY_MAYFAIL, the OOM killer is bypassed. Returning
   VM_FAULT_OOM leaks to the #PF handler, which cannot make progress and
   retries the faulting instruction indefinitely. Fix this by returning
   -ENOSPC instead of -ENOMEM on charge failure, which maps to
   VM_FAULT_SIGBUS, terminating the process cleanly.

And while running the reproducer, I checked this routinely

    head /sys/kernel/mm/hugepages/hugepages-2048kB/*

and found that there was a resv_hugepages leak, 1 per execution of the
reproducer, unveiling (4.) below.

4. Move memcg charge earlier to prevent reservation leak:

   When mem_cgroup_charge_hugetlb() fails, the error path historically
   bypassed vma_end_reservation(). Since the reservation had already been
   committed via vma_commit_reservation(), this left the reservation map in
   an inconsistent state, leaking resv_huge_pages when the process
   exited. Fix this by moving mem_cgroup_charge_hugetlb() earlier in
   alloc_hugetlb_folio(), before vma_commit_reservation() is called.

Testing:

+ libhugetlbfs tests pass
+ ./tools/testing/selftests/mm/ksft_hugetlb.sh passes

v1: https://lore.kernel.org/r/20260707-hugetlb-alloc-failure-fixes-v1-0-5bbd3a4b836d@google.com

The series [1] has some changes that are dependent on these fixes, so I'll wait
for your reviews before continuing on [1].

Thank you!

[1] https://lore.kernel.org/all/20260702-hugetlb-open-up-v4-0-d53cefcccf34@google.com/T/
[2] https://sashiko.dev/#/patchset/20260518-hugetlb-open-up-v3-0-e14b302477f8%40google.com?part=5
[3] https://sashiko.dev/#/patchset/20260702-hugetlb-open-up-v4-0-d53cefcccf34%40google.com?part=6

Signed-off-by: Ackerley Tng <ackerleytng@google.com>
---
Ackerley Tng (5):
      mm: hugetlb: Track used_hpages when getting/putting pages from subpool
      mm: hugetlb: Fix subpool usage leak on allocation failure
      mm: hugetlb: Fix folio refcount mismatch on memcg charge failure
      mm: hugetlb: Return -ENOSPC on memcg charge failure
      mm: hugetlb: Move memcg charge earlier to prevent reservation leak

 fs/hugetlbfs/inode.c    |  8 +++--
 include/linux/hugetlb.h |  4 +--
 mm/hugetlb.c            | 79 +++++++++++++++++++++++++++----------------------
 3 files changed, 52 insertions(+), 39 deletions(-)
---
base-commit: 0e35b9b6ec0ffcc5e23cbdec09f5c622ad532b53
change-id: 20260706-hugetlb-alloc-failure-fixes-c7f775eca29f

Best regards,
--
Ackerley Tng <ackerleytng@google.com>

[PATCH v2 0/5] Fix bugs on HugeTLB folio allocation failure paths

[Ackerley Tng]

When mem_cgroup_charge_hugetlb() fails (e.g. when hitting memcg limits),
the HugeTLB allocation path currently handles the errors incorrectly.

This series fixes 5 issues related to HugeTLB allocation failure paths.

Some of these issues were pointed out by Sashiko while I was working on
[1].

0. The fix for (1.) below was in v1 of this series was incomplete, because it
   fixed the case where used_hpages was not restored, but introduced another
   issue, where subpool reservations are always restored even when a subpool
   reservation was not used. I looked into this more, I believe there's a deeper
   conceptual conflict in the way hugepage_subpool_put_pages() is used. I added
   a new patch to this series, which addresses the issue.

With the additional change, (1.) now does the correct thing. It always calls
hugepage_subpool_put_pages(), which restores used_hpages, and is able to
correctly skip updating reservations as well. The other patches fall in place
after this additional patch.

1. Fix subpool usage leak on allocation failure: (Sashiko pointed this out
   in [2])

   When alloc_hugetlb_folio() fails early (e.g. buddy allocation failure or
   hugetlb cgroup charging failure) and gbl_chg == 1, the error path skips
   restoring the page to the subpool, leaking the subpool's used_hpages
   counter. Fix this by calling hugepage_subpool_put_pages()
   unconditionally if map_chg is true.

(1.) above could technically be a separate patch series, but I'm relying on that
for fix (4.) below.

2. Fix folio refcount mismatch on memcg charge failure: (Sashiko pointed this
   out in [3])

   The error path in alloc_hugetlb_folio() calls free_huge_folio() directly
   on a folio with a refcount of 1 (set via folio_ref_unfreeze() earlier).
   This triggers VM_BUG_ON_FOLIO(folio_ref_count(folio), folio) if
   CONFIG_DEBUG_VM is enabled, and can corrupt allocator state otherwise.
   Fix this by using folio_put() instead of free_huge_folio() to correctly
   drop the refcount before freeing.

While fixing the above, the reproducer (I'll send these separately) was causing
an infinite loop during allocation, unveiling issue (3.).

3. Return -ENOSPC on memcg charge failure to prevent infinite loop:

   alloc_hugetlb_folio() propagates -ENOMEM on charge failure, which maps
   to VM_FAULT_OOM. Because HugeTLB physical allocations are high-order and
   use __GFP_RETRY_MAYFAIL, the OOM killer is bypassed. Returning
   VM_FAULT_OOM leaks to the #PF handler, which cannot make progress and
   retries the faulting instruction indefinitely. Fix this by returning
   -ENOSPC instead of -ENOMEM on charge failure, which maps to
   VM_FAULT_SIGBUS, terminating the process cleanly.

And while running the reproducer, I checked this routinely

    head /sys/kernel/mm/hugepages/hugepages-2048kB/*

and found that there was a resv_hugepages leak, 1 per execution of the
reproducer, unveiling (4.) below.

4. Move memcg charge earlier to prevent reservation leak:

   When mem_cgroup_charge_hugetlb() fails, the error path historically
   bypassed vma_end_reservation(). Since the reservation had already been
   committed via vma_commit_reservation(), this left the reservation map in
   an inconsistent state, leaking resv_huge_pages when the process
   exited. Fix this by moving mem_cgroup_charge_hugetlb() earlier in
   alloc_hugetlb_folio(), before vma_commit_reservation() is called.

Testing:

+ libhugetlbfs tests pass
+ ./tools/testing/selftests/mm/ksft_hugetlb.sh passes

v1: https://lore.kernel.org/r/20260707-hugetlb-alloc-failure-fixes-v1-0-5bbd3a4b836d@google.com

The series [1] has some changes that are dependent on these fixes, so I'll wait
for your reviews before continuing on [1].

Thank you!

[1] https://lore.kernel.org/all/20260702-hugetlb-open-up-v4-0-d53cefcccf34@google.com/T/
[2] https://sashiko.dev/#/patchset/20260518-hugetlb-open-up-v3-0-e14b302477f8%40google.com?part=5
[3] https://sashiko.dev/#/patchset/20260702-hugetlb-open-up-v4-0-d53cefcccf34%40google.com?part=6

Signed-off-by: Ackerley Tng <ackerleytng@google.com>
---
Ackerley Tng (5):
      mm: hugetlb: Track used_hpages when getting/putting pages from subpool
      mm: hugetlb: Fix subpool usage leak on allocation failure
      mm: hugetlb: Fix folio refcount mismatch on memcg charge failure
      mm: hugetlb: Return -ENOSPC on memcg charge failure
      mm: hugetlb: Move memcg charge earlier to prevent reservation leak

 fs/hugetlbfs/inode.c    |  8 +++--
 include/linux/hugetlb.h |  4 +--
 mm/hugetlb.c            | 79 +++++++++++++++++++++++++++----------------------
 3 files changed, 52 insertions(+), 39 deletions(-)
---
base-commit: 0e35b9b6ec0ffcc5e23cbdec09f5c622ad532b53
change-id: 20260706-hugetlb-alloc-failure-fixes-c7f775eca29f

Best regards,
--
Ackerley Tng <ackerleytng@google.com>

[PATCH v2 1/5] mm: hugetlb: Track used_hpages when getting/putting pages from subpool

[Ackerley Tng via B4 Relay]

From: Ackerley Tng <ackerleytng@google.com>

hugepage_subpool_put_pages() currently has two distinct responsibilities
that conflict:

1. When size is specified for the mount, max_hpages != -1: Keep track of
   total active pages (allocated + reserved) and decrement this count
   (used_hpages) when a page is freed or allocation fails.
2. When min_size is specified for the mount, min_hpages != -1: Ensure we
   don't drop below the guaranteed minimum, and restore a reservation
   (rsv_hpages) if we do.

This causes trouble because when allocation fails (refer to
alloc_hugetlb_folio()) if gbl_chg = 1 (i.e. no subpool reservation was
taken):

+ To keep used_hpages consistent, HugeTLB needs to call
  hugepage_subpool_put_pages() to restore undo used_hpages being
  incremented
+ But can't call hugepage_subpool_put_pages() if no reservation was
  consumed.

One option would be to conditionally do subpool tracking updates outside of
the hugepage_subpool_put_pages() function, but that would spread logic all
over.

Instead, always track used_hpages, regardless of whether a max_size was
requested for the mount, so that the subpool always knows how many pages
were allocated through it. Every page allocated through the subpool
increments used_hpages, regardless of whether a reservation was taken from
it.

Conceptually, now, every allocation involving a subpool uses a page from
the subpool, which must be returned to the subpool. Every page taken from
the subpool tries to use a subpool reservation. Restoring a page to the
subpool reservations only if the page was taken from subpool
reservations. (If used_hpages >= min_hpages, the page must have not have
been taken from the reservations.)

Always tracking used_hpages provides the subpool with information of both
used and reserved counts to make the correct decision for both max_size and
min_size correctly.

With used_hpages always tracked, subpool_is_free() can be simplified, such
that the subpool can be declared free if there are no more pages in use.

Also update the documentation for used_hpages, since it no longer matters
whether the used pages count against the maximum.

Also update statfs reporting. Previously, if max_hpages is negative,
used_hpages is static at 0, so returning max_hpages - used_hpages returns
-1 and is always correct. Now, if the subpool doesn't have a maximum
requested size, indicate no limit for free pages (-1). If it does have a
maximum size, report the difference between the requested size and the
number of used pages. This difference is always positive, because if the
mount does have a maximum size, hugepage_subpool_get_pages() ensures that
the subpool usage never exceeds the maximum.

This fixes a bug in hugetlb_unreserve_pages(), where pages are returned to
the subpool regardless of whether it consumed a reservation. The
corresponding bug in the failure handling path of alloc_hugetlb_folio() was
fixed in a833a693a490e.

Fixes: 1c5ecae3a93fa ("hugetlbfs: add minimum size accounting to subpools")
Cc: stable@vger.kernel.org
Signed-off-by: Ackerley Tng <ackerleytng@google.com>
---
 fs/hugetlbfs/inode.c    |  8 ++++++--
 include/linux/hugetlb.h |  4 ++--
 mm/hugetlb.c            | 22 ++++++++--------------
 3 files changed, 16 insertions(+), 18 deletions(-)

diff --git a/fs/hugetlbfs/inode.c b/fs/hugetlbfs/inode.c
index 216e1a0dd0b23..26c0187340636 100644
--- a/fs/hugetlbfs/inode.c
+++ b/fs/hugetlbfs/inode.c
@@ -1109,8 +1109,12 @@ static int hugetlbfs_statfs(struct dentry *dentry, struct kstatfs *buf)
 
 			spin_lock_irq(&sbinfo->spool->lock);
 			buf->f_blocks = sbinfo->spool->max_hpages;
-			free_pages = sbinfo->spool->max_hpages
-				- sbinfo->spool->used_hpages;
+			if (sbinfo->spool->max_hpages == -1) {
+				free_pages = -1;
+			} else {
+				free_pages = sbinfo->spool->max_hpages -
+					     sbinfo->spool->used_hpages;
+			}
 			buf->f_bavail = buf->f_bfree = free_pages;
 			spin_unlock_irq(&sbinfo->spool->lock);
 			buf->f_files = sbinfo->max_inodes;
diff --git a/include/linux/hugetlb.h b/include/linux/hugetlb.h
index 2abaf99321e90..34b9a3e1be0fa 100644
--- a/include/linux/hugetlb.h
+++ b/include/linux/hugetlb.h
@@ -38,8 +38,8 @@ struct hugepage_subpool {
 	spinlock_t lock;
 	long count;
 	long max_hpages;	/* Maximum huge pages or -1 if no maximum. */
-	long used_hpages;	/* Used count against maximum, includes */
-				/* both allocated and reserved pages. */
+	long used_hpages;	/* Used page count, includes both */
+				/* allocated and reserved pages. */
 	struct hstate *hstate;
 	long min_hpages;	/* Minimum huge pages or -1 if no minimum. */
 	long rsv_hpages;	/* Pages reserved against global pool to */
diff --git a/mm/hugetlb.c b/mm/hugetlb.c
index 571212b80835e..ee5e99c1894b9 100644
--- a/mm/hugetlb.c
+++ b/mm/hugetlb.c
@@ -129,12 +129,8 @@ static inline bool subpool_is_free(struct hugepage_subpool *spool)
 {
 	if (spool->count)
 		return false;
-	if (spool->max_hpages != -1)
-		return spool->used_hpages == 0;
-	if (spool->min_hpages != -1)
-		return spool->rsv_hpages == spool->min_hpages;
 
-	return true;
+	return spool->used_hpages == 0;
 }
 
 static inline void unlock_or_release_subpool(struct hugepage_subpool *spool,
@@ -205,15 +201,14 @@ static long hugepage_subpool_get_pages(struct hugepage_subpool *spool,
 
 	spin_lock_irq(&spool->lock);
 
-	if (spool->max_hpages != -1) {		/* maximum size accounting */
-		if ((spool->used_hpages + delta) <= spool->max_hpages)
-			spool->used_hpages += delta;
-		else {
-			ret = -ENOMEM;
-			goto unlock_ret;
-		}
+	if (spool->max_hpages != -1 &&
+	    spool->used_hpages + delta > spool->max_hpages) {
+		ret = -ENOMEM;
+		goto unlock_ret;
 	}
 
+	spool->used_hpages += delta;
+
 	/* minimum size accounting */
 	if (spool->min_hpages != -1 && spool->rsv_hpages) {
 		if (delta > spool->rsv_hpages) {
@@ -251,8 +246,7 @@ static long hugepage_subpool_put_pages(struct hugepage_subpool *spool,
 
 	spin_lock_irqsave(&spool->lock, flags);
 
-	if (spool->max_hpages != -1)		/* maximum size accounting */
-		spool->used_hpages -= delta;
+	spool->used_hpages -= delta;
 
 	 /* minimum size accounting */
 	if (spool->min_hpages != -1 && spool->used_hpages < spool->min_hpages) {

-- 
2.55.0.795.g602f6c329a-goog

[PATCH v2 1/5] mm: hugetlb: Track used_hpages when getting/putting pages from subpool

[Ackerley Tng]

hugepage_subpool_put_pages() currently has two distinct responsibilities
that conflict:

1. When size is specified for the mount, max_hpages != -1: Keep track of
   total active pages (allocated + reserved) and decrement this count
   (used_hpages) when a page is freed or allocation fails.
2. When min_size is specified for the mount, min_hpages != -1: Ensure we
   don't drop below the guaranteed minimum, and restore a reservation
   (rsv_hpages) if we do.

This causes trouble because when allocation fails (refer to
alloc_hugetlb_folio()) if gbl_chg = 1 (i.e. no subpool reservation was
taken):

+ To keep used_hpages consistent, HugeTLB needs to call
  hugepage_subpool_put_pages() to restore undo used_hpages being
  incremented
+ But can't call hugepage_subpool_put_pages() if no reservation was
  consumed.

One option would be to conditionally do subpool tracking updates outside of
the hugepage_subpool_put_pages() function, but that would spread logic all
over.

Instead, always track used_hpages, regardless of whether a max_size was
requested for the mount, so that the subpool always knows how many pages
were allocated through it. Every page allocated through the subpool
increments used_hpages, regardless of whether a reservation was taken from
it.

Conceptually, now, every allocation involving a subpool uses a page from
the subpool, which must be returned to the subpool. Every page taken from
the subpool tries to use a subpool reservation. Restoring a page to the
subpool reservations only if the page was taken from subpool
reservations. (If used_hpages >= min_hpages, the page must have not have
been taken from the reservations.)

Always tracking used_hpages provides the subpool with information of both
used and reserved counts to make the correct decision for both max_size and
min_size correctly.

With used_hpages always tracked, subpool_is_free() can be simplified, such
that the subpool can be declared free if there are no more pages in use.

Also update the documentation for used_hpages, since it no longer matters
whether the used pages count against the maximum.

Also update statfs reporting. Previously, if max_hpages is negative,
used_hpages is static at 0, so returning max_hpages - used_hpages returns
-1 and is always correct. Now, if the subpool doesn't have a maximum
requested size, indicate no limit for free pages (-1). If it does have a
maximum size, report the difference between the requested size and the
number of used pages. This difference is always positive, because if the
mount does have a maximum size, hugepage_subpool_get_pages() ensures that
the subpool usage never exceeds the maximum.

This fixes a bug in hugetlb_unreserve_pages(), where pages are returned to
the subpool regardless of whether it consumed a reservation. The
corresponding bug in the failure handling path of alloc_hugetlb_folio() was
fixed in a833a693a490e.

Fixes: 1c5ecae3a93fa ("hugetlbfs: add minimum size accounting to subpools")
Cc: stable@vger.kernel.org
Signed-off-by: Ackerley Tng <ackerleytng@google.com>
---
 fs/hugetlbfs/inode.c    |  8 ++++++--
 include/linux/hugetlb.h |  4 ++--
 mm/hugetlb.c            | 22 ++++++++--------------
 3 files changed, 16 insertions(+), 18 deletions(-)

diff --git a/fs/hugetlbfs/inode.c b/fs/hugetlbfs/inode.c
index 216e1a0dd0b23..26c0187340636 100644
--- a/fs/hugetlbfs/inode.c
+++ b/fs/hugetlbfs/inode.c
@@ -1109,8 +1109,12 @@ static int hugetlbfs_statfs(struct dentry *dentry, struct kstatfs *buf)
 
 			spin_lock_irq(&sbinfo->spool->lock);
 			buf->f_blocks = sbinfo->spool->max_hpages;
-			free_pages = sbinfo->spool->max_hpages
-				- sbinfo->spool->used_hpages;
+			if (sbinfo->spool->max_hpages == -1) {
+				free_pages = -1;
+			} else {
+				free_pages = sbinfo->spool->max_hpages -
+					     sbinfo->spool->used_hpages;
+			}
 			buf->f_bavail = buf->f_bfree = free_pages;
 			spin_unlock_irq(&sbinfo->spool->lock);
 			buf->f_files = sbinfo->max_inodes;
diff --git a/include/linux/hugetlb.h b/include/linux/hugetlb.h
index 2abaf99321e90..34b9a3e1be0fa 100644
--- a/include/linux/hugetlb.h
+++ b/include/linux/hugetlb.h
@@ -38,8 +38,8 @@ struct hugepage_subpool {
 	spinlock_t lock;
 	long count;
 	long max_hpages;	/* Maximum huge pages or -1 if no maximum. */
-	long used_hpages;	/* Used count against maximum, includes */
-				/* both allocated and reserved pages. */
+	long used_hpages;	/* Used page count, includes both */
+				/* allocated and reserved pages. */
 	struct hstate *hstate;
 	long min_hpages;	/* Minimum huge pages or -1 if no minimum. */
 	long rsv_hpages;	/* Pages reserved against global pool to */
diff --git a/mm/hugetlb.c b/mm/hugetlb.c
index 571212b80835e..ee5e99c1894b9 100644
--- a/mm/hugetlb.c
+++ b/mm/hugetlb.c
@@ -129,12 +129,8 @@ static inline bool subpool_is_free(struct hugepage_subpool *spool)
 {
 	if (spool->count)
 		return false;
-	if (spool->max_hpages != -1)
-		return spool->used_hpages == 0;
-	if (spool->min_hpages != -1)
-		return spool->rsv_hpages == spool->min_hpages;
 
-	return true;
+	return spool->used_hpages == 0;
 }
 
 static inline void unlock_or_release_subpool(struct hugepage_subpool *spool,
@@ -205,15 +201,14 @@ static long hugepage_subpool_get_pages(struct hugepage_subpool *spool,
 
 	spin_lock_irq(&spool->lock);
 
-	if (spool->max_hpages != -1) {		/* maximum size accounting */
-		if ((spool->used_hpages + delta) <= spool->max_hpages)
-			spool->used_hpages += delta;
-		else {
-			ret = -ENOMEM;
-			goto unlock_ret;
-		}
+	if (spool->max_hpages != -1 &&
+	    spool->used_hpages + delta > spool->max_hpages) {
+		ret = -ENOMEM;
+		goto unlock_ret;
 	}
 
+	spool->used_hpages += delta;
+
 	/* minimum size accounting */
 	if (spool->min_hpages != -1 && spool->rsv_hpages) {
 		if (delta > spool->rsv_hpages) {
@@ -251,8 +246,7 @@ static long hugepage_subpool_put_pages(struct hugepage_subpool *spool,
 
 	spin_lock_irqsave(&spool->lock, flags);
 
-	if (spool->max_hpages != -1)		/* maximum size accounting */
-		spool->used_hpages -= delta;
+	spool->used_hpages -= delta;
 
 	 /* minimum size accounting */
 	if (spool->min_hpages != -1 && spool->used_hpages < spool->min_hpages) {

-- 
2.55.0.795.g602f6c329a-goog

[PATCH v2 2/5] mm: hugetlb: Fix subpool usage leak on allocation failure

[Ackerley Tng via B4 Relay]

From: Ackerley Tng <ackerleytng@google.com>

When alloc_hugetlb_folio() fails early (e.g. buddy allocation failure or
hugetlb cgroup charging failure) and gbl_chg == 1 (meaning a reservation
was not used, but a global page was allocated instead), the subpool page
acquired via hugepage_subpool_get_pages() must still be returned.

Currently, the error path out_subpool_put: only calls
hugepage_subpool_put_pages() if !gbl_chg is true. If gbl_chg is 1, it
skips it, permanently leaking the subpool's used_hpages counter.

With the earlier patch to always track used_hpages in the subpool, always
call hugepage_subpool_put_pages() if map_chg is true to consistently
restore the page to the subpool. Only call hugetlb_acct_memory() to adjust
global reservations if gbl_chg == 0 since gbl_chg == 0 indicates a
subpool (and global) reservation was used.

Fixes: a833a693a490e ("mm: hugetlb: fix incorrect fallback for subpool")
Cc: stable@vger.kernel.org
Signed-off-by: Ackerley Tng <ackerleytng@google.com>
---
 mm/hugetlb.c | 14 ++++++--------
 1 file changed, 6 insertions(+), 8 deletions(-)

diff --git a/mm/hugetlb.c b/mm/hugetlb.c
index ee5e99c1894b9..4093c1c0a4a1d 100644
--- a/mm/hugetlb.c
+++ b/mm/hugetlb.c
@@ -2852,7 +2852,7 @@ struct folio *alloc_hugetlb_folio(struct vm_area_struct *vma,
 	struct hugepage_subpool *spool = subpool_vma(vma);
 	struct hstate *h = hstate_vma(vma);
 	struct folio *folio;
-	long retval, gbl_chg, gbl_reserve;
+	long retval, gbl_chg;
 	map_chg_state map_chg;
 	int ret, idx;
 	struct hugetlb_cgroup *h_cg = NULL;
@@ -3003,13 +3003,11 @@ struct folio *alloc_hugetlb_folio(struct vm_area_struct *vma,
 		hugetlb_cgroup_uncharge_cgroup_rsvd(idx, pages_per_huge_page(h),
 						    h_cg_rsvd);
 out_subpool_put:
-	/*
-	 * put page to subpool iff the quota of subpool's rsv_hpages is used
-	 * during hugepage_subpool_get_pages.
-	 */
-	if (map_chg && !gbl_chg) {
-		gbl_reserve = hugepage_subpool_put_pages(spool, 1);
-		hugetlb_acct_memory(h, -gbl_reserve);
+	if (map_chg) {
+		long gbl_reserve = hugepage_subpool_put_pages(spool, 1);
+
+		if (!gbl_chg)
+			hugetlb_acct_memory(h, -gbl_reserve);
 	}
 
 

-- 
2.55.0.795.g602f6c329a-goog

[PATCH v2 2/5] mm: hugetlb: Fix subpool usage leak on allocation failure

[Ackerley Tng]

When alloc_hugetlb_folio() fails early (e.g. buddy allocation failure or
hugetlb cgroup charging failure) and gbl_chg == 1 (meaning a reservation
was not used, but a global page was allocated instead), the subpool page
acquired via hugepage_subpool_get_pages() must still be returned.

Currently, the error path out_subpool_put: only calls
hugepage_subpool_put_pages() if !gbl_chg is true. If gbl_chg is 1, it
skips it, permanently leaking the subpool's used_hpages counter.

With the earlier patch to always track used_hpages in the subpool, always
call hugepage_subpool_put_pages() if map_chg is true to consistently
restore the page to the subpool. Only call hugetlb_acct_memory() to adjust
global reservations if gbl_chg == 0 since gbl_chg == 0 indicates a
subpool (and global) reservation was used.

Fixes: a833a693a490e ("mm: hugetlb: fix incorrect fallback for subpool")
Cc: stable@vger.kernel.org
Signed-off-by: Ackerley Tng <ackerleytng@google.com>
---
 mm/hugetlb.c | 14 ++++++--------
 1 file changed, 6 insertions(+), 8 deletions(-)

diff --git a/mm/hugetlb.c b/mm/hugetlb.c
index ee5e99c1894b9..4093c1c0a4a1d 100644
--- a/mm/hugetlb.c
+++ b/mm/hugetlb.c
@@ -2852,7 +2852,7 @@ struct folio *alloc_hugetlb_folio(struct vm_area_struct *vma,
 	struct hugepage_subpool *spool = subpool_vma(vma);
 	struct hstate *h = hstate_vma(vma);
 	struct folio *folio;
-	long retval, gbl_chg, gbl_reserve;
+	long retval, gbl_chg;
 	map_chg_state map_chg;
 	int ret, idx;
 	struct hugetlb_cgroup *h_cg = NULL;
@@ -3003,13 +3003,11 @@ struct folio *alloc_hugetlb_folio(struct vm_area_struct *vma,
 		hugetlb_cgroup_uncharge_cgroup_rsvd(idx, pages_per_huge_page(h),
 						    h_cg_rsvd);
 out_subpool_put:
-	/*
-	 * put page to subpool iff the quota of subpool's rsv_hpages is used
-	 * during hugepage_subpool_get_pages.
-	 */
-	if (map_chg && !gbl_chg) {
-		gbl_reserve = hugepage_subpool_put_pages(spool, 1);
-		hugetlb_acct_memory(h, -gbl_reserve);
+	if (map_chg) {
+		long gbl_reserve = hugepage_subpool_put_pages(spool, 1);
+
+		if (!gbl_chg)
+			hugetlb_acct_memory(h, -gbl_reserve);
 	}
 
 

-- 
2.55.0.795.g602f6c329a-goog

Re: [PATCH v2 2/5] mm: hugetlb: Fix subpool usage leak on allocation failure

[Joshua Hahn]

Hi Ackerley,

Thank you for this series. I really wanted to work on hugeTLB accounting
fixes but never got the time to get to it. I'm very grateful that you
are taking a look!!

> From: Ackerley Tng <ackerleytng@google.com>
> 
> When alloc_hugetlb_folio() fails early (e.g. buddy allocation failure or
> hugetlb cgroup charging failure) and gbl_chg == 1 (meaning a reservation
> was not used, but a global page was allocated instead), the subpool page
> acquired via hugepage_subpool_get_pages() must still be returned.
> 
> Currently, the error path out_subpool_put: only calls
> hugepage_subpool_put_pages() if !gbl_chg is true. If gbl_chg is 1, it
> skips it, permanently leaking the subpool's used_hpages counter.
> 
> With the earlier patch to always track used_hpages in the subpool, always
> call hugepage_subpool_put_pages() if map_chg is true to consistently
> restore the page to the subpool. Only call hugetlb_acct_memory() to adjust
> global reservations if gbl_chg == 0 since gbl_chg == 0 indicates a
> subpool (and global) reservation was used.

So I think that I've seen that this part of the accounting specifically
is a bit suspicious. There have been two attempts in the past to fix
this area [1] [2]. I think functionally they are quite similar to this
fix, they just open-code the contents of the put_pages function inside
the condition. I've Cc-ed the authors of those two patches in case
they wanted to chime in.

I reference these fixes because I think they handle the minimum
subpage case a bit differently. To be honest, I recall reading those
fixes a while back and getting a bit confused on what exactly happens
when the page is absorbed to fulfill the minimum size...

It does seem like Sashiko also notes this as a possible concern.
WDYT? Does your reproducer for this issue also work when a minimum
size is set (let's say, to 1?)

Thanks again. I hope you have a great day!!!
Joshua

> Fixes: a833a693a490e ("mm: hugetlb: fix incorrect fallback for subpool")
> Cc: stable@vger.kernel.org
> Signed-off-by: Ackerley Tng <ackerleytng@google.com>
> ---
>  mm/hugetlb.c | 14 ++++++--------
>  1 file changed, 6 insertions(+), 8 deletions(-)
> 
> diff --git a/mm/hugetlb.c b/mm/hugetlb.c
> index ee5e99c1894b9..4093c1c0a4a1d 100644
> --- a/mm/hugetlb.c
> +++ b/mm/hugetlb.c
> @@ -2852,7 +2852,7 @@ struct folio *alloc_hugetlb_folio(struct vm_area_struct *vma,
>  	struct hugepage_subpool *spool = subpool_vma(vma);
>  	struct hstate *h = hstate_vma(vma);
>  	struct folio *folio;
> -	long retval, gbl_chg, gbl_reserve;
> +	long retval, gbl_chg;
>  	map_chg_state map_chg;
>  	int ret, idx;
>  	struct hugetlb_cgroup *h_cg = NULL;
> @@ -3003,13 +3003,11 @@ struct folio *alloc_hugetlb_folio(struct vm_area_struct *vma,
>  		hugetlb_cgroup_uncharge_cgroup_rsvd(idx, pages_per_huge_page(h),
>  						    h_cg_rsvd);
>  out_subpool_put:
> -	/*
> -	 * put page to subpool iff the quota of subpool's rsv_hpages is used
> -	 * during hugepage_subpool_get_pages.
> -	 */
> -	if (map_chg && !gbl_chg) {
> -		gbl_reserve = hugepage_subpool_put_pages(spool, 1);
> -		hugetlb_acct_memory(h, -gbl_reserve);
> +	if (map_chg) {
> +		long gbl_reserve = hugepage_subpool_put_pages(spool, 1);
> +
> +		if (!gbl_chg)
> +			hugetlb_acct_memory(h, -gbl_reserve);
>  	}

[1] https://lore.kernel.org/linux-mm/20260428113037.88766-2-enderaoelyther@gmail.com/
[2] https://lore.kernel.org/linux-mm/20260515202902.461539-1-devnexen@gmail.com/

Re: [PATCH v2 2/5] mm: hugetlb: Fix subpool usage leak on allocation failure

[Ackerley Tng]

Joshua Hahn <joshua.hahnjy@gmail.com> writes:

> Hi Ackerley,
>
> Thank you for this series. I really wanted to work on hugeTLB accounting
> fixes but never got the time to get to it. I'm very grateful that you
> are taking a look!!
>
>> From: Ackerley Tng <ackerleytng@google.com>
>>
>> When alloc_hugetlb_folio() fails early (e.g. buddy allocation failure or
>> hugetlb cgroup charging failure) and gbl_chg == 1 (meaning a reservation
>> was not used, but a global page was allocated instead), the subpool page
>> acquired via hugepage_subpool_get_pages() must still be returned.
>>
>> Currently, the error path out_subpool_put: only calls
>> hugepage_subpool_put_pages() if !gbl_chg is true. If gbl_chg is 1, it
>> skips it, permanently leaking the subpool's used_hpages counter.
>>
>> With the earlier patch to always track used_hpages in the subpool, always
>> call hugepage_subpool_put_pages() if map_chg is true to consistently
>> restore the page to the subpool. Only call hugetlb_acct_memory() to adjust
>> global reservations if gbl_chg == 0 since gbl_chg == 0 indicates a
>> subpool (and global) reservation was used.
>
> So I think that I've seen that this part of the accounting specifically
> is a bit suspicious. There have been two attempts in the past to fix
> this area [1] [2]. I think functionally they are quite similar to this
> fix, they just open-code the contents of the put_pages function inside
> the condition. I've Cc-ed the authors of those two patches in case
> they wanted to chime in.
>

Thanks for connecting us! I didn't realize this was already being worked
on. Also adding Lance, who commented at [3].

> I reference these fixes because I think they handle the minimum
> subpage case a bit differently. To be honest, I recall reading those
> fixes a while back and getting a bit confused on what exactly happens
> when the page is absorbed to fulfill the minimum size...
>
> It does seem like Sashiko also notes this as a possible concern.
> WDYT? Does your reproducer for this issue also work when a minimum
> size is set (let's say, to 1?)
>

Let me look into this more!

> Thanks again. I hope you have a great day!!!
> Joshua
>
>> Fixes: a833a693a490e ("mm: hugetlb: fix incorrect fallback for subpool")
>> Cc: stable@vger.kernel.org
>> Signed-off-by: Ackerley Tng <ackerleytng@google.com>
>> ---
>>  mm/hugetlb.c | 14 ++++++--------
>>  1 file changed, 6 insertions(+), 8 deletions(-)
>>
>> diff --git a/mm/hugetlb.c b/mm/hugetlb.c
>> index ee5e99c1894b9..4093c1c0a4a1d 100644
>> --- a/mm/hugetlb.c
>> +++ b/mm/hugetlb.c
>> @@ -2852,7 +2852,7 @@ struct folio *alloc_hugetlb_folio(struct vm_area_struct *vma,
>>  	struct hugepage_subpool *spool = subpool_vma(vma);
>>  	struct hstate *h = hstate_vma(vma);
>>  	struct folio *folio;
>> -	long retval, gbl_chg, gbl_reserve;
>> +	long retval, gbl_chg;
>>  	map_chg_state map_chg;
>>  	int ret, idx;
>>  	struct hugetlb_cgroup *h_cg = NULL;
>> @@ -3003,13 +3003,11 @@ struct folio *alloc_hugetlb_folio(struct vm_area_struct *vma,
>>  		hugetlb_cgroup_uncharge_cgroup_rsvd(idx, pages_per_huge_page(h),
>>  						    h_cg_rsvd);
>>  out_subpool_put:
>> -	/*
>> -	 * put page to subpool iff the quota of subpool's rsv_hpages is used
>> -	 * during hugepage_subpool_get_pages.
>> -	 */
>> -	if (map_chg && !gbl_chg) {
>> -		gbl_reserve = hugepage_subpool_put_pages(spool, 1);
>> -		hugetlb_acct_memory(h, -gbl_reserve);
>> +	if (map_chg) {
>> +		long gbl_reserve = hugepage_subpool_put_pages(spool, 1);
>> +
>> +		if (!gbl_chg)
>> +			hugetlb_acct_memory(h, -gbl_reserve);
>>  	}
>
> [1] https://lore.kernel.org/linux-mm/20260428113037.88766-2-enderaoelyther@gmail.com/
> [2] https://lore.kernel.org/linux-mm/20260515202902.461539-1-devnexen@gmail.com/
[3] https://lore.kernel.org/linux-mm/20260428113059.79001-1-lance.yang@linux.dev/

Re: [PATCH v2 2/5] mm: hugetlb: Fix subpool usage leak on allocation failure

[Ackerley Tng]

Ackerley Tng <ackerleytng@google.com> writes:

> Joshua Hahn <joshua.hahnjy@gmail.com> writes:
>
>> Hi Ackerley,
>>
>> Thank you for this series. I really wanted to work on hugeTLB accounting
>> fixes but never got the time to get to it. I'm very grateful that you
>> are taking a look!!
>>
>>> From: Ackerley Tng <ackerleytng@google.com>
>>>
>>> When alloc_hugetlb_folio() fails early (e.g. buddy allocation failure or
>>> hugetlb cgroup charging failure) and gbl_chg == 1 (meaning a reservation
>>> was not used, but a global page was allocated instead), the subpool page
>>> acquired via hugepage_subpool_get_pages() must still be returned.
>>>
>>> Currently, the error path out_subpool_put: only calls
>>> hugepage_subpool_put_pages() if !gbl_chg is true. If gbl_chg is 1, it
>>> skips it, permanently leaking the subpool's used_hpages counter.
>>>
>>> With the earlier patch to always track used_hpages in the subpool, always
>>> call hugepage_subpool_put_pages() if map_chg is true to consistently
>>> restore the page to the subpool. Only call hugetlb_acct_memory() to adjust
>>> global reservations if gbl_chg == 0 since gbl_chg == 0 indicates a
>>> subpool (and global) reservation was used.
>>
>> So I think that I've seen that this part of the accounting specifically
>> is a bit suspicious. There have been two attempts in the past to fix
>> this area [1] [2]. I think functionally they are quite similar to this
>> fix, they just open-code the contents of the put_pages function inside
>> the condition. I've Cc-ed the authors of those two patches in case
>> they wanted to chime in.
>>
>
> Thanks for connecting us! I didn't realize this was already being worked
> on. Also adding Lance, who commented at [3].
>
>> I reference these fixes because I think they handle the minimum
>> subpage case a bit differently. To be honest, I recall reading those
>> fixes a while back and getting a bit confused on what exactly happens
>> when the page is absorbed to fulfill the minimum size...
>>
>> It does seem like Sashiko also notes this as a possible concern.
>> WDYT? Does your reproducer for this issue also work when a minimum
>> size is set (let's say, to 1?)
>>
>
> Let me look into this more!
>

I was looking at Sashiko's comment [4] and Sashiko is right that there
could be a race. Specifically, at the time of
hugepage_subpool_get_page(), I might be using a reservation, hence
returning 1 (thread A), but at the time of hugepage_subpool_put_page(),
some other thread (B) might already have restored a reservation to the
subpool.

With used_hpages tracking within the subpool, we can rely on the return value of
hugepage_subpool_put_page():

+ Return 1: Reservation should be returned elsewhere (pool has enough pages to
  satisfy minimum, or doesn't track minimums)
+ Return 0: Reservation does not need to be returned elsewhere

But calling hugepage_subpool_put_pages(1) doesn't say anything about whether the
1 (in this case, 1 doesn't always represent a page, since specifically on the
error cases, the page wasn't allocated) consumed a reservation or not.

+ gbl_chg == 0 means a reservation was used, and
+ gbl_chg == 1 means a reservation *should be used*, but
+ gbl_chg == 1 does not necessarily mean a reservation should be used, since
  h->resv_huge_pages may not have been decremented yet. (See goto
  out_subpool_put and goto out_uncharge_cgroup_reservation in
  alloc_hugetlb_folio())

So, we need to track if h->resv_huge_pages-- happened (with some local variable
like reservation_consumed)

| reservation_consumed | hugepage_subpool_put_pages() return value |
h->resv_huge_pages++ |
|----------------------|-------------------------------------------|----------------------|
| 1                    | 1                                         |
Yes                  |
| 1                    | 0                                         |
No                   |
| 0                    | 1                                         |
No                   |
| 0                    | 0                                         |
No                   |

TLDR: replace !gbl_chg with reservation_consumed?

    <<== reservation_consumed = true

	if (!gbl_chg) {
		folio_set_hugetlb_restore_reserve(folio);
		h->resv_huge_pages--;
        <<== reservation_consumed = true
	}

	if (map_chg) {
		long gbl_reserve = hugepage_subpool_put_pages(spool, 1);

		if (!gbl_chg)  <<== this should be reservation_consumed
			hugetlb_acct_memory(h, -gbl_reserve);
	}

I believe this depends on always tracking used_hpages in subpools, in order to
know for sure, just based on a number (with no page information), if the page is
supposed to use a reservation.


Perhaps related: does it make sense to flipping the reservation tracking
to instead track an "available" page count? This is basically the
mathematical opposite of rsvd_huge_pages, as in,

  available_pages = free_huge_pages - resv_huge_pages

This way, there's no need to synchronize the number of reserved pages in
both the subpool and global hstate.

It's quite confusing now, that reserved_pages-- can be either "used a
reservation" or "unreserve". If we track the opposite, available_pages--
will definitely mean the page's availability is removed from the global
hstate and transferred to the subpool, and allocating page doesn't
involve updating both the subpool and hstate.

One downside I can think of is that creating hugepages (surplus, or
promote/demote, or hotplug, or runtime echo 1 > nr_hugepages) would need
to update the global hstate's available_pages count, but I imagine that
would be less frequent than allocating huge pages?

>>
>> [...snip...]
>>
>>
>> [1] https://lore.kernel.org/linux-mm/20260428113037.88766-2-enderaoelyther@gmail.com/
>> [2] https://lore.kernel.org/linux-mm/20260515202902.461539-1-devnexen@gmail.com/
> [3] https://lore.kernel.org/linux-mm/20260428113059.79001-1-lance.yang@linux.dev/
[4] https://sashiko.dev/#/patchset/20260708-hugetlb-alloc-failure-fixes-v2-0-c7f27cbb462b%40google.com?part=2,

Re: [PATCH v2 2/5] mm: hugetlb: Fix subpool usage leak on allocation failure

[Joshua Hahn]

On Thu, 9 Jul 2026 15:15:50 -0700 Ackerley Tng <ackerleytng@google.com> wrote:

> Ackerley Tng <ackerleytng@google.com> writes:
> 
> > Joshua Hahn <joshua.hahnjy@gmail.com> writes:
> >
> >> Hi Ackerley,
> >>
> >> Thank you for this series. I really wanted to work on hugeTLB accounting
> >> fixes but never got the time to get to it. I'm very grateful that you
> >> are taking a look!!
> >>
> >>> From: Ackerley Tng <ackerleytng@google.com>
> >>>
> >>> When alloc_hugetlb_folio() fails early (e.g. buddy allocation failure or
> >>> hugetlb cgroup charging failure) and gbl_chg == 1 (meaning a reservation
> >>> was not used, but a global page was allocated instead), the subpool page
> >>> acquired via hugepage_subpool_get_pages() must still be returned.
> >>>
> >>> Currently, the error path out_subpool_put: only calls
> >>> hugepage_subpool_put_pages() if !gbl_chg is true. If gbl_chg is 1, it
> >>> skips it, permanently leaking the subpool's used_hpages counter.
> >>>
> >>> With the earlier patch to always track used_hpages in the subpool, always
> >>> call hugepage_subpool_put_pages() if map_chg is true to consistently
> >>> restore the page to the subpool. Only call hugetlb_acct_memory() to adjust
> >>> global reservations if gbl_chg == 0 since gbl_chg == 0 indicates a
> >>> subpool (and global) reservation was used.
> >>
> >> So I think that I've seen that this part of the accounting specifically
> >> is a bit suspicious. There have been two attempts in the past to fix
> >> this area [1] [2]. I think functionally they are quite similar to this
> >> fix, they just open-code the contents of the put_pages function inside
> >> the condition. I've Cc-ed the authors of those two patches in case
> >> they wanted to chime in.
> >>
> >
> > Thanks for connecting us! I didn't realize this was already being worked
> > on. Also adding Lance, who commented at [3].
> >
> >> I reference these fixes because I think they handle the minimum
> >> subpage case a bit differently. To be honest, I recall reading those
> >> fixes a while back and getting a bit confused on what exactly happens
> >> when the page is absorbed to fulfill the minimum size...
> >>
> >> It does seem like Sashiko also notes this as a possible concern.
> >> WDYT? Does your reproducer for this issue also work when a minimum
> >> size is set (let's say, to 1?)
> >>
> >
> > Let me look into this more!
> >

Hi Ackerley, sorry that the reply took a while : -(

> I was looking at Sashiko's comment [4] and Sashiko is right that there
> could be a race. Specifically, at the time of
> hugepage_subpool_get_page(), I might be using a reservation, hence
> returning 1 (thread A), but at the time of hugepage_subpool_put_page(),
> some other thread (B) might already have restored a reservation to the
> subpool.
> 
> With used_hpages tracking within the subpool, we can rely on the return value of
> hugepage_subpool_put_page():
> 
> + Return 1: Reservation should be returned elsewhere (pool has enough pages to
>   satisfy minimum, or doesn't track minimums)
> + Return 0: Reservation does not need to be returned elsewhere
> 
> But calling hugepage_subpool_put_pages(1) doesn't say anything about whether the
> 1 (in this case, 1 doesn't always represent a page, since specifically on the
> error cases, the page wasn't allocated) consumed a reservation or not.
> 
> + gbl_chg == 0 means a reservation was used, and
> + gbl_chg == 1 means a reservation *should be used*, but
> + gbl_chg == 1 does not necessarily mean a reservation should be used, since
>   h->resv_huge_pages may not have been decremented yet. (See goto
>   out_subpool_put and goto out_uncharge_cgroup_reservation in
>   alloc_hugetlb_folio())

Yeah... this kind of tripped me up.

> So, we need to track if h->resv_huge_pages-- happened (with some local variable
> like reservation_consumed)
> 
> | reservation_consumed | hugepage_subpool_put_pages() return value |
> h->resv_huge_pages++ |
> |----------------------|-------------------------------------------|----------------------|
> | 1                    | 1                                         |
> Yes                  |
> | 1                    | 0                                         |
> No                   |
> | 0                    | 1                                         |
> No                   |
> | 0                    | 0                                         |
> No                   |
> 
> TLDR: replace !gbl_chg with reservation_consumed?

This seems like quite a reasonable approach. It's new information that
we can't derive from the existing flags we have.

>     <<== reservation_consumed = true
> 
> 	if (!gbl_chg) {
> 		folio_set_hugetlb_restore_reserve(folio);
> 		h->resv_huge_pages--;
>         <<== reservation_consumed = true
> 	}
> 
> 	if (map_chg) {
> 		long gbl_reserve = hugepage_subpool_put_pages(spool, 1);
> 
> 		if (!gbl_chg)  <<== this should be reservation_consumed
> 			hugetlb_acct_memory(h, -gbl_reserve);
> 	}
> 
> I believe this depends on always tracking used_hpages in subpools, in order to
> know for sure, just based on a number (with no page information), if the page is
> supposed to use a reservation.
> 
> 
> Perhaps related: does it make sense to flipping the reservation tracking
> to instead track an "available" page count? This is basically the
> mathematical opposite of rsvd_huge_pages, as in,
> 
>   available_pages = free_huge_pages - resv_huge_pages
> 
> This way, there's no need to synchronize the number of reserved pages in
> both the subpool and global hstate.
> 
> It's quite confusing now, that reserved_pages-- can be either "used a
> reservation" or "unreserve". If we track the opposite, available_pages--
> will definitely mean the page's availability is removed from the global
> hstate and transferred to the subpool, and allocating page doesn't
> involve updating both the subpool and hstate.

I think I like the semantic change, since it does make the code a lot
more obvious on why and where we are making the accounting changes.
One fear that I have is that this churn could introduce some new
subtle accounting errors, since we have been fixing this area for
quite some time now. Let's audit very carefully to make sure that
there are no new accounting errors that will be introduced : -)

> One downside I can think of is that creating hugepages (surplus, or
> promote/demote, or hotplug, or runtime echo 1 > nr_hugepages) would need
> to update the global hstate's available_pages count, but I imagine that
> would be less frequent than allocating huge pages?

Yup I think so. You might be more familiar with the new usecase you
are adding as part of the guest_memfd, but I have not really seen a
usecase for hugeTLB that would require userspace to do many hugeTLB
allocations (that THP wouldn't be able to serve). Having this accuracy
seems to be much more important than the rare contention we will see.

Thanks again for the fixes and ideas Ackerley! Looking forward to your
new approach : -)
Joshua

> >>
> >> [...snip...]
> >>
> >>
> >> [1] https://lore.kernel.org/linux-mm/20260428113037.88766-2-enderaoelyther@gmail.com/
> >> [2] https://lore.kernel.org/linux-mm/20260515202902.461539-1-devnexen@gmail.com/
> > [3] https://lore.kernel.org/linux-mm/20260428113059.79001-1-lance.yang@linux.dev/
> [4] https://sashiko.dev/#/patchset/20260708-hugetlb-alloc-failure-fixes-v2-0-c7f27cbb462b%40google.com?part=2,

[PATCH v2 3/5] mm: hugetlb: Fix folio refcount mismatch on memcg charge failure

[Ackerley Tng via B4 Relay]

From: Ackerley Tng <ackerleytng@google.com>

When mem_cgroup_charge_hugetlb(folio, gfp) returns -ENOMEM, the folio has
its refcount set to 1 via folio_ref_unfreeze(folio, 1).

The error path calls free_huge_folio(folio) directly, which expects a
refcount of 0. Hence, VM_BUG_ON_FOLIO(folio_ref_count(folio), folio) is
triggered.

Even with CONFIG_DEBUG_VM disabled, returning a folio with refcount 1 to
the freelist can corrupt allocator state later.

Use folio_put(folio) instead of free_huge_folio(folio) to properly drop the
reference before freeing it.

Fixes: 991135774c0e0 ("memcg/hugetlb: introduce mem_cgroup_charge_hugetlb")
Cc: stable@vger.kernel.org
Signed-off-by: Ackerley Tng <ackerleytng@google.com>
Reviewed-by: Muchun Song <muchun.song@linux.dev>
---
 mm/hugetlb.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/mm/hugetlb.c b/mm/hugetlb.c
index 4093c1c0a4a1d..1f3f4b964b153 100644
--- a/mm/hugetlb.c
+++ b/mm/hugetlb.c
@@ -2990,7 +2990,7 @@ struct folio *alloc_hugetlb_folio(struct vm_area_struct *vma,
 	lruvec_stat_mod_folio(folio, NR_HUGETLB, pages_per_huge_page(h));
 
 	if (ret == -ENOMEM) {
-		free_huge_folio(folio);
+		folio_put(folio);
 		return ERR_PTR(-ENOMEM);
 	}
 

-- 
2.55.0.795.g602f6c329a-goog

[PATCH v2 3/5] mm: hugetlb: Fix folio refcount mismatch on memcg charge failure

[Ackerley Tng]

When mem_cgroup_charge_hugetlb(folio, gfp) returns -ENOMEM, the folio has
its refcount set to 1 via folio_ref_unfreeze(folio, 1).

The error path calls free_huge_folio(folio) directly, which expects a
refcount of 0. Hence, VM_BUG_ON_FOLIO(folio_ref_count(folio), folio) is
triggered.

Even with CONFIG_DEBUG_VM disabled, returning a folio with refcount 1 to
the freelist can corrupt allocator state later.

Use folio_put(folio) instead of free_huge_folio(folio) to properly drop the
reference before freeing it.

Fixes: 991135774c0e0 ("memcg/hugetlb: introduce mem_cgroup_charge_hugetlb")
Cc: stable@vger.kernel.org
Signed-off-by: Ackerley Tng <ackerleytng@google.com>
Reviewed-by: Muchun Song <muchun.song@linux.dev>
---
 mm/hugetlb.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/mm/hugetlb.c b/mm/hugetlb.c
index 4093c1c0a4a1d..1f3f4b964b153 100644
--- a/mm/hugetlb.c
+++ b/mm/hugetlb.c
@@ -2990,7 +2990,7 @@ struct folio *alloc_hugetlb_folio(struct vm_area_struct *vma,
 	lruvec_stat_mod_folio(folio, NR_HUGETLB, pages_per_huge_page(h));
 
 	if (ret == -ENOMEM) {
-		free_huge_folio(folio);
+		folio_put(folio);
 		return ERR_PTR(-ENOMEM);
 	}
 

-- 
2.55.0.795.g602f6c329a-goog

Re: [PATCH v2 3/5] mm: hugetlb: Fix folio refcount mismatch on memcg charge failure

[Joshua Hahn]

On Wed, 08 Jul 2026 15:12:51 -0700 Ackerley Tng via B4 Relay <devnull+ackerleytng.google.com@kernel.org> wrote:

Hi Ackerley, 

Thanks again for the fix. Thank you also for adding the reproducers,
they made testing this really easy for me! : -D

> From: Ackerley Tng <ackerleytng@google.com>
> 
> When mem_cgroup_charge_hugetlb(folio, gfp) returns -ENOMEM, the folio has
> its refcount set to 1 via folio_ref_unfreeze(folio, 1).
> 
> The error path calls free_huge_folio(folio) directly, which expects a
> refcount of 0. Hence, VM_BUG_ON_FOLIO(folio_ref_count(folio), folio) is
> triggered.

So yeah, I built mm-new (with your hugetlb open code series) and ran
reproducer #3, and got the following output:

...
[   23.855813] Call Trace:
[   23.855838]  <TASK>
[   23.855877]  hugetlb_alloc_folio+0x190/0x360
[   23.855930]  alloc_hugetlb_folio+0xf9/0x310
[   23.855972]  hugetlb_no_page+0x590/0xa00
[   23.856016]  hugetlb_fault+0x194/0x770
[   23.856060]  handle_mm_fault+0x29b/0x2c0
[   23.856103]  do_user_addr_fault+0x208/0x6e0
[   23.856146]  exc_page_fault+0x67/0x140
[   23.856191]  asm_exc_page_fault+0x22/0x30
[   23.856234] RIP: 0033:0x401734
...

Running it with your fix, I don't get a kernel crash. So this change
looks good to me, please feel free to add:

Tested-by: Joshua Hahn <joshua.hahnjy@gmail.com>
Reviewed-by: Joshua Hahn <joshua.hahnjy@gmail.com>

But.....

It seems like fixing this bug actually surfaces a different bug.
I don't see a kernel crash anymore, but my kernel gets stuck in a 
different loop:

...
[   42.526726] pagefault_out_of_memory: 120780 callbacks suppressed
[   42.526732] Huh VM_FAULT_OOM leaked out to the #PF handler. Retrying PF
[   42.526955] Huh VM_FAULT_OOM leaked out to the #PF handler. Retrying PF
[   42.527084] Huh VM_FAULT_OOM leaked out to the #PF handler. Retrying PF
[   42.527192] Huh VM_FAULT_OOM leaked out to the #PF handler. Retrying PF
[   42.527288] Huh VM_FAULT_OOM leaked out to the #PF handler. Retrying PF
[   42.527382] Huh VM_FAULT_OOM leaked out to the #PF handler. Retrying PF
[   42.527476] Huh VM_FAULT_OOM leaked out to the #PF handler. Retrying PF
...

mem_cgroup_charge_hugetlb() fails
--> hugetlb_alloc_folio() returns ERR_PTR(-ENOMEM)
--> propagated to alloc_hugetlb_folio()
--> hugetlb_no_page converts it to vmf_error(PTR_ERROR(folio))
--> propagates up to the pagefault handler..

But there's no OOM handler to resolve, I think. So it just keeps retrying
the fault and cycling over and over and over again... I think we need
to be returning ENOSPC instead of ENOMEM like the other failure paths
in alloc_hugetlb_folio (or hugetlb_alloc_folio... I think we should
change this name, it's a bit confusing).

diff --git a/mm/hugetlb.c b/mm/hugetlb.c
index 288535838a48b..05214b1cd491a 100644
--- a/mm/hugetlb.c
+++ b/mm/hugetlb.c
@@ -2903,7 +2903,7 @@ struct folio *hugetlb_alloc_folio(struct hstate *h,
                 * were committed to the folio and freeing the folio
                 * would have cleared those up.
                 */
-               return ERR_PTR(ret);
+               return ERR_PTR(-ENOSPC);
        }

        return folio;

This gives me the following result from your reproducer, which I think
is the expected behavior:

attempting to remount cgroup2 with memory_hugetlb_accounting...
Successfully enabled memory_hugetlb_accounting
Child: Attempting to allocate and touch 2MB hugepage...
Child: mmap succeeded at 0x7f0023600000, touching it now (should trigger fault)...
Parent: Child exited. Cleaning up.
Parent: Child killed by signal 7 (Bus error)
Parent: Child got SIGBUS as expected (if kernel didn't crash).

What do you think? Happy to send it out as a separate fix or feel free
to fold it into your series / fix, whatever you prefer.

Thanks Ackerley!

> Even with CONFIG_DEBUG_VM disabled, returning a folio with refcount 1 to
> the freelist can corrupt allocator state later.
> 
> Use folio_put(folio) instead of free_huge_folio(folio) to properly drop the
> reference before freeing it.
> Fixes: 991135774c0e0 ("memcg/hugetlb: introduce mem_cgroup_charge_hugetlb")
> Cc: stable@vger.kernel.org
> Signed-off-by: Ackerley Tng <ackerleytng@google.com>
> Reviewed-by: Muchun Song <muchun.song@linux.dev>
> ---
>  mm/hugetlb.c | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/mm/hugetlb.c b/mm/hugetlb.c
> index 4093c1c0a4a1d..1f3f4b964b153 100644
> --- a/mm/hugetlb.c
> +++ b/mm/hugetlb.c
> @@ -2990,7 +2990,7 @@ struct folio *alloc_hugetlb_folio(struct vm_area_struct *vma,
>  	lruvec_stat_mod_folio(folio, NR_HUGETLB, pages_per_huge_page(h));
>  
>  	if (ret == -ENOMEM) {
> -		free_huge_folio(folio);
> +		folio_put(folio);
>  		return ERR_PTR(-ENOMEM);
>  	}
>  
> 
> -- 
> 2.55.0.795.g602f6c329a-goog

Re: [PATCH v2 3/5] mm: hugetlb: Fix folio refcount mismatch on memcg charge failure

[Joshua Hahn]

On Thu,  9 Jul 2026 11:54:55 -0700 Joshua Hahn <joshua.hahnjy@gmail.com> wrote:

> On Wed, 08 Jul 2026 15:12:51 -0700 Ackerley Tng via B4 Relay <devnull+ackerleytng.google.com@kernel.org> wrote:
> 
> Hi Ackerley, 
> 
> Thanks again for the fix. Thank you also for adding the reproducers,
> they made testing this really easy for me! : -D
> 
> > From: Ackerley Tng <ackerleytng@google.com>
> > 
> > When mem_cgroup_charge_hugetlb(folio, gfp) returns -ENOMEM, the folio has
> > its refcount set to 1 via folio_ref_unfreeze(folio, 1).
> > 
> > The error path calls free_huge_folio(folio) directly, which expects a
> > refcount of 0. Hence, VM_BUG_ON_FOLIO(folio_ref_count(folio), folio) is
> > triggered.
> 
> So yeah, I built mm-new (with your hugetlb open code series) and ran
> reproducer #3, and got the following output:
> 
> ...
> [   23.855813] Call Trace:
> [   23.855838]  <TASK>
> [   23.855877]  hugetlb_alloc_folio+0x190/0x360
> [   23.855930]  alloc_hugetlb_folio+0xf9/0x310
> [   23.855972]  hugetlb_no_page+0x590/0xa00
> [   23.856016]  hugetlb_fault+0x194/0x770
> [   23.856060]  handle_mm_fault+0x29b/0x2c0
> [   23.856103]  do_user_addr_fault+0x208/0x6e0
> [   23.856146]  exc_page_fault+0x67/0x140
> [   23.856191]  asm_exc_page_fault+0x22/0x30
> [   23.856234] RIP: 0033:0x401734
> ...
> 
> Running it with your fix, I don't get a kernel crash. So this change
> looks good to me, please feel free to add:
> 
> Tested-by: Joshua Hahn <joshua.hahnjy@gmail.com>
> Reviewed-by: Joshua Hahn <joshua.hahnjy@gmail.com>
> 
> But.....
> 
> It seems like fixing this bug actually surfaces a different bug.
> I don't see a kernel crash anymore, but my kernel gets stuck in a 
> different loop:
> 
> ...
> [   42.526726] pagefault_out_of_memory: 120780 callbacks suppressed
> [   42.526732] Huh VM_FAULT_OOM leaked out to the #PF handler. Retrying PF
> [   42.526955] Huh VM_FAULT_OOM leaked out to the #PF handler. Retrying PF
> [   42.527084] Huh VM_FAULT_OOM leaked out to the #PF handler. Retrying PF
> [   42.527192] Huh VM_FAULT_OOM leaked out to the #PF handler. Retrying PF
> [   42.527288] Huh VM_FAULT_OOM leaked out to the #PF handler. Retrying PF
> [   42.527382] Huh VM_FAULT_OOM leaked out to the #PF handler. Retrying PF
> [   42.527476] Huh VM_FAULT_OOM leaked out to the #PF handler. Retrying PF
> ...
> 
> mem_cgroup_charge_hugetlb() fails
> --> hugetlb_alloc_folio() returns ERR_PTR(-ENOMEM)
> --> propagated to alloc_hugetlb_folio()
> --> hugetlb_no_page converts it to vmf_error(PTR_ERROR(folio))
> --> propagates up to the pagefault handler..
> 
> But there's no OOM handler to resolve, I think. So it just keeps retrying
> the fault and cycling over and over and over again... I think we need
> to be returning ENOSPC instead of ENOMEM like the other failure paths
> in alloc_hugetlb_folio (or hugetlb_alloc_folio... I think we should
> change this name, it's a bit confusing).
> 
> diff --git a/mm/hugetlb.c b/mm/hugetlb.c
> index 288535838a48b..05214b1cd491a 100644
> --- a/mm/hugetlb.c
> +++ b/mm/hugetlb.c
> @@ -2903,7 +2903,7 @@ struct folio *hugetlb_alloc_folio(struct hstate *h,
>                  * were committed to the folio and freeing the folio
>                  * would have cleared those up.
>                  */
> -               return ERR_PTR(ret);
> +               return ERR_PTR(-ENOSPC);
>         }
> 
>         return folio;

Ah, immediately as I sent this out and went to 4/5 to review it, I
see that this is the exact change you have... oops, please disregard
the message : -) I do think we should 3/5 and 4/5 together for
bisectability purposes.

Thanks again. Sorry for the noise!
Joshua

[PATCH v2 4/5] mm: hugetlb: Return -ENOSPC on memcg charge failure

[Ackerley Tng via B4 Relay]

From: Ackerley Tng <ackerleytng@google.com>

When mem_cgroup_charge_hugetlb() fails with -ENOMEM, alloc_hugetlb_folio()
currently propagates this error. This results in the page fault handler
returning VM_FAULT_OOM.

Because HugeTLB allocations are high-order and use __GFP_RETRY_MAYFAIL,
they bypass the OOM killer. Returning VM_FAULT_OOM to the #PF handler
without triggering the OOM killer (or having it make progress) leads to
an infinite loop of retrying the fault.

Avoid this loop by returning -ENOSPC when charging fails, which maps to
VM_FAULT_SIGBUS, terminating the process cleanly.

Make mem_cgroup_charge_hugetlb() fault handling use a common error handling
path, the same handling used for hugetlb_cgroup_uncharge_cgroup{,_rsvd}(),
which also don't trigger the OOM killer and hence opt to terminate the
process with a SIGBUS.

Fixes: 991135774c0e0 ("memcg/hugetlb: introduce mem_cgroup_charge_hugetlb")
Cc: stable@vger.kernel.org
Signed-off-by: Ackerley Tng <ackerleytng@google.com>
Reviewed-by: Muchun Song <muchun.song@linux.dev>
---
 mm/hugetlb.c | 13 ++++++++++++-
 1 file changed, 12 insertions(+), 1 deletion(-)

diff --git a/mm/hugetlb.c b/mm/hugetlb.c
index 1f3f4b964b153..3e1d99f03c70e 100644
--- a/mm/hugetlb.c
+++ b/mm/hugetlb.c
@@ -2991,7 +2991,7 @@ struct folio *alloc_hugetlb_folio(struct vm_area_struct *vma,
 
 	if (ret == -ENOMEM) {
 		folio_put(folio);
-		return ERR_PTR(-ENOMEM);
+		goto err;
 	}
 
 	return folio;
@@ -3014,6 +3014,17 @@ struct folio *alloc_hugetlb_folio(struct vm_area_struct *vma,
 out_end_reservation:
 	if (map_chg != MAP_CHG_ENFORCED)
 		vma_end_reservation(h, vma, addr);
+err:
+	/*
+	 * Return -ENOSPC when this function fails to allocate or
+	 * charge a huge page. If a standard (PAGE_SIZE) page
+	 * allocation fails, the OOM killer is given a chance to run,
+	 * which may resolve the failure on retry. However, for
+	 * HugeTLB allocations, the OOM killer is not triggered.
+	 * Returning -ENOMEM (or anything resulting in VM_FAULT_OOM)
+	 * would leak to the #PF handler, causing it to loop
+	 * indefinitely retrying the fault.
+	 */
 	return ERR_PTR(-ENOSPC);
 }
 

-- 
2.55.0.795.g602f6c329a-goog

[PATCH v2 4/5] mm: hugetlb: Return -ENOSPC on memcg charge failure

[Ackerley Tng]

When mem_cgroup_charge_hugetlb() fails with -ENOMEM, alloc_hugetlb_folio()
currently propagates this error. This results in the page fault handler
returning VM_FAULT_OOM.

Because HugeTLB allocations are high-order and use __GFP_RETRY_MAYFAIL,
they bypass the OOM killer. Returning VM_FAULT_OOM to the #PF handler
without triggering the OOM killer (or having it make progress) leads to
an infinite loop of retrying the fault.

Avoid this loop by returning -ENOSPC when charging fails, which maps to
VM_FAULT_SIGBUS, terminating the process cleanly.

Make mem_cgroup_charge_hugetlb() fault handling use a common error handling
path, the same handling used for hugetlb_cgroup_uncharge_cgroup{,_rsvd}(),
which also don't trigger the OOM killer and hence opt to terminate the
process with a SIGBUS.

Fixes: 991135774c0e0 ("memcg/hugetlb: introduce mem_cgroup_charge_hugetlb")
Cc: stable@vger.kernel.org
Signed-off-by: Ackerley Tng <ackerleytng@google.com>
Reviewed-by: Muchun Song <muchun.song@linux.dev>
---
 mm/hugetlb.c | 13 ++++++++++++-
 1 file changed, 12 insertions(+), 1 deletion(-)

diff --git a/mm/hugetlb.c b/mm/hugetlb.c
index 1f3f4b964b153..3e1d99f03c70e 100644
--- a/mm/hugetlb.c
+++ b/mm/hugetlb.c
@@ -2991,7 +2991,7 @@ struct folio *alloc_hugetlb_folio(struct vm_area_struct *vma,
 
 	if (ret == -ENOMEM) {
 		folio_put(folio);
-		return ERR_PTR(-ENOMEM);
+		goto err;
 	}
 
 	return folio;
@@ -3014,6 +3014,17 @@ struct folio *alloc_hugetlb_folio(struct vm_area_struct *vma,
 out_end_reservation:
 	if (map_chg != MAP_CHG_ENFORCED)
 		vma_end_reservation(h, vma, addr);
+err:
+	/*
+	 * Return -ENOSPC when this function fails to allocate or
+	 * charge a huge page. If a standard (PAGE_SIZE) page
+	 * allocation fails, the OOM killer is given a chance to run,
+	 * which may resolve the failure on retry. However, for
+	 * HugeTLB allocations, the OOM killer is not triggered.
+	 * Returning -ENOMEM (or anything resulting in VM_FAULT_OOM)
+	 * would leak to the #PF handler, causing it to loop
+	 * indefinitely retrying the fault.
+	 */
 	return ERR_PTR(-ENOSPC);
 }
 

-- 
2.55.0.795.g602f6c329a-goog

Re: [PATCH v2 4/5] mm: hugetlb: Return -ENOSPC on memcg charge failure

[Joshua Hahn]

On Wed, 08 Jul 2026 15:12:52 -0700 Ackerley Tng via B4 Relay <devnull+ackerleytng.google.com@kernel.org> wrote:

> From: Ackerley Tng <ackerleytng@google.com>
> 
> When mem_cgroup_charge_hugetlb() fails with -ENOMEM, alloc_hugetlb_folio()
> currently propagates this error. This results in the page fault handler
> returning VM_FAULT_OOM.
> 
> Because HugeTLB allocations are high-order and use __GFP_RETRY_MAYFAIL,
> they bypass the OOM killer. Returning VM_FAULT_OOM to the #PF handler
> without triggering the OOM killer (or having it make progress) leads to
> an infinite loop of retrying the fault.
> 
> Avoid this loop by returning -ENOSPC when charging fails, which maps to
> VM_FAULT_SIGBUS, terminating the process cleanly.
> 
> Make mem_cgroup_charge_hugetlb() fault handling use a common error handling
> path, the same handling used for hugetlb_cgroup_uncharge_cgroup{,_rsvd}(),
> which also don't trigger the OOM killer and hence opt to terminate the
> process with a SIGBUS.
> 
> Fixes: 991135774c0e0 ("memcg/hugetlb: introduce mem_cgroup_charge_hugetlb")
> Cc: stable@vger.kernel.org

Tested-by: Joshua Hahn <joshua.hahnjy@gmail.com>
Reviewed-by: Joshua Hahn <joshua.hahnjy@gmail.com>

> Signed-off-by: Ackerley Tng <ackerleytng@google.com>
> Reviewed-by: Muchun Song <muchun.song@linux.dev>
> ---
>  mm/hugetlb.c | 13 ++++++++++++-
>  1 file changed, 12 insertions(+), 1 deletion(-)
> 
> diff --git a/mm/hugetlb.c b/mm/hugetlb.c
> index 1f3f4b964b153..3e1d99f03c70e 100644
> --- a/mm/hugetlb.c
> +++ b/mm/hugetlb.c
> @@ -2991,7 +2991,7 @@ struct folio *alloc_hugetlb_folio(struct vm_area_struct *vma,
>  
>  	if (ret == -ENOMEM) {
>  		folio_put(folio);
> -		return ERR_PTR(-ENOMEM);
> +		goto err;
>  	}
>  
>  	return folio;
> @@ -3014,6 +3014,17 @@ struct folio *alloc_hugetlb_folio(struct vm_area_struct *vma,
>  out_end_reservation:
>  	if (map_chg != MAP_CHG_ENFORCED)
>  		vma_end_reservation(h, vma, addr);

NIT: is there a reason why the comment block below doesn't go all the
way up to 80 columns? I also think that we can achieve the same effect
with a shorter comment block : -) I think the essence is to answer
"Why -ENOSPC and not -ENOMEM?"

> +err:
> +	/*
> +	 * Return -ENOSPC when this function fails to allocate or
> +	 * charge a huge page. If a standard (PAGE_SIZE) page
> +	 * allocation fails, the OOM killer is given a chance to run,
> +	 * which may resolve the failure on retry. However, for
> +	 * HugeTLB allocations, the OOM killer is not triggered.
> +	 * Returning -ENOMEM (or anything resulting in VM_FAULT_OOM)
> +	 * would leak to the #PF handler, causing it to loop
> +	 * indefinitely retrying the fault.
> +	 */
>  	return ERR_PTR(-ENOSPC);
>  }

But this fix looks good otherwise and I don't think this a big deal : -)

Thanks Ackerley!
Joshua

[PATCH v2 5/5] mm: hugetlb: Move memcg charge earlier to prevent reservation leak

[Ackerley Tng via B4 Relay]

From: Ackerley Tng <ackerleytng@google.com>

When mem_cgroup_charge_hugetlb() fails, alloc_hugetlb_folio() clears the
charges and frees the folio. The reservation committed via
vma_commit_reservation() was not undone, leaving the reservation map in an
inconsistent state, causing resv_huge_pages to leak when the process
exited.

Fix this by moving the mem_cgroup_charge_hugetlb() call earlier in
alloc_hugetlb_folio(), before vma_commit_reservation() is called.

If the charge fails now, the reservation is not yet committed. Jump to
out_subpool_put, which will then call vma_end_reservation() to abort the
reservation and keep the reservation map and resv_huge_pages counter
consistent.

Fixes: 991135774c0e0 ("memcg/hugetlb: introduce mem_cgroup_charge_hugetlb")
Cc: stable@vger.kernel.org
Signed-off-by: Ackerley Tng <ackerleytng@google.com>
---
 mm/hugetlb.c | 34 ++++++++++++++++++++--------------
 1 file changed, 20 insertions(+), 14 deletions(-)

diff --git a/mm/hugetlb.c b/mm/hugetlb.c
index 3e1d99f03c70e..e000af6f28585 100644
--- a/mm/hugetlb.c
+++ b/mm/hugetlb.c
@@ -2954,6 +2954,25 @@ struct folio *alloc_hugetlb_folio(struct vm_area_struct *vma,
 
 	spin_unlock_irq(&hugetlb_lock);
 
+	ret = mem_cgroup_charge_hugetlb(folio, gfp);
+	/*
+	 * Unconditionally increment NR_HUGETLB here. If it turns out that
+	 * mem_cgroup_charge_hugetlb failed, then immediately free the page and
+	 * decrement NR_HUGETLB.
+	 */
+	lruvec_stat_mod_folio(folio, NR_HUGETLB, pages_per_huge_page(h));
+
+	if (ret == -ENOMEM) {
+		folio_put(folio);
+		/*
+		 * Charges to hugetlb_cgroup for usage and
+		 * reservations were already committed, so folio_put()
+		 * would have uncharged those. Go straight to undoing
+		 * subpool charges.
+		 */
+		goto out_subpool_put;
+	}
+
 	hugetlb_set_folio_subpool(folio, spool);
 
 	if (map_chg != MAP_CHG_ENFORCED) {
@@ -2981,19 +3000,6 @@ struct folio *alloc_hugetlb_folio(struct vm_area_struct *vma,
 		}
 	}
 
-	ret = mem_cgroup_charge_hugetlb(folio, gfp);
-	/*
-	 * Unconditionally increment NR_HUGETLB here. If it turns out that
-	 * mem_cgroup_charge_hugetlb failed, then immediately free the page and
-	 * decrement NR_HUGETLB.
-	 */
-	lruvec_stat_mod_folio(folio, NR_HUGETLB, pages_per_huge_page(h));
-
-	if (ret == -ENOMEM) {
-		folio_put(folio);
-		goto err;
-	}
-
 	return folio;
 
 out_uncharge_cgroup:
@@ -3014,7 +3020,7 @@ struct folio *alloc_hugetlb_folio(struct vm_area_struct *vma,
 out_end_reservation:
 	if (map_chg != MAP_CHG_ENFORCED)
 		vma_end_reservation(h, vma, addr);
-err:
+
 	/*
 	 * Return -ENOSPC when this function fails to allocate or
 	 * charge a huge page. If a standard (PAGE_SIZE) page

-- 
2.55.0.795.g602f6c329a-goog

[PATCH v2 5/5] mm: hugetlb: Move memcg charge earlier to prevent reservation leak

[Ackerley Tng]

When mem_cgroup_charge_hugetlb() fails, alloc_hugetlb_folio() clears the
charges and frees the folio. The reservation committed via
vma_commit_reservation() was not undone, leaving the reservation map in an
inconsistent state, causing resv_huge_pages to leak when the process
exited.

Fix this by moving the mem_cgroup_charge_hugetlb() call earlier in
alloc_hugetlb_folio(), before vma_commit_reservation() is called.

If the charge fails now, the reservation is not yet committed. Jump to
out_subpool_put, which will then call vma_end_reservation() to abort the
reservation and keep the reservation map and resv_huge_pages counter
consistent.

Fixes: 991135774c0e0 ("memcg/hugetlb: introduce mem_cgroup_charge_hugetlb")
Cc: stable@vger.kernel.org
Signed-off-by: Ackerley Tng <ackerleytng@google.com>
---
 mm/hugetlb.c | 34 ++++++++++++++++++++--------------
 1 file changed, 20 insertions(+), 14 deletions(-)

diff --git a/mm/hugetlb.c b/mm/hugetlb.c
index 3e1d99f03c70e..e000af6f28585 100644
--- a/mm/hugetlb.c
+++ b/mm/hugetlb.c
@@ -2954,6 +2954,25 @@ struct folio *alloc_hugetlb_folio(struct vm_area_struct *vma,
 
 	spin_unlock_irq(&hugetlb_lock);
 
+	ret = mem_cgroup_charge_hugetlb(folio, gfp);
+	/*
+	 * Unconditionally increment NR_HUGETLB here. If it turns out that
+	 * mem_cgroup_charge_hugetlb failed, then immediately free the page and
+	 * decrement NR_HUGETLB.
+	 */
+	lruvec_stat_mod_folio(folio, NR_HUGETLB, pages_per_huge_page(h));
+
+	if (ret == -ENOMEM) {
+		folio_put(folio);
+		/*
+		 * Charges to hugetlb_cgroup for usage and
+		 * reservations were already committed, so folio_put()
+		 * would have uncharged those. Go straight to undoing
+		 * subpool charges.
+		 */
+		goto out_subpool_put;
+	}
+
 	hugetlb_set_folio_subpool(folio, spool);
 
 	if (map_chg != MAP_CHG_ENFORCED) {
@@ -2981,19 +3000,6 @@ struct folio *alloc_hugetlb_folio(struct vm_area_struct *vma,
 		}
 	}
 
-	ret = mem_cgroup_charge_hugetlb(folio, gfp);
-	/*
-	 * Unconditionally increment NR_HUGETLB here. If it turns out that
-	 * mem_cgroup_charge_hugetlb failed, then immediately free the page and
-	 * decrement NR_HUGETLB.
-	 */
-	lruvec_stat_mod_folio(folio, NR_HUGETLB, pages_per_huge_page(h));
-
-	if (ret == -ENOMEM) {
-		folio_put(folio);
-		goto err;
-	}
-
 	return folio;
 
 out_uncharge_cgroup:
@@ -3014,7 +3020,7 @@ struct folio *alloc_hugetlb_folio(struct vm_area_struct *vma,
 out_end_reservation:
 	if (map_chg != MAP_CHG_ENFORCED)
 		vma_end_reservation(h, vma, addr);
-err:
+
 	/*
 	 * Return -ENOSPC when this function fails to allocate or
 	 * charge a huge page. If a standard (PAGE_SIZE) page

-- 
2.55.0.795.g602f6c329a-goog

[POC PATCH 0/3] Reproducers for hugetlb allocation failure issues

[Ackerley Tng]

Please see separate patches for details!

Ackerley Tng (3):
  Reproducer for false restoration on shared HugeTLB mappings
  Reproducer for subpool usage leak
  Reproducer for allocation failure due to cgroup v2 memory limits

 cgroup_v2_allocation_failure.c | 160 +++++++++++++++++++++++++++++++++
 subpool_leak_max_size.sh       |  71 +++++++++++++++
 subpool_shared_leak.c          |  29 ++++++
 subpool_shared_leak.sh         |  86 ++++++++++++++++++
 4 files changed, 346 insertions(+)
 create mode 100644 cgroup_v2_allocation_failure.c
 create mode 100755 subpool_leak_max_size.sh
 create mode 100644 subpool_shared_leak.c
 create mode 100755 subpool_shared_leak.sh

--
2.55.0.795.g602f6c329a-goog

[POC PATCH 1/3] Reproducer for false restoration on shared HugeTLB mappings

[Ackerley Tng]

(This reproducer was hacked up and not meant to be merged.)

hugetlb_unreserve_pages() unconditionally returns reservations to the subpool
(via hugepage_subpool_put_pages()). This means that regardless of whether a
subpool reservation was actually used, the reservation is processed by the
subpool structure.

To create a false restoration, the reproducer performs these steps:

1.  Mount with min_size=2M (1 page). Global resv_hugepages becomes 1.
2.  The program maps 4MB (2 pages) shared (which also grows the file to
    4MB). Global resv_hugepages becomes 2 (1 from the mount, 1 new global
    reservation).
3.  The program populates only the first page. Global resv_hugepages decrements
    to 1 (reservation consumed by allocation).
4.  The program exits (closing VMAs/fds). For shared mappings, reservations are
    associated with the file inode, so they remain active. Global resv_hugepages
    remains 1.
5.  The script truncates the file to 2MB (truncate -s 2M).
    +   This synchronously triggers hugetlb_unreserve_pages() to release the
        reservation of the truncated range (the unallocated 2nd page).
    +   It calls hugepage_subpool_put_pages(spool, 1).
    +   On Vanilla Kernel (Buggy):
        +   used_hpages is 0 (not tracked).
        +   used_hpages (0) < min_hpages (1) is TRUE.
        +   The subpool incorrectly restores the reservation (spool->rsv_hpages
            becomes 1), even though Page 0 is still allocated and satisfies the
            mount's minimum guarantee.
        +   hugepage_subpool_put_pages() returns 0, skipping
            hugetlb_acct_memory(h, -1).
        +   Result: Global resv_hugepages remains stuck at 1 (Leak).
    +   On Fixed Kernel:
        +   used_hpages is tracked and is initially 2.
        +   hugepage_subpool_put_pages(1) decrements used_hpages to 1.
        +   used_hpages (1) < min_hpages (1) is FALSE.
        +   The subpool does not restore the reservation.
        +   hugepage_subpool_put_pages() returns 1.
        +   hugetlb_acct_memory(h, -1) is called.
        +   Result: Global resv_hugepages decrements to 0 (No leak).

When the filesystem is unmounted, hugetlbfs_put_super drops the subpool
reference. Since the filesystem is being unmounted, the reference count drops to
0, triggering unlock_or_release_subpool.

Inside unlock_or_release_subpool, the kernel checks if the subpool is free using
subpool_is_free.
+ On the buggy kernel, subpool_is_free checks if spool->rsv_hpages is equal to
  spool->min_hpages. Because of the phantom reservation, spool->rsv_hpages was
  restored to 1. Since min_hpages is 1, the check (1 == 1) returns true.
+ Since the subpool is considered free, the kernel releases the initial
  mount-time reservation by calling hugetlb_acct_memory to decrement
  resv_huge_pages by spool->min_hpages (which is 1).
+ This decrement reduces resv_huge_pages from 1 (the leaked state) to 0.

As a result, the leaked reservation is cleaned up during unmount and does not
persist afterward.

Signed-off-by: Ackerley Tng <ackerleytng@google.com>
---
 subpool_shared_leak.c  | 29 ++++++++++++++
 subpool_shared_leak.sh | 86 ++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 115 insertions(+)
 create mode 100644 subpool_shared_leak.c
 create mode 100755 subpool_shared_leak.sh

diff --git a/subpool_shared_leak.c b/subpool_shared_leak.c
new file mode 100644
index 0000000000000..5811e18d7f8be
--- /dev/null
+++ b/subpool_shared_leak.c
@@ -0,0 +1,29 @@
+#include <stdio.h>
+#include <stdlib.h>
+#include <unistd.h>
+#include <sys/mman.h>
+#include <sys/stat.h>
+#include <fcntl.h>
+
+#define HPAGE_SIZE (2 * 1024 * 1024)
+
+int main(int argc, char **argv) {
+    if (argc < 2) {
+        fprintf(stderr, "Usage: %s <file_path>\n", argv[0]);
+        return 1;
+    }
+    const char *file_path = argv[1];
+
+    int fd = open(file_path, O_CREAT | O_RDWR, 0666);
+    if (fd < 0) { perror("open"); return 1; }
+
+    void *addr = mmap(NULL, 2 * HPAGE_SIZE, PROT_READ | PROT_WRITE, MAP_SHARED, fd, 0);
+    if (addr == MAP_FAILED) { perror("mmap"); close(fd); return 1; }
+
+    *(volatile char *)addr = 1; // Allocate 1st page only. 2nd page remains unallocated (but reserved).
+
+    munmap(addr, 2 * HPAGE_SIZE);
+    close(fd);
+
+    return 0;
+}
diff --git a/subpool_shared_leak.sh b/subpool_shared_leak.sh
new file mode 100755
index 0000000000000..46c622b18559a
--- /dev/null
+++ b/subpool_shared_leak.sh
@@ -0,0 +1,86 @@
+#!/bin/bash
+
+if [ "$EUID" -ne 0 ]; then
+    echo "Please run as root"
+    exit 1
+fi
+
+MNT_PATH="/tmp/mnt_hugetlb_shared_leak"
+FILE_PATH="$MNT_PATH/test_file"
+
+# Save original values
+orig_nr=$(cat /sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages)
+
+cleanup() {
+    echo "Cleaning up..."
+    rm -f "$FILE_PATH"
+    umount "$MNT_PATH" 2>/dev/null
+    rmdir "$MNT_PATH" 2>/dev/null
+    echo "$orig_nr" > /sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages
+    echo "Cleanup done."
+}
+trap cleanup EXIT
+
+# 1. Set nr_hugepages to 2
+echo 2 > /sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages
+
+# 2. Mount hugetlbfs with min_size=2M (1 page)
+mkdir -p "$MNT_PATH"
+if ! mount -t hugetlbfs -o min_size=2M none "$MNT_PATH"; then
+    echo "Failed to mount hugetlbfs"
+    exit 1
+fi
+
+# Check resv_hugepages after mount (should be 1)
+initial_resv=$(cat /sys/kernel/mm/hugepages/hugepages-2048kB/resv_hugepages)
+echo "Initial resv_hugepages (after mount): $initial_resv"
+if [ "$initial_resv" -ne 1 ]; then
+    echo "ERROR: Initial resv_hugepages is not 1!"
+    exit 1
+fi
+
+# Verify reproducer binary exists
+if [ ! -x ./subpool_shared_leak ]; then
+    echo "reproducer binary './subpool_shared_leak' not found or not executable."
+    echo "Please compile it first: gcc -static -o subpool_shared_leak subpool_shared_leak.c"
+    exit 1
+fi
+
+# 3. Run helper to map 4MB, allocate 2MB, and close.
+# This creates 2 reservations, consumes 1 (by allocating Page 0).
+# The unallocated Page 1 reservation remains active in the inode's resv_map.
+echo "Running helper..."
+./subpool_shared_leak "$FILE_PATH"
+
+resv_after_helper=$(cat /sys/kernel/mm/hugepages/hugepages-2048kB/resv_hugepages)
+echo "resv_hugepages after helper (should be 1): $resv_after_helper"
+# Page 0 is allocated (no longer reserved). Page 1 is reserved.
+# So resv_hugepages should be 1.
+if [ "$resv_after_helper" -ne 1 ]; then
+    echo "ERROR: resv_hugepages is not 1 after helper run!"
+    exit 1
+fi
+
+# 4. Truncate file to 2MB (releases Page 1 reservation)
+echo "Truncating file to 2MB (releasing 1 page reservation)..."
+truncate -s 2M "$FILE_PATH"
+
+# Check resv_hugepages after truncate.
+# Since Page 0 is still allocated (and in page cache), and satisfies the
+# min_size=2M guarantee, we should have 0 reservations remaining.
+# If the bug is present, the truncate path will incorrectly restore the
+# reservation to the subpool and skip releasing it globally, leaving
+# resv_hugepages at 1.
+final_resv=$(cat /sys/kernel/mm/hugepages/hugepages-2048kB/resv_hugepages)
+echo "Final resv_hugepages (after 2MB truncate): $final_resv"
+
+if [ "$final_resv" -eq 1 ]; then
+    echo "RESULT: LEAK DETECTED (FAIL)"
+    exit 1
+elif [ "$final_resv" -eq 0 ]; then
+    echo "RESULT: NO LEAK (PASS)"
+    exit 0
+else
+    echo "RESULT: UNEXPECTED STATE ($final_resv)"
+    exit 2
+fi
-- 
2.55.0.795.g602f6c329a-goog

[POC PATCH 2/3] Reproducer for subpool usage leak

[Ackerley Tng]

(This reproducer was hacked up and not meant to be merged.)

The kernel leaks subpool usage and the subpool structure itself if a HugeTLBfs
mount specifying size (which sets max_hpages on the subpool) is created.

subpool_leak_max_size.sh reproduces this with the following steps:

1. Create mount, specifying size=2M (1 page). This sets max_hpages = 1 on the
   subpool, but does not reserve any pages.
2. Set nr_hugepages = 0 and nr_overcommit_hugepages = 0 so that physical
   allocations will fail.
3. Run fallocate -l 2M on a file in the mount.
     + This calls hugetlbfs_fallocate, which attempts to allocate a page by
       calling alloc_hugetlb_folio.
     + alloc_hugetlb_folio calls hugepage_subpool_get_pages to track the
       allocation against the subpool limit. This increments used_hpages to 1.
     + Physical allocation fails because nr_hugepages is 0.
     + Before patch (Buggy):
         + The error path in alloc_hugetlb_folio sees gbl_chg is 1 (indicating
           we tried to allocate a global page) and incorrectly skips calling
           hugepage_subpool_put_pages.
         + fallocate fails and returns to userspace, but the subpool used_hpages
           counter remains leaked at 1.
     + After patch:
         + The error path always calls hugepage_subpool_put_pages if map_chg is
           true, restoring used_hpages to 0.

4. Unmount the filesystem.
     + During unmount, the kernel calls unlock_or_release_subpool to clean up
       the subpool.
     + It checks if the subpool is free using subpool_is_free, which returns
       whether used_hpages is 0.
     + Before patch (Buggy):
         + Since used_hpages leaked and is 1, subpool_is_free returns false.
         + The kernel skips freeing the subpool structure, leaking the
           hugepage_subpool structure in kernel memory.
     + After patch:
         + Since used_hpages is 0, subpool_is_free returns true, and the subpool
           structure is correctly freed.

Signed-off-by: Ackerley Tng <ackerleytng@google.com>
---
 subpool_leak_max_size.sh | 71 ++++++++++++++++++++++++++++++++++++++++
 1 file changed, 71 insertions(+)
 create mode 100755 subpool_leak_max_size.sh

diff --git a/subpool_leak_max_size.sh b/subpool_leak_max_size.sh
new file mode 100755
index 0000000000000..bfafa1ba074ea
--- /dev/null
+++ b/subpool_leak_max_size.sh
@@ -0,0 +1,71 @@
+#!/bin/bash
+
+if [ "$EUID" -ne 0 ]; then
+    echo "Please run as root"
+    exit 1
+fi
+
+MNT_PATH="/tmp/mnt_hugetlb"
+FILE_PATH="$MNT_PATH/test_file"
+
+# Save original values
+orig_nr=$(cat /sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages)
+orig_overcommit=$(cat /sys/kernel/mm/hugepages/hugepages-2048kB/nr_overcommit_hugepages)
+
+cleanup() {
+    echo "Cleaning up..."
+    rm -f "$FILE_PATH"
+    umount "$MNT_PATH" 2>/dev/null
+    rmdir "$MNT_PATH" 2>/dev/null
+    echo "$orig_nr" > /sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages
+    echo "$orig_overcommit" > /sys/kernel/mm/hugepages/hugepages-2048kB/nr_overcommit_hugepages
+    echo "Cleanup done."
+}
+trap cleanup EXIT
+
+# 1. Mount hugetlbfs with size=2M (1 page)
+mkdir -p "$MNT_PATH"
+if ! mount -t hugetlbfs -o size=2M none "$MNT_PATH"; then
+    echo "Failed to mount hugetlbfs"
+    exit 1
+fi
+
+# 2. Set nr_hugepages to 0, overcommit to 0
+echo 0 > /sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages
+echo 0 > /sys/kernel/mm/hugepages/hugepages-2048kB/nr_overcommit_hugepages
+
+# Check subpool usage before running
+read total free < <(stat -f -c "%b %f" "$MNT_PATH")
+used_before=$((total - free))
+echo "Before test - Subpool total blocks: $total"
+echo "Before test - Subpool free blocks:  $free"
+echo "Before test - Subpool used blocks:  $used_before"
+if [ "$used_before" -ne 0 ]; then
+    echo "ERROR: Subpool is not clean before test starts!"
+    exit 1
+fi
+
+# Run fallocate (expecting failure)
+echo "Running fallocate (expecting failure)..."
+if fallocate -l 2M "$FILE_PATH" 2>/dev/null; then
+    echo "ERROR: fallocate succeeded but should have failed (nr_hugepages is 0)"
+    exit 1
+fi
+
+# Check subpool usage via statfs
+# %b: Total blocks
+# %f: Free blocks
+read total free < <(stat -f -c "%b %f" "$MNT_PATH")
+used=$((total - free))
+
+echo "Subpool total blocks: $total"
+echo "Subpool free blocks:  $free"
+echo "Subpool used blocks (leaked if > 0): $used"
+
+if [ "$used" -gt 0 ]; then
+    echo "RESULT: LEAK DETECTED (FAIL)"
+    exit 1
+else
+    echo "RESULT: NO LEAK (PASS)"
+    exit 0
+fi
-- 
2.55.0.795.g602f6c329a-goog

[POC PATCH 3/3] Reproducer for allocation failure due to cgroup v2 memory limits

[Ackerley Tng]

(This reproducer was hacked up and not meant to be merged.)

cgroup_v2_allocation_failure.c triggers HugeTLB allocation failure by exploiting
cgroup v2 memory limits. This allows testing the error paths in the kernel when
memory control charging fails, even when physical huge pages are available.

The program performs the following steps to trigger the failure:

1. Enable hugetlb accounting in cgroup v2.
     + The program checks if memory_hugetlb_accounting is enabled in the cgroup2
       mount options. If not, it remounts /sys/fs/cgroup with this option
       enabled. This ensures that HugeTLB allocations are charged against the
       cgroup memory limits.

2. Create a test cgroup and set limits.
     + The program creates a new cgroup subdirectory named test_reproducer under
       /sys/fs/cgroup.
     + It sets the memory.max limit of this cgroup to 1MB (which is less than
       the 2MB huge page size).

3. Fork a child process and move it to the test cgroup.
     + The program forks a child process.
     + The child process moves itself into the test_reproducer cgroup by writing
       its PID (using 0 for current process) to cgroup.procs in the test cgroup
       directory.

4. Attempt to allocate and touch a 2MB huge page.
     + The child process maps a 2MB anonymous huge page using mmap with
       MAP_PRIVATE, MAP_ANONYMOUS, and MAP_HUGETLB.
     + The child process writes to the mapped address, triggering a page fault.

5. Triggering the kernel bugs.
     + The page fault handler calls alloc_hugetlb_folio to allocate the huge
       page.
     + The allocation of the physical page from buddy allocator succeeds
       (assuming nr_hugepages is sufficient).
     + The kernel then attempts to charge this allocation to the child process's
       cgroup by calling mem_cgroup_charge_hugetlb.
     + Since the child's cgroup memory limit is 1MB and the page is 2MB, the
       charge fails and mem_cgroup_charge_hugetlb returns -ENOMEM.
     + This triggers the error path in alloc_hugetlb_folio where the bugs (folio
       refcount mismatch, infinite loop on ENOMEM, and reservation leaks) are
       handled.
---
 cgroup_v2_allocation_failure.c | 160 +++++++++++++++++++++++++++++++++
 1 file changed, 160 insertions(+)
 create mode 100644 cgroup_v2_allocation_failure.c

diff --git a/cgroup_v2_allocation_failure.c b/cgroup_v2_allocation_failure.c
new file mode 100644
index 0000000000000..938cbf02ae6f7
--- /dev/null
+++ b/cgroup_v2_allocation_failure.c
@@ -0,0 +1,160 @@
+#define _GNU_SOURCE
+#include <stdio.h>
+#include <stdlib.h>
+#include <unistd.h>
+#include <fcntl.h>
+#include <sys/mman.h>
+#include <sys/stat.h>
+#include <sys/types.h>
+#include <sys/wait.h>
+#include <string.h>
+#include <errno.h>
+
+#define CGROUP_PATH "/sys/fs/cgroup"
+#define TEST_CGROUP "test_reproducer"
+#define TEST_CGROUP_PATH CGROUP_PATH "/" TEST_CGROUP
+
+void write_file(const char *path, const char *val) {
+    int fd = open(path, O_WRONLY);
+    if (fd < 0) {
+        fprintf(stderr, "Failed to open %s: %s\n", path, strerror(errno));
+        exit(1);
+    }
+    if (write(fd, val, strlen(val)) < 0) {
+        fprintf(stderr, "Failed to write %s to %s: %s\n", val, path, strerror(errno));
+        close(fd);
+        exit(1);
+    }
+    close(fd);
+}
+
+int is_hugetlb_accounting_enabled() {
+    FILE *fp = fopen("/proc/mounts", "r");
+    if (!fp) {
+        perror("fopen /proc/mounts");
+        return -1;
+    }
+
+    char line[1024];
+    int enabled = 0;
+    while (fgets(line, sizeof(line), fp)) {
+        char spec[256], file[256], type[256], opts[512];
+        if (sscanf(line, "%255s %255s %255s %511s", spec, file, type, opts) == 4) {
+            if (strcmp(file, CGROUP_PATH) == 0 && strcmp(type, "cgroup2") == 0) {
+                if (strstr(opts, "memory_hugetlb_accounting") != NULL) {
+                    enabled = 1;
+                }
+                break;
+            }
+        }
+    }
+    fclose(fp);
+    return enabled;
+}
+
+int enable_hugetlb_accounting() {
+    printf("Attempting to remount cgroup2 with memory_hugetlb_accounting...\n");
+    int ret = system("mount -o remount,memory_hugetlb_accounting " CGROUP_PATH);
+    if (ret != 0) {
+        fprintf(stderr, "Failed to remount: system() returned %d\n", ret);
+        return -1;
+    }
+    return 0;
+}
+
+int main() {
+    struct stat st;
+    if (stat(CGROUP_PATH, &st) != 0 || !S_ISDIR(st.st_mode)) {
+        fprintf(stderr, "cgroup v2 not mounted at %s\n", CGROUP_PATH);
+        return 1;
+    }
+
+    int enabled = is_hugetlb_accounting_enabled();
+    if (enabled < 0) {
+        return 1;
+    }
+    if (!enabled) {
+        if (enable_hugetlb_accounting() != 0) {
+            fprintf(stderr, "Could not enable memory_hugetlb_accounting\n");
+            return 1;
+        }
+        // Re-check
+        enabled = is_hugetlb_accounting_enabled();
+        if (enabled <= 0) {
+            fprintf(stderr, "Failed to enable memory_hugetlb_accounting (re-check failed)\n");
+            return 1;
+        }
+        printf("Successfully enabled memory_hugetlb_accounting\n");
+    } else {
+        printf("memory_hugetlb_accounting is already enabled\n");
+    }
+
+    // Enable memory controller in subtree
+    int fd = open(CGROUP_PATH "/cgroup.subtree_control", O_WRONLY);
+    if (fd >= 0) {
+        if (write(fd, "+memory", 7) < 0) {
+            // Might fail if already enabled or not supported, ignore for now
+        }
+        close(fd);
+    }
+
+    if (mkdir(TEST_CGROUP_PATH, 0755) != 0) {
+        if (errno != EEXIST) {
+            perror("mkdir test_reproducer");
+            return 1;
+        }
+    }
+
+    // Set memory limit to 1MB (less than 2MB hugepage)
+    write_file(TEST_CGROUP_PATH "/memory.max", "1M");
+
+    pid_t pid = fork();
+    if (pid < 0) {
+        perror("fork");
+        return 1;
+    }
+
+    if (pid == 0) {
+        // Child
+        // Move to cgroup
+        write_file(TEST_CGROUP_PATH "/cgroup.procs", "0");
+
+        printf("Child: Attempting to allocate and touch 2MB hugepage...\n");
+        // Allocate 2MB hugepage
+        size_t size = 2 * 1024 * 1024;
+        void *addr = mmap(NULL, size, PROT_READ | PROT_WRITE,
+                          MAP_PRIVATE | MAP_ANONYMOUS | MAP_HUGETLB, -1, 0);
+        if (addr == MAP_FAILED) {
+            perror("Child: mmap MAP_HUGETLB");
+            exit(1);
+        }
+
+        printf("Child: mmap succeeded at %p, touching it now (should trigger fault)...\n", addr);
+        // This should trigger the fault and call alloc_hugetlb_folio -> mem_cgroup_charge_hugetlb
+        // which should fail and trigger the bug.
+        *(volatile char *)addr = 1;
+
+        printf("Child: Successfully touched page (bug not triggered?).\n");
+        munmap(addr, size);
+        exit(0);
+    }
+
+    // Parent
+    int status;
+    waitpid(pid, &status, 0);
+
+    printf("Parent: Child exited. Cleaning up.\n");
+    rmdir(TEST_CGROUP_PATH);
+
+    if (WIFSIGNALED(status)) {
+        printf("Parent: Child killed by signal %d (%s)\n",
+               WTERMSIG(status), strsignal(WTERMSIG(status)));
+        if (WTERMSIG(status) == SIGBUS) {
+            printf("Parent: Child got SIGBUS as expected (if kernel didn't crash).\n");
+        }
+    } else if (WIFEXITED(status)) {
+        printf("Parent: Child exited with status %d\n", WEXITSTATUS(status));
+    }
+
+    return 0;
+}
-- 
2.55.0.795.g602f6c329a-goog

Re: [PATCH v2 0/5] Fix bugs on HugeTLB folio allocation failure paths

[Ackerley Tng]

Ackerley Tng via B4 Relay <devnull+ackerleytng.google.com@kernel.org>
writes:

Looking at Sashiko's comments [1] again, it looks like there are even
more issues. Seems like these bugs are quite deep into the weeds of
accounting, would definitely like some advice from reviewers and
maintainers!

1. Are the issues I've described the correct issues or did I get the
   wrong root causes? Is this the right direction?

2. Is there a good way to split up this series so at least some bugs are
   fixed? I kind of clumped them together because one fix revealed
   another bug and so on... Is it okay to merge some patches that we're
   sure fixes something and leave other bugs to be fixed separately? (If
   we get to a nice fix within a few iterations we don't have to split
   up the series)

[1] https://sashiko.dev/#/patchset/20260708-hugetlb-alloc-failure-fixes-v2-0-c7f27cbb462b%40google.com