[PATCH 6.6] mm: replace free hugepage folios after migration

[PATCH 6.6] mm: replace free hugepage folios after migration

[Mark Bundschuh]

From: yangge <yangge1116@126.com>

[ Upstream commit 04f13d241b8b146b23038bffd907cb8278391d07 ]

My machine has 4 NUMA nodes, each equipped with 32GB of memory.  I have
configured each NUMA node with 16GB of CMA and 16GB of in-use hugetlb
pages.  The allocation of contiguous memory via cma_alloc() can fail
probabilistically.

When there are free hugetlb folios in the hugetlb pool, during the
migration of in-use hugetlb folios, new folios are allocated from the free
hugetlb pool.  After the migration is completed, the old folios are
released back to the free hugetlb pool instead of being returned to the
buddy system.  This can cause test_pages_isolated() check to fail,
ultimately leading to the failure of cma_alloc().

Call trace:

cma_alloc()
    __alloc_contig_migrate_range() // migrate in-use hugepage
    test_pages_isolated()
        __test_page_isolated_in_pageblock()
             PageBuddy(page) // check if the page is in buddy

To address this issue, we introduce a function named
replace_free_hugepage_folios().  This function will replace the hugepage
in the free hugepage pool with a new one and release the old one to the
buddy system.  After the migration of in-use hugetlb pages is completed,
we will invoke replace_free_hugepage_folios() to ensure that these
hugepages are properly released to the buddy system.  Following this step,
when test_pages_isolated() is executed for inspection, it will
successfully pass.

Additionally, when alloc_contig_range() is used to migrate multiple in-use
hugetlb pages, it can result in some in-use hugetlb pages being released
back to the free hugetlb pool and subsequently being reallocated and used
again.  For example:

[huge 0] [huge 1]

To migrate huge 0, we obtain huge x from the pool.  After the migration is
completed, we return the now-freed huge 0 back to the pool.  When it's
time to migrate huge 1, we can simply reuse the now-freed huge 0 from the
pool.  As a result, when replace_free_hugepage_folios() is executed, it
cannot release huge 0 back to the buddy system.  To address this issue, we
should prevent the reuse of isolated free hugepages during the migration
process.

Link: https://lkml.kernel.org/r/1734503588-16254-1-git-send-email-yangge1116@126.com
Link: https://lkml.kernel.org/r/1736582300-11364-1-git-send-email-yangge1116@126.com
Signed-off-by: yangge <yangge1116@126.com>
Cc: Baolin Wang <baolin.wang@linux.alibaba.com>
Cc: Barry Song <21cnbao@gmail.com>
Cc: David Hildenbrand <david@redhat.com>
Cc: SeongJae Park <sj@kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
[ 6.6: Take only the dequeue_hugetlb_folio_node_exact() hunk, so memory
  offline stops migrating huge pages back into the isolated range. ]
Signed-off-by: Mark Bundschuh <mkbund@amazon.com>
---
Without this, offlining a memory block holding mapped huge pages keeps
migrating them back into the block until the offline is interrupted.
Reproduced on arm64 on 5.10 and 6.12 (stock AL2023 6.12.113): the offline
was still migrating after 60 s. With this patch on 5.10 it completes in
0.15 s with the data intact. Also sent for 5.10, 5.15, 6.1, 6.6 and 6.12.

 mm/hugetlb.c | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/mm/hugetlb.c b/mm/hugetlb.c
index 8bfaaa84bb3f..8c1a032a6f53 100644
--- a/mm/hugetlb.c
+++ b/mm/hugetlb.c
@@ -47,6 +47,7 @@
 #include <linux/page_owner.h>
 #include "internal.h"
 #include "hugetlb_vmemmap.h"
+#include <linux/page-isolation.h>
 
 int hugetlb_max_hstate __read_mostly;
 unsigned int default_hstate_idx;
@@ -1344,6 +1345,9 @@ static struct folio *dequeue_hugetlb_folio_node_exact(struct hstate *h,
 		if (folio_test_hwpoison(folio))
 			continue;
 
+		if (is_migrate_isolate_page(&folio->page))
+			continue;
+
 		list_move(&folio->lru, &h->hugepage_activelist);
 		folio_ref_unfreeze(folio, 1);
 		folio_clear_hugetlb_freed(folio);
-- 
2.50.1

Re: [PATCH 6.6] mm: replace free hugepage folios after migration

[sashiko-bot]

Thank you for your contribution! Sashiko AI review found 2 potential
issues to consider:
- [High] mm: missing replace_free_hugepage_folios() breaks
  alloc_contig_range()
- [High] mm: hugetlb: skipping isolated folios in dequeue breaks resv
  accounting
--

commit bc6da6c8bf105f6ed6558b1313cb2fed9d8b04a8
Author: yangge <yangge1116@126.com>

mm: replace free hugepage folios after migration

This backport updates dequeue_hugetlb_folio_node_exact() in mm/hugetlb.c to
skip free hugetlb folios located in MIGRATE_ISOLATE pageblocks so migration
does not reuse isolated hugepages. However, it omits the
replace_free_hugepage_folios() implementation and call site from the upstream
commit while retaining the upstream commit title and changelog.

Link: https://lkml.kernel.org/r/1734503588-16254-1-git-send-email-yangge1116@126.com
Link: https://lkml.kernel.org/r/1736582300-11364-1-git-send-email-yangge1116@126.com

[Severity: High]
In mm/hugetlb.c at dequeue_hugetlb_folio_node_exact() (line 1348), this
backport only takes the check for is_migrate_isolate_page(&folio->page) and
omits replace_free_hugepage_folios() along with its call after
__alloc_contig_migrate_range() in alloc_contig_range() (mm/page_alloc.c, line
6267). Does omitting replace_free_hugepage_folios() leave
alloc_contig_range() and cma_alloc() broken while also shrinking the
persistent hugetlb pool during memory offlining?

When alloc_contig_range() calls start_isolate_page_range() and
__alloc_contig_migrate_range() on a range containing in-use hugetlb folios,
alloc_migration_target() -> alloc_hugetlb_folio_nodemask() (mm/hugetlb.c,
line 2488) checks available_huge_pages(h) and dequeues a free folio from
h->hugepage_freelists[nid] instead of allocating a temporary folio via
alloc_migrate_hugetlb_folio(). Because HPG_temporary is not set on the source
folio, when migration finishes and free_huge_folio() (mm/hugetlb.c, line
1945) releases the old folio in the isolated range, enqueue_hugetlb_folio()
places it back onto h->hugepage_freelists[nid] rather than freeing it to the
buddy allocator.

Because isolate_migratepages_block() in mm/compaction.c already advanced past
the migrated folio and replace_free_hugepage_folios() is never called (even
though alloc_and_dissolve_hugetlb_folio() is already available in
mm/hugetlb.c at line 3050), the freed source folio remains on
h->hugepage_freelists when test_pages_isolated() runs in alloc_contig_range()
at line 6297, causing alloc_contig_range() and cma_alloc() to still fail with
-EBUSY. Moreover, when multiple in-use hugetlb folios are migrated, skipping
MIGRATE_ISOLATE folios in dequeue_hugetlb_folio_node_exact() without
replacing the freed source folios causes each migration to pull another free
pool folio from outside the range into use and leave multiple freed folios on
hugepage_freelists inside the isolated range before test_pages_isolated()
fails.

Similarly, during offline_pages() in mm/memory_hotplug.c, when an in-use
hugetlb folio in the offlined range is migrated into a free persistent folio
outside the range, the old source folio is returned to h->hugepage_freelists
inside the offlined range and then dissolved by dissolve_free_huge_pages() ->
dissolve_free_huge_page() (mm/hugetlb.c, line 2305), which decrements
h->max_huge_pages and shrinks the persistent hugetlb pool. Finally, the
commit subject and changelog still state that replace_free_hugepage_folios()
is introduced and called after migration, which no longer matches the patch.

[Severity: High]
In mm/hugetlb.c at dequeue_hugetlb_folio_node_exact() (line 1348),
is_migrate_isolate_page(&folio->page) skips isolated folios on
h->hugepage_freelists[nid], but those folios remain counted in
h->free_huge_pages. Can this mismatch with available_huge_pages()
(mm/hugetlb.c, line 1398) cause migration to consume reserved free folios,
breaking offline_pages() and causing SIGBUS on reserved hugetlb faults?

Because available_huge_pages(h) returns h->free_huge_pages -
h->resv_huge_pages, any free hugetlb folio in a MIGRATE_ISOLATE pageblock
(either isolated beforehand by start_isolate_page_range() or placed on
h->hugepage_freelists[nid] by free_huge_folio() after migrating an in-use
folio) still inflates available_huge_pages(h) even though
dequeue_hugetlb_folio_node_exact() refuses to dequeue it.

Suppose h->resv_huge_pages is R (backed by R non-isolated free folios) and
1 unreserved free folio is in a MIGRATE_ISOLATE pageblock, so
h->free_huge_pages is R + 1 and available_huge_pages(h) is 1. When
alloc_hugetlb_folio_nodemask() (mm/hugetlb.c, line 2490) runs to migrate the
next in-use hugetlb folio, available_huge_pages(h) is non-zero, so it calls
dequeue_hugetlb_folio_nodemask() -> dequeue_hugetlb_folio_node_exact(), which
skips the MIGRATE_ISOLATE folio and dequeues one of the R non-isolated folios
backing h->resv_huge_pages instead of falling back to
alloc_migrate_hugetlb_folio(). Once the migrated source folio is freed into
the MIGRATE_ISOLATE range, available_huge_pages(h) remains 1, allowing
subsequent migrations to move all R reserved folios into the MIGRATE_ISOLATE
range.

When offline_pages() later calls dissolve_free_huge_pages(),
dissolve_free_huge_page() (mm/hugetlb.c, line 2305) dissolves the first
unreserved folio (reducing available_huge_pages(h) to 0 and decrementing
h->max_huge_pages) and then fails with -EBUSY on the remaining isolated free
folios because available_huge_pages(h) is now 0, aborting memory offlining.

In addition, while the free folios backing h->resv_huge_pages sit in
MIGRATE_ISOLATE pageblocks, a page fault on a VMA with a valid reservation
calls alloc_hugetlb_folio() -> dequeue_hugetlb_folio_vma() (mm/hugetlb.c,
line 3113), which returns NULL because dequeue_hugetlb_folio_node_exact()
skips the isolated folios. alloc_hugetlb_folio() then falls back to
alloc_buddy_hugetlb_folio_with_mpol() -> alloc_surplus_hugetlb_folio(), which
fails when h->surplus_huge_pages >= h->nr_overcommit_huge_pages (0 by
default, and always 0 for gigantic pages), returning -ENOSPC and delivering
SIGBUS to a process faulting on validly reserved hugepages.

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20261105070431.3100950-1-mkbund@amazon.com?part=1

[PATCH v2 6.6 0/2] mm: replace free hugepage folios after migration

[Mark Bundschuh]

Full backport of 04f13d241b8b ("mm: replace free hugepage folios after
migration") with its fix 113ed54ad276.

Tested in a VM on 6.6.161: memory offline of a block with mapped huge
pages no longer livelocks, and cma_alloc() over mapped huge pages now
succeeds (it failed on 6.6.161 and with v1).

v1: https://lore.kernel.org/all/20261105070431.3100950-1-mkbund@amazon.com/

Ge Yang (1):
  mm/hugetlb: fix kernel NULL pointer dereference when replacing free
    hugetlb folios

yangge (1):
  mm: replace free hugepage folios after migration

 include/linux/hugetlb.h |  7 ++++++
 mm/hugetlb.c            | 50 +++++++++++++++++++++++++++++++++++++++++
 mm/page_alloc.c         | 12 +++++++++-
 3 files changed, 68 insertions(+), 1 deletion(-)

-- 
2.50.1

[PATCH v2 6.6 1/2] mm: replace free hugepage folios after migration

[Mark Bundschuh]

From: yangge <yangge1116@126.com>

[ Upstream commit 04f13d241b8b146b23038bffd907cb8278391d07 ]

My machine has 4 NUMA nodes, each equipped with 32GB of memory.  I have
configured each NUMA node with 16GB of CMA and 16GB of in-use hugetlb
pages.  The allocation of contiguous memory via cma_alloc() can fail
probabilistically.

When there are free hugetlb folios in the hugetlb pool, during the
migration of in-use hugetlb folios, new folios are allocated from the free
hugetlb pool.  After the migration is completed, the old folios are
released back to the free hugetlb pool instead of being returned to the
buddy system.  This can cause test_pages_isolated() check to fail,
ultimately leading to the failure of cma_alloc().

Call trace:

cma_alloc()
    __alloc_contig_migrate_range() // migrate in-use hugepage
    test_pages_isolated()
        __test_page_isolated_in_pageblock()
             PageBuddy(page) // check if the page is in buddy

To address this issue, we introduce a function named
replace_free_hugepage_folios().  This function will replace the hugepage
in the free hugepage pool with a new one and release the old one to the
buddy system.  After the migration of in-use hugetlb pages is completed,
we will invoke replace_free_hugepage_folios() to ensure that these
hugepages are properly released to the buddy system.  Following this step,
when test_pages_isolated() is executed for inspection, it will
successfully pass.

Additionally, when alloc_contig_range() is used to migrate multiple in-use
hugetlb pages, it can result in some in-use hugetlb pages being released
back to the free hugetlb pool and subsequently being reallocated and used
again.  For example:

[huge 0] [huge 1]

To migrate huge 0, we obtain huge x from the pool.  After the migration is
completed, we return the now-freed huge 0 back to the pool.  When it's
time to migrate huge 1, we can simply reuse the now-freed huge 0 from the
pool.  As a result, when replace_free_hugepage_folios() is executed, it
cannot release huge 0 back to the buddy system.  To address this issue, we
should prevent the reuse of isolated free hugepages during the migration
process.

Link: https://lkml.kernel.org/r/1734503588-16254-1-git-send-email-yangge1116@126.com
Link: https://lkml.kernel.org/r/1736582300-11364-1-git-send-email-yangge1116@126.com
Signed-off-by: yangge <yangge1116@126.com>
Cc: Baolin Wang <baolin.wang@linux.alibaba.com>
Cc: Barry Song <21cnbao@gmail.com>
Cc: David Hildenbrand <david@redhat.com>
Cc: SeongJae Park <sj@kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Mark Bundschuh <mkbund@amazon.com>
---
 include/linux/hugetlb.h |  7 +++++++
 mm/hugetlb.c            | 42 +++++++++++++++++++++++++++++++++++++++++
 mm/page_alloc.c         | 12 +++++++++++-
 3 files changed, 60 insertions(+), 1 deletion(-)

diff --git a/include/linux/hugetlb.h b/include/linux/hugetlb.h
index 6ecf2907172b..996fee996517 100644
--- a/include/linux/hugetlb.h
+++ b/include/linux/hugetlb.h
@@ -745,6 +745,7 @@ struct huge_bootmem_page {
 };
 
 int isolate_or_dissolve_huge_page(struct page *page, struct list_head *list);
+int replace_free_hugepage_folios(unsigned long start_pfn, unsigned long end_pfn);
 struct folio *alloc_hugetlb_folio(struct vm_area_struct *vma,
 				unsigned long addr, int avoid_reserve);
 struct folio *alloc_hugetlb_folio_nodemask(struct hstate *h, int preferred_nid,
@@ -1049,6 +1050,12 @@ static inline int isolate_or_dissolve_huge_page(struct page *page,
 	return -ENOMEM;
 }
 
+static inline int replace_free_hugepage_folios(unsigned long start_pfn,
+		unsigned long end_pfn)
+{
+	return 0;
+}
+
 static inline struct folio *alloc_hugetlb_folio(struct vm_area_struct *vma,
 					   unsigned long addr,
 					   int avoid_reserve)
diff --git a/mm/hugetlb.c b/mm/hugetlb.c
index 8bfaaa84bb3f..2ef28c49b9d0 100644
--- a/mm/hugetlb.c
+++ b/mm/hugetlb.c
@@ -47,6 +47,7 @@
 #include <linux/page_owner.h>
 #include "internal.h"
 #include "hugetlb_vmemmap.h"
+#include <linux/page-isolation.h>
 
 int hugetlb_max_hstate __read_mostly;
 unsigned int default_hstate_idx;
@@ -1344,6 +1345,9 @@ static struct folio *dequeue_hugetlb_folio_node_exact(struct hstate *h,
 		if (folio_test_hwpoison(folio))
 			continue;
 
+		if (is_migrate_isolate_page(&folio->page))
+			continue;
+
 		list_move(&folio->lru, &h->hugepage_activelist);
 		folio_ref_unfreeze(folio, 1);
 		folio_clear_hugetlb_freed(folio);
@@ -3048,6 +3052,44 @@ int isolate_or_dissolve_huge_page(struct page *page, struct list_head *list)
 	return ret;
 }
 
+/*
+ *  replace_free_hugepage_folios - Replace free hugepage folios in a given pfn
+ *  range with new folios.
+ *  @start_pfn: start pfn of the given pfn range
+ *  @end_pfn: end pfn of the given pfn range
+ *  Returns 0 on success, otherwise negated error.
+ */
+int replace_free_hugepage_folios(unsigned long start_pfn, unsigned long end_pfn)
+{
+	struct hstate *h;
+	struct folio *folio;
+	int ret = 0;
+
+	LIST_HEAD(isolate_list);
+
+	while (start_pfn < end_pfn) {
+		folio = pfn_folio(start_pfn);
+		if (folio_test_hugetlb(folio)) {
+			h = folio_hstate(folio);
+		} else {
+			start_pfn++;
+			continue;
+		}
+
+		if (!folio_ref_count(folio)) {
+			ret = alloc_and_dissolve_hugetlb_folio(h, folio,
+							       &isolate_list);
+			if (ret)
+				break;
+
+			putback_movable_pages(&isolate_list);
+		}
+		start_pfn++;
+	}
+
+	return ret;
+}
+
 struct folio *alloc_hugetlb_folio(struct vm_area_struct *vma,
 				    unsigned long addr, int avoid_reserve)
 {
diff --git a/mm/page_alloc.c b/mm/page_alloc.c
index 06c493b908ed..7c8dffb7f05f 100644
--- a/mm/page_alloc.c
+++ b/mm/page_alloc.c
@@ -6254,7 +6254,17 @@ int alloc_contig_range(unsigned long start, unsigned long end,
 	ret = __alloc_contig_migrate_range(&cc, start, end);
 	if (ret && ret != -EBUSY)
 		goto done;
-	ret = 0;
+
+	/*
+	 * When in-use hugetlb pages are migrated, they may simply be released
+	 * back into the free hugepage pool instead of being returned to the
+	 * buddy system.  After the migration of in-use huge pages is completed,
+	 * we will invoke replace_free_hugepage_folios() to ensure that these
+	 * hugepages are properly released to the buddy system.
+	 */
+	ret = replace_free_hugepage_folios(start, end);
+	if (ret)
+		goto done;
 
 	/*
 	 * Pages from [start, end) are within a pageblock_nr_pages
-- 
2.50.1

[PATCH v2 6.6 2/2] mm/hugetlb: fix kernel NULL pointer dereference when replacing free hugetlb folios

[Mark Bundschuh]

From: Ge Yang <yangge1116@126.com>

[ Upstream commit 113ed54ad276c352ee5ce109bdcf0df118a43bda ]

A kernel crash was observed when replacing free hugetlb folios:

BUG: kernel NULL pointer dereference, address: 0000000000000028
PGD 0 P4D 0
Oops: Oops: 0000 [#1] SMP NOPTI
CPU: 28 UID: 0 PID: 29639 Comm: test_cma.sh Tainted 6.15.0-rc6-zp #41 PREEMPT(voluntary)
RIP: 0010:alloc_and_dissolve_hugetlb_folio+0x1d/0x1f0
RSP: 0018:ffffc9000b30fa90 EFLAGS: 00010286
RAX: 0000000000000000 RBX: 0000000000342cca RCX: ffffea0043000000
RDX: ffffc9000b30fb08 RSI: ffffea0043000000 RDI: 0000000000000000
RBP: ffffc9000b30fb20 R08: 0000000000001000 R09: 0000000000000000
R10: ffff88886f92eb00 R11: 0000000000000000 R12: ffffea0043000000
R13: 0000000000000000 R14: 00000000010c0200 R15: 0000000000000004
FS:  00007fcda5f14740(0000) GS:ffff8888ec1d8000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000000000000028 CR3: 0000000391402000 CR4: 0000000000350ef0
Call Trace:
<TASK>
 replace_free_hugepage_folios+0xb6/0x100
 alloc_contig_range_noprof+0x18a/0x590
 ? srso_return_thunk+0x5/0x5f
 ? down_read+0x12/0xa0
 ? srso_return_thunk+0x5/0x5f
 cma_range_alloc.constprop.0+0x131/0x290
 __cma_alloc+0xcf/0x2c0
 cma_alloc_write+0x43/0xb0
 simple_attr_write_xsigned.constprop.0.isra.0+0xb2/0x110
 debugfs_attr_write+0x46/0x70
 full_proxy_write+0x62/0xa0
 vfs_write+0xf8/0x420
 ? srso_return_thunk+0x5/0x5f
 ? filp_flush+0x86/0xa0
 ? srso_return_thunk+0x5/0x5f
 ? filp_close+0x1f/0x30
 ? srso_return_thunk+0x5/0x5f
 ? do_dup2+0xaf/0x160
 ? srso_return_thunk+0x5/0x5f
 ksys_write+0x65/0xe0
 do_syscall_64+0x64/0x170
 entry_SYSCALL_64_after_hwframe+0x76/0x7e

There is a potential race between __update_and_free_hugetlb_folio() and
replace_free_hugepage_folios():

CPU1                              CPU2
__update_and_free_hugetlb_folio   replace_free_hugepage_folios
                                    folio_test_hugetlb(folio)
                                    -- It's still hugetlb folio.

  __folio_clear_hugetlb(folio)
  hugetlb_free_folio(folio)
                                    h = folio_hstate(folio)
                                    -- Here, h is NULL pointer

When the above race condition occurs, folio_hstate(folio) returns NULL,
and subsequent access to this NULL pointer will cause the system to crash.
To resolve this issue, execute folio_hstate(folio) under the protection
of the hugetlb_lock lock, ensuring that folio_hstate(folio) does not
return NULL.

Link: https://lkml.kernel.org/r/1747884137-26685-1-git-send-email-yangge1116@126.com
Fixes: 04f13d241b8b ("mm: replace free hugepage folios after migration")
Signed-off-by: Ge Yang <yangge1116@126.com>
Reviewed-by: Muchun Song <muchun.song@linux.dev>
Reviewed-by: Oscar Salvador <osalvador@suse.de>
Cc: Baolin Wang <baolin.wang@linux.alibaba.com>
Cc: Barry Song <21cnbao@gmail.com>
Cc: David Hildenbrand <david@redhat.com>
Cc: <stable@vger.kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Mark Bundschuh <mkbund@amazon.com>
---
 mm/hugetlb.c | 8 ++++++++
 1 file changed, 8 insertions(+)

diff --git a/mm/hugetlb.c b/mm/hugetlb.c
index 2ef28c49b9d0..b755fe95f9fc 100644
--- a/mm/hugetlb.c
+++ b/mm/hugetlb.c
@@ -3069,12 +3069,20 @@ int replace_free_hugepage_folios(unsigned long start_pfn, unsigned long end_pfn)
 
 	while (start_pfn < end_pfn) {
 		folio = pfn_folio(start_pfn);
+
+		/*
+		 * The folio might have been dissolved from under our feet, so make sure
+		 * to carefully check the state under the lock.
+		 */
+		spin_lock_irq(&hugetlb_lock);
 		if (folio_test_hugetlb(folio)) {
 			h = folio_hstate(folio);
 		} else {
+			spin_unlock_irq(&hugetlb_lock);
 			start_pfn++;
 			continue;
 		}
+		spin_unlock_irq(&hugetlb_lock);
 
 		if (!folio_ref_count(folio)) {
 			ret = alloc_and_dissolve_hugetlb_folio(h, folio,
-- 
2.50.1

Re: [PATCH v2 6.6 1/2] mm: replace free hugepage folios after migration

[sashiko-bot]

Thank you for your contribution! Sashiko AI review found 3 potential
issues to consider:
- [High] mm: hugetlb: skipping isolated folios in dequeue breaks resv
  accounting
- [Medium] mm: hugetlb: pfn step in replace_free_hugepage_folios()
  over-allocates
- [Medium] mm: hugetlb: missing free_hpage_work flush after hugetlb
  migration
--

commit 001656b3d65502344f54f0dc762e32f45c183341
Author: yangge <yangge1116@126.com>

mm: replace free hugepage folios after migration

This commit skips isolated free hugetlb folios during pool dequeuing and
adds replace_free_hugepage_folios() to replace and dissolve free hugetlb
folios in the target range after migration in alloc_contig_range().

Link: https://lkml.kernel.org/r/1734503588-16254-1-git-send-email-yangge1116@126.com
Link: https://lkml.kernel.org/r/1736582300-11364-1-git-send-email-yangge1116@126.com

> diff --git a/mm/hugetlb.c b/mm/hugetlb.c
> index 8bfaaa84bb3f7..2ef28c49b9d0f 100644
> --- a/mm/hugetlb.c
> +++ b/mm/hugetlb.c

[ ... ]

> @@ -1344,6 +1345,9 @@ static struct folio *dequeue_hugetlb_folio_node_exact(struct hstate *h,
>  		if (folio_test_hwpoison(folio))
>  			continue;
>  
> +		if (is_migrate_isolate_page(&folio->page))
> +			continue;
> +
>  		list_move(&folio->lru, &h->hugepage_activelist);
>  		folio_ref_unfreeze(folio, 1);
>  		folio_clear_hugetlb_freed(folio);

[Severity: High]
Does skipping isolated folios in dequeue_hugetlb_folio_node_exact() while
still counting them in h->free_huge_pages break hugetlb reservation
accounting?

available_huge_pages(h) computes h->free_huge_pages - h->resv_huge_pages,
so a free folio in a MIGRATE_ISOLATE pageblock still appears as an
available unreserved page even though it cannot be dequeued.  When
alloc_hugetlb_folio_nodemask() or an unreserved fault sees
available_huge_pages(h) > 0, dequeue_hugetlb_folio_node_exact() skips the
isolated folio and consumes a non-isolated free folio that was backing
h->resv_huge_pages instead of allocating a temporary migration folio via
alloc_migrate_hugetlb_folio().

Once the reserved free folios have been replaced by isolated free folios:

1. A page fault on a VMA with a valid reservation sees
   dequeue_hugetlb_folio_vma() return NULL and falls back to
   alloc_buddy_hugetlb_folio_with_mpol(), which fails with -ENOSPC (and
   delivers SIGBUS) whenever surplus overcommit is disabled
   (nr_overcommit_huge_pages == 0 by default, or gigantic pages).

2. In offline_pages(), dissolve_free_huge_pages() dissolves the unreserved
   folio (dropping available_huge_pages(h) to 0 and shrinking
   h->max_huge_pages) and then fails with -EBUSY on the remaining isolated
   free folios because !available_huge_pages(h).

Also, is_migrate_isolate_page(&folio->page) only checks the head pageblock
and misses isolated tail pageblocks when huge_page_order(h) >
pageblock_order.

> @@ -3048,6 +3052,44 @@ int isolate_or_dissolve_huge_page(struct page *page, struct list_head *list)
>  	return ret;
>  }
>  
> +/*
> + *  replace_free_hugepage_folios - Replace free hugepage folios in a given pfn
> + *  range with new folios.
> + *  @start_pfn: start pfn of the given pfn range
> + *  @end_pfn: end pfn of the given pfn range
> + *  Returns 0 on success, otherwise negated error.
> + */
> +int replace_free_hugepage_folios(unsigned long start_pfn, unsigned long end_pfn)
> +{
> +	struct hstate *h;
> +	struct folio *folio;
> +	int ret = 0;
> +
> +	LIST_HEAD(isolate_list);
> +
> +	while (start_pfn < end_pfn) {
> +		folio = pfn_folio(start_pfn);
> +		if (folio_test_hugetlb(folio)) {
> +			h = folio_hstate(folio);
> +		} else {
> +			start_pfn++;
> +			continue;
> +		}
> +
> +		if (!folio_ref_count(folio)) {
> +			ret = alloc_and_dissolve_hugetlb_folio(h, folio,
> +							       &isolate_list);

[Severity: Medium]
When HugeTLB vmemmap optimization (HVO) is active, can
replace_free_hugepage_folios() miss temporary or surplus folios that were
just freed by __alloc_contig_migrate_range()?

Because dequeue_hugetlb_folio_node_exact() now skips isolated free folios,
hugetlb migrations in the range fall back to alloc_migrate_hugetlb_folio(),
which sets HPG_temporary.  When migration finishes, free_huge_folio() frees
the old folio via update_and_free_hugetlb_folio(h, folio, true); for an HVO
folio, this keeps a reference count of 1 and queues the folio onto
hpage_freelist for free_hpage_work to restore vmemmap asynchronously.

Without flushing free_hpage_work (for example via
wait_for_freed_hugetlb_folios() or flush_free_hpage_work(h)) before
scanning the range, folio_ref_count(folio) is still 1 here and
test_pages_isolated() subsequently fails with -EBUSY because the folio has
not yet been returned to the buddy allocator.

> +			if (ret)
> +				break;
> +
> +			putback_movable_pages(&isolate_list);
> +		}
> +		start_pfn++;
> +	}

[Severity: Medium]
Can advancing start_pfn by 1 cause replace_free_hugepage_folios() to call
alloc_and_dissolve_hugetlb_folio() on every tail page of the same hugepage
if it was not dissolved?

In alloc_and_dissolve_hugetlb_folio(), __update_and_free_hugetlb_folio()
can fail to restore vmemmap under memory pressure and put old_folio back on
the hugetlb freelist as a surplus page with refcount 0 while still
returning 0:

mm/hugetlb.c:__update_and_free_hugetlb_folio() {
    ...
	if (hugetlb_vmemmap_restore(h, &folio->page)) {
		spin_lock_irq(&hugetlb_lock);
		add_hugetlb_folio(h, folio, true);
		spin_unlock_irq(&hugetlb_lock);
		return;
	}
    ...
}

Because old_folio remains a free hugetlb folio and start_pfn only increments
by 1, the loop hits the same folio on each of its remaining tail PFNs,
allocating a new replacement hugepage and incrementing h->nr_huge_pages and
h->surplus_huge_pages on every tail PFN (up to 511 extra hugepages for a
2MB folio).  The same repeated scan happens if
alloc_and_dissolve_hugetlb_folio() isolates a concurrently referenced folio
into isolate_list and putback_movable_pages(&isolate_list) immediately
releases it.

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20261106220812.2874571-1-mkbund@amazon.com?part=1