[PATCH 6.12] mm: replace free hugepage folios after migration

[PATCH 6.12] mm: replace free hugepage folios after migration

[Mark Bundschuh]

From: yangge <yangge1116@126.com>

[ Upstream commit 04f13d241b8b146b23038bffd907cb8278391d07 ]

My machine has 4 NUMA nodes, each equipped with 32GB of memory.  I have
configured each NUMA node with 16GB of CMA and 16GB of in-use hugetlb
pages.  The allocation of contiguous memory via cma_alloc() can fail
probabilistically.

When there are free hugetlb folios in the hugetlb pool, during the
migration of in-use hugetlb folios, new folios are allocated from the free
hugetlb pool.  After the migration is completed, the old folios are
released back to the free hugetlb pool instead of being returned to the
buddy system.  This can cause test_pages_isolated() check to fail,
ultimately leading to the failure of cma_alloc().

Call trace:

cma_alloc()
    __alloc_contig_migrate_range() // migrate in-use hugepage
    test_pages_isolated()
        __test_page_isolated_in_pageblock()
             PageBuddy(page) // check if the page is in buddy

To address this issue, we introduce a function named
replace_free_hugepage_folios().  This function will replace the hugepage
in the free hugepage pool with a new one and release the old one to the
buddy system.  After the migration of in-use hugetlb pages is completed,
we will invoke replace_free_hugepage_folios() to ensure that these
hugepages are properly released to the buddy system.  Following this step,
when test_pages_isolated() is executed for inspection, it will
successfully pass.

Additionally, when alloc_contig_range() is used to migrate multiple in-use
hugetlb pages, it can result in some in-use hugetlb pages being released
back to the free hugetlb pool and subsequently being reallocated and used
again.  For example:

[huge 0] [huge 1]

To migrate huge 0, we obtain huge x from the pool.  After the migration is
completed, we return the now-freed huge 0 back to the pool.  When it's
time to migrate huge 1, we can simply reuse the now-freed huge 0 from the
pool.  As a result, when replace_free_hugepage_folios() is executed, it
cannot release huge 0 back to the buddy system.  To address this issue, we
should prevent the reuse of isolated free hugepages during the migration
process.

Link: https://lkml.kernel.org/r/1734503588-16254-1-git-send-email-yangge1116@126.com
Link: https://lkml.kernel.org/r/1736582300-11364-1-git-send-email-yangge1116@126.com
Signed-off-by: yangge <yangge1116@126.com>
Cc: Baolin Wang <baolin.wang@linux.alibaba.com>
Cc: Barry Song <21cnbao@gmail.com>
Cc: David Hildenbrand <david@redhat.com>
Cc: SeongJae Park <sj@kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
[ 6.12: Take only the dequeue_hugetlb_folio_node_exact() hunk, so memory
  offline stops migrating huge pages back into the isolated range. ]
Signed-off-by: Mark Bundschuh <mkbund@amazon.com>
---
Without this, offlining a memory block holding mapped huge pages keeps
migrating them back into the block until the offline is interrupted.
Reproduced on arm64 on 5.10 and 6.12 (stock AL2023 6.12.113): the offline
was still migrating after 60 s. With this patch on 5.10 it completes in
0.15 s with the data intact. Also sent for 5.10, 5.15, 6.1, 6.6 and 6.12.

 mm/hugetlb.c | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/mm/hugetlb.c b/mm/hugetlb.c
index 68e0cc0748ca..b4c6c926a71e 100644
--- a/mm/hugetlb.c
+++ b/mm/hugetlb.c
@@ -48,6 +48,7 @@
 #include <linux/page_owner.h>
 #include "internal.h"
 #include "hugetlb_vmemmap.h"
+#include <linux/page-isolation.h>
 
 int hugetlb_max_hstate __read_mostly;
 unsigned int default_hstate_idx;
@@ -1350,6 +1351,9 @@ static struct folio *dequeue_hugetlb_folio_node_exact(struct hstate *h,
 		if (folio_test_hwpoison(folio))
 			continue;
 
+		if (is_migrate_isolate_page(&folio->page))
+			continue;
+
 		list_move(&folio->lru, &h->hugepage_activelist);
 		folio_ref_unfreeze(folio, 1);
 		folio_clear_hugetlb_freed(folio);
-- 
2.50.1

Re: [PATCH 6.12] mm: replace free hugepage folios after migration

[sashiko-bot]

Thank you for your contribution! Sashiko AI review found 2 potential
issues to consider:
- [High] mm: dequeue_hugetlb_folio_node_exact() skips isolated folios
  counted as free
- [Medium] mm: missing replace_free_hugepage_folios() breaks
  alloc_contig_range()
--

commit 90894e199378334e1089f01dd3d151f3b4957700
Author: yangge <yangge1116@126.com>

mm: replace free hugepage folios after migration

This backport takes only the dequeue_hugetlb_folio_node_exact() check from the
upstream commit to skip isolated free hugetlb folios when dequeuing from the
hugetlb freelist, while omitting replace_free_hugepage_folios() and its call
site in alloc_contig_range_noprof().

Link: https://lkml.kernel.org/r/1734503588-16254-1-git-send-email-yangge1116@126.com
Link: https://lkml.kernel.org/r/1736582300-11364-1-git-send-email-yangge1116@126.com

>     mm: replace free hugepage folios after migration
>     
>     [ Upstream commit 04f13d241b8b146b23038bffd907cb8278391d07 ]
>     
>     My machine has 4 NUMA nodes, each equipped with 32GB of memory.  I have
>     configured each NUMA node with 16GB of CMA and 16GB of in-use hugetlb
>     pages.  The allocation of contiguous memory via cma_alloc() can fail
>     probabilistically.
>     
>     When there are free hugetlb folios in the hugetlb pool, during the
>     migration of in-use hugetlb folios, new folios are allocated from the free
>     hugetlb pool.  After the migration is completed, the old folios are
>     released back to the free hugetlb pool instead of being returned to the
>     buddy system.  This can cause test_pages_isolated() check to fail,
>     ultimately leading to the failure of cma_alloc().
>     
>     Call trace:
>     
>     cma_alloc()
>         __alloc_contig_migrate_range() // migrate in-use hugepage
>         test_pages_isolated()
>             __test_page_isolated_in_pageblock()
>                  PageBuddy(page) // check if the page is in buddy
>     
>     To address this issue, we introduce a function named
>     replace_free_hugepage_folios().  This function will replace the hugepage
>     in the free hugepage pool with a new one and release the old one to the
>     buddy system.  After the migration of in-use hugetlb pages is completed,
>     we will invoke replace_free_hugepage_folios() to ensure that these
>     hugepages are properly released to the buddy system.  Following this step,
>     when test_pages_isolated() is executed for inspection, it will
>     successfully pass.
>     
>     Additionally, when alloc_contig_range() is used to migrate multiple in-use
>     hugetlb pages, it can result in some in-use hugetlb pages being released
>     back to the free hugetlb pool and subsequently being reallocated and used
>     again.  For example:
>     
>     [huge 0] [huge 1]
>     
>     To migrate huge 0, we obtain huge x from the pool.  After the migration is
>     completed, we return the now-freed huge 0 back to the pool.  When it's
>     time to migrate huge 1, we can simply reuse the now-freed huge 0 from the
>     pool.  As a result, when replace_free_hugepage_folios() is executed, it
>     cannot release huge 0 back to the buddy system.  To address this issue, we
>     should prevent the reuse of isolated free hugepages during the migration
>     process.
>     
>     Link: https://lkml.kernel.org/r/1734503588-16254-1-git-send-email-yangge1116@126.com
>     Link: https://lkml.kernel.org/r/1736582300-11364-1-git-send-email-yangge1116@126.com
>     Signed-off-by: yangge <yangge1116@126.com>
>     Cc: Baolin Wang <baolin.wang@linux.alibaba.com>
>     Cc: Barry Song <21cnbao@gmail.com>
>     Cc: David Hildenbrand <david@redhat.com>
>     Cc: SeongJae Park <sj@kernel.org>
>     Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
>     [ 6.12: Take only the dequeue_hugetlb_folio_node_exact() hunk, so memory
>       offline stops migrating huge pages back into the isolated range. ]
>     Signed-off-by: Mark Bundschuh <mkbund@amazon.com>

[Severity: Medium]
The commit subject and message state that this patch introduces
replace_free_hugepage_folios() and invokes it after migrating in-use hugetlb
pages so that cma_alloc() and test_pages_isolated() succeed, but this backport
drops replace_free_hugepage_folios() and its call in
alloc_contig_range_noprof() even though the helper
alloc_and_dissolve_hugetlb_folio() is already present in mm/hugetlb.c.

When alloc_contig_range_noprof() runs on a range containing multiple in-use
hugetlb folios while non-isolated free hugetlb folios exist in
h->hugepage_freelists, migrate_pages() -> alloc_migration_target() ->
alloc_hugetlb_folio_nodemask() dequeues a persistent free pool folio (which
does not have HPG_temporary set). After migration, free_huge_folio() returns
the old source folio to h->hugepage_freelists instead of the buddy allocator:

mm/hugetlb.c:free_huge_folio() {
    ...
	} else {
		arch_clear_hugetlb_flags(folio);
		enqueue_hugetlb_folio(h, folio);
		spin_unlock_irqrestore(&hugetlb_lock, flags);
	}
}

With the new check in dequeue_hugetlb_folio_node_exact() skipping that newly
freed isolated folio, each subsequent in-use hugetlb folio migrated out of the
range consumes another non-isolated free pool folio from outside the range and
leaves its old folio on h->hugepage_freelists inside the isolated range.

Because alloc_contig_range_noprof() never calls
replace_free_hugepage_folios(start, end) after __alloc_contig_migrate_range():

mm/page_alloc.c:alloc_contig_range_noprof() {
    ...
	ret = __alloc_contig_migrate_range(&cc, start, end, migratetype);
	if (ret && ret != -EBUSY)
		goto done;
	ret = 0;

	outer_start = find_large_buddy(start);

	/* Make sure the range is really isolated. */
	if (test_pages_isolated(outer_start, end, 0)) {
		ret = -EBUSY;
		goto done;
	}
}

doesn't test_pages_isolated() still fail with -EBUSY on those freed hugetlb
folios, while pulling additional free hugetlb pool folios into the isolated
CMA/contiguous range?

> diff --git a/mm/hugetlb.c b/mm/hugetlb.c
> index 68e0cc0748cac..b4c6c926a71e2 100644
> --- a/mm/hugetlb.c
> +++ b/mm/hugetlb.c

[ ... ]

> @@ -1350,6 +1351,9 @@ static struct folio *dequeue_hugetlb_folio_node_exact(struct hstate *h,
>  		if (folio_test_hwpoison(folio))
>  			continue;
>  
> +		if (is_migrate_isolate_page(&folio->page))
> +			continue;

[Severity: High]
Can unconditionally skipping isolated folios in
dequeue_hugetlb_folio_node_exact() while they remain counted in
h->free_huge_pages break hugetlb reservation accounting during memory
offlining and page faults?

Specifically, available_huge_pages(h) computes:

    h->free_huge_pages - h->resv_huge_pages

which still counts MIGRATE_ISOLATE folios on h->hugepage_freelists[nid] as
available.

During offline_pages(), start_isolate_page_range() marks the range
MIGRATE_ISOLATE and do_migrate_range() runs before
dissolve_free_hugetlb_folios(). If there are unreserved free hugetlb folios in
the isolated range (either initially free or placed on h->hugepage_freelists
when an earlier in-use hugetlb folio in the range was migrated into an
unreserved free pool folio) and non-isolated free folios outside the range
backing h->resv_huge_pages, available_huge_pages(h) is greater than 0. When
do_migrate_range() migrates another in-use hugetlb folio in the range:

mm/hugetlb.c:alloc_hugetlb_folio_nodemask() {
    ...
	spin_lock_irq(&hugetlb_lock);
	if (available_huge_pages(h)) {
		struct folio *folio;

		folio = dequeue_hugetlb_folio_nodemask(h, gfp_mask,
						preferred_nid, nmask);
    ...
}

dequeue_hugetlb_folio_node_exact() skips the isolated free folios and steals
one of the non-isolated free folios backing h->resv_huge_pages, while
free_huge_folio() places the old folio onto h->hugepage_freelists inside the
isolated range. When dissolve_free_hugetlb_folios() then runs on the isolated
range, after dissolving the unreserved count, the remaining isolated free
folio hits:

mm/hugetlb.c:dissolve_free_hugetlb_folio() {
    ...
		if (!available_huge_pages(h))
			goto out;
    ...
}

returning -EBUSY and failing offline_pages() after persistent hugepages have
already been dissolved.

Similarly, if the free folios backing h->resv_huge_pages happen to lie in a
MIGRATE_ISOLATE pageblock, alloc_hugetlb_folio() ->
dequeue_hugetlb_folio_vma() -> dequeue_hugetlb_folio_node_exact() and
alloc_hugetlb_folio_reserve() will skip them and return NULL:

mm/hugetlb.c:alloc_hugetlb_folio() {
    ...
	folio = dequeue_hugetlb_folio_vma(h, vma, addr, gbl_chg);
	if (!folio) {
		spin_unlock_irq(&hugetlb_lock);\
		folio = alloc_buddy_hugetlb_folio_with_mpol(h, vma, addr);
		if (!folio)
			goto out_uncharge_cgroup;
    ...
}

Won't that cause reserved page faults to fail with -ENOSPC (SIGBUS) whenever
surplus overcommit is disabled (nr_overcommit_huge_pages == 0) or the hstate
is gigantic?

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20261105070442.3101877-1-mkbund@amazon.com?part=1

[PATCH v2 6.12 0/2] mm: replace free hugepage folios after migration

[Mark Bundschuh]

Full backport of 04f13d241b8b ("mm: replace free hugepage folios after
migration") with its fix 113ed54ad276.

Tested in a VM on 6.12.115: memory offline of a block with mapped huge
pages no longer livelocks, and cma_alloc() over mapped huge pages now
succeeds (it failed on 6.12.115 and with v1).

v1: https://lore.kernel.org/all/20261105070442.3101877-1-mkbund@amazon.com/

Ge Yang (1):
  mm/hugetlb: fix kernel NULL pointer dereference when replacing free
    hugetlb folios

yangge (1):
  mm: replace free hugepage folios after migration

 include/linux/hugetlb.h |  7 ++++++
 mm/hugetlb.c            | 50 +++++++++++++++++++++++++++++++++++++++++
 mm/page_alloc.c         | 12 +++++++++-
 3 files changed, 68 insertions(+), 1 deletion(-)

-- 
2.50.1

[PATCH v2 6.12 1/2] mm: replace free hugepage folios after migration

[Mark Bundschuh]

From: yangge <yangge1116@126.com>

[ Upstream commit 04f13d241b8b146b23038bffd907cb8278391d07 ]

My machine has 4 NUMA nodes, each equipped with 32GB of memory.  I have
configured each NUMA node with 16GB of CMA and 16GB of in-use hugetlb
pages.  The allocation of contiguous memory via cma_alloc() can fail
probabilistically.

When there are free hugetlb folios in the hugetlb pool, during the
migration of in-use hugetlb folios, new folios are allocated from the free
hugetlb pool.  After the migration is completed, the old folios are
released back to the free hugetlb pool instead of being returned to the
buddy system.  This can cause test_pages_isolated() check to fail,
ultimately leading to the failure of cma_alloc().

Call trace:

cma_alloc()
    __alloc_contig_migrate_range() // migrate in-use hugepage
    test_pages_isolated()
        __test_page_isolated_in_pageblock()
             PageBuddy(page) // check if the page is in buddy

To address this issue, we introduce a function named
replace_free_hugepage_folios().  This function will replace the hugepage
in the free hugepage pool with a new one and release the old one to the
buddy system.  After the migration of in-use hugetlb pages is completed,
we will invoke replace_free_hugepage_folios() to ensure that these
hugepages are properly released to the buddy system.  Following this step,
when test_pages_isolated() is executed for inspection, it will
successfully pass.

Additionally, when alloc_contig_range() is used to migrate multiple in-use
hugetlb pages, it can result in some in-use hugetlb pages being released
back to the free hugetlb pool and subsequently being reallocated and used
again.  For example:

[huge 0] [huge 1]

To migrate huge 0, we obtain huge x from the pool.  After the migration is
completed, we return the now-freed huge 0 back to the pool.  When it's
time to migrate huge 1, we can simply reuse the now-freed huge 0 from the
pool.  As a result, when replace_free_hugepage_folios() is executed, it
cannot release huge 0 back to the buddy system.  To address this issue, we
should prevent the reuse of isolated free hugepages during the migration
process.

Link: https://lkml.kernel.org/r/1734503588-16254-1-git-send-email-yangge1116@126.com
Link: https://lkml.kernel.org/r/1736582300-11364-1-git-send-email-yangge1116@126.com
Signed-off-by: yangge <yangge1116@126.com>
Cc: Baolin Wang <baolin.wang@linux.alibaba.com>
Cc: Barry Song <21cnbao@gmail.com>
Cc: David Hildenbrand <david@redhat.com>
Cc: SeongJae Park <sj@kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Mark Bundschuh <mkbund@amazon.com>
---
 include/linux/hugetlb.h |  7 +++++++
 mm/hugetlb.c            | 42 +++++++++++++++++++++++++++++++++++++++++
 mm/page_alloc.c         | 12 +++++++++++-
 3 files changed, 60 insertions(+), 1 deletion(-)

diff --git a/include/linux/hugetlb.h b/include/linux/hugetlb.h
index 7eaabacdf110..7b5395e361d5 100644
--- a/include/linux/hugetlb.h
+++ b/include/linux/hugetlb.h
@@ -690,6 +690,7 @@ struct huge_bootmem_page {
 
 int isolate_or_dissolve_huge_page(struct page *page, struct list_head *list);
 void wait_for_freed_hugetlb_folios(void);
+int replace_free_hugepage_folios(unsigned long start_pfn, unsigned long end_pfn);
 struct folio *alloc_hugetlb_folio(struct vm_area_struct *vma,
 				unsigned long addr, int avoid_reserve);
 struct folio *alloc_hugetlb_folio_nodemask(struct hstate *h, int preferred_nid,
@@ -1081,6 +1082,12 @@ static inline void wait_for_freed_hugetlb_folios(void)
 {
 }
 
+static inline int replace_free_hugepage_folios(unsigned long start_pfn,
+		unsigned long end_pfn)
+{
+	return 0;
+}
+
 static inline struct folio *alloc_hugetlb_folio(struct vm_area_struct *vma,
 					   unsigned long addr,
 					   int avoid_reserve)
diff --git a/mm/hugetlb.c b/mm/hugetlb.c
index 68e0cc0748ca..a3d3fb1de3ec 100644
--- a/mm/hugetlb.c
+++ b/mm/hugetlb.c
@@ -48,6 +48,7 @@
 #include <linux/page_owner.h>
 #include "internal.h"
 #include "hugetlb_vmemmap.h"
+#include <linux/page-isolation.h>
 
 int hugetlb_max_hstate __read_mostly;
 unsigned int default_hstate_idx;
@@ -1350,6 +1351,9 @@ static struct folio *dequeue_hugetlb_folio_node_exact(struct hstate *h,
 		if (folio_test_hwpoison(folio))
 			continue;
 
+		if (is_migrate_isolate_page(&folio->page))
+			continue;
+
 		list_move(&folio->lru, &h->hugepage_activelist);
 		folio_ref_unfreeze(folio, 1);
 		folio_clear_hugetlb_freed(folio);
@@ -2991,6 +2995,44 @@ void wait_for_freed_hugetlb_folios(void)
 	flush_work(&free_hpage_work);
 }
 
+/*
+ *  replace_free_hugepage_folios - Replace free hugepage folios in a given pfn
+ *  range with new folios.
+ *  @start_pfn: start pfn of the given pfn range
+ *  @end_pfn: end pfn of the given pfn range
+ *  Returns 0 on success, otherwise negated error.
+ */
+int replace_free_hugepage_folios(unsigned long start_pfn, unsigned long end_pfn)
+{
+	struct hstate *h;
+	struct folio *folio;
+	int ret = 0;
+
+	LIST_HEAD(isolate_list);
+
+	while (start_pfn < end_pfn) {
+		folio = pfn_folio(start_pfn);
+		if (folio_test_hugetlb(folio)) {
+			h = folio_hstate(folio);
+		} else {
+			start_pfn++;
+			continue;
+		}
+
+		if (!folio_ref_count(folio)) {
+			ret = alloc_and_dissolve_hugetlb_folio(h, folio,
+							       &isolate_list);
+			if (ret)
+				break;
+
+			putback_movable_pages(&isolate_list);
+		}
+		start_pfn++;
+	}
+
+	return ret;
+}
+
 struct folio *alloc_hugetlb_folio(struct vm_area_struct *vma,
 				    unsigned long addr, int avoid_reserve)
 {
diff --git a/mm/page_alloc.c b/mm/page_alloc.c
index d8209a512019..3686b787c14e 100644
--- a/mm/page_alloc.c
+++ b/mm/page_alloc.c
@@ -6730,7 +6730,17 @@ int alloc_contig_range_noprof(unsigned long start, unsigned long end,
 	ret = __alloc_contig_migrate_range(&cc, start, end, migratetype);
 	if (ret && ret != -EBUSY)
 		goto done;
-	ret = 0;
+
+	/*
+	 * When in-use hugetlb pages are migrated, they may simply be released
+	 * back into the free hugepage pool instead of being returned to the
+	 * buddy system.  After the migration of in-use huge pages is completed,
+	 * we will invoke replace_free_hugepage_folios() to ensure that these
+	 * hugepages are properly released to the buddy system.
+	 */
+	ret = replace_free_hugepage_folios(start, end);
+	if (ret)
+		goto done;
 
 	/*
 	 * Pages from [start, end) are within a pageblock_nr_pages
-- 
2.50.1

Re: [PATCH v2 6.12 1/2] mm: replace free hugepage folios after migration

[sashiko-bot]

Thank you for your contribution! Sashiko AI review found 3 potential
issues to consider:
- [High] mm/hugetlb: dequeue_hugetlb_folio_node_exact() skips isolated
  folios
- [Medium] mm: soft lockup in replace_free_hugepage_folios() on deferred
  HVO folios
- [Medium] mm/hugetlb: repeated dissolution of undissolved folio in
  replace_free_hugepage_folios()
--

commit 10e41795aa4490543b53bcfb3054aebb335f7640
Author: yangge <yangge1116@126.com>

mm: replace free hugepage folios after migration

This commit skips isolated free hugetlb folios during hugetlb pool
allocation and introduces replace_free_hugepage_folios() so free hugetlb
folios left behind after migration in alloc_contig_range_noprof() are
replaced and released to the buddy allocator.

Link: https://lkml.kernel.org/r/1734503588-16254-1-git-send-email-yangge1116@126.com
Link: https://lkml.kernel.org/r/1736582300-11364-1-git-send-email-yangge1116@126.com

[Severity: High]
In mm/hugetlb.c at line 1354, does skipping MIGRATE_ISOLATE folios in
dequeue_hugetlb_folio_node_exact() break hugetlb reservation accounting
because those folios remain counted in h->free_huge_pages and
available_huge_pages(h)?

When free hugetlb folios sit in an isolated range, available_huge_pages(h)
still counts them as unreserved excess.  During migration in
alloc_hugetlb_folio_nodemask() (line 2441), dequeue_hugetlb_folio_node_exact()
skips the isolated free folios and instead dequeues non-isolated free
folios that were backing h->resv_huge_pages, leaving the freed old folio
queued in the isolated range.  In offline_pages(), a subsequent call to
dissolve_free_hugetlb_folios() dissolves the initial unreserved folios
until available_huge_pages(h) reaches 0 and then fails with -EBUSY in
dissolve_free_hugetlb_folio() (line 2230) after persistent hugepages have
already been dissolved.

In addition, when a task faults on a mapping with a valid reservation in
alloc_hugetlb_folio() (line 3105) while the free folios backing
h->resv_huge_pages are in a MIGRATE_ISOLATE pageblock,
dequeue_hugetlb_folio_vma() returns NULL and falls back to surplus
allocation, which fails with -ENOSPC (delivering SIGBUS) when overcommit
is disabled or the hstate is gigantic.

[Severity: Medium]
In mm/hugetlb.c at line 3022, can replace_free_hugepage_folios() loop on
 every tail PFN of the same hugetlb folio if
__update_and_free_hugetlb_folio() does not dissolve it into buddy pages?

After alloc_and_dissolve_hugetlb_folio() returns 0,
replace_free_hugepage_folios() advances start_pfn by 1 rather than
skipping the rest of the hugetlb folio.  When old_folio is
vmemmap-optimized and hugetlb_vmemmap_restore_folio() fails in
__update_and_free_hugetlb_folio() (line 1689), old_folio is put back on
the hugetlb freelist as a surplus compound folio via
add_hugetlb_folio(h, folio, true).  The subsequent tail PFNs resolve back
to the same old_folio via pfn_folio(start_pfn) and call
alloc_and_dissolve_hugetlb_folio() again, allocating extra replacement
hugepages and permanently inflating h->nr_huge_pages and
h->surplus_huge_pages.  Similarly, if an HVO folio has
HPG_raw_hwp_unreliable set (line 1681), __update_and_free_hugetlb_folio()
returns early with PG_hugetlb still set and HPG_freed clear, causing the
next PFN iteration to spin indefinitely in the
!folio_test_hugetlb_freed(old_folio) retry loop in
alloc_and_dissolve_hugetlb_folio() (line 2901).

[Severity: Medium]
In mm/page_alloc.c at line 6741, should alloc_contig_range_noprof() call
wait_for_freed_hugetlb_folios() before replace_free_hugepage_folios(start,
end)?

When HugeTLB vmemmap optimization (HVO) is enabled and
__alloc_contig_migrate_range() migrates an in-use hugetlb folio that is
freed to the buddy allocator (for example, when migrated to a temporary
folio or when surplus pages exist on the node),
update_and_free_hugetlb_folio() queues the old folio onto hpage_freelist
for free_hpage_work with PG_hugetlb still set, refcount 0, and HPG_freed
clear.  Because wait_for_freed_hugetlb_folios() is only called later in
test_pages_isolated() (mm/page_isolation.c, line 622),
replace_free_hugepage_folios() (mm/hugetlb.c, line 3022) passes that
deferred-free folio to alloc_and_dissolve_hugetlb_folio() (line 2901),
which busy-spins on !folio_test_hugetlb_freed(old_folio) taking and
dropping hugetlb_lock while free_hpage_workfn() sleeps in
synchronize_rcu(), and triggers a soft lockup if the caller is a
SCHED_FIFO or SCHED_RR task on the CPU where free_hpage_work was queued.

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20261106220835.2878368-1-mkbund@amazon.com?part=1

[PATCH v2 6.12 2/2] mm/hugetlb: fix kernel NULL pointer dereference when replacing free hugetlb folios

[Mark Bundschuh]

From: Ge Yang <yangge1116@126.com>

[ Upstream commit 113ed54ad276c352ee5ce109bdcf0df118a43bda ]

A kernel crash was observed when replacing free hugetlb folios:

BUG: kernel NULL pointer dereference, address: 0000000000000028
PGD 0 P4D 0
Oops: Oops: 0000 [#1] SMP NOPTI
CPU: 28 UID: 0 PID: 29639 Comm: test_cma.sh Tainted 6.15.0-rc6-zp #41 PREEMPT(voluntary)
RIP: 0010:alloc_and_dissolve_hugetlb_folio+0x1d/0x1f0
RSP: 0018:ffffc9000b30fa90 EFLAGS: 00010286
RAX: 0000000000000000 RBX: 0000000000342cca RCX: ffffea0043000000
RDX: ffffc9000b30fb08 RSI: ffffea0043000000 RDI: 0000000000000000
RBP: ffffc9000b30fb20 R08: 0000000000001000 R09: 0000000000000000
R10: ffff88886f92eb00 R11: 0000000000000000 R12: ffffea0043000000
R13: 0000000000000000 R14: 00000000010c0200 R15: 0000000000000004
FS:  00007fcda5f14740(0000) GS:ffff8888ec1d8000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000000000000028 CR3: 0000000391402000 CR4: 0000000000350ef0
Call Trace:
<TASK>
 replace_free_hugepage_folios+0xb6/0x100
 alloc_contig_range_noprof+0x18a/0x590
 ? srso_return_thunk+0x5/0x5f
 ? down_read+0x12/0xa0
 ? srso_return_thunk+0x5/0x5f
 cma_range_alloc.constprop.0+0x131/0x290
 __cma_alloc+0xcf/0x2c0
 cma_alloc_write+0x43/0xb0
 simple_attr_write_xsigned.constprop.0.isra.0+0xb2/0x110
 debugfs_attr_write+0x46/0x70
 full_proxy_write+0x62/0xa0
 vfs_write+0xf8/0x420
 ? srso_return_thunk+0x5/0x5f
 ? filp_flush+0x86/0xa0
 ? srso_return_thunk+0x5/0x5f
 ? filp_close+0x1f/0x30
 ? srso_return_thunk+0x5/0x5f
 ? do_dup2+0xaf/0x160
 ? srso_return_thunk+0x5/0x5f
 ksys_write+0x65/0xe0
 do_syscall_64+0x64/0x170
 entry_SYSCALL_64_after_hwframe+0x76/0x7e

There is a potential race between __update_and_free_hugetlb_folio() and
replace_free_hugepage_folios():

CPU1                              CPU2
__update_and_free_hugetlb_folio   replace_free_hugepage_folios
                                    folio_test_hugetlb(folio)
                                    -- It's still hugetlb folio.

  __folio_clear_hugetlb(folio)
  hugetlb_free_folio(folio)
                                    h = folio_hstate(folio)
                                    -- Here, h is NULL pointer

When the above race condition occurs, folio_hstate(folio) returns NULL,
and subsequent access to this NULL pointer will cause the system to crash.
To resolve this issue, execute folio_hstate(folio) under the protection
of the hugetlb_lock lock, ensuring that folio_hstate(folio) does not
return NULL.

Link: https://lkml.kernel.org/r/1747884137-26685-1-git-send-email-yangge1116@126.com
Fixes: 04f13d241b8b ("mm: replace free hugepage folios after migration")
Signed-off-by: Ge Yang <yangge1116@126.com>
Reviewed-by: Muchun Song <muchun.song@linux.dev>
Reviewed-by: Oscar Salvador <osalvador@suse.de>
Cc: Baolin Wang <baolin.wang@linux.alibaba.com>
Cc: Barry Song <21cnbao@gmail.com>
Cc: David Hildenbrand <david@redhat.com>
Cc: <stable@vger.kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Mark Bundschuh <mkbund@amazon.com>
---
 mm/hugetlb.c | 8 ++++++++
 1 file changed, 8 insertions(+)

diff --git a/mm/hugetlb.c b/mm/hugetlb.c
index a3d3fb1de3ec..43a96477904a 100644
--- a/mm/hugetlb.c
+++ b/mm/hugetlb.c
@@ -3012,12 +3012,20 @@ int replace_free_hugepage_folios(unsigned long start_pfn, unsigned long end_pfn)
 
 	while (start_pfn < end_pfn) {
 		folio = pfn_folio(start_pfn);
+
+		/*
+		 * The folio might have been dissolved from under our feet, so make sure
+		 * to carefully check the state under the lock.
+		 */
+		spin_lock_irq(&hugetlb_lock);
 		if (folio_test_hugetlb(folio)) {
 			h = folio_hstate(folio);
 		} else {
+			spin_unlock_irq(&hugetlb_lock);
 			start_pfn++;
 			continue;
 		}
+		spin_unlock_irq(&hugetlb_lock);
 
 		if (!folio_ref_count(folio)) {
 			ret = alloc_and_dissolve_hugetlb_folio(h, folio,
-- 
2.50.1