[PATCH v2 0/3] mm/hugetlb_vmemmap: Fix HVO races and deadlock

[PATCH v2 0/3] mm/hugetlb_vmemmap: Fix HVO races and deadlock

[Muchun Song]

This series fixes three HVO issues reported by Sashiko.

The first patch serializes HWPoison updates against HVO remapping so
raw_hwp_page state is not lost while a replacement vmemmap head page is
being installed.  The second patch uses an HVO-private rwsem to prevent
deferred PMD splits from racing with PTE remapping before the TLB flush.
The last patch switches the vmemmap walk to the lockless walker so a PMD
split allocation cannot recurse into init_mm.mmap_lock through reclaim.

v2:
- Add the HWPoison serialization fix reported by Sashiko.
- Use an HVO-private rwsem for deferred PMD split serialization.
- Add the lockless page-table walk fix for the PMD split deadlock.

v1: https://lore.kernel.org/20261110112456.62827-1-songmuchun@bytedance.com/

Muchun Song (3):
  mm/hugetlb_vmemmap: fix lost HWPoison state during optimization
  mm/hugetlb_vmemmap: fix PMD split TLB flush race
  mm/hugetlb_vmemmap: fix deadlock during PMD split

 mm/hugetlb_vmemmap.c | 28 +++++++++++++++++++++++-----
 mm/hugetlb_vmemmap.h | 24 ++++++++++++++++++++++++
 mm/memory-failure.c  |  2 ++
 3 files changed, 49 insertions(+), 5 deletions(-)


base-commit: 2dc0136b1fdac153434ab02778fcdb32db5d0dae
-- 
2.54.0

[PATCH v2 1/3] mm/hugetlb_vmemmap: fix lost HWPoison state during optimization

[Muchun Song]

HVO copies the vmemmap head page before remapping it to a newly allocated
backing page. Memory failure can update the old head page after the copy
but before the remap's TLB invalidation completes:

CPU 0                            CPU 1
init_new_hugetlb_folio()
__hugetlb_vmemmap_optimize_folio()
  vmemmap_optimize_pte()
    copy_page(old_head, new_head)
                                 try_memory_failure_hugetlb()
                                   get_huge_page_for_hwpoison()
                                     hugetlb_update_hwpoison(old_head)
  set_pte_at(..., new_head)
  flush_tlb_all()
free_vmemmap_page_list(old_head)

The raw_hwp_page linked to the old head is then leaked and the folio's
HWPoison state is lost when the obsolete backing page is freed, allowing
the poisoned huge page to be allocated later.

Serialize HVO remapping against HWPoison updates. This ensures that an
update is either copied to the new vmemmap head page or applied after HVO
installs the new mapping and completes the TLB flush.

Fixes: 11aad2631bf7 ("mm/hugetlb_vmemmap: remap head page to newly allocated page")
Cc: stable@vger.kernel.org
Signed-off-by: Muchun Song <songmuchun@bytedance.com>
Reported-by: Sashiko <sashiko-bot@kernel.org>
Closes: https://sashiko.dev/#/bug/linux-18e93f17-529b-4d84-ba44-a41829f4be64
---
 mm/hugetlb_vmemmap.c | 10 ++++++++++
 mm/hugetlb_vmemmap.h | 24 ++++++++++++++++++++++++
 mm/memory-failure.c  |  2 ++
 3 files changed, 36 insertions(+)

diff --git a/mm/hugetlb_vmemmap.c b/mm/hugetlb_vmemmap.c
index ba8e11e399b2..79ca68dff635 100644
--- a/mm/hugetlb_vmemmap.c
+++ b/mm/hugetlb_vmemmap.c
@@ -15,6 +15,7 @@
 #include <linux/mmdebug.h>
 #include <linux/pagewalk.h>
 #include <linux/pgalloc.h>
+#include <linux/rwsem.h>
 #include <linux/vmemmap-optimization.h>
 
 #include <asm/tlbflush.h>
@@ -41,6 +42,8 @@ struct vmemmap_remap_walk {
 	unsigned long		flags;
 };
 
+DECLARE_RWSEM(vmemmap_remap_rwsem);
+
 static int vmemmap_split_pmd(pmd_t *pmd, struct page *head, unsigned long start,
 			     struct vmemmap_remap_walk *walk)
 {
@@ -333,6 +336,9 @@ static int vmemmap_pteify_folio(const struct folio *folio, unsigned long flags)
  * responsible for freeing the pages left on the @vmemmap_pages list, even on
  * failure.
  *
+ * Note: The caller must hold vmemmap_remap_rwsem. If TLB flushing is deferred,
+ * the lock must remain held until the final TLB flush.
+ *
  * Return: %0 on success, negative error code otherwise.
  */
 static int vmemmap_optimize_folio(const struct folio *folio,
@@ -348,6 +354,8 @@ static int vmemmap_optimize_folio(const struct folio *folio,
 		.flags		= flags,
 	};
 
+	rwsem_assert_held(&vmemmap_remap_rwsem);
+
 	VM_WARN_ON_ONCE_FOLIO(folio_ref_count(folio), folio);
 
 	if (!vmemmap_optimizable_order(folio_order(folio)))
@@ -536,6 +544,7 @@ void hugetlb_vmemmap_optimize_folio(const struct hstate *h, struct folio *folio)
 {
 	LIST_HEAD(vmemmap_pages);
 
+	guard(rwsem_read)(&vmemmap_remap_rwsem);
 	__hugetlb_vmemmap_optimize_folio(h, folio, &vmemmap_pages, 0);
 	free_vmemmap_page_list(&vmemmap_pages);
 }
@@ -588,6 +597,7 @@ void hugetlb_vmemmap_optimize_folios(struct hstate *h, struct list_head *folio_l
 
 	flush_tlb_all();
 
+	guard(rwsem_read)(&vmemmap_remap_rwsem);
 	list_for_each_entry(folio, folio_list, lru) {
 		int ret;
 
diff --git a/mm/hugetlb_vmemmap.h b/mm/hugetlb_vmemmap.h
index d057e8b12bf1..8b28063d78cc 100644
--- a/mm/hugetlb_vmemmap.h
+++ b/mm/hugetlb_vmemmap.h
@@ -9,9 +9,12 @@
 #ifndef _LINUX_HUGETLB_VMEMMAP_H
 #define _LINUX_HUGETLB_VMEMMAP_H
 #include <linux/hugetlb.h>
+#include <linux/rwsem.h>
 #include <linux/vmemmap-optimization.h>
 
 #ifdef CONFIG_HUGETLB_PAGE_OPTIMIZE_VMEMMAP
+extern struct rw_semaphore vmemmap_remap_rwsem;
+
 int hugetlb_vmemmap_restore_folio(const struct hstate *h, struct folio *folio);
 long hugetlb_vmemmap_restore_folios(const struct hstate *h,
 					struct list_head *folio_list,
@@ -20,6 +23,16 @@ void hugetlb_vmemmap_optimize_folio(const struct hstate *h, struct folio *folio)
 void hugetlb_vmemmap_optimize_folios(struct hstate *h, struct list_head *folio_list);
 void hugetlb_vmemmap_optimize_bootmem_page(unsigned long pfn, unsigned int order);
 
+static inline void hugetlb_vmemmap_write_lock(void)
+{
+	down_write(&vmemmap_remap_rwsem);
+}
+
+static inline void hugetlb_vmemmap_write_unlock(void)
+{
+	up_write(&vmemmap_remap_rwsem);
+}
+
 static inline unsigned int hugetlb_vmemmap_size(const struct hstate *h)
 {
 	return pages_per_huge_page(h) * sizeof(struct page);
@@ -59,6 +72,14 @@ static inline void hugetlb_vmemmap_optimize_folios(struct hstate *h, struct list
 {
 }
 
+static inline void hugetlb_vmemmap_write_lock(void)
+{
+}
+
+static inline void hugetlb_vmemmap_write_unlock(void)
+{
+}
+
 static inline unsigned int hugetlb_vmemmap_optimizable_size(const struct hstate *h)
 {
 	return 0;
@@ -69,6 +90,9 @@ static inline void hugetlb_vmemmap_optimize_bootmem_page(unsigned long pfn, unsi
 }
 #endif /* CONFIG_HUGETLB_PAGE_OPTIMIZE_VMEMMAP */
 
+DEFINE_LOCK_GUARD_0(vmemmap_remap,
+		    hugetlb_vmemmap_write_lock(), hugetlb_vmemmap_write_unlock())
+
 static inline bool hugetlb_vmemmap_optimizable(const struct hstate *h)
 {
 	return hugetlb_vmemmap_optimizable_size(h) != 0;
diff --git a/mm/memory-failure.c b/mm/memory-failure.c
index 78f0a13349cb..164a02866fd5 100644
--- a/mm/memory-failure.c
+++ b/mm/memory-failure.c
@@ -69,6 +69,7 @@
 #include "page_alloc.h"
 #include "internal.h"
 #include "memory-failure.h"
+#include "hugetlb_vmemmap.h"
 
 static int sysctl_memory_failure_early_kill __read_mostly;
 
@@ -2038,6 +2039,7 @@ static int get_huge_page_for_hwpoison(unsigned long pfn, int flags,
 	bool count_increased = false;
 	int ret, rc;
 
+	guard(vmemmap_remap)();
 	spin_lock_irq(&hugetlb_lock);
 	folio = page_folio(page);
 	if (!folio_test_hugetlb(folio)) {
-- 
2.54.0

Re: [PATCH v2 1/3] mm/hugetlb_vmemmap: fix lost HWPoison state during optimization

[Jose A. Perez de Azpillaga]

On 11/11/26 10:37 AM, Muchun Song wrote:
> HVO copies the vmemmap head page before remapping it to a newly allocated
> backing page. Memory failure can update the old head page after the copy
> but before the remap's TLB invalidation completes:
> 
> CPU 0                            CPU 1
> init_new_hugetlb_folio()
> __hugetlb_vmemmap_optimize_folio()
>    vmemmap_optimize_pte()
>      copy_page(old_head, new_head)
>                                   try_memory_failure_hugetlb()
>                                     get_huge_page_for_hwpoison()
>                                       hugetlb_update_hwpoison(old_head)
>    set_pte_at(..., new_head)
>    flush_tlb_all()
> free_vmemmap_page_list(old_head)
> 
> The raw_hwp_page linked to the old head is then leaked and the folio's
> HWPoison state is lost when the obsolete backing page is freed, allowing
> the poisoned huge page to be allocated later.
> 
> Serialize HVO remapping against HWPoison updates. This ensures that an
> update is either copied to the new vmemmap head page or applied after HVO
> installs the new mapping and completes the TLB flush.
> 
> Fixes: 11aad2631bf7 ("mm/hugetlb_vmemmap: remap head page to newly allocated page")
> Cc: stable@vger.kernel.org
> Signed-off-by: Muchun Song <songmuchun@bytedance.com>
> Reported-by: Sashiko <sashiko-bot@kernel.org>
> Closes: https://sashiko.dev/#/bug/linux-18e93f17-529b-4d84-ba44-a41829f4be64

LGTM:

Reviewed-by: Jose A. Perez de Azpillaga <azpijr@gmail.com>

-- 
cheers,
jose a. p-a

Re: [PATCH v2 1/3] mm/hugetlb_vmemmap: fix lost HWPoison state during optimization

[Jiaqi Yan]

On Wed, Nov 11, 2026 at 1:38 AM Muchun Song <songmuchun@bytedance.com> wrote:
>
> HVO copies the vmemmap head page before remapping it to a newly allocated
> backing page. Memory failure can update the old head page after the copy
> but before the remap's TLB invalidation completes:
>
> CPU 0                            CPU 1
> init_new_hugetlb_folio()
> __hugetlb_vmemmap_optimize_folio()
>   vmemmap_optimize_pte()
>     copy_page(old_head, new_head)
>                                  try_memory_failure_hugetlb()
>                                    get_huge_page_for_hwpoison()
>                                      hugetlb_update_hwpoison(old_head)
>   set_pte_at(..., new_head)
>   flush_tlb_all()
> free_vmemmap_page_list(old_head)
>
> The raw_hwp_page linked to the old head is then leaked and the folio's
> HWPoison state is lost when the obsolete backing page is freed, allowing
> the poisoned huge page to be allocated later.
>
> Serialize HVO remapping against HWPoison updates. This ensures that an
> update is either copied to the new vmemmap head page or applied after HVO
> installs the new mapping and completes the TLB flush.
>
> Fixes: 11aad2631bf7 ("mm/hugetlb_vmemmap: remap head page to newly allocated page")
> Cc: stable@vger.kernel.org
> Signed-off-by: Muchun Song <songmuchun@bytedance.com>
> Reported-by: Sashiko <sashiko-bot@kernel.org>
> Closes: https://sashiko.dev/#/bug/linux-18e93f17-529b-4d84-ba44-a41829f4be64
> ---
>  mm/hugetlb_vmemmap.c | 10 ++++++++++
>  mm/hugetlb_vmemmap.h | 24 ++++++++++++++++++++++++
>  mm/memory-failure.c  |  2 ++
>  3 files changed, 36 insertions(+)
>
> diff --git a/mm/hugetlb_vmemmap.c b/mm/hugetlb_vmemmap.c
> index ba8e11e399b2..79ca68dff635 100644
> --- a/mm/hugetlb_vmemmap.c
> +++ b/mm/hugetlb_vmemmap.c
> @@ -15,6 +15,7 @@
>  #include <linux/mmdebug.h>
>  #include <linux/pagewalk.h>
>  #include <linux/pgalloc.h>
> +#include <linux/rwsem.h>
>  #include <linux/vmemmap-optimization.h>
>
>  #include <asm/tlbflush.h>
> @@ -41,6 +42,8 @@ struct vmemmap_remap_walk {
>         unsigned long           flags;
>  };
>
> +DECLARE_RWSEM(vmemmap_remap_rwsem);
> +
>  static int vmemmap_split_pmd(pmd_t *pmd, struct page *head, unsigned long start,
>                              struct vmemmap_remap_walk *walk)
>  {
> @@ -333,6 +336,9 @@ static int vmemmap_pteify_folio(const struct folio *folio, unsigned long flags)
>   * responsible for freeing the pages left on the @vmemmap_pages list, even on
>   * failure.
>   *
> + * Note: The caller must hold vmemmap_remap_rwsem. If TLB flushing is deferred,
> + * the lock must remain held until the final TLB flush.
> + *
>   * Return: %0 on success, negative error code otherwise.
>   */
>  static int vmemmap_optimize_folio(const struct folio *folio,
> @@ -348,6 +354,8 @@ static int vmemmap_optimize_folio(const struct folio *folio,
>                 .flags          = flags,
>         };
>
> +       rwsem_assert_held(&vmemmap_remap_rwsem);
> +
>         VM_WARN_ON_ONCE_FOLIO(folio_ref_count(folio), folio);
>
>         if (!vmemmap_optimizable_order(folio_order(folio)))
> @@ -536,6 +544,7 @@ void hugetlb_vmemmap_optimize_folio(const struct hstate *h, struct folio *folio)
>  {
>         LIST_HEAD(vmemmap_pages);
>
> +       guard(rwsem_read)(&vmemmap_remap_rwsem);
>         __hugetlb_vmemmap_optimize_folio(h, folio, &vmemmap_pages, 0);
>         free_vmemmap_page_list(&vmemmap_pages);
>  }
> @@ -588,6 +597,7 @@ void hugetlb_vmemmap_optimize_folios(struct hstate *h, struct list_head *folio_l
>
>         flush_tlb_all();
>
> +       guard(rwsem_read)(&vmemmap_remap_rwsem);
>         list_for_each_entry(folio, folio_list, lru) {
>                 int ret;
>
> diff --git a/mm/hugetlb_vmemmap.h b/mm/hugetlb_vmemmap.h
> index d057e8b12bf1..8b28063d78cc 100644
> --- a/mm/hugetlb_vmemmap.h
> +++ b/mm/hugetlb_vmemmap.h
> @@ -9,9 +9,12 @@
>  #ifndef _LINUX_HUGETLB_VMEMMAP_H
>  #define _LINUX_HUGETLB_VMEMMAP_H
>  #include <linux/hugetlb.h>
> +#include <linux/rwsem.h>
>  #include <linux/vmemmap-optimization.h>
>
>  #ifdef CONFIG_HUGETLB_PAGE_OPTIMIZE_VMEMMAP
> +extern struct rw_semaphore vmemmap_remap_rwsem;
> +
>  int hugetlb_vmemmap_restore_folio(const struct hstate *h, struct folio *folio);
>  long hugetlb_vmemmap_restore_folios(const struct hstate *h,
>                                         struct list_head *folio_list,
> @@ -20,6 +23,16 @@ void hugetlb_vmemmap_optimize_folio(const struct hstate *h, struct folio *folio)
>  void hugetlb_vmemmap_optimize_folios(struct hstate *h, struct list_head *folio_list);
>  void hugetlb_vmemmap_optimize_bootmem_page(unsigned long pfn, unsigned int order);
>
> +static inline void hugetlb_vmemmap_write_lock(void)
> +{
> +       down_write(&vmemmap_remap_rwsem);
> +}
> +
> +static inline void hugetlb_vmemmap_write_unlock(void)
> +{
> +       up_write(&vmemmap_remap_rwsem);
> +}
> +
>  static inline unsigned int hugetlb_vmemmap_size(const struct hstate *h)
>  {
>         return pages_per_huge_page(h) * sizeof(struct page);
> @@ -59,6 +72,14 @@ static inline void hugetlb_vmemmap_optimize_folios(struct hstate *h, struct list
>  {
>  }
>
> +static inline void hugetlb_vmemmap_write_lock(void)
> +{
> +}
> +
> +static inline void hugetlb_vmemmap_write_unlock(void)
> +{
> +}
> +
>  static inline unsigned int hugetlb_vmemmap_optimizable_size(const struct hstate *h)
>  {
>         return 0;
> @@ -69,6 +90,9 @@ static inline void hugetlb_vmemmap_optimize_bootmem_page(unsigned long pfn, unsi
>  }
>  #endif /* CONFIG_HUGETLB_PAGE_OPTIMIZE_VMEMMAP */
>
> +DEFINE_LOCK_GUARD_0(vmemmap_remap,
> +                   hugetlb_vmemmap_write_lock(), hugetlb_vmemmap_write_unlock())
> +
>  static inline bool hugetlb_vmemmap_optimizable(const struct hstate *h)
>  {
>         return hugetlb_vmemmap_optimizable_size(h) != 0;
> diff --git a/mm/memory-failure.c b/mm/memory-failure.c
> index 78f0a13349cb..164a02866fd5 100644
> --- a/mm/memory-failure.c
> +++ b/mm/memory-failure.c
> @@ -69,6 +69,7 @@
>  #include "page_alloc.h"
>  #include "internal.h"
>  #include "memory-failure.h"
> +#include "hugetlb_vmemmap.h"
>
>  static int sysctl_memory_failure_early_kill __read_mostly;
>
> @@ -2038,6 +2039,7 @@ static int get_huge_page_for_hwpoison(unsigned long pfn, int flags,
>         bool count_increased = false;
>         int ret, rc;
>
> +       guard(vmemmap_remap)();
>         spin_lock_irq(&hugetlb_lock);
>         folio = page_folio(page);
>         if (!folio_test_hugetlb(folio)) {
> --
> 2.54.0
>
>

LGTM :)

Reviewed-by: Jiaqi Yan <jiaqiyan@google.com>

Re: [PATCH v2 1/3] mm/hugetlb_vmemmap: fix lost HWPoison state during optimization

[sashiko-bot]

Thank you for your contribution! Sashiko AI review found 1 potential
issue to consider:
- [Medium] mm/memory-failure: unserialized unpoison_memory() races with
  HVO head remap
--

commit c0c1b7e9976626bf636152889fd9f5d88944d78b
Author: Muchun Song <muchun.song@linux.dev>

mm/hugetlb_vmemmap: fix lost HWPoison state during optimization

This commit introduces vmemmap_remap_rwsem to serialize HVO head page
remapping against HWPoison state updates in get_huge_page_for_hwpoison().

> diff --git a/mm/memory-failure.c b/mm/memory-failure.c
> index 78f0a13349cb3..164a02866fd59 100644
> --- a/mm/memory-failure.c
> +++ b/mm/memory-failure.c

[ ... ]

> @@ -2038,6 +2039,7 @@ static int get_huge_page_for_hwpoison(unsigned long pfn, int flags,
>  	bool count_increased = false;
>  	int ret, rc;
>  
> +	guard(vmemmap_remap)();
>  	spin_lock_irq(&hugetlb_lock);
>  	folio = page_folio(page);
>  	if (!folio_test_hugetlb(folio)) {

[Severity: Medium]
Does unpoison_memory() also need vmemmap_remap_rwsem when clearing
PG_hwpoison and freeing folio->_hugetlb_hwpoison?

If a newly initialized hugetlb folio is poisoned by a software-simulated
memory failure before HVO acquires vmemmap_remap_rwsem, unpoison_memory()
can race with the head page remap:

CPU0                                CPU1
hugetlb_vmemmap_optimize_folio()
  down_read(&vmemmap_remap_rwsem)
  ...
    vmemmap_optimize_pte()
      vmemmap_dup_page_from_list(old_head, ...)
                                    unpoison_memory()
                                      get_hwpoison_page(..., MF_UNPOISON)
                                      folio_free_raw_hwp(old_head, false)
                                      folio_test_clear_hwpoison(old_head)
      set_pte_at(..., new_head)
      flush_tlb_all()
                                    /* new_head retains PG_hwpoison and
                                     * freed _hugetlb_hwpoison pointer */

In get_hwpoison_hugetlb_folio(), passing unpoison == true takes the
folio_try_get(folio) branch even when !folio_test_hugetlb_migratable(folio),
returning 0 on the frozen folio instead of -EBUSY:

mm/hugetlb.c:get_hwpoison_hugetlb_folio() {
    ...
		if (folio_test_hugetlb_freed(folio))
			ret = 0;
		else if (folio_test_hugetlb_migratable(folio) || unpoison)
			ret = folio_try_get(folio);
		else
			ret = -EBUSY;
    ...
}

unpoison_memory() then sees ghp == 0 and calls folio_free_raw_hwp(folio,
false), which skips the folio_test_hugetlb_vmemmap_optimized(folio) check
used when move_flag is true and frees the struct raw_hwp_page entries from
old_head.  Once CPU0 installs new_head and flushes the TLB, the folio still
has PG_hwpoison set and folio->_hugetlb_hwpoison pointing to freed memory,
leading to a use-after-free or double free when the list is later walked or
freed.

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20261111093747.50048-1-songmuchun@bytedance.com?part=1

Re: [PATCH v2 1/3] mm/hugetlb_vmemmap: fix lost HWPoison state during optimization

[Muchun Song]

> On Nov 12, 2026, at 06:38, sashiko-bot@kernel.org wrote:
> 
> Thank you for your contribution! Sashiko AI review found 1 potential
> issue to consider:
> - [Medium] mm/memory-failure: unserialized unpoison_memory() races with
>  HVO head remap
> --
> 
> commit c0c1b7e9976626bf636152889fd9f5d88944d78b
> Author: Muchun Song <muchun.song@linux.dev>
> 
> mm/hugetlb_vmemmap: fix lost HWPoison state during optimization
> 
> This commit introduces vmemmap_remap_rwsem to serialize HVO head page
> remapping against HWPoison state updates in get_huge_page_for_hwpoison().
> 
>> diff --git a/mm/memory-failure.c b/mm/memory-failure.c
>> index 78f0a13349cb3..164a02866fd59 100644
>> --- a/mm/memory-failure.c
>> +++ b/mm/memory-failure.c
> 
> [ ... ]
> 
>> @@ -2038,6 +2039,7 @@ static int get_huge_page_for_hwpoison(unsigned long pfn, int flags,
>> 	bool count_increased = false;
>> 	int ret, rc;
>> 
>> +	guard(vmemmap_remap)();
>> 	spin_lock_irq(&hugetlb_lock);
>> 	folio = page_folio(page);
>> 	if (!folio_test_hugetlb(folio)) {
> 
> [Severity: Medium]
> Does unpoison_memory() also need vmemmap_remap_rwsem when clearing
> PG_hwpoison and freeing folio->_hugetlb_hwpoison?

I think the following diff will fix this.

diff --git a/mm/memory-failure.c b/mm/memory-failure.c
index 164a02866fd5..abe92db14fbc 100644
--- a/mm/memory-failure.c
+++ b/mm/memory-failure.c
@@ -2814,6 +2814,7 @@ int unpoison_memory(unsigned long pfn)

        ghp = get_hwpoison_page(p, MF_UNPOISON);
        if (!ghp) {
+               guard(vmemmap_remap)();
                if (folio_test_hugetlb(folio)) {
                        huge = true;
                        count = folio_free_raw_hwp(folio, false);
@@ -2830,6 +2831,7 @@ int unpoison_memory(unsigned long pfn)
                                         pfn, &unpoison_rs);
                }
        } else {
+               guard(vmemmap_remap)();
                if (folio_test_hugetlb(folio)) {
                        huge = true;
                        count = folio_free_raw_hwp(folio, false);

> 
> If a newly initialized hugetlb folio is poisoned by a software-simulated
> memory failure before HVO acquires vmemmap_remap_rwsem, unpoison_memory()
> can race with the head page remap:
> 
> CPU0                                CPU1
> hugetlb_vmemmap_optimize_folio()
>  down_read(&vmemmap_remap_rwsem)
>  ...
>    vmemmap_optimize_pte()
>      vmemmap_dup_page_from_list(old_head, ...)
>                                    unpoison_memory()
>                                      get_hwpoison_page(..., MF_UNPOISON)
>                                      folio_free_raw_hwp(old_head, false)
>                                      folio_test_clear_hwpoison(old_head)
>      set_pte_at(..., new_head)
>      flush_tlb_all()
>                                    /* new_head retains PG_hwpoison and
>                                     * freed _hugetlb_hwpoison pointer */
> 
> In get_hwpoison_hugetlb_folio(), passing unpoison == true takes the
> folio_try_get(folio) branch even when !folio_test_hugetlb_migratable(folio),
> returning 0 on the frozen folio instead of -EBUSY:
> 
> mm/hugetlb.c:get_hwpoison_hugetlb_folio() {
>    ...
> 		if (folio_test_hugetlb_freed(folio))
> 			ret = 0;
> 		else if (folio_test_hugetlb_migratable(folio) || unpoison)
> 			ret = folio_try_get(folio);
> 		else
> 			ret = -EBUSY;
>    ...
> }
> 
> unpoison_memory() then sees ghp == 0 and calls folio_free_raw_hwp(folio,
> false), which skips the folio_test_hugetlb_vmemmap_optimized(folio) check
> used when move_flag is true and frees the struct raw_hwp_page entries from
> old_head.  Once CPU0 installs new_head and flushes the TLB, the folio still
> has PG_hwpoison set and folio->_hugetlb_hwpoison pointing to freed memory,
> leading to a use-after-free or double free when the list is later walked or
> freed.
> 
> -- 
> Sashiko AI review · https://sashiko.dev/#/patchset/20261111093747.50048-1-songmuchun@bytedance.com?part=1

Re: [PATCH v2 1/3] mm/hugetlb_vmemmap: fix lost HWPoison state during optimization

[Muchun Song]

[-- Attachment #1: Type: text/plain, Size: 4335 bytes --]



> On Nov 12, 2026, at 10:59, Muchun Song <muchun.song@linux.dev> wrote:
> 
> 
> 
>> On Nov 12, 2026, at 06:38, sashiko-bot@kernel.org wrote:
>> 
>> Thank you for your contribution! Sashiko AI review found 1 potential
>> issue to consider:
>> - [Medium] mm/memory-failure: unserialized unpoison_memory() races with
>> HVO head remap
>> --
>> 
>> commit c0c1b7e9976626bf636152889fd9f5d88944d78b
>> Author: Muchun Song <muchun.song@linux.dev>
>> 
>> mm/hugetlb_vmemmap: fix lost HWPoison state during optimization
>> 
>> This commit introduces vmemmap_remap_rwsem to serialize HVO head page
>> remapping against HWPoison state updates in get_huge_page_for_hwpoison().
>> 
>>> diff --git a/mm/memory-failure.c b/mm/memory-failure.c
>>> index 78f0a13349cb3..164a02866fd59 100644
>>> --- a/mm/memory-failure.c
>>> +++ b/mm/memory-failure.c
>> 
>> [ ... ]
>> 
>>> @@ -2038,6 +2039,7 @@ static int get_huge_page_for_hwpoison(unsigned long pfn, int flags,
>>> 	bool count_increased = false;
>>> 	int ret, rc;
>>> 
>>> +	guard(vmemmap_remap)();
>>> 	spin_lock_irq(&hugetlb_lock);
>>> 	folio = page_folio(page);
>>> 	if (!folio_test_hugetlb(folio)) {
>> 
>> [Severity: Medium]
>> Does unpoison_memory() also need vmemmap_remap_rwsem when clearing
>> PG_hwpoison and freeing folio->_hugetlb_hwpoison?
> 
> I think the following diff will fix this.
> 
> diff --git a/mm/memory-failure.c b/mm/memory-failure.c
> index 164a02866fd5..abe92db14fbc 100644
> --- a/mm/memory-failure.c
> +++ b/mm/memory-failure.c
> @@ -2814,6 +2814,7 @@ int unpoison_memory(unsigned long pfn)
> 
>        ghp = get_hwpoison_page(p, MF_UNPOISON);
>        if (!ghp) {
> +               guard(vmemmap_remap)();
>                if (folio_test_hugetlb(folio)) {
>                        huge = true;
>                        count = folio_free_raw_hwp(folio, false);
> @@ -2830,6 +2831,7 @@ int unpoison_memory(unsigned long pfn)
>                                         pfn, &unpoison_rs);
>                }
>        } else {
> +               guard(vmemmap_remap)();

Sorry. This is unnecessary since we already hold a refcount to the folio.

>                if (folio_test_hugetlb(folio)) {
>                        huge = true;
>                        count = folio_free_raw_hwp(folio, false);
> 
>> 
>> If a newly initialized hugetlb folio is poisoned by a software-simulated
>> memory failure before HVO acquires vmemmap_remap_rwsem, unpoison_memory()
>> can race with the head page remap:
>> 
>> CPU0                                CPU1
>> hugetlb_vmemmap_optimize_folio()
>> down_read(&vmemmap_remap_rwsem)
>> ...
>>   vmemmap_optimize_pte()
>>     vmemmap_dup_page_from_list(old_head, ...)
>>                                   unpoison_memory()
>>                                     get_hwpoison_page(..., MF_UNPOISON)
>>                                     folio_free_raw_hwp(old_head, false)
>>                                     folio_test_clear_hwpoison(old_head)
>>     set_pte_at(..., new_head)
>>     flush_tlb_all()
>>                                   /* new_head retains PG_hwpoison and
>>                                    * freed _hugetlb_hwpoison pointer */
>> 
>> In get_hwpoison_hugetlb_folio(), passing unpoison == true takes the
>> folio_try_get(folio) branch even when !folio_test_hugetlb_migratable(folio),
>> returning 0 on the frozen folio instead of -EBUSY:
>> 
>> mm/hugetlb.c:get_hwpoison_hugetlb_folio() {
>>   ...
>> 		if (folio_test_hugetlb_freed(folio))
>> 			ret = 0;
>> 		else if (folio_test_hugetlb_migratable(folio) || unpoison)
>> 			ret = folio_try_get(folio);
>> 		else
>> 			ret = -EBUSY;
>>   ...
>> }
>> 
>> unpoison_memory() then sees ghp == 0 and calls folio_free_raw_hwp(folio,
>> false), which skips the folio_test_hugetlb_vmemmap_optimized(folio) check
>> used when move_flag is true and frees the struct raw_hwp_page entries from
>> old_head.  Once CPU0 installs new_head and flushes the TLB, the folio still
>> has PG_hwpoison set and folio->_hugetlb_hwpoison pointing to freed memory,
>> leading to a use-after-free or double free when the list is later walked or
>> freed.
>> 
>> -- 
>> Sashiko AI review · https://sashiko.dev/#/patchset/20261111093747.50048-1-songmuchun@bytedance.com?part=1


[-- Attachment #2: Type: text/html, Size: 26775 bytes --]

[PATCH v2 2/3] mm/hugetlb_vmemmap: fix PMD split TLB flush race

[Muchun Song]

Bulk HVO optimization converts vmemmap PMDs for a list of folios into
PTE tables before issuing a single global TLB flush. After publishing
a PTE table, a concurrent single-folio optimization can observe it and
remap its PTEs before the deferred flush:

CPU0                             CPU1
hugetlb_vmemmap_optimize_folios()
  hugetlb_vmemmap_pteify_folio()
    vmemmap_split_pmd()
      pmd_populate_kernel()
      /* TLB flush deferred */
                                 hugetlb_vmemmap_optimize_folio()
                                   vmemmap_pmd_entry()
                                   /* sees a PTE table */
                                   vmemmap_optimize_pte()
                                     set_pte_at()
  flush_tlb_all()

Modern MMUs permit overlapping TLB entries with different mapping
granularities only when they translate to the same physical address. In
this race, the stale PMD translation still points at the original vmemmap
backing range while the new PTE can point at a shared or replacement
vmemmap page for the same virtual address.

Serialize deferred PMD splitting against PTE remapping with the HVO
rwsem. Hold it for write across the split-only pteify batch and its TLB
flush, then downgrade it before remapping the folios' vmemmap pages.
This guarantees that no PTE remapper can observe a newly split PMD until
the stale PMD translations have been invalidated.

Fixes: f4b7e3efaddb ("hugetlb: batch PMD split for bulk vmemmap dedup")
Cc: stable@vger.kernel.org
Signed-off-by: Muchun Song <songmuchun@bytedance.com>
Reported-by: Sashiko <sashiko-bot@kernel.org>
Closes: https://sashiko.dev/#/bug/linux-fbdc3410-a4cf-457e-88e6-684380fbadfe
---
 mm/hugetlb_vmemmap.c | 14 ++++++++++++--
 1 file changed, 12 insertions(+), 2 deletions(-)

diff --git a/mm/hugetlb_vmemmap.c b/mm/hugetlb_vmemmap.c
index 79ca68dff635..6fba3184356c 100644
--- a/mm/hugetlb_vmemmap.c
+++ b/mm/hugetlb_vmemmap.c
@@ -311,6 +311,9 @@ static inline int vmemmap_restore_range(unsigned long start, unsigned long end,
  * is skipped, the caller must flush TLBs after all pteify operations are complete
  * before remapping the folio's vmemmap pages.
  *
+ * Note: The caller must hold vmemmap_remap_rwsem for write from the first
+ * pteify operation through the final TLB flush.
+ *
  * Return: %0 on success, negative error code otherwise.
  */
 static int vmemmap_pteify_folio(const struct folio *folio, unsigned long flags)
@@ -321,6 +324,8 @@ static int vmemmap_pteify_folio(const struct folio *folio, unsigned long flags)
 		.flags		= flags,
 	};
 
+	rwsem_assert_held_write(&vmemmap_remap_rwsem);
+
 	return vmemmap_remap_range(start, end, &walk);
 }
 
@@ -400,6 +405,7 @@ static int vmemmap_restore_folio(const struct folio *folio, unsigned long flags)
 	if (ret)
 		return ret;
 
+	guard(rwsem_read)(&vmemmap_remap_rwsem);
 	return vmemmap_restore_range(start, end, &vmemmap_pages, flags);
 }
 
@@ -568,6 +574,7 @@ void hugetlb_vmemmap_optimize_folios(struct hstate *h, struct list_head *folio_l
 	if (list_empty(folio_list))
 		return;
 
+	down_write(&vmemmap_remap_rwsem);
 	list_for_each_entry(folio, folio_list, lru) {
 		int ret;
 
@@ -592,12 +599,14 @@ void hugetlb_vmemmap_optimize_folios(struct hstate *h, struct list_head *folio_l
 			break;
 	}
 
-	if (!nr_to_optimize)
+	if (!nr_to_optimize) {
+		up_write(&vmemmap_remap_rwsem);
 		return;
+	}
 
 	flush_tlb_all();
+	downgrade_write(&vmemmap_remap_rwsem);
 
-	guard(rwsem_read)(&vmemmap_remap_rwsem);
 	list_for_each_entry(folio, folio_list, lru) {
 		int ret;
 
@@ -620,6 +629,7 @@ void hugetlb_vmemmap_optimize_folios(struct hstate *h, struct list_head *folio_l
 	}
 
 	flush_tlb_all();
+	up_read(&vmemmap_remap_rwsem);
 	free_vmemmap_page_list(&vmemmap_pages);
 }
 
-- 
2.54.0

Re: [PATCH v2 2/3] mm/hugetlb_vmemmap: fix PMD split TLB flush race

[Jose A. Perez de Azpillaga]

On 11/11/26 10:37 AM, Muchun Song wrote:
> Bulk HVO optimization converts vmemmap PMDs for a list of folios into
> PTE tables before issuing a single global TLB flush. After publishing
> a PTE table, a concurrent single-folio optimization can observe it and
> remap its PTEs before the deferred flush:
> 
> CPU0                             CPU1
> hugetlb_vmemmap_optimize_folios()
>    hugetlb_vmemmap_pteify_folio()
>      vmemmap_split_pmd()
>        pmd_populate_kernel()
>        /* TLB flush deferred */
>                                   hugetlb_vmemmap_optimize_folio()
>                                     vmemmap_pmd_entry()
>                                     /* sees a PTE table */
>                                     vmemmap_optimize_pte()
>                                       set_pte_at()
>    flush_tlb_all()
> 
> Modern MMUs permit overlapping TLB entries with different mapping
> granularities only when they translate to the same physical address. In
> this race, the stale PMD translation still points at the original vmemmap
> backing range while the new PTE can point at a shared or replacement
> vmemmap page for the same virtual address.
> 
> Serialize deferred PMD splitting against PTE remapping with the HVO
> rwsem. Hold it for write across the split-only pteify batch and its TLB
> flush, then downgrade it before remapping the folios' vmemmap pages.
> This guarantees that no PTE remapper can observe a newly split PMD until
> the stale PMD translations have been invalidated.
> 
> Fixes: f4b7e3efaddb ("hugetlb: batch PMD split for bulk vmemmap dedup")
> Cc: stable@vger.kernel.org
> Signed-off-by: Muchun Song <songmuchun@bytedance.com>
> Reported-by: Sashiko <sashiko-bot@kernel.org>
> Closes: https://sashiko.dev/#/bug/linux-fbdc3410-a4cf-457e-88e6-684380fbadfe


LGTM:

Reviewed-by: Jose A. Perez de Azpillaga <azpijr@gmail.com>

-- 
cheers,
jose a. p-a

[PATCH v2 3/3] mm/hugetlb_vmemmap: fix deadlock during PMD split

[Muchun Song]

vmemmap_remap_range() currently walks vmemmap page tables under
init_mm.mmap_lock. If the walk reaches a leaf PMD, vmemmap_split_pmd()
allocates a PTE table with GFP_KERNEL and can enter direct reclaim:

hugetlb_vmemmap_optimize_folio()
  vmemmap_remap_range()
    mmap_read_lock(&init_mm)  <-- outer read lock
    ...
      vmemmap_split_pmd()
        pte_alloc_one_kernel(&init_mm)  /* GFP_KERNEL */
          <direct reclaim>
            ttm_pool_shrinker_scan()
              ...
                change_page_attr_set_clr()
                  mmap_read_lock(&init_mm)  <-- self-deadlock

Since commit d5d8b8662e6e ("x86/mm/pat: Acquire init_mm read lock on
attribute changes to avoid UAF"), reclaim through TTM can enter CPA,
which also takes init_mm.mmap_lock. If a writer is pending, the nested
read blocks behind it while the outer read cannot be released, causing a
self-deadlock.

HVO walks vmemmap ranges whose intermediate page tables remain present.
vmemmap_split_pmd() revalidates the PMD under init_mm.page_table_lock
before installing the PTE table. Use the lockless page-table walker so the
allocation no longer runs under init_mm.mmap_lock.

Fixes: d5d8b8662e6e ("x86/mm/pat: Acquire init_mm read lock on attribute changes to avoid UAF")
Cc: stable@vger.kernel.org
Signed-off-by: Muchun Song <songmuchun@bytedance.com>
Reported-by: Sashiko <sashiko-bot@kernel.org>
Closes: https://lore.kernel.org/sashiko-outbox-206969@kernel.org/
---
 mm/hugetlb_vmemmap.c | 6 ++----
 1 file changed, 2 insertions(+), 4 deletions(-)

diff --git a/mm/hugetlb_vmemmap.c b/mm/hugetlb_vmemmap.c
index 6fba3184356c..5852b0a1c479 100644
--- a/mm/hugetlb_vmemmap.c
+++ b/mm/hugetlb_vmemmap.c
@@ -161,10 +161,8 @@ static int vmemmap_remap_range(unsigned long start, unsigned long end,
 
 	VM_BUG_ON(!PAGE_ALIGNED(start | end));
 
-	mmap_read_lock(&init_mm);
-	ret = walk_kernel_page_table_range(start, end, &vmemmap_remap_ops,
-				    NULL, walk);
-	mmap_read_unlock(&init_mm);
+	ret = walk_kernel_page_table_range_lockless(start, end, &vmemmap_remap_ops,
+						    NULL, walk);
 	if (ret)
 		return ret;
 
-- 
2.54.0

Re: [PATCH v2 3/3] mm/hugetlb_vmemmap: fix deadlock during PMD split

[Jose A. Perez de Azpillaga]

On 11/11/26 10:37 AM, Muchun Song wrote:
> vmemmap_remap_range() currently walks vmemmap page tables under
> init_mm.mmap_lock. If the walk reaches a leaf PMD, vmemmap_split_pmd()
> allocates a PTE table with GFP_KERNEL and can enter direct reclaim:
> 
> hugetlb_vmemmap_optimize_folio()
>    vmemmap_remap_range()
>      mmap_read_lock(&init_mm)  <-- outer read lock
>      ...
>        vmemmap_split_pmd()
>          pte_alloc_one_kernel(&init_mm)  /* GFP_KERNEL */
>            <direct reclaim>
>              ttm_pool_shrinker_scan()
>                ...
>                  change_page_attr_set_clr()
>                    mmap_read_lock(&init_mm)  <-- self-deadlock
> 
> Since commit d5d8b8662e6e ("x86/mm/pat: Acquire init_mm read lock on
> attribute changes to avoid UAF"), reclaim through TTM can enter CPA,
> which also takes init_mm.mmap_lock. If a writer is pending, the nested
> read blocks behind it while the outer read cannot be released, causing a
> self-deadlock.
> 
> HVO walks vmemmap ranges whose intermediate page tables remain present.
> vmemmap_split_pmd() revalidates the PMD under init_mm.page_table_lock
> before installing the PTE table. Use the lockless page-table walker so the
> allocation no longer runs under init_mm.mmap_lock.
> 
> Fixes: d5d8b8662e6e ("x86/mm/pat: Acquire init_mm read lock on attribute changes to avoid UAF")
> Cc: stable@vger.kernel.org
> Signed-off-by: Muchun Song <songmuchun@bytedance.com>
> Reported-by: Sashiko <sashiko-bot@kernel.org>
> Closes: https://lore.kernel.org/sashiko-outbox-206969@kernel.org/


LGTM:

Reviewed-by: Jose A. Perez de Azpillaga <azpijr@gmail.com>

-- 
cheers,
jose a. p-a