[PATCH v4] mm/huge_memory: fix folio isn't locked in softleaf_to_folio()

[PATCH v4] mm/huge_memory: fix folio isn't locked in softleaf_to_folio()

[Jinjiang Tu]

On arm64 server, we found folio that get from migration entry isn't locked
in softleaf_to_folio(). This issue triggers when mTHP splitting and
zap_nonpresent_ptes() races, and the root cause is lack of memory barrier
in softleaf_to_folio(). The race is as follows:

	CPU0                                             CPU1

deferred_split_scan()                              zap_nonpresent_ptes()
  lock folio
  split_folio()
    unmap_folio()
      change ptes to migration entries
    __split_folio_to_order()                         softleaf_to_folio()
      set flags(including PG_locked) for tail pages    folio = pfn_folio(softleaf_to_pfn(entry))
      smp_wmb()                                        VM_WARN_ON_ONCE(!folio_test_locked(folio))
      prep_compound_page() for tail pages

In __split_folio_to_order(), smp_wmb() guarantees page flags of tail pages
are visible before the tail page becomes non-compound. smp_wmb() should
be paired with smp_rmb() in softleaf_to_folio(), which is missed. As a
result, if zap_nonpresent_ptes() accesses migration entry that stores
tail pfn, softleaf_to_folio() may see the updated compound_head of tail
page before page->flags.

To fix it, add missing smp_rmb() if the softleaf entry is migration entry
in softleaf_to_folio() and softleaf_to_page().

Fixes: e9b61f19858a ("thp: reintroduce split_huge_page()")
Cc: stable@kernel.org
Acked-by: David Hildenbrand (Arm) <david@kernel.org>
Reviewed-by: Lorenzo Stoakes (Oracle) <ljs@kernel.org>
Signed-off-by: Jinjiang Tu <tujinjiang@huawei.com>
---
Changes in v4:
 * update function name and comments.
 * collect Acked-by and Reviewed-by.

 include/linux/leafops.h | 32 +++++++++++++++++++++-----------
 1 file changed, 21 insertions(+), 11 deletions(-)

diff --git a/include/linux/leafops.h b/include/linux/leafops.h
index a9ff94b744f2..05673d3529e7 100644
--- a/include/linux/leafops.h
+++ b/include/linux/leafops.h
@@ -363,6 +363,23 @@ static inline unsigned long softleaf_to_pfn(softleaf_t entry)
 	return swp_offset(entry) & SWP_PFN_MASK;
 }
 
+static inline void softleaf_migration_sync(softleaf_t entry,
+		struct folio *folio)
+{
+	/*
+	 * Ensure we do not race with split, which might alter tail pages into new
+	 * folios and thus result in observing an unlocked folio.
+	 * This matches the write barrier in __split_folio_to_order().
+	 */
+	smp_rmb();
+
+	/*
+	 * Any use of migration entries may only occur while the
+	 * corresponding page is locked
+	 */
+	VM_WARN_ON_ONCE(!folio_test_locked(folio));
+}
+
 /**
  * softleaf_to_page() - Obtains struct page for PFN encoded within leaf entry.
  * @entry: Leaf entry, softleaf_has_pfn(@entry) must return true.
@@ -374,11 +391,8 @@ static inline struct page *softleaf_to_page(softleaf_t entry)
 	struct page *page = pfn_to_page(softleaf_to_pfn(entry));
 
 	VM_WARN_ON_ONCE(!softleaf_has_pfn(entry));
-	/*
-	 * Any use of migration entries may only occur while the
-	 * corresponding page is locked
-	 */
-	VM_WARN_ON_ONCE(softleaf_is_migration(entry) && !PageLocked(page));
+	if (softleaf_is_migration(entry))
+		softleaf_migration_sync(entry, page_folio(page));
 
 	return page;
 }
@@ -394,12 +408,8 @@ static inline struct folio *softleaf_to_folio(softleaf_t entry)
 	struct folio *folio = pfn_folio(softleaf_to_pfn(entry));
 
 	VM_WARN_ON_ONCE(!softleaf_has_pfn(entry));
-	/*
-	 * Any use of migration entries may only occur while the
-	 * corresponding folio is locked.
-	 */
-	VM_WARN_ON_ONCE(softleaf_is_migration(entry) &&
-			!folio_test_locked(folio));
+	if (softleaf_is_migration(entry))
+		softleaf_migration_sync(entry, folio);
 
 	return folio;
 }
-- 
2.43.0

Re: [PATCH v4] mm/huge_memory: fix folio isn't locked in softleaf_to_folio()

[Andrew Morton]

On Sat, 21 Mar 2026 15:52:14 +0800 Jinjiang Tu <tujinjiang@huawei.com> wrote:

> On arm64 server, we found folio that get from migration entry isn't locked
> in softleaf_to_folio(). This issue triggers when mTHP splitting and
> zap_nonpresent_ptes() races, and the root cause is lack of memory barrier
> in softleaf_to_folio(). The race is as follows:
> 
> 	CPU0                                             CPU1
> 
> deferred_split_scan()                              zap_nonpresent_ptes()
>   lock folio
>   split_folio()
>     unmap_folio()
>       change ptes to migration entries
>     __split_folio_to_order()                         softleaf_to_folio()
>       set flags(including PG_locked) for tail pages    folio = pfn_folio(softleaf_to_pfn(entry))
>       smp_wmb()                                        VM_WARN_ON_ONCE(!folio_test_locked(folio))
>       prep_compound_page() for tail pages
> 
> In __split_folio_to_order(), smp_wmb() guarantees page flags of tail pages
> are visible before the tail page becomes non-compound. smp_wmb() should
> be paired with smp_rmb() in softleaf_to_folio(), which is missed. As a
> result, if zap_nonpresent_ptes() accesses migration entry that stores
> tail pfn, softleaf_to_folio() may see the updated compound_head of tail
> page before page->flags.
> 
> To fix it, add missing smp_rmb() if the softleaf entry is migration entry
> in softleaf_to_folio() and softleaf_to_page().

AI review isn't entirely happy:
	https://sashiko.dev/#/patchset/20260321075214.3305564-1-tujinjiang@huawei.com

Re: [PATCH v4] mm/huge_memory: fix folio isn't locked in softleaf_to_folio()

[Jinjiang Tu]

在 2026/3/22 2:07, Andrew Morton 写道:
> On Sat, 21 Mar 2026 15:52:14 +0800 Jinjiang Tu <tujinjiang@huawei.com> wrote:
>
>> On arm64 server, we found folio that get from migration entry isn't locked
>> in softleaf_to_folio(). This issue triggers when mTHP splitting and
>> zap_nonpresent_ptes() races, and the root cause is lack of memory barrier
>> in softleaf_to_folio(). The race is as follows:
>>
>> 	CPU0                                             CPU1
>>
>> deferred_split_scan()                              zap_nonpresent_ptes()
>>    lock folio
>>    split_folio()
>>      unmap_folio()
>>        change ptes to migration entries
>>      __split_folio_to_order()                         softleaf_to_folio()
>>        set flags(including PG_locked) for tail pages    folio = pfn_folio(softleaf_to_pfn(entry))
>>        smp_wmb()                                        VM_WARN_ON_ONCE(!folio_test_locked(folio))
>>        prep_compound_page() for tail pages
>>
>> In __split_folio_to_order(), smp_wmb() guarantees page flags of tail pages
>> are visible before the tail page becomes non-compound. smp_wmb() should
>> be paired with smp_rmb() in softleaf_to_folio(), which is missed. As a
>> result, if zap_nonpresent_ptes() accesses migration entry that stores
>> tail pfn, softleaf_to_folio() may see the updated compound_head of tail
>> page before page->flags.
>>
>> To fix it, add missing smp_rmb() if the softleaf entry is migration entry
>> in softleaf_to_folio() and softleaf_to_page().
> AI review isn't entirely happy:
> 	https://sashiko.dev/#/patchset/20260321075214.3305564-1-tujinjiang@huawei.com

commit 54150f26b76fbe8cbf90c19d0052add5199851f0
Author: Jinjiang Tu <tujinjiang@huawei.com>
mm/huge_memory: fix folio isn't locked in softleaf_to_folio()
This commit adds a memory barrier to softleaf_to_folio() and
softleaf_to_page() to prevent a race condition during mTHP splitting
where the updated compound_head of a tail page might be observed before
its page flags.
> diff --git a/include/linux/leafops.h b/include/linux/leafops.h
> --- a/include/linux/leafops.h
> +++ b/include/linux/leafops.h
> @@ -363,6 +363,23 @@ static inline unsigned long softleaf_to_pfn(softleaf_t entry)
>      return swp_offset(entry) & SWP_PFN_MASK;
>  }
>  
> +static inline void softleaf_migration_sync(softleaf_t entry,
> +        struct folio *folio)
"This isn't a bug, but is the entry parameter needed here? It doesn't appear
to be used in the function body."

Indeed, the previous version move softleaf_is_migration() check out of
softleaf_migration_sync(), the parameter could be removed.

> +{
> +    /*
> +     * Ensure we do not race with split, which might alter tail pages into new
> +     * folios and thus result in observing an unlocked folio.
> +     * This matches the write barrier in __split_folio_to_order().
> +     */
> +    smp_rmb();
> +
> +    /*
> +     * Any use of migration entries may only occur while the
> +     * corresponding page is locked
> +     */
> +    VM_WARN_ON_ONCE(!folio_test_locked(folio));
> +}
> +
>  /**
>   * softleaf_to_page() - Obtains struct page for PFN encoded within leaf entry.
>   * @entry: Leaf entry, softleaf_has_pfn(@entry) must return true.
> @@ -374,11 +391,8 @@ static inline struct page *softleaf_to_page(softleaf_t entry)
>      struct page *page = pfn_to_page(softleaf_to_pfn(entry));
>  
>      VM_WARN_ON_ONCE(!softleaf_has_pfn(entry));
> -    /*
> -     * Any use of migration entries may only occur while the
> -     * corresponding page is locked
> -     */
> -    VM_WARN_ON_ONCE(softleaf_is_migration(entry) && !PageLocked(page));
> +    if (softleaf_is_migration(entry))
> +        softleaf_migration_sync(entry, page_folio(page));
>  
>      return page;
>  }
"Can this still race if callers use softleaf_to_page() followed by
page_folio()?
If a caller does something similar to:
page = softleaf_to_page(entry);
folio = page_folio(page);
if (folio_test_locked(folio)) { ... }
The call to page_folio() performs a new read of compound_head. Because
this new read and the subsequent reads of folio->flags both execute after
the smp_rmb() inside softleaf_to_page(), there is no memory barrier
between them.
If a weakly-ordered CPU speculatively reads folio->flags before the new
read of compound_head evaluates as order-0, could the caller still see
stale flags and reproduce the crash this patch intends to fix?
Would it be safer to require vulnerable callers to use softleaf_to_folio()
directly?
[ ... ]
"

This issue only exist when the softleaf_entry is migration entry and is
pte-level. Check all callsites of softleaf_to_page(), the softleaf entries
are almost SOFTLEAF_DEVICE_* or pmd-level. The exceptions are smaps_pte_entry()
and pte_to_pagemap_entry(), they only collect information for userspace, and
I think such race doesn't matter. So, I think this AI review comment is false positive.

Thanks.