[PATCH] mm/hugetlb: fix swap entry corruption when clearing uffd-wp at fork()

[PATCH] mm/hugetlb: fix swap entry corruption when clearing uffd-wp at fork()

[Kiryl Shutsemau]

From: "Kiryl Shutsemau (Meta)" <kas@kernel.org>

copy_hugetlb_page_range() clears the uffd-wp bit of hwpoison and
migration entries with huge_pte_clear_uffd_wp(), which operates on the
present-PTE bit position. Swap entries keep the uffd-wp state elsewhere
-- the same branches read and set it with pte_swp_uffd_wp() and
pte_swp_mkuffd_wp() -- and the present-PTE position falls into the swap
payload. On x86-64 it lands in the inverted swap offset, where a
naturally-aligned hugetlb PFN always has the affected bit set, so the
clear advances the encoded PFN by two pages.

No userfaultfd needs to be involved: the clear is guarded only by the
child VMA not being uffd-wp registered, so a plain fork() with an
in-flight hugetlb migration entry (or a poisoned hugetlb page) corrupts
the entry copied into the child. Instrumenting the hwpoison branch and
forking after MADV_HWPOISON on a 2MB anon hugetlb page shows:

  offset before=120e00
  offset after =120e02

The fallout is mostly latent: rmap walks match migration entries by
folio range and remove_migration_pte() rebuilds the PTE from the folio,
so a within-folio PFN skew heals once migration completes. But any path
that re-encodes the corrupted offset -- e.g. hugetlb_change_protection()
rewriting a writable migration entry via
make_readable_migration_entry(swp_offset(entry)) -- propagates it, and
an hwpoison entry misidentifies which page is poisoned.

Use pte_swp_clear_uffd_wp(), matching copy_nonpresent_pte() and
move_huge_pte().

Reported-by: Sashiko AI review <sashiko-bot@kernel.org>
Closes: https://lore.kernel.org/all/20260703140011.99E601F000E9@smtp.kernel.org/
Fixes: bc70fbf269fd ("mm/hugetlb: handle uffd-wp during fork()")
Cc: stable@vger.kernel.org
Signed-off-by: Kiryl Shutsemau <kas@kernel.org>
Assisted-by: Claude:claude-fable-5
---
 mm/hugetlb.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/mm/hugetlb.c b/mm/hugetlb.c
index 571212b80835..a4e6dd3a82f4 100644
--- a/mm/hugetlb.c
+++ b/mm/hugetlb.c
@@ -4918,7 +4918,7 @@ int copy_hugetlb_page_range(struct mm_struct *dst, struct mm_struct *src,
 		softleaf = softleaf_from_pte(entry);
 		if (unlikely(softleaf_is_hwpoison(softleaf))) {
 			if (!userfaultfd_wp(dst_vma))
-				entry = huge_pte_clear_uffd_wp(entry);
+				entry = pte_swp_clear_uffd_wp(entry);
 			set_huge_pte_at(dst, addr, dst_pte, entry, sz);
 		} else if (unlikely(softleaf_is_migration(softleaf))) {
 			bool uffd_wp = pte_swp_uffd_wp(entry);
@@ -4936,7 +4936,7 @@ int copy_hugetlb_page_range(struct mm_struct *dst, struct mm_struct *src,
 				set_huge_pte_at(src, addr, src_pte, entry, sz);
 			}
 			if (!userfaultfd_wp(dst_vma))
-				entry = huge_pte_clear_uffd_wp(entry);
+				entry = pte_swp_clear_uffd_wp(entry);
 			set_huge_pte_at(dst, addr, dst_pte, entry, sz);
 		} else if (unlikely(pte_is_marker(entry))) {
 			const pte_marker marker = copy_pte_marker(softleaf, dst_vma);

base-commit: dc59e4fea9d83f03bad6bddf3fa2e52491777482
-- 
2.54.0

Re: [PATCH] mm/hugetlb: fix swap entry corruption when clearing uffd-wp at fork()

[David Hildenbrand (Arm)]

On 7/3/26 18:18, Kiryl Shutsemau wrote:
> From: "Kiryl Shutsemau (Meta)" <kas@kernel.org>
> 
> copy_hugetlb_page_range() clears the uffd-wp bit of hwpoison and
> migration entries with huge_pte_clear_uffd_wp(), which operates on the
> present-PTE bit position. Swap entries keep the uffd-wp state elsewhere
> -- the same branches read and set it with pte_swp_uffd_wp() and
> pte_swp_mkuffd_wp() -- and the present-PTE position falls into the swap
> payload. On x86-64 it lands in the inverted swap offset, where a
> naturally-aligned hugetlb PFN always has the affected bit set, so the
> clear advances the encoded PFN by two pages.
> 
> No userfaultfd needs to be involved: the clear is guarded only by the
> child VMA not being uffd-wp registered, so a plain fork() with an
> in-flight hugetlb migration entry (or a poisoned hugetlb page) corrupts
> the entry copied into the child. Instrumenting the hwpoison branch and
> forking after MADV_HWPOISON on a 2MB anon hugetlb page shows:
> 
>   offset before=120e00
>   offset after =120e02
> 
> The fallout is mostly latent: rmap walks match migration entries by
> folio range and remove_migration_pte() rebuilds the PTE from the folio,
> so a within-folio PFN skew heals once migration completes. But any path
> that re-encodes the corrupted offset -- e.g. hugetlb_change_protection()
> rewriting a writable migration entry via
> make_readable_migration_entry(swp_offset(entry)) -- propagates it, and
> an hwpoison entry misidentifies which page is poisoned.
> 
> Use pte_swp_clear_uffd_wp(), matching copy_nonpresent_pte() and
> move_huge_pte().
> 
> Reported-by: Sashiko AI review <sashiko-bot@kernel.org>
> Closes: https://lore.kernel.org/all/20260703140011.99E601F000E9@smtp.kernel.org/
> Fixes: bc70fbf269fd ("mm/hugetlb: handle uffd-wp during fork()")
> Cc: stable@vger.kernel.org
> Signed-off-by: Kiryl Shutsemau <kas@kernel.org>
> Assisted-by: Claude:claude-fable-5
> ---
>  mm/hugetlb.c | 4 ++--
>  1 file changed, 2 insertions(+), 2 deletions(-)
> 
> diff --git a/mm/hugetlb.c b/mm/hugetlb.c
> index 571212b80835..a4e6dd3a82f4 100644
> --- a/mm/hugetlb.c
> +++ b/mm/hugetlb.c
> @@ -4918,7 +4918,7 @@ int copy_hugetlb_page_range(struct mm_struct *dst, struct mm_struct *src,
>  		softleaf = softleaf_from_pte(entry);
>  		if (unlikely(softleaf_is_hwpoison(softleaf))) {
>  			if (!userfaultfd_wp(dst_vma))
> -				entry = huge_pte_clear_uffd_wp(entry);
> +				entry = pte_swp_clear_uffd_wp(entry);

I think installing a hwpoison pte will actually drop the uffd marker.

hugetlb_change_protection() does nothing on hwpoison entrues.

So how could be possibly get a hwpoison entry with an uffd-wp bit set here?

If we indeed can't, Id assume there is nothing to clear here at all.

>  			set_huge_pte_at(dst, addr, dst_pte, entry, sz);
>  		} else if (unlikely(softleaf_is_migration(softleaf))) {
>  			bool uffd_wp = pte_swp_uffd_wp(entry);
> @@ -4936,7 +4936,7 @@ int copy_hugetlb_page_range(struct mm_struct *dst, struct mm_struct *src,
>  				set_huge_pte_at(src, addr, src_pte, entry, sz);
>  			}
>  			if (!userfaultfd_wp(dst_vma))
> -				entry = huge_pte_clear_uffd_wp(entry);
> +				entry = pte_swp_clear_uffd_wp(entry);


That looks correct.


-- 
Cheers,

David

Re: [PATCH] mm/hugetlb: fix swap entry corruption when clearing uffd-wp at fork()

[Kiryl Shutsemau]

On Tue, Jul 07, 2026 at 05:05:49PM +0200, David Hildenbrand (Arm) wrote:
> On 7/3/26 18:18, Kiryl Shutsemau wrote:
> > From: "Kiryl Shutsemau (Meta)" <kas@kernel.org>
> > 
> > copy_hugetlb_page_range() clears the uffd-wp bit of hwpoison and
> > migration entries with huge_pte_clear_uffd_wp(), which operates on the
> > present-PTE bit position. Swap entries keep the uffd-wp state elsewhere
> > -- the same branches read and set it with pte_swp_uffd_wp() and
> > pte_swp_mkuffd_wp() -- and the present-PTE position falls into the swap
> > payload. On x86-64 it lands in the inverted swap offset, where a
> > naturally-aligned hugetlb PFN always has the affected bit set, so the
> > clear advances the encoded PFN by two pages.
> > 
> > No userfaultfd needs to be involved: the clear is guarded only by the
> > child VMA not being uffd-wp registered, so a plain fork() with an
> > in-flight hugetlb migration entry (or a poisoned hugetlb page) corrupts
> > the entry copied into the child. Instrumenting the hwpoison branch and
> > forking after MADV_HWPOISON on a 2MB anon hugetlb page shows:
> > 
> >   offset before=120e00
> >   offset after =120e02
> > 
> > The fallout is mostly latent: rmap walks match migration entries by
> > folio range and remove_migration_pte() rebuilds the PTE from the folio,
> > so a within-folio PFN skew heals once migration completes. But any path
> > that re-encodes the corrupted offset -- e.g. hugetlb_change_protection()
> > rewriting a writable migration entry via
> > make_readable_migration_entry(swp_offset(entry)) -- propagates it, and
> > an hwpoison entry misidentifies which page is poisoned.
> > 
> > Use pte_swp_clear_uffd_wp(), matching copy_nonpresent_pte() and
> > move_huge_pte().
> > 
> > Reported-by: Sashiko AI review <sashiko-bot@kernel.org>
> > Closes: https://lore.kernel.org/all/20260703140011.99E601F000E9@smtp.kernel.org/
> > Fixes: bc70fbf269fd ("mm/hugetlb: handle uffd-wp during fork()")
> > Cc: stable@vger.kernel.org
> > Signed-off-by: Kiryl Shutsemau <kas@kernel.org>
> > Assisted-by: Claude:claude-fable-5
> > ---
> >  mm/hugetlb.c | 4 ++--
> >  1 file changed, 2 insertions(+), 2 deletions(-)
> > 
> > diff --git a/mm/hugetlb.c b/mm/hugetlb.c
> > index 571212b80835..a4e6dd3a82f4 100644
> > --- a/mm/hugetlb.c
> > +++ b/mm/hugetlb.c
> > @@ -4918,7 +4918,7 @@ int copy_hugetlb_page_range(struct mm_struct *dst, struct mm_struct *src,
> >  		softleaf = softleaf_from_pte(entry);
> >  		if (unlikely(softleaf_is_hwpoison(softleaf))) {
> >  			if (!userfaultfd_wp(dst_vma))
> > -				entry = huge_pte_clear_uffd_wp(entry);
> > +				entry = pte_swp_clear_uffd_wp(entry);
> 
> I think installing a hwpoison pte will actually drop the uffd marker.
> 
> hugetlb_change_protection() does nothing on hwpoison entrues.
> 
> So how could be possibly get a hwpoison entry with an uffd-wp bit set here?
> 
> If we indeed can't, Id assume there is nothing to clear here at all.

You are right.

I am inclined to remove it in a separate cleanup patch. Any objections?

-- 
  Kiryl Shutsemau / Kirill A. Shutemov

Re: [PATCH] mm/hugetlb: fix swap entry corruption when clearing uffd-wp at fork()

[David Hildenbrand (Arm)]

On 7/7/26 19:05, Kiryl Shutsemau wrote:
> On Tue, Jul 07, 2026 at 05:05:49PM +0200, David Hildenbrand (Arm) wrote:
>> On 7/3/26 18:18, Kiryl Shutsemau wrote:
>>> From: "Kiryl Shutsemau (Meta)" <kas@kernel.org>
>>>
>>> copy_hugetlb_page_range() clears the uffd-wp bit of hwpoison and
>>> migration entries with huge_pte_clear_uffd_wp(), which operates on the
>>> present-PTE bit position. Swap entries keep the uffd-wp state elsewhere
>>> -- the same branches read and set it with pte_swp_uffd_wp() and
>>> pte_swp_mkuffd_wp() -- and the present-PTE position falls into the swap
>>> payload. On x86-64 it lands in the inverted swap offset, where a
>>> naturally-aligned hugetlb PFN always has the affected bit set, so the
>>> clear advances the encoded PFN by two pages.
>>>
>>> No userfaultfd needs to be involved: the clear is guarded only by the
>>> child VMA not being uffd-wp registered, so a plain fork() with an
>>> in-flight hugetlb migration entry (or a poisoned hugetlb page) corrupts
>>> the entry copied into the child. Instrumenting the hwpoison branch and
>>> forking after MADV_HWPOISON on a 2MB anon hugetlb page shows:
>>>
>>>   offset before=120e00
>>>   offset after =120e02
>>>
>>> The fallout is mostly latent: rmap walks match migration entries by
>>> folio range and remove_migration_pte() rebuilds the PTE from the folio,
>>> so a within-folio PFN skew heals once migration completes. But any path
>>> that re-encodes the corrupted offset -- e.g. hugetlb_change_protection()
>>> rewriting a writable migration entry via
>>> make_readable_migration_entry(swp_offset(entry)) -- propagates it, and
>>> an hwpoison entry misidentifies which page is poisoned.
>>>
>>> Use pte_swp_clear_uffd_wp(), matching copy_nonpresent_pte() and
>>> move_huge_pte().
>>>
>>> Reported-by: Sashiko AI review <sashiko-bot@kernel.org>
>>> Closes: https://lore.kernel.org/all/20260703140011.99E601F000E9@smtp.kernel.org/
>>> Fixes: bc70fbf269fd ("mm/hugetlb: handle uffd-wp during fork()")
>>> Cc: stable@vger.kernel.org
>>> Signed-off-by: Kiryl Shutsemau <kas@kernel.org>
>>> Assisted-by: Claude:claude-fable-5
>>> ---
>>>  mm/hugetlb.c | 4 ++--
>>>  1 file changed, 2 insertions(+), 2 deletions(-)
>>>
>>> diff --git a/mm/hugetlb.c b/mm/hugetlb.c
>>> index 571212b80835..a4e6dd3a82f4 100644
>>> --- a/mm/hugetlb.c
>>> +++ b/mm/hugetlb.c
>>> @@ -4918,7 +4918,7 @@ int copy_hugetlb_page_range(struct mm_struct *dst, struct mm_struct *src,
>>>  		softleaf = softleaf_from_pte(entry);
>>>  		if (unlikely(softleaf_is_hwpoison(softleaf))) {
>>>  			if (!userfaultfd_wp(dst_vma))
>>> -				entry = huge_pte_clear_uffd_wp(entry);
>>> +				entry = pte_swp_clear_uffd_wp(entry);
>>
>> I think installing a hwpoison pte will actually drop the uffd marker.
>>
>> hugetlb_change_protection() does nothing on hwpoison entrues.
>>
>> So how could be possibly get a hwpoison entry with an uffd-wp bit set here?
>>
>> If we indeed can't, Id assume there is nothing to clear here at all.
> 
> You are right.
> 
> I am inclined to remove it in a separate cleanup patch. Any objections?

I'd just do it in this patch -- removing unnecessary code instead of fixing it.

However, if there are strong opinions, I don't care that much.

-- 
Cheers,

David