[PATCH] mm/huge_memory: fix huge zero folio publication race

[PATCH] mm/huge_memory: fix huge zero folio publication race

[xueyuan.chen]

From: Xueyuan Chen <xueyuan.chen@vivo.com>

get_huge_zero_folio() uses huge_zero_refcount as the publication gate for
lockless users. However, it publishes huge_zero_folio with cmpxchg()
before storing huge_zero_pfn and initializes the refcount with an unordered
atomic_set().

On weakly ordered systems, a racing atomic_inc_not_zero() can observe the
nonzero refcount without observing the huge_zero_pfn store.
is_huge_zero_pmd() can then misidentify a huge zero PMD as a regular
anonymous THP. The write-protect fault path may consequently mark the
global huge zero folio anonymous, exclusive and writable.

Use atomic_set_release() to publish the initialized folio and PFN. A
successful atomic_inc_not_zero() is fully ordered, so a reader that obtains
a reference also observes the state published before the refcount became
nonzero.

Fixes: 97ae17497e99 ("thp: implement refcounting for huge zero page")
Cc: stable@vger.kernel.org
Signed-off-by: Xueyuan Chen <xueyuan.chen@vivo.com>
---
 mm/huge_memory.c | 7 +++++--
 1 file changed, 5 insertions(+), 2 deletions(-)

diff --git a/mm/huge_memory.c b/mm/huge_memory.c
index 032702a4637b..6c74d0375377 100644
--- a/mm/huge_memory.c
+++ b/mm/huge_memory.c
@@ -245,8 +245,11 @@ static bool get_huge_zero_folio(void)
 	}
 	WRITE_ONCE(huge_zero_pfn, folio_pfn(zero_folio));
 
-	/* We take additional reference here. It will be put back by shrinker */
-	atomic_set(&huge_zero_refcount, 2);
+	/*
+	 * Publish the folio and PFN before making them available to lockless
+	 * users. Pairs with a successful atomic_inc_not_zero() above.
+	 */
+	atomic_set_release(&huge_zero_refcount, 2);
 	preempt_enable();
 	count_vm_event(THP_ZERO_PAGE_ALLOC);
 	return true;
-- 
2.47.3

Re: [PATCH] mm/huge_memory: fix huge zero folio publication race

[David Hildenbrand (Arm)]

On 7/29/26 06:50, xueyuan.chen@vivo.com wrote:
> From: Xueyuan Chen <xueyuan.chen@vivo.com>
> 
> get_huge_zero_folio() uses huge_zero_refcount as the publication gate for
> lockless users. However, it publishes huge_zero_folio with cmpxchg()
> before storing huge_zero_pfn and initializes the refcount with an unordered
> atomic_set().
> 
> On weakly ordered systems, a racing atomic_inc_not_zero() can observe the
> nonzero refcount without observing the huge_zero_pfn store.
> is_huge_zero_pmd() can then misidentify a huge zero PMD as a regular
> anonymous THP. The write-protect fault path may consequently mark the
> global huge zero folio anonymous, exclusive and writable.
> 
> Use atomic_set_release() to publish the initialized folio and PFN. A
> successful atomic_inc_not_zero() is fully ordered, so a reader that obtains
> a reference also observes the state published before the refcount became
> nonzero.
> 
> Fixes: 97ae17497e99 ("thp: implement refcounting for huge zero page")
> Cc: stable@vger.kernel.org
> Signed-off-by: Xueyuan Chen <xueyuan.chen@vivo.com>
> ---
>  mm/huge_memory.c | 7 +++++--
>  1 file changed, 5 insertions(+), 2 deletions(-)
> 
> diff --git a/mm/huge_memory.c b/mm/huge_memory.c
> index 032702a4637b..6c74d0375377 100644
> --- a/mm/huge_memory.c
> +++ b/mm/huge_memory.c
> @@ -245,8 +245,11 @@ static bool get_huge_zero_folio(void)
>  	}
>  	WRITE_ONCE(huge_zero_pfn, folio_pfn(zero_folio));
>  
> -	/* We take additional reference here. It will be put back by shrinker */
> -	atomic_set(&huge_zero_refcount, 2);
> +	/*
> +	 * Publish the folio and PFN before making them available to lockless
> +	 * users. Pairs with a successful atomic_inc_not_zero() above.
> +	 */
> +	atomic_set_release(&huge_zero_refcount, 2);
>  	preempt_enable();
>  	count_vm_event(THP_ZERO_PAGE_ALLOC);
>  	return true;

https://lore.kernel.org/r/20260728-fix-refcounted-huge-zero-v1-0-3f261f5447b4@kernel.org


-- 
Cheers,

David

Re: [PATCH] mm/huge_memory: fix huge zero folio publication race

[Andrew Morton]

On Wed, 29 Jul 2026 08:44:12 +0200 "David Hildenbrand (Arm)" <david@kernel.org> wrote:

> > --- a/mm/huge_memory.c
> > +++ b/mm/huge_memory.c
> > @@ -245,8 +245,11 @@ static bool get_huge_zero_folio(void)
> >  	}
> >  	WRITE_ONCE(huge_zero_pfn, folio_pfn(zero_folio));
> >  
> > -	/* We take additional reference here. It will be put back by shrinker */
> > -	atomic_set(&huge_zero_refcount, 2);
> > +	/*
> > +	 * Publish the folio and PFN before making them available to lockless
> > +	 * users. Pairs with a successful atomic_inc_not_zero() above.
> > +	 */
> > +	atomic_set_release(&huge_zero_refcount, 2);
> >  	preempt_enable();
> >  	count_vm_event(THP_ZERO_PAGE_ALLOC);
> >  	return true;
> 
> https://lore.kernel.org/r/20260728-fix-refcounted-huge-zero-v1-0-3f261f5447b4@kernel.org

Does that actually fix the same thing?

Is so, Xueyuan's Fixes: is 97ae17497e99 ("thp: implement refcounting
for huge zero page").  Which to believe?

Re: [PATCH] mm/huge_memory: fix huge zero folio publication race

[Lorenzo Stoakes (ARM)]

On Wed, Jul 29, 2026 at 06:02:41PM -0700, Andrew Morton wrote:
> On Wed, 29 Jul 2026 08:44:12 +0200 "David Hildenbrand (Arm)" <david@kernel.org> wrote:
>
> > > --- a/mm/huge_memory.c
> > > +++ b/mm/huge_memory.c
> > > @@ -245,8 +245,11 @@ static bool get_huge_zero_folio(void)
> > >  	}
> > >  	WRITE_ONCE(huge_zero_pfn, folio_pfn(zero_folio));
> > >
> > > -	/* We take additional reference here. It will be put back by shrinker */
> > > -	atomic_set(&huge_zero_refcount, 2);
> > > +	/*
> > > +	 * Publish the folio and PFN before making them available to lockless
> > > +	 * users. Pairs with a successful atomic_inc_not_zero() above.
> > > +	 */
> > > +	atomic_set_release(&huge_zero_refcount, 2);
> > >  	preempt_enable();
> > >  	count_vm_event(THP_ZERO_PAGE_ALLOC);
> > >  	return true;
> >
> > https://lore.kernel.org/r/20260728-fix-refcounted-huge-zero-v1-0-3f261f5447b4@kernel.org
>
> Does that actually fix the same thing?

???

This patch was sent after mine and tries to change code that doesn't exist?

If you look at [0] you will see:

+	/* Paired with atomic_inc_not_zero(). +1 for shrinker pin. */
+	atomic_set_release(&huge_zero_refcount, 2);

Which is... doing what this does anyway?

(I don't actually think this patch is correct anyway since the cmpxchg() is
fully-ordered with a try loop and the issue with PFN vs folio publication still
exists.)

BUT it doesn't matter anyway.

The race is fixed and I went into great detail in [0] and [1] explaining why and
how. If somebody can poke holes in that then go ahead, otherwise I'm not sure
why David's response didn't suffice here.

>
> Is so, Xueyuan's Fixes: is 97ae17497e99 ("thp: implement refcounting
> for huge zero page").  Which to believe?
>

Again ???

3b77e8c8cde5 is where the PFN actually started to matter, before that the huge
zero page was compared directly.

The patch proposed here says the PFN is the issue so it already had an incorrect
fixes tag.

Cheers, Lorenzo

[0]:https://lore.kernel.org/all/20260728-fix-refcounted-huge-zero-v1-2-3f261f5447b4@kernel.org/
[1]:https://lore.kernel.org/all/ameC4IWCOVBGlXZu@lucifer/