* [PATCH mm-hotfixes v2 1/2] mm/huge_memory: fix huge_zero_pfn race
2026-07-30 10:55 [PATCH mm-hotfixes v2 0/2] mm/huge_memory: fix huge_zero_pfn race Lorenzo Stoakes (ARM)
@ 2026-07-30 10:55 ` Lorenzo Stoakes (ARM)
2026-07-30 14:18 ` David Hildenbrand (Arm)
2026-07-30 10:55 ` [PATCH mm-hotfixes v2 2/2] mm/huge_memory: separate out CONFIG_PERSISTENT_HUGE_ZERO_FOLIO logic Lorenzo Stoakes (ARM)
1 sibling, 1 reply; 4+ messages in thread
From: Lorenzo Stoakes (ARM) @ 2026-07-30 10:55 UTC (permalink / raw)
To: Andrew Morton, David Hildenbrand, Zi Yan, Baolin Wang,
Liam R. Howlett, Nico Pache, Ryan Roberts, Dev Jain, Barry Song,
Lance Yang, Usama Arif, Pankaj Raghav, Hannes Reinecke,
Hugh Dickins, Yang Shi, Kiryl Shutsemau
Cc: linux-mm, linux-kernel, Hengbin Zhang, Lorenzo Stoakes (ARM),
stable
If !CONFIG_PERSISTENT_HUGE_ZERO_FOLIO, the huge_zero_folio is refcounted by
huge_zero_refcount and returned by mm_get_huge_zero_folio().
When the caller is done with the huge zero page, its reference count is
decremented. Only a shrinker can set the reference count to zero.
A race can unfortunately occur between a shrinker decrementing the
reference count to zero and a concurrent page fault.
This is because shrink_huge_zero_folio_scan() might, if very
unlucky, be preempted between setting huge_zero_refcount to zero
and writing an invalid value.
During this time get_huge_zero_folio() could write to huge_zero_pfn
before shrink_huge_zero_folio_scan() resumes.
In this event the huge zero folio will be persistently
misidentified causing the THP code path to be entered
inappropriately for the huge zero folio:
CPU 0 CPU 1
=======================================|=================================
shrink_huge_zero_folio_scan() |
atomic_cmpxchg() sets refcount to 0 |
xchg() sets huge_zero_folio to NULL | get_huge_zero_folio()
| | atomic_inc_not_zero() -> zero
preempted for a long time | Allocate new huge zero folio
| | Write valid huge_zero_folio
v | Write valid huge_zero_pfn
Overwrite huge_zero_pfn with ~0UL <--- Invalid overwrite!
This results in is_huge_zero_pfn() and is_huge_zero_pmd() incorrectly
returning false for a huge zero page which could result in issues like the
huge zero folio being incorrectly split.
Note that the issue is with huge_zero_pfn not huge_zero_folio, as
get_huge_zero_folio() uses cmpxchg() gated on huge_zero_folio being NULL
with a retry loop and shrink_huge_zero_folio_scan() uses xchg() to set
huge_zero_folio.
Fix the issue by introducing a spinlock, huge_zero_lock, to prevent
concurrent write of huge_zero_folio, huge_zero_pfn and huge_zero_refcount.
There needs to be significant care taken here to ensure correctness:
The fast path in get_huge_zero_folio() uses atomic_inc_not_zero(), which is
outside of the critical section, and means huge zero allocation is gated on
zero huge_zero_refcount.
The fast path doesn't use huge_zero_lock, so the critical section is
irrelevant to it.
So invariants are required - huge_zero_refcount MUST:
* Only be set in the huge_zero_lock critical section to ensure
serialisation of huge_zero_pfn, huge_zero_folio and huge_zero_refcount
writes.
* Be set non-zero only AFTER huge_zero_[pfn, folio] are set to valid values
so installation of the huge zero folio on read page fault ensures
concurrent is_huge_zero_*() calls correctly identify the huge zero folio.
* Be set zero only BEFORE huge_zero_[pfn, folio] are set to NULL and ~0UL
respectively, and atomically.
Establish these by:
* Only setting huge_zero_refcount to zero or an absolute value in the
huge_zero_lock critical section in get_huge_zero_folio() and
shrink_huge_zero_folio_scan(), and always updating atomically there
and elsewhere.
* Using atomic_set_release(&huge_zero_refcount) in get_huge_zero_folio()
after huge_zero_[pfn, folio] are set. This is paired with
atomic_inc_not_zero() to ensure atomic_inc_not_zero() only observes a
non-zero value if huge_zero_[pfn, folio] are set.
* Using atomic_cmpxchg() in shrink_huge_zero_folio_scan() (as before) to
ensure that it is set zero only when equal to 1 and set atomically.
* atomic_cmpxchg() being fully ordered ensures this is done prior to
huge_zero_[folio, pfn] being set to NULL and ~0UL respectively.
Eliminate the retry loop in get_huge_zero_folio() as the atomic_cmpxchg()
in shrink_huge_zero_folio_scan() is now performed under the lock, and
replace with an equally locked atomic_inc() to set the reference count
should the caller be raced on huge zero folio installation.
folio_put() naturally implies a full memory barrier so its ordering is
maintained correctly.
The huge zero folio also cannot be released except when the shrinker does
so as it is non-LRU and non-rmappable.
Note that only the huge zero shrinker (via shrink_huge_zero_folio_scan())
can actually set huge_zero_refcount to zero, which is the count of mm's
which have at least one huge zero folio installed plus one shrinker pin.
Additionally convert a BUG_ON() to a VM_WARN_ON_ONCE().
Suggested-by: David Hildenbrand (Arm) <david@kernel.org>
Reported-by: Hengbin Zhang <uqbarz@gmail.com>
Closes: https://lore.kernel.org/linux-mm/20260727154001.4102341-1-uqbarz@gmail.com/
Fixes: 3b77e8c8cde5 ("mm/thp: make is_huge_zero_pmd() safe and quicker")
Cc: stable@vger.kernel.org
Signed-off-by: Lorenzo Stoakes (ARM) <ljs@kernel.org>
---
mm/huge_memory.c | 43 +++++++++++++++++++++++++++++--------------
1 file changed, 29 insertions(+), 14 deletions(-)
diff --git a/mm/huge_memory.c b/mm/huge_memory.c
index 032702a4637b..1f0535721652 100644
--- a/mm/huge_memory.c
+++ b/mm/huge_memory.c
@@ -41,6 +41,7 @@
#include <linux/pgalloc.h>
#include <linux/pgalloc_tag.h>
#include <linux/pagewalk.h>
+#include <linux/cleanup.h>
#include <asm/tlb.h>
#include "internal.h"
@@ -78,6 +79,7 @@ static unsigned long deferred_split_scan(struct shrinker *shrink,
static bool split_underused_thp = true;
static atomic_t huge_zero_refcount;
+static DEFINE_SPINLOCK(huge_zero_lock);
struct folio *huge_zero_folio __read_mostly;
unsigned long huge_zero_pfn __read_mostly = ~0UL;
unsigned long huge_anon_orders_always __read_mostly;
@@ -224,7 +226,8 @@ unsigned long __thp_vma_allowable_orders(struct vm_area_struct *vma,
static bool get_huge_zero_folio(void)
{
struct folio *zero_folio;
-retry:
+
+ /* Paired with atomic_set_release(). */
if (likely(atomic_inc_not_zero(&huge_zero_refcount)))
return true;
@@ -237,17 +240,22 @@ static bool get_huge_zero_folio(void)
}
/* Ensure zero folio won't have large_rmappable flag set. */
folio_clear_large_rmappable(zero_folio);
- preempt_disable();
- if (cmpxchg(&huge_zero_folio, NULL, zero_folio)) {
- preempt_enable();
+
+ /* Paired with critical section in shrink_huge_zero_folio_scan(). */
+ spin_lock(&huge_zero_lock);
+ if (huge_zero_folio) {
+ /* Somebody else already installed it. */
+ atomic_inc(&huge_zero_refcount);
+ spin_unlock(&huge_zero_lock);
folio_put(zero_folio);
- goto retry;
+ return true;
}
+ WRITE_ONCE(huge_zero_folio, zero_folio);
WRITE_ONCE(huge_zero_pfn, folio_pfn(zero_folio));
+ /* Paired with atomic_inc_not_zero(). +1 for shrinker pin. */
+ atomic_set_release(&huge_zero_refcount, 2);
+ spin_unlock(&huge_zero_lock);
- /* We take additional reference here. It will be put back by shrinker */
- atomic_set(&huge_zero_refcount, 2);
- preempt_enable();
count_vm_event(THP_ZERO_PAGE_ALLOC);
return true;
}
@@ -297,15 +305,22 @@ static unsigned long shrink_huge_zero_folio_count(struct shrinker *shrink,
static unsigned long shrink_huge_zero_folio_scan(struct shrinker *shrink,
struct shrink_control *sc)
{
- if (atomic_cmpxchg(&huge_zero_refcount, 1, 0) == 1) {
- struct folio *zero_folio = xchg(&huge_zero_folio, NULL);
- BUG_ON(zero_folio == NULL);
+ struct folio *zero_folio;
+
+ /* Paired with critical section in get_huge_zero_folio(). */
+ scoped_guard(spinlock, &huge_zero_lock) {
+ /* Paired with atomic_inc_not_zero() in get_huge_zero_folio(). */
+ if (atomic_cmpxchg(&huge_zero_refcount, 1, 0) != 1)
+ return 0;
+
+ zero_folio = huge_zero_folio;
+ VM_WARN_ON_ONCE(!zero_folio);
+ WRITE_ONCE(huge_zero_folio, NULL);
WRITE_ONCE(huge_zero_pfn, ~0UL);
- folio_put(zero_folio);
- return HPAGE_PMD_NR;
}
- return 0;
+ folio_put(zero_folio);
+ return HPAGE_PMD_NR;
}
static struct shrinker *huge_zero_folio_shrinker;
--
2.55.0
^ permalink raw reply related [flat|nested] 4+ messages in thread* [PATCH mm-hotfixes v2 2/2] mm/huge_memory: separate out CONFIG_PERSISTENT_HUGE_ZERO_FOLIO logic
2026-07-30 10:55 [PATCH mm-hotfixes v2 0/2] mm/huge_memory: fix huge_zero_pfn race Lorenzo Stoakes (ARM)
2026-07-30 10:55 ` [PATCH mm-hotfixes v2 1/2] " Lorenzo Stoakes (ARM)
@ 2026-07-30 10:55 ` Lorenzo Stoakes (ARM)
1 sibling, 0 replies; 4+ messages in thread
From: Lorenzo Stoakes (ARM) @ 2026-07-30 10:55 UTC (permalink / raw)
To: Andrew Morton, David Hildenbrand, Zi Yan, Baolin Wang,
Liam R. Howlett, Nico Pache, Ryan Roberts, Dev Jain, Barry Song,
Lance Yang, Usama Arif, Pankaj Raghav, Hannes Reinecke,
Hugh Dickins, Yang Shi, Kiryl Shutsemau
Cc: linux-mm, linux-kernel, Hengbin Zhang, Lorenzo Stoakes (ARM)
Rather than mixing the refcounted and non-refcounted
CONFIG_PERSISTENT_HUGE_ZERO_FOLIO logic, separate the two out cleanly
so it is clear what happens when this configuration option is set and what
happens when it is not.
Introduce HUGE_ZERO_UNSET_PFN to abstract the ~0UL assignment, only
introduce the refcount, lock and shrinker if
!CONFIG_PERSISTENT_HUGE_ZERO_FOLIO, abstract initialisation and teardown,
abstract the huge zero folio allocation from refcounting.
Also change a BUG_ON() to WARN_ON_ONCE() while we're at it.
No functional change intended.
Signed-off-by: Lorenzo Stoakes (ARM) <ljs@kernel.org>
---
mm/huge_memory.c | 160 ++++++++++++++++++++++++++++++++-----------------------
1 file changed, 94 insertions(+), 66 deletions(-)
diff --git a/mm/huge_memory.c b/mm/huge_memory.c
index 1f0535721652..4d2c891b1968 100644
--- a/mm/huge_memory.c
+++ b/mm/huge_memory.c
@@ -78,10 +78,15 @@ static unsigned long deferred_split_scan(struct shrinker *shrink,
struct shrink_control *sc);
static bool split_underused_thp = true;
+#define HUGE_ZERO_UNSET_PFN (~0UL)
+struct folio *huge_zero_folio __read_mostly;
+unsigned long huge_zero_pfn __read_mostly = HUGE_ZERO_UNSET_PFN;
+#ifndef CONFIG_PERSISTENT_HUGE_ZERO_FOLIO
static atomic_t huge_zero_refcount;
static DEFINE_SPINLOCK(huge_zero_lock);
-struct folio *huge_zero_folio __read_mostly;
-unsigned long huge_zero_pfn __read_mostly = ~0UL;
+static struct shrinker *huge_zero_folio_shrinker;
+#endif
+
unsigned long huge_anon_orders_always __read_mostly;
unsigned long huge_anon_orders_madvise __read_mostly;
unsigned long huge_anon_orders_inherit __read_mostly;
@@ -223,23 +228,58 @@ unsigned long __thp_vma_allowable_orders(struct vm_area_struct *vma,
return orders;
}
-static bool get_huge_zero_folio(void)
+static struct folio *alloc_huge_zero_folio(void)
{
struct folio *zero_folio;
- /* Paired with atomic_set_release(). */
- if (likely(atomic_inc_not_zero(&huge_zero_refcount)))
- return true;
-
zero_folio = folio_alloc((GFP_TRANSHUGE | __GFP_ZERO | __GFP_ZEROTAGS) &
~__GFP_MOVABLE,
HPAGE_PMD_ORDER);
if (!zero_folio) {
count_vm_event(THP_ZERO_PAGE_ALLOC_FAILED);
- return false;
+ return NULL;
+ }
+ folio_clear_large_rmappable(zero_folio); /* Explicitly not rmappable. */
+ return zero_folio;
+}
+
+#ifdef CONFIG_PERSISTENT_HUGE_ZERO_FOLIO
+static int __init huge_zero_init(void)
+{
+ huge_zero_folio = alloc_huge_zero_folio();
+ if (!huge_zero_folio) {
+ pr_warn("Allocating persistent huge zero folio failed\n");
+ } else {
+ huge_zero_pfn = folio_pfn(huge_zero_folio);
+ count_vm_event(THP_ZERO_PAGE_ALLOC);
}
- /* Ensure zero folio won't have large_rmappable flag set. */
- folio_clear_large_rmappable(zero_folio);
+ return 0;
+}
+
+static void __init huge_zero_shrinker_exit(void)
+{
+}
+
+struct folio *mm_get_huge_zero_folio(struct mm_struct *mm)
+{
+ return huge_zero_folio;
+}
+
+void mm_put_huge_zero_folio(struct mm_struct *mm)
+{
+}
+#else
+static bool get_huge_zero_folio(void)
+{
+ struct folio *zero_folio;
+
+ /* Paired with atomic_set_release(). */
+ if (likely(atomic_inc_not_zero(&huge_zero_refcount)))
+ return true;
+
+ zero_folio = alloc_huge_zero_folio();
+ if (unlikely(!zero_folio))
+ return false;
/* Paired with critical section in shrink_huge_zero_folio_scan(). */
spin_lock(&huge_zero_lock);
@@ -266,33 +306,7 @@ static void put_huge_zero_folio(void)
* Counter should never go to zero here. Only shrinker can put
* last reference.
*/
- BUG_ON(atomic_dec_and_test(&huge_zero_refcount));
-}
-
-struct folio *mm_get_huge_zero_folio(struct mm_struct *mm)
-{
- if (IS_ENABLED(CONFIG_PERSISTENT_HUGE_ZERO_FOLIO))
- return huge_zero_folio;
-
- if (mm_flags_test(MMF_HUGE_ZERO_FOLIO, mm))
- return READ_ONCE(huge_zero_folio);
-
- if (!get_huge_zero_folio())
- return NULL;
-
- if (mm_flags_test_and_set(MMF_HUGE_ZERO_FOLIO, mm))
- put_huge_zero_folio();
-
- return READ_ONCE(huge_zero_folio);
-}
-
-void mm_put_huge_zero_folio(struct mm_struct *mm)
-{
- if (IS_ENABLED(CONFIG_PERSISTENT_HUGE_ZERO_FOLIO))
- return;
-
- if (mm_flags_test(MMF_HUGE_ZERO_FOLIO, mm))
- put_huge_zero_folio();
+ WARN_ON_ONCE(atomic_dec_and_test(&huge_zero_refcount));
}
static unsigned long shrink_huge_zero_folio_count(struct shrinker *shrink,
@@ -316,14 +330,53 @@ static unsigned long shrink_huge_zero_folio_scan(struct shrinker *shrink,
zero_folio = huge_zero_folio;
VM_WARN_ON_ONCE(!zero_folio);
WRITE_ONCE(huge_zero_folio, NULL);
- WRITE_ONCE(huge_zero_pfn, ~0UL);
+ WRITE_ONCE(huge_zero_pfn, HUGE_ZERO_UNSET_PFN);
}
folio_put(zero_folio);
return HPAGE_PMD_NR;
}
-static struct shrinker *huge_zero_folio_shrinker;
+static int __init huge_zero_init(void)
+{
+ huge_zero_folio_shrinker = shrinker_alloc(0, "thp-zero");
+ if (!huge_zero_folio_shrinker) {
+ shrinker_free(deferred_split_shrinker);
+ list_lru_destroy(&deferred_split_lru);
+ return -ENOMEM;
+ }
+
+ huge_zero_folio_shrinker->count_objects = shrink_huge_zero_folio_count;
+ huge_zero_folio_shrinker->scan_objects = shrink_huge_zero_folio_scan;
+ shrinker_register(huge_zero_folio_shrinker);
+ return 0;
+}
+
+static void __init huge_zero_shrinker_exit(void)
+{
+ shrinker_free(huge_zero_folio_shrinker);
+}
+
+struct folio *mm_get_huge_zero_folio(struct mm_struct *mm)
+{
+ if (mm_flags_test(MMF_HUGE_ZERO_FOLIO, mm))
+ return READ_ONCE(huge_zero_folio);
+
+ if (!get_huge_zero_folio())
+ return NULL;
+
+ if (mm_flags_test_and_set(MMF_HUGE_ZERO_FOLIO, mm))
+ put_huge_zero_folio();
+
+ return READ_ONCE(huge_zero_folio);
+}
+
+void mm_put_huge_zero_folio(struct mm_struct *mm)
+{
+ if (mm_flags_test(MMF_HUGE_ZERO_FOLIO, mm))
+ put_huge_zero_folio();
+}
+#endif /* CONFIG_PERSISTENT_HUGE_ZERO_FOLIO */
#ifdef CONFIG_SYSFS
static ssize_t enabled_show(struct kobject *kobj,
@@ -987,39 +1040,14 @@ static int __init thp_shrinker_init(void)
deferred_split_shrinker->scan_objects = deferred_split_scan;
shrinker_register(deferred_split_shrinker);
- if (IS_ENABLED(CONFIG_PERSISTENT_HUGE_ZERO_FOLIO)) {
- /*
- * Bump the reference of the huge_zero_folio and do not
- * initialize the shrinker.
- *
- * huge_zero_folio will always be NULL on failure. We assume
- * that get_huge_zero_folio() will most likely not fail as
- * thp_shrinker_init() is invoked early on during boot.
- */
- if (!get_huge_zero_folio())
- pr_warn("Allocating persistent huge zero folio failed\n");
- return 0;
- }
-
- huge_zero_folio_shrinker = shrinker_alloc(0, "thp-zero");
- if (!huge_zero_folio_shrinker) {
- shrinker_free(deferred_split_shrinker);
- list_lru_destroy(&deferred_split_lru);
- return -ENOMEM;
- }
-
- huge_zero_folio_shrinker->count_objects = shrink_huge_zero_folio_count;
- huge_zero_folio_shrinker->scan_objects = shrink_huge_zero_folio_scan;
- shrinker_register(huge_zero_folio_shrinker);
-
- return 0;
+ return huge_zero_init();
}
static void __init thp_shrinker_exit(void)
{
- shrinker_free(huge_zero_folio_shrinker);
shrinker_free(deferred_split_shrinker);
list_lru_destroy(&deferred_split_lru);
+ huge_zero_shrinker_exit();
}
static int __init hugepage_init(void)
--
2.55.0
^ permalink raw reply related [flat|nested] 4+ messages in thread