[PATCH for 7.4-rc2+ 0/7] Re-enable discarding lazyfree THPs at PMD level

[PATCH for 7.4-rc2+ 0/7] Re-enable discarding lazyfree THPs at PMD level

[Zi Yan]

For maintainers, Patch 6 requires first 5 patches to work correctly, so it
might be better to get ack from arch maintainers for first 5 patches and
merge the series via mm tree. The patch disabling lazyfree THP discard at
PMD level has been picked up [2] and would take some time to test, so this
patchset probably can be considered after the merge window closes. I am
sending it out for early feedback.

Motivation
===

A bug was found in the PMD-level discard of lazyfree THPs [1]. The
proposed fix keeps the PMD invalidated with pmdp_invalidate() instead of
clearing it, but that does not work on every architecture:

1. riscv, LoongArch and s390 let GUP-fast follow an invalidated PMD;
2. powerpc/book3s64 page table check counts an invalidated PMD as mapped;
3. sparc64's thp_pte_count becomes unbalanced around pmdp_invalidate().

So the PMD-level discard was disabled instead [2]. This series fixes the
arch code and re-enables it.

The discard decides from the folio refcount, so GUP-fast must not take a
reference through an invalidated PMD, and arch accounting must stay
balanced when an invalidated PMD is set again or cleared.

Details
===

Patches 1-5 fix the arch code, patch 6 from Kyle keeps the PMD invalidated
instead of cleared until the discard is committed, and patch 7 enables the
PMD-level discard again:

1. powerpc/book3s64: page table check counts an invalidated PMD as a live
   mapping, so setting it again or clearing it decrements the map count a
   second time and hits BUG_ON(). This is already reachable via MADV_COLD
   and mprotect() with CONFIG_PAGE_TABLE_CHECK.

2. riscv: an invalidated PMD still passes pmd_present() and the generic
   pmd_access_permitted(), so GUP-fast follows it. Add a
   pmd_access_permitted() that requires _PAGE_PRESENT.

3. LoongArch: same problem and fix as riscv.

4. s390: _SEGMENT_ENTRY_INVALID also marks an old PMD, so GUP-fast follows
   an invalidated PMD. Always clear _SEGMENT_ENTRY_READ in
   pmdp_invalidate(), which makes it pmd_protnone(), so GUP-fast rejects
   it.

5. sparc64: pmdp_invalidate() decrements thp_pte_count assuming a THP
   split follows, so setting the PMD again loses a count. Count only
   valid huge PMDs in __set_pmd_acct() instead, which keeps invalidate,
   restore, clear and split balanced.

6. mm/huge_memory: use pmdp_invalidate() in the discard, so munmap() and
   mremap() wait on the PMD lock instead of skipping a none PMD.

7. mm/rmap: discard lazyfree THPs at PMD level again.

Note
===

x86, arm64 and powerpc already make GUP-fast reject an invalidated PMD,
and sparc64 has no GUP-fast. 32-bit MIPS would not reject it, but it
cannot have THP: _PAGE_HUGE makes PFN_PTE_SHIFT exceed PAGE_SHIFT, so the
BUILD_BUG_ON() in arch_mm_preinit() fires, which is why Ingenic dropped
huge pages in commit 69b686ddd579 ("MIPS: ingenic: Remove
CPU_SUPPORTS_HUGEPAGES").

Link: https://lore.kernel.org/all/20261009165214.40212-2-kylebot@openai.com/ [1]
Link: https://lore.kernel.org/all/20261012-lazyfree-stable-fix-v2-1-6387d6e30063@nvidia.com/ [2]

Signed-off-by: Zi Yan <ziy@nvidia.com>
---
Kyle Zeng (1):
      mm/huge_memory: avoid transient none PMDs during lazyfree reclaim

Zi Yan (6):
      powerpc/book3s64/mm: don't count invalidated PMDs in page table check
      riscv: mm: don't let GUP-fast follow invalidated THP PMDs
      LoongArch: mm: don't let GUP-fast follow invalidated THP PMDs
      s390/mm: always clear _SEGMENT_ENTRY_READ in pmdp_invalidate()
      sparc64: mm: fix thp_pte_count accounting around pmdp_invalidate()
      mm/rmap: re-enable discarding lazyfree THPs at PMD level

 arch/loongarch/include/asm/pgtable.h         |  6 ++++++
 arch/powerpc/include/asm/book3s/64/pgtable.h |  3 ++-
 arch/riscv/include/asm/pgtable.h             |  6 ++++++
 arch/s390/include/asm/pgtable.h              |  8 ++++++--
 arch/sparc/mm/tlb.c                          | 26 ++++++++++++--------------
 mm/huge_memory.c                             |  5 +++--
 mm/rmap.c                                    | 11 +++++++++++
 7 files changed, 46 insertions(+), 19 deletions(-)
---
base-commit: 71a9b024444eba184335b67867897714e4da143c
change-id: 20261007-anon_vma_uaf_fix-e7accd1d2ee1

Best regards,
--  
Yan, Zi

[PATCH for 7.4-rc2+ 0/7] Re-enable discarding lazyfree THPs at PMD level

[Zi Yan]

For maintainers, Patch 6 requires first 5 patches to work correctly, so it
might be better to get ack from arch maintainers for first 5 patches and
merge the series via mm tree. The patch disabling lazyfree THP discard at
PMD level has been picked up [2] and would take some time to test, so this
patchset probably can be considered after the merge window closes. I am
sending it out for early feedback.

Motivation
===

A bug was found in the PMD-level discard of lazyfree THPs [1]. The
proposed fix keeps the PMD invalidated with pmdp_invalidate() instead of
clearing it, but that does not work on every architecture:

1. riscv, LoongArch and s390 let GUP-fast follow an invalidated PMD;
2. powerpc/book3s64 page table check counts an invalidated PMD as mapped;
3. sparc64's thp_pte_count becomes unbalanced around pmdp_invalidate().

So the PMD-level discard was disabled instead [2]. This series fixes the
arch code and re-enables it.

The discard decides from the folio refcount, so GUP-fast must not take a
reference through an invalidated PMD, and arch accounting must stay
balanced when an invalidated PMD is set again or cleared.

Details
===

Patches 1-5 fix the arch code, patch 6 from Kyle keeps the PMD invalidated
instead of cleared until the discard is committed, and patch 7 enables the
PMD-level discard again:

1. powerpc/book3s64: page table check counts an invalidated PMD as a live
   mapping, so setting it again or clearing it decrements the map count a
   second time and hits BUG_ON(). This is already reachable via MADV_COLD
   and mprotect() with CONFIG_PAGE_TABLE_CHECK.

2. riscv: an invalidated PMD still passes pmd_present() and the generic
   pmd_access_permitted(), so GUP-fast follows it. Add a
   pmd_access_permitted() that requires _PAGE_PRESENT.

3. LoongArch: same problem and fix as riscv.

4. s390: _SEGMENT_ENTRY_INVALID also marks an old PMD, so GUP-fast follows
   an invalidated PMD. Always clear _SEGMENT_ENTRY_READ in
   pmdp_invalidate(), which makes it pmd_protnone(), so GUP-fast rejects
   it.

5. sparc64: pmdp_invalidate() decrements thp_pte_count assuming a THP
   split follows, so setting the PMD again loses a count. Count only
   valid huge PMDs in __set_pmd_acct() instead, which keeps invalidate,
   restore, clear and split balanced.

6. mm/huge_memory: use pmdp_invalidate() in the discard, so munmap() and
   mremap() wait on the PMD lock instead of skipping a none PMD.

7. mm/rmap: discard lazyfree THPs at PMD level again.

Note
===

x86, arm64 and powerpc already make GUP-fast reject an invalidated PMD,
and sparc64 has no GUP-fast. 32-bit MIPS would not reject it, but it
cannot have THP: _PAGE_HUGE makes PFN_PTE_SHIFT exceed PAGE_SHIFT, so the
BUILD_BUG_ON() in arch_mm_preinit() fires, which is why Ingenic dropped
huge pages in commit 69b686ddd579 ("MIPS: ingenic: Remove
CPU_SUPPORTS_HUGEPAGES").

Link: https://lore.kernel.org/all/20261009165214.40212-2-kylebot@openai.com/ [1]
Link: https://lore.kernel.org/all/20261012-lazyfree-stable-fix-v2-1-6387d6e30063@nvidia.com/ [2]

Signed-off-by: Zi Yan <ziy@nvidia.com>
---
Kyle Zeng (1):
      mm/huge_memory: avoid transient none PMDs during lazyfree reclaim

Zi Yan (6):
      powerpc/book3s64/mm: don't count invalidated PMDs in page table check
      riscv: mm: don't let GUP-fast follow invalidated THP PMDs
      LoongArch: mm: don't let GUP-fast follow invalidated THP PMDs
      s390/mm: always clear _SEGMENT_ENTRY_READ in pmdp_invalidate()
      sparc64: mm: fix thp_pte_count accounting around pmdp_invalidate()
      mm/rmap: re-enable discarding lazyfree THPs at PMD level

 arch/loongarch/include/asm/pgtable.h         |  6 ++++++
 arch/powerpc/include/asm/book3s/64/pgtable.h |  3 ++-
 arch/riscv/include/asm/pgtable.h             |  6 ++++++
 arch/s390/include/asm/pgtable.h              |  8 ++++++--
 arch/sparc/mm/tlb.c                          | 26 ++++++++++++--------------
 mm/huge_memory.c                             |  5 +++--
 mm/rmap.c                                    | 11 +++++++++++
 7 files changed, 46 insertions(+), 19 deletions(-)
---
base-commit: 71a9b024444eba184335b67867897714e4da143c
change-id: 20261007-anon_vma_uaf_fix-e7accd1d2ee1

Best regards,
--  
Yan, Zi


_______________________________________________
linux-riscv mailing list
linux-riscv@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-riscv

[PATCH for 7.4-rc2+ 1/7] powerpc/book3s64/mm: don't count invalidated PMDs in page table check

[Zi Yan]

pmdp_invalidate() replaces a huge PMD with an invalidated one
(_PAGE_PRESENT cleared and _PAGE_INVALID set) and calls
page_table_check_pmd_clear() on the old entry. This decreases the page
table check map count. When the invalidated entry is later replaced with
set_pmd_at(), as MADV_COLD, page_mkclean_one() and change_huge_pmd() do,
page_table_check_pmds_set() in set_pmd_at() clears it a second time and
decreases the map count because pmd_user_accessible_page() still counts it
as a live user mapping. The same happens when it is cleared with
pmdp_huge_get_and_clear(). Then the map count underflows and hits BUG_ON()
in page_table_check_clear().

Found by code inspection and confirmed in QEMU (pseries, POWER9, 64K
pages, CONFIG_PAGE_TABLE_CHECK_ENFORCED=y): MADV_COLD or mprotect() on
an anonymous THP hits the BUG_ON() with both the radix and the hash MMU.

Fix it by only counting entries that are hardware valid, so an
invalidated entry is skipped, as the patch "riscv: mm: exclude invalid
THP PMDs from page table check" did for riscv.

Fixes: 641d47d4c963 ("powerpc/mm: support page table check")
Cc: stable@vger.kernel.org
Assisted-by: LLM
Signed-off-by: Zi Yan <ziy@nvidia.com>
To: Madhavan Srinivasan <maddy@linux.ibm.com>
To: Andrew Donnellan <andrew+kernel@donnellan.id.au>
To: Rohan McLure <rmclure@linux.ibm.com>
To: Pasha Tatashin <pasha.tatashin@soleen.com>
To: Andrew Morton <akpm@linux-foundation.org>
Cc: Michael Ellerman <mpe@ellerman.id.au>
Cc: Nicholas Piggin <npiggin@gmail.com>
Cc: "Christophe Leroy (CS GROUP)" <chleroy@kernel.org>
Cc: "Ritesh Harjani (IBM)" <ritesh.list@gmail.com>
Cc: Shrikanth Hegde <sshegde@linux.ibm.com>
Cc: linuxppc-dev@lists.ozlabs.org
Cc: linux-kernel@vger.kernel.org
---
 arch/powerpc/include/asm/book3s/64/pgtable.h | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/arch/powerpc/include/asm/book3s/64/pgtable.h b/arch/powerpc/include/asm/book3s/64/pgtable.h
index dff8790a047db..392474972777c 100644
--- a/arch/powerpc/include/asm/book3s/64/pgtable.h
+++ b/arch/powerpc/include/asm/book3s/64/pgtable.h
@@ -1091,7 +1091,8 @@ static inline bool pmd_access_permitted(pmd_t pmd, bool write)
 #define pmd_user_accessible_page pmd_user_accessible_page
 static inline bool pmd_user_accessible_page(struct mm_struct *mm, unsigned long addr, pmd_t pmd)
 {
-	return pmd_leaf(pmd) && pte_user_accessible_page(mm, addr, pmd_pte(pmd));
+	return pmd_leaf(pmd) && pte_hw_valid(pmd_pte(pmd)) &&
+	       pte_user_accessible_page(mm, addr, pmd_pte(pmd));
 }
 
 #ifdef CONFIG_TRANSPARENT_HUGEPAGE

-- 
2.53.0

[PATCH for 7.4-rc2+ 2/7] riscv: mm: don't let GUP-fast follow invalidated THP PMDs

[Zi Yan]

pmdp_invalidate() leaves a PMD whose pmd_present() is still true, so the
generic pmd_access_permitted() returns true for an invalidated PMD. Then
GUP-fast can grab a reference on the folio and, as the PMD value doesn't
change until the caller finishes, the recheck after taking the reference
passes too.

Code that invalidates a PMD and then decides, from the folio refcount,
whether it may drop the mapping relies on GUP-fast backing off from an
invalidated PMD, like it does from a cleared one. x86, arm64 and powerpc
already reject invalidated PMDs in pmd_access_permitted().

Provide a riscv pmd_access_permitted() that requires _PAGE_PRESENT, so
invalidated (and PROT_NONE) PMDs are not followed by GUP-fast.

Assisted-by: LLM
Signed-off-by: Zi Yan <ziy@nvidia.com>
To: Paul Walmsley <pjw@kernel.org>
To: Palmer Dabbelt <palmer@dabbelt.com>
To: Albert Ou <aou@eecs.berkeley.edu>
Cc: Alexandre Ghiti <alex@ghiti.fr>
Cc: Jason Gunthorpe <jgg@nvidia.com>
Cc: John Hubbard <jhubbard@nvidia.com>
Cc: Peter Xu <peterx@redhat.com>
Cc: linux-riscv@lists.infradead.org
Cc: linux-kernel@vger.kernel.org
---
 arch/riscv/include/asm/pgtable.h | 6 ++++++
 1 file changed, 6 insertions(+)

diff --git a/arch/riscv/include/asm/pgtable.h b/arch/riscv/include/asm/pgtable.h
index d48f90140841e..66aac56631f0c 100644
--- a/arch/riscv/include/asm/pgtable.h
+++ b/arch/riscv/include/asm/pgtable.h
@@ -816,6 +816,12 @@ static inline int pmd_write(pmd_t pmd)
 	return pte_write(pmd_pte(pmd));
 }
 
+#define pmd_access_permitted pmd_access_permitted
+static inline bool pmd_access_permitted(pmd_t pmd, bool write)
+{
+	return (pmd_val(pmd) & _PAGE_PRESENT) && (!write || pmd_write(pmd));
+}
+
 #define pud_write pud_write
 static inline int pud_write(pud_t pud)
 {

-- 
2.53.0

[PATCH for 7.4-rc2+ 2/7] riscv: mm: don't let GUP-fast follow invalidated THP PMDs

[Zi Yan]

pmdp_invalidate() leaves a PMD whose pmd_present() is still true, so the
generic pmd_access_permitted() returns true for an invalidated PMD. Then
GUP-fast can grab a reference on the folio and, as the PMD value doesn't
change until the caller finishes, the recheck after taking the reference
passes too.

Code that invalidates a PMD and then decides, from the folio refcount,
whether it may drop the mapping relies on GUP-fast backing off from an
invalidated PMD, like it does from a cleared one. x86, arm64 and powerpc
already reject invalidated PMDs in pmd_access_permitted().

Provide a riscv pmd_access_permitted() that requires _PAGE_PRESENT, so
invalidated (and PROT_NONE) PMDs are not followed by GUP-fast.

Assisted-by: LLM
Signed-off-by: Zi Yan <ziy@nvidia.com>
To: Paul Walmsley <pjw@kernel.org>
To: Palmer Dabbelt <palmer@dabbelt.com>
To: Albert Ou <aou@eecs.berkeley.edu>
Cc: Alexandre Ghiti <alex@ghiti.fr>
Cc: Jason Gunthorpe <jgg@nvidia.com>
Cc: John Hubbard <jhubbard@nvidia.com>
Cc: Peter Xu <peterx@redhat.com>
Cc: linux-riscv@lists.infradead.org
Cc: linux-kernel@vger.kernel.org
---
 arch/riscv/include/asm/pgtable.h | 6 ++++++
 1 file changed, 6 insertions(+)

diff --git a/arch/riscv/include/asm/pgtable.h b/arch/riscv/include/asm/pgtable.h
index d48f90140841e..66aac56631f0c 100644
--- a/arch/riscv/include/asm/pgtable.h
+++ b/arch/riscv/include/asm/pgtable.h
@@ -816,6 +816,12 @@ static inline int pmd_write(pmd_t pmd)
 	return pte_write(pmd_pte(pmd));
 }
 
+#define pmd_access_permitted pmd_access_permitted
+static inline bool pmd_access_permitted(pmd_t pmd, bool write)
+{
+	return (pmd_val(pmd) & _PAGE_PRESENT) && (!write || pmd_write(pmd));
+}
+
 #define pud_write pud_write
 static inline int pud_write(pud_t pud)
 {

-- 
2.53.0


_______________________________________________
linux-riscv mailing list
linux-riscv@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-riscv

[PATCH for 7.4-rc2+ 3/7] LoongArch: mm: don't let GUP-fast follow invalidated THP PMDs

[Zi Yan]

pmdp_invalidate() leaves a PMD whose pmd_present() is still true, so the
generic pmd_access_permitted() returns true for an invalidated PMD. Then
GUP-fast can grab a reference on the folio and, as the PMD value doesn't
change until the caller finishes, the recheck after taking the reference
passes too.

Code that invalidates a PMD and then decides, from the folio refcount,
whether it may drop the mapping relies on GUP-fast backing off from an
invalidated PMD, like it does from a cleared one. x86, arm64 and powerpc
already reject invalidated PMDs in pmd_access_permitted().

Provide a LoongArch pmd_access_permitted() that requires _PAGE_PRESENT,
so invalidated (and PROT_NONE) PMDs are not followed by GUP-fast.

Assisted-by: LLM
Signed-off-by: Zi Yan <ziy@nvidia.com>
To: Huacai Chen <chenhuacai@kernel.org>
Cc: WANG Xuerui <kernel@xen0n.name>
Cc: Jason Gunthorpe <jgg@nvidia.com>
Cc: John Hubbard <jhubbard@nvidia.com>
Cc: Peter Xu <peterx@redhat.com>
Cc: loongarch@lists.linux.dev
Cc: linux-kernel@vger.kernel.org
---
 arch/loongarch/include/asm/pgtable.h | 6 ++++++
 1 file changed, 6 insertions(+)

diff --git a/arch/loongarch/include/asm/pgtable.h b/arch/loongarch/include/asm/pgtable.h
index f876031351319..bcba7bce71d5d 100644
--- a/arch/loongarch/include/asm/pgtable.h
+++ b/arch/loongarch/include/asm/pgtable.h
@@ -522,6 +522,12 @@ static inline int pmd_write(pmd_t pmd)
 	return !!(pmd_val(pmd) & _PAGE_WRITE);
 }
 
+#define pmd_access_permitted pmd_access_permitted
+static inline bool pmd_access_permitted(pmd_t pmd, bool write)
+{
+	return (pmd_val(pmd) & _PAGE_PRESENT) && (!write || pmd_write(pmd));
+}
+
 static inline pmd_t pmd_mkwrite_novma(pmd_t pmd)
 {
 	pmd_val(pmd) |= _PAGE_WRITE;

-- 
2.53.0

[PATCH for 7.4-rc2+ 4/7] s390/mm: always clear _SEGMENT_ENTRY_READ in pmdp_invalidate()

[Zi Yan]

pmdp_invalidate() sets _SEGMENT_ENTRY_INVALID, but on s390 that bit
alone also marks an old (not young) PMD, which is still a valid mapping
that GUP-fast follows. Without CONFIG_PAGE_TABLE_CHECK, an invalidated
PMD is therefore not pmd_protnone(), so GUP-fast can grab a reference on
the folio and, as the PMD value doesn't change until the caller
finishes, the recheck after taking the reference passes too.

Code that invalidates a PMD and then decides, from the folio refcount,
whether it may drop the mapping relies on GUP-fast backing off from an
invalidated PMD, like it does from a cleared one. x86, arm64 and powerpc
already make GUP-fast reject invalidated PMDs.

With CONFIG_PAGE_TABLE_CHECK, pmdp_invalidate() already clears
_SEGMENT_ENTRY_READ, which makes the entry pmd_protnone(), so GUP-fast
does not follow it. Do that unconditionally. As noted in commit
7b4dde5e40ad6 ("s390/pgtable: Add s390 support for page table check"),
an invalidated PMD that is pmd_protnone() is fine for the remaining
pmd_protnone() users, which run under the page table lock or are
lockless walkers that should not follow it anyway.

Assisted-by: LLM
Signed-off-by: Zi Yan <ziy@nvidia.com>
To: Alexander Gordeev <agordeev@linux.ibm.com>
To: Gerald Schaefer <gerald.schaefer@linux.ibm.com>
To: Heiko Carstens <hca@linux.ibm.com>
To: Vasily Gorbik <gor@linux.ibm.com>
Cc: Christian Borntraeger <borntraeger@linux.ibm.com>
Cc: Sven Schnelle <svens@linux.ibm.com>
Cc: Jason Gunthorpe <jgg@nvidia.com>
Cc: John Hubbard <jhubbard@nvidia.com>
Cc: Peter Xu <peterx@redhat.com>
Cc: linux-s390@vger.kernel.org
Cc: linux-kernel@vger.kernel.org
---
 arch/s390/include/asm/pgtable.h | 8 ++++++--
 1 file changed, 6 insertions(+), 2 deletions(-)

diff --git a/arch/s390/include/asm/pgtable.h b/arch/s390/include/asm/pgtable.h
index 2d5c2ab06de98..fe1ba746203ab 100644
--- a/arch/s390/include/asm/pgtable.h
+++ b/arch/s390/include/asm/pgtable.h
@@ -1781,10 +1781,14 @@ static inline pmd_t pmdp_invalidate(struct vm_area_struct *vma,
 	pmd_t pmd = *pmdp;
 
 	VM_WARN_ON_ONCE(!pmd_present(pmd));
+	/*
+	 * _SEGMENT_ENTRY_INVALID alone also marks an old (not young) entry,
+	 * which lockless walkers like GUP-fast may follow. Clear
+	 * _SEGMENT_ENTRY_READ too, so the invalidated entry is pmd_protnone()
+	 * and is not followed, and page table check does not count it.
+	 */
 	pmd = set_pmd_bit(pmd, __pgprot(_SEGMENT_ENTRY_INVALID));
-#ifdef CONFIG_PAGE_TABLE_CHECK
 	pmd = clear_pmd_bit(pmd, __pgprot(_SEGMENT_ENTRY_READ));
-#endif
 	page_table_check_pmd_set(vma->vm_mm, addr, pmdp, pmd);
 	pmd = pmdp_xchg_direct(vma->vm_mm, addr, pmdp, pmd);
 	return pmd;

-- 
2.53.0

[PATCH for 7.4-rc2+ 5/7] sparc64: mm: fix thp_pte_count accounting around pmdp_invalidate()

[Zi Yan]

thp_pte_count is maintained in __set_pmd_acct() whenever _PAGE_PMD_HUGE
flips. pmdp_invalidate() only clears _PAGE_VALID, so it does not flip
that bit, but it also decrements thp_pte_count on its own, because THP
split replaces the invalidated PMD with a page table via pmd_populate(),
which does no accounting.

That is only right if the invalidated PMD is never set back. Several
callers invalidate a PMD and then install a valid huge PMD again with
set_pmd_at(): MADV_FREE, MADV_COLD clearing the young bit,
page_mkclean_one() and change_huge_pmd(). The valid PMD is not counted
again, as _PAGE_PMD_HUGE did not flip, so each such call loses one count.

Count only valid huge PMDs instead: in __set_pmd_acct(), adjust the
count when an entry starts or stops being both _PAGE_PMD_HUGE and
_PAGE_VALID, and drop the separate decrement from pmdp_invalidate(),
whose pmdp_establish() now accounts for the invalidation itself. An
invalidated PMD that is set again, cleared or replaced by a page table
then stays balanced. A PROT_NONE THP is no longer counted either, which
suits the TSB sizing the count is used for, as only valid entries are
inserted into the TSB.

Fixes: 1e953d846ac0 ("sparc64 mm: Fix more TSB sizing issues")
Assisted-by: LLM
Signed-off-by: Zi Yan <ziy@nvidia.com>
To: "David S. Miller" <davem@davemloft.net>
To: Andreas Larsson <andreas@gaisler.com>
Cc: sparclinux@vger.kernel.org
Cc: linux-kernel@vger.kernel.org
---
 arch/sparc/mm/tlb.c | 26 ++++++++++++--------------
 1 file changed, 12 insertions(+), 14 deletions(-)

diff --git a/arch/sparc/mm/tlb.c b/arch/sparc/mm/tlb.c
index 6d9dd5eb13287..0a941d497525f 100644
--- a/arch/sparc/mm/tlb.c
+++ b/arch/sparc/mm/tlb.c
@@ -176,13 +176,23 @@ static void tlb_batch_pmd_scan(struct mm_struct *mm, unsigned long vaddr,
 }
 
 
+/*
+ * Only valid huge PMDs are counted: an invalidated or PROT_NONE PMD gives no
+ * access and never goes into the TSB.
+ */
+static bool pmd_huge_valid(pmd_t pmd)
+{
+	return (pmd_val(pmd) & (_PAGE_PMD_HUGE | _PAGE_VALID)) ==
+	       (_PAGE_PMD_HUGE | _PAGE_VALID);
+}
+
 static void __set_pmd_acct(struct mm_struct *mm, unsigned long addr,
 			   pmd_t orig, pmd_t pmd)
 {
 	if (mm == &init_mm)
 		return;
 
-	if ((pmd_val(pmd) ^ pmd_val(orig)) & _PAGE_PMD_HUGE) {
+	if (pmd_huge_valid(pmd) != pmd_huge_valid(orig)) {
 		/*
 		 * Note that this routine only sets pmds for THP pages.
 		 * Hugetlb pages are handled elsewhere.  We need to check
@@ -191,7 +201,7 @@ static void __set_pmd_acct(struct mm_struct *mm, unsigned long addr,
 		 * for TSB entries.  So, huge zero page counts go into
 		 * hugetlb_pte_count.
 		 */
-		if (pmd_val(pmd) & _PAGE_PMD_HUGE) {
+		if (pmd_huge_valid(pmd)) {
 			if (is_huge_zero_pmd(pmd))
 				mm->context.hugetlb_pte_count++;
 			else
@@ -250,9 +260,6 @@ static inline pmd_t pmdp_establish(struct vm_area_struct *vma,
 	return old;
 }
 
-/*
- * This routine is only called when splitting a THP
- */
 pmd_t pmdp_invalidate(struct vm_area_struct *vma, unsigned long address,
 		     pmd_t *pmdp)
 {
@@ -263,15 +270,6 @@ pmd_t pmdp_invalidate(struct vm_area_struct *vma, unsigned long address,
 	old = pmdp_establish(vma, address, pmdp, entry);
 	flush_tlb_range(vma, address, address + HPAGE_PMD_SIZE);
 
-	/*
-	 * set_pmd_at() will not be called in a way to decrement
-	 * thp_pte_count when splitting a THP, so do it now.
-	 * Sanity check pmd before doing the actual decrement.
-	 */
-	if ((pmd_val(entry) & _PAGE_PMD_HUGE) &&
-	    !is_huge_zero_pmd(entry))
-		(vma->vm_mm)->context.thp_pte_count--;
-
 	return old;
 }
 

-- 
2.53.0

[PATCH for 7.4-rc2+ 6/7] mm/huge_memory: avoid transient none PMDs during lazyfree reclaim

[Zi Yan]

From: Kyle Zeng <kylebot@openai.com>

__discard_anon_folio_pmd_locked() clears and flushes a huge PMD before
checking whether the folio can be discarded. If the folio was redirtied
or has unexpected references, it restores the original PMD. Reclaim holds
the PMD lock and the anon_vma read lock, but not the owning mmap_lock.

Both zap_pmd_range() and mremap's get_old_pmd() can skip a none PMD
without taking its lock. A concurrent munmap() or whole-VMA
MREMAP_DONTUNMAP can therefore miss the PMD and later unlink the source
VMA from its anon_vma, leaving a restored mapping behind. The anon_vma
write lock taken by unlink_anon_vmas() waits for reclaim to finish, but
does not repeat the skipped page-table walk. Once the remaining VMA
links are removed, the folio's positive mapcount no longer guarantees a
live anon_vma. Racing lazyfree reclaim against MREMAP_DONTUNMAP as an
unprivileged user reproduces a KASAN use-after-free in
folio_lock_anon_vma_read().

Use pmdp_invalidate() to keep a recognizable, non-none huge PMD while
the discard can still fail. Concurrent unmap and move operations then
have to synchronize on the PMD lock instead of skipping the mapping.
Clear the invalidated PMD only after the dirty and reference checks have
succeeded, before removing the rmap and withdrawing the deposited page
table. The full invalidation retains the TLB flush required for the
dirty and GUP-fast checks.

Note: this patch relies on the following patches to work correctly:
1. powerpc/book3s64/mm: don't count invalidated PMDs in page table check
2. riscv: mm: don't let GUP-fast follow invalidated THP PMDs
3. LoongArch: mm: don't let GUP-fast follow invalidated THP PMDs
4. s390/mm: always clear _SEGMENT_ENTRY_READ in pmdp_invalidate()
5. sparc64: mm: fix thp_pte_count accounting around pmdp_invalidate()

Assisted-by: LLM
Signed-off-by: Kyle Zeng <kylebot@openai.com>
Signed-off-by: Zi Yan <ziy@nvidia.com>
To: Andrew Morton <akpm@linux-foundation.org>
To: David Hildenbrand <david@kernel.org>
To: Lance Yang <lance.yang@linux.dev>
Cc: Zi Yan <ziy@nvidia.com>
Cc: Baolin Wang <baolin.wang@linux.alibaba.com>
Cc: "Liam R. Howlett" <liam@infradead.org>
Cc: Nico Pache <nico.pache@linux.dev>
Cc: Ryan Roberts <ryan.roberts@arm.com>
Cc: Dev Jain <dev.jain@arm.com>
Cc: Barry Song <baohua@kernel.org>
Cc: Usama Arif <usama.arif@linux.dev>
Cc: Kiryl Shutsemau <kas@kernel.org>
Cc: Jason Gunthorpe <jgg@nvidia.com>
Cc: John Hubbard <jhubbard@nvidia.com>
Cc: Peter Xu <peterx@redhat.com>
Cc: linux-mm@kvack.org
Cc: linux-kernel@vger.kernel.org
---
 mm/huge_memory.c | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)

diff --git a/mm/huge_memory.c b/mm/huge_memory.c
index bd44d58e86845..39d56b5032d59 100644
--- a/mm/huge_memory.c
+++ b/mm/huge_memory.c
@@ -3571,11 +3571,11 @@ static bool __discard_anon_folio_pmd_locked(struct vm_area_struct *vma,
 		return false;
 	}
 
-	orig_pmd = pmdp_huge_clear_flush(vma, addr, pmdp);
+	orig_pmd = pmdp_invalidate(vma, addr, pmdp);
 
 	/*
 	 * Syncing against concurrent GUP-fast:
-	 * - clear PMD; barrier; read refcount
+	 * - invalidate PMD; barrier; read refcount
 	 * - inc refcount; barrier; read PMD
 	 */
 	smp_mb();
@@ -3609,6 +3609,7 @@ static bool __discard_anon_folio_pmd_locked(struct vm_area_struct *vma,
 		return false;
 	}
 
+	pmdp_huge_get_and_clear(mm, addr, pmdp);
 	folio_remove_rmap_pmd(folio, pmd_page(orig_pmd), vma);
 	zap_deposited_table(mm, pmdp);
 	add_mm_counter(mm, MM_ANONPAGES, -HPAGE_PMD_NR);

-- 
2.53.0

[PATCH for 7.4-rc2+ 7/7] mm/rmap: re-enable discarding lazyfree THPs at PMD level

[Zi Yan]

The patch "mm/rmap: don't discard lazyfree THPs at PMD level" disabled
the PMD-level discard, because it cleared the PMD before checking whether
the folio could be discarded, and munmap() or mremap() could skip that
transiently none PMD.

__discard_anon_folio_pmd_locked() now keeps the PMD invalidated, not
cleared, until the discard is committed, GUP-fast no longer follows an
invalidated PMD on any architecture, powerpc page table check no longer
counts an invalidated PMD as mapped, and sparc64 keeps thp_pte_count
balanced when an invalidated PMD is set again or cleared. Discard lazyfree
THPs at PMD level again.

Assisted-by: LLM
Signed-off-by: Zi Yan <ziy@nvidia.com>
To: Andrew Morton <akpm@linux-foundation.org>
To: David Hildenbrand <david@kernel.org>
To: Lorenzo Stoakes <ljs@kernel.org>
Cc: Rik van Riel <riel@surriel.com>
Cc: "Liam R. Howlett" <liam@infradead.org>
Cc: Vlastimil Babka <vbabka@kernel.org>
Cc: Harry Yoo <harry@kernel.org>
Cc: Jann Horn <jannh@google.com>
Cc: Lance Yang <lance.yang@linux.dev>
Cc: Kyle Zeng <kylebot@openai.com>
Cc: linux-mm@kvack.org
Cc: linux-kernel@vger.kernel.org
---
 mm/rmap.c | 11 +++++++++++
 1 file changed, 11 insertions(+)

diff --git a/mm/rmap.c b/mm/rmap.c
index 1131b76bbbc28..805db93fe0428 100644
--- a/mm/rmap.c
+++ b/mm/rmap.c
@@ -2275,6 +2275,17 @@ static bool try_to_unmap_one(struct folio *folio, struct vm_area_struct *vma,
 		}
 
 		if (!pvmw.pte) {
+			if (folio_test_lazyfree(folio)) {
+				if (unmap_huge_pmd_locked(vma, pvmw.address, pvmw.pmd, folio))
+					goto walk_done;
+				/*
+				 * unmap_huge_pmd_locked has either already marked
+				 * the folio as swap-backed or decided to retain it
+				 * due to GUP or speculative references.
+				 */
+				goto walk_abort;
+			}
+
 			if (flags & TTU_SPLIT_HUGE_PMD) {
 				/*
 				 * We temporarily have to drop the PTL and

-- 
2.53.0

Re: [PATCH for 7.4-rc2+ 4/7] s390/mm: always clear _SEGMENT_ENTRY_READ in pmdp_invalidate()

[sashiko-bot]

> pmdp_invalidate() sets _SEGMENT_ENTRY_INVALID, but on s390 that bit
> alone also marks an old (not young) PMD, which is still a valid mapping
> that GUP-fast follows. Without CONFIG_PAGE_TABLE_CHECK, an invalidated
> PMD is therefore not pmd_protnone(), so GUP-fast can grab a reference on
> the folio and, as the PMD value doesn't change until the caller
> finishes, the recheck after taking the reference passes too.
> 
> Code that invalidates a PMD and then decides, from the folio refcount,
> whether it may drop the mapping relies on GUP-fast backing off from an
> invalidated PMD, like it does from a cleared one. x86, arm64 and powerpc
> already make GUP-fast reject invalidated PMDs.
> 
> With CONFIG_PAGE_TABLE_CHECK, pmdp_invalidate() already clears
> _SEGMENT_ENTRY_READ, which makes the entry pmd_protnone(), so GUP-fast
> does not follow it. Do that unconditionally. As noted in commit
> 7b4dde5e40ad6 ("s390/pgtable: Add s390 support for page table check"),
> an invalidated PMD that is pmd_protnone() is fine for the remaining
> pmd_protnone() users, which run under the page table lock or are
> lockless walkers that should not follow it anyway.
> 
> Assisted-by: LLM
> Signed-off-by: Zi Yan <ziy@nvidia.com>

Sashiko has reviewed this patch and found no issues. It looks great!

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20261015-anon_vma_uaf_fix-v1-0-7b977303b42b@nvidia.com?part=4

Re: [PATCH for 7.4-rc2+ 2/7] riscv: mm: don't let GUP-fast follow invalidated THP PMDs

[Zi Yan]

On Thu Oct 15, 2026 at 2:13 PM EDT, Zi Yan wrote:
> pmdp_invalidate() leaves a PMD whose pmd_present() is still true, so the
> generic pmd_access_permitted() returns true for an invalidated PMD. Then
> GUP-fast can grab a reference on the folio and, as the PMD value doesn't
> change until the caller finishes, the recheck after taking the reference
> passes too.
>
> Code that invalidates a PMD and then decides, from the folio refcount,
> whether it may drop the mapping relies on GUP-fast backing off from an
> invalidated PMD, like it does from a cleared one. x86, arm64 and powerpc
> already reject invalidated PMDs in pmd_access_permitted().
>
> Provide a riscv pmd_access_permitted() that requires _PAGE_PRESENT, so
> invalidated (and PROT_NONE) PMDs are not followed by GUP-fast.
>
> Assisted-by: LLM
> Signed-off-by: Zi Yan <ziy@nvidia.com>
> To: Paul Walmsley <pjw@kernel.org>
> To: Palmer Dabbelt <palmer@dabbelt.com>
> To: Albert Ou <aou@eecs.berkeley.edu>
> Cc: Alexandre Ghiti <alex@ghiti.fr>
> Cc: Jason Gunthorpe <jgg@nvidia.com>
> Cc: John Hubbard <jhubbard@nvidia.com>
> Cc: Peter Xu <peterx@redhat.com>
> Cc: linux-riscv@lists.infradead.org
> Cc: linux-kernel@vger.kernel.org
> ---
>  arch/riscv/include/asm/pgtable.h | 6 ++++++
>  1 file changed, 6 insertions(+)
>
> diff --git a/arch/riscv/include/asm/pgtable.h b/arch/riscv/include/asm/pgtable.h
> index d48f90140841e..66aac56631f0c 100644
> --- a/arch/riscv/include/asm/pgtable.h
> +++ b/arch/riscv/include/asm/pgtable.h
> @@ -816,6 +816,12 @@ static inline int pmd_write(pmd_t pmd)
>  	return pte_write(pmd_pte(pmd));
>  }
>  
> +#define pmd_access_permitted pmd_access_permitted
> +static inline bool pmd_access_permitted(pmd_t pmd, bool write)
> +{
> +	return (pmd_val(pmd) & _PAGE_PRESENT) && (!write || pmd_write(pmd));
> +}
> +

Sashiko asked if this function allows GUP-fast to follow execute-only
pages so that the page can be read.

Answer:

yes, and the new patch is at[1] and fixes both issues.

[1] https://lore.kernel.org/all/20261015-riscv-gup-read-fix-v1-1-8cf7f2504029@nvidia.com/

-- 
Best Regards,
Yan, Zi


_______________________________________________
linux-riscv mailing list
linux-riscv@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-riscv

Re: [PATCH for 7.4-rc2+ 2/7] riscv: mm: don't let GUP-fast follow invalidated THP PMDs

[Zi Yan]

On Thu Oct 15, 2026 at 2:13 PM EDT, Zi Yan wrote:
> pmdp_invalidate() leaves a PMD whose pmd_present() is still true, so the
> generic pmd_access_permitted() returns true for an invalidated PMD. Then
> GUP-fast can grab a reference on the folio and, as the PMD value doesn't
> change until the caller finishes, the recheck after taking the reference
> passes too.
>
> Code that invalidates a PMD and then decides, from the folio refcount,
> whether it may drop the mapping relies on GUP-fast backing off from an
> invalidated PMD, like it does from a cleared one. x86, arm64 and powerpc
> already reject invalidated PMDs in pmd_access_permitted().
>
> Provide a riscv pmd_access_permitted() that requires _PAGE_PRESENT, so
> invalidated (and PROT_NONE) PMDs are not followed by GUP-fast.
>
> Assisted-by: LLM
> Signed-off-by: Zi Yan <ziy@nvidia.com>
> To: Paul Walmsley <pjw@kernel.org>
> To: Palmer Dabbelt <palmer@dabbelt.com>
> To: Albert Ou <aou@eecs.berkeley.edu>
> Cc: Alexandre Ghiti <alex@ghiti.fr>
> Cc: Jason Gunthorpe <jgg@nvidia.com>
> Cc: John Hubbard <jhubbard@nvidia.com>
> Cc: Peter Xu <peterx@redhat.com>
> Cc: linux-riscv@lists.infradead.org
> Cc: linux-kernel@vger.kernel.org
> ---
>  arch/riscv/include/asm/pgtable.h | 6 ++++++
>  1 file changed, 6 insertions(+)
>
> diff --git a/arch/riscv/include/asm/pgtable.h b/arch/riscv/include/asm/pgtable.h
> index d48f90140841e..66aac56631f0c 100644
> --- a/arch/riscv/include/asm/pgtable.h
> +++ b/arch/riscv/include/asm/pgtable.h
> @@ -816,6 +816,12 @@ static inline int pmd_write(pmd_t pmd)
>  	return pte_write(pmd_pte(pmd));
>  }
>  
> +#define pmd_access_permitted pmd_access_permitted
> +static inline bool pmd_access_permitted(pmd_t pmd, bool write)
> +{
> +	return (pmd_val(pmd) & _PAGE_PRESENT) && (!write || pmd_write(pmd));
> +}
> +

Sashiko asked if this function allows GUP-fast to follow execute-only
pages so that the page can be read.

Answer:

yes, and the new patch is at[1] and fixes both issues.

[1] https://lore.kernel.org/all/20261015-riscv-gup-read-fix-v1-1-8cf7f2504029@nvidia.com/

-- 
Best Regards,
Yan, Zi

Re: [PATCH for 7.4-rc2+ 7/7] mm/rmap: re-enable discarding lazyfree THPs at PMD level

[Zi Yan]

On Thu Oct 15, 2026 at 2:13 PM EDT, Zi Yan wrote:
> The patch "mm/rmap: don't discard lazyfree THPs at PMD level" disabled
> the PMD-level discard, because it cleared the PMD before checking whether
> the folio could be discarded, and munmap() or mremap() could skip that
> transiently none PMD.
>
> __discard_anon_folio_pmd_locked() now keeps the PMD invalidated, not
> cleared, until the discard is committed, GUP-fast no longer follows an
> invalidated PMD on any architecture, powerpc page table check no longer
> counts an invalidated PMD as mapped, and sparc64 keeps thp_pte_count
> balanced when an invalidated PMD is set again or cleared. Discard lazyfree
> THPs at PMD level again.
>
> Assisted-by: LLM
> Signed-off-by: Zi Yan <ziy@nvidia.com>
> To: Andrew Morton <akpm@linux-foundation.org>
> To: David Hildenbrand <david@kernel.org>
> To: Lorenzo Stoakes <ljs@kernel.org>
> Cc: Rik van Riel <riel@surriel.com>
> Cc: "Liam R. Howlett" <liam@infradead.org>
> Cc: Vlastimil Babka <vbabka@kernel.org>
> Cc: Harry Yoo <harry@kernel.org>
> Cc: Jann Horn <jannh@google.com>
> Cc: Lance Yang <lance.yang@linux.dev>
> Cc: Kyle Zeng <kylebot@openai.com>
> Cc: linux-mm@kvack.org
> Cc: linux-kernel@vger.kernel.org
> ---
>  mm/rmap.c | 11 +++++++++++
>  1 file changed, 11 insertions(+)
>
> diff --git a/mm/rmap.c b/mm/rmap.c
> index 1131b76bbbc28..805db93fe0428 100644
> --- a/mm/rmap.c
> +++ b/mm/rmap.c
> @@ -2275,6 +2275,17 @@ static bool try_to_unmap_one(struct folio *folio, struct vm_area_struct *vma,
>  		}
>  
>  		if (!pvmw.pte) {
> +			if (folio_test_lazyfree(folio)) {
> +				if (unmap_huge_pmd_locked(vma, pvmw.address, pvmw.pmd, folio))

Sashiko mentioned that unmap_huge_pmd_locked() does add_mm_counter()
without calling update_hiwater_rss() first so that peak RSS can be
under-reported.

Answer:

yes, I fixed it locally by calling update_hiwater_rss() before
unmap_huge_pmd_locked().

> +					goto walk_done;
> +				/*
> +				 * unmap_huge_pmd_locked has either already marked
> +				 * the folio as swap-backed or decided to retain it
> +				 * due to GUP or speculative references.
> +				 */
> +				goto walk_abort;
> +			}
> +
>  			if (flags & TTU_SPLIT_HUGE_PMD) {
>  				/*
>  				 * We temporarily have to drop the PTL and




-- 
Best Regards,
Yan, Zi