[PATCH for 7.4-rc2+ 0/7] Re-enable discarding lazyfree THPs at PMD level

[PATCH for 7.4-rc2+ 0/7] Re-enable discarding lazyfree THPs at PMD level

[Zi Yan]

For maintainers, Patch 6 requires first 5 patches to work correctly, so it
might be better to get ack from arch maintainers for first 5 patches and
merge the series via mm tree. The patch disabling lazyfree THP discard at
PMD level has been picked up [2] and would take some time to test, so this
patchset probably can be considered after the merge window closes. I am
sending it out for early feedback.

Motivation
===

A bug was found in the PMD-level discard of lazyfree THPs [1]. The
proposed fix keeps the PMD invalidated with pmdp_invalidate() instead of
clearing it, but that does not work on every architecture:

1. riscv, LoongArch and s390 let GUP-fast follow an invalidated PMD;
2. powerpc/book3s64 page table check counts an invalidated PMD as mapped;
3. sparc64's thp_pte_count becomes unbalanced around pmdp_invalidate().

So the PMD-level discard was disabled instead [2]. This series fixes the
arch code and re-enables it.

The discard decides from the folio refcount, so GUP-fast must not take a
reference through an invalidated PMD, and arch accounting must stay
balanced when an invalidated PMD is set again or cleared.

Details
===

Patches 1-5 fix the arch code, patch 6 from Kyle keeps the PMD invalidated
instead of cleared until the discard is committed, and patch 7 enables the
PMD-level discard again:

1. powerpc/book3s64: page table check counts an invalidated PMD as a live
   mapping, so setting it again or clearing it decrements the map count a
   second time and hits BUG_ON(). This is already reachable via MADV_COLD
   and mprotect() with CONFIG_PAGE_TABLE_CHECK.

2. riscv: an invalidated PMD still passes pmd_present() and the generic
   pmd_access_permitted(), so GUP-fast follows it. Add a
   pmd_access_permitted() that requires _PAGE_PRESENT.

3. LoongArch: same problem and fix as riscv.

4. s390: _SEGMENT_ENTRY_INVALID also marks an old PMD, so GUP-fast follows
   an invalidated PMD. Always clear _SEGMENT_ENTRY_READ in
   pmdp_invalidate(), which makes it pmd_protnone(), so GUP-fast rejects
   it.

5. sparc64: pmdp_invalidate() decrements thp_pte_count assuming a THP
   split follows, so setting the PMD again loses a count. Count only
   valid huge PMDs in __set_pmd_acct() instead, which keeps invalidate,
   restore, clear and split balanced.

6. mm/huge_memory: use pmdp_invalidate() in the discard, so munmap() and
   mremap() wait on the PMD lock instead of skipping a none PMD.

7. mm/rmap: discard lazyfree THPs at PMD level again.

Note
===

x86, arm64 and powerpc already make GUP-fast reject an invalidated PMD,
and sparc64 has no GUP-fast. 32-bit MIPS would not reject it, but it
cannot have THP: _PAGE_HUGE makes PFN_PTE_SHIFT exceed PAGE_SHIFT, so the
BUILD_BUG_ON() in arch_mm_preinit() fires, which is why Ingenic dropped
huge pages in commit 69b686ddd579 ("MIPS: ingenic: Remove
CPU_SUPPORTS_HUGEPAGES").

Link: https://lore.kernel.org/all/20261009165214.40212-2-kylebot@openai.com/ [1]
Link: https://lore.kernel.org/all/20261012-lazyfree-stable-fix-v2-1-6387d6e30063@nvidia.com/ [2]

Signed-off-by: Zi Yan <ziy@nvidia.com>
---
Kyle Zeng (1):
      mm/huge_memory: avoid transient none PMDs during lazyfree reclaim

Zi Yan (6):
      powerpc/book3s64/mm: don't count invalidated PMDs in page table check
      riscv: mm: don't let GUP-fast follow invalidated THP PMDs
      LoongArch: mm: don't let GUP-fast follow invalidated THP PMDs
      s390/mm: always clear _SEGMENT_ENTRY_READ in pmdp_invalidate()
      sparc64: mm: fix thp_pte_count accounting around pmdp_invalidate()
      mm/rmap: re-enable discarding lazyfree THPs at PMD level

 arch/loongarch/include/asm/pgtable.h         |  6 ++++++
 arch/powerpc/include/asm/book3s/64/pgtable.h |  3 ++-
 arch/riscv/include/asm/pgtable.h             |  6 ++++++
 arch/s390/include/asm/pgtable.h              |  8 ++++++--
 arch/sparc/mm/tlb.c                          | 26 ++++++++++++--------------
 mm/huge_memory.c                             |  5 +++--
 mm/rmap.c                                    | 11 +++++++++++
 7 files changed, 46 insertions(+), 19 deletions(-)
---
base-commit: 71a9b024444eba184335b67867897714e4da143c
change-id: 20261007-anon_vma_uaf_fix-e7accd1d2ee1

Best regards,
--  
Yan, Zi

[PATCH for 7.4-rc2+ 4/7] s390/mm: always clear _SEGMENT_ENTRY_READ in pmdp_invalidate()

[Zi Yan]

pmdp_invalidate() sets _SEGMENT_ENTRY_INVALID, but on s390 that bit
alone also marks an old (not young) PMD, which is still a valid mapping
that GUP-fast follows. Without CONFIG_PAGE_TABLE_CHECK, an invalidated
PMD is therefore not pmd_protnone(), so GUP-fast can grab a reference on
the folio and, as the PMD value doesn't change until the caller
finishes, the recheck after taking the reference passes too.

Code that invalidates a PMD and then decides, from the folio refcount,
whether it may drop the mapping relies on GUP-fast backing off from an
invalidated PMD, like it does from a cleared one. x86, arm64 and powerpc
already make GUP-fast reject invalidated PMDs.

With CONFIG_PAGE_TABLE_CHECK, pmdp_invalidate() already clears
_SEGMENT_ENTRY_READ, which makes the entry pmd_protnone(), so GUP-fast
does not follow it. Do that unconditionally. As noted in commit
7b4dde5e40ad6 ("s390/pgtable: Add s390 support for page table check"),
an invalidated PMD that is pmd_protnone() is fine for the remaining
pmd_protnone() users, which run under the page table lock or are
lockless walkers that should not follow it anyway.

Assisted-by: LLM
Signed-off-by: Zi Yan <ziy@nvidia.com>
To: Alexander Gordeev <agordeev@linux.ibm.com>
To: Gerald Schaefer <gerald.schaefer@linux.ibm.com>
To: Heiko Carstens <hca@linux.ibm.com>
To: Vasily Gorbik <gor@linux.ibm.com>
Cc: Christian Borntraeger <borntraeger@linux.ibm.com>
Cc: Sven Schnelle <svens@linux.ibm.com>
Cc: Jason Gunthorpe <jgg@nvidia.com>
Cc: John Hubbard <jhubbard@nvidia.com>
Cc: Peter Xu <peterx@redhat.com>
Cc: linux-s390@vger.kernel.org
Cc: linux-kernel@vger.kernel.org
---
 arch/s390/include/asm/pgtable.h | 8 ++++++--
 1 file changed, 6 insertions(+), 2 deletions(-)

diff --git a/arch/s390/include/asm/pgtable.h b/arch/s390/include/asm/pgtable.h
index 2d5c2ab06de98..fe1ba746203ab 100644
--- a/arch/s390/include/asm/pgtable.h
+++ b/arch/s390/include/asm/pgtable.h
@@ -1781,10 +1781,14 @@ static inline pmd_t pmdp_invalidate(struct vm_area_struct *vma,
 	pmd_t pmd = *pmdp;
 
 	VM_WARN_ON_ONCE(!pmd_present(pmd));
+	/*
+	 * _SEGMENT_ENTRY_INVALID alone also marks an old (not young) entry,
+	 * which lockless walkers like GUP-fast may follow. Clear
+	 * _SEGMENT_ENTRY_READ too, so the invalidated entry is pmd_protnone()
+	 * and is not followed, and page table check does not count it.
+	 */
 	pmd = set_pmd_bit(pmd, __pgprot(_SEGMENT_ENTRY_INVALID));
-#ifdef CONFIG_PAGE_TABLE_CHECK
 	pmd = clear_pmd_bit(pmd, __pgprot(_SEGMENT_ENTRY_READ));
-#endif
 	page_table_check_pmd_set(vma->vm_mm, addr, pmdp, pmd);
 	pmd = pmdp_xchg_direct(vma->vm_mm, addr, pmdp, pmd);
 	return pmd;

-- 
2.53.0

Re: [PATCH for 7.4-rc2+ 4/7] s390/mm: always clear _SEGMENT_ENTRY_READ in pmdp_invalidate()

[sashiko-bot]

> pmdp_invalidate() sets _SEGMENT_ENTRY_INVALID, but on s390 that bit
> alone also marks an old (not young) PMD, which is still a valid mapping
> that GUP-fast follows. Without CONFIG_PAGE_TABLE_CHECK, an invalidated
> PMD is therefore not pmd_protnone(), so GUP-fast can grab a reference on
> the folio and, as the PMD value doesn't change until the caller
> finishes, the recheck after taking the reference passes too.
> 
> Code that invalidates a PMD and then decides, from the folio refcount,
> whether it may drop the mapping relies on GUP-fast backing off from an
> invalidated PMD, like it does from a cleared one. x86, arm64 and powerpc
> already make GUP-fast reject invalidated PMDs.
> 
> With CONFIG_PAGE_TABLE_CHECK, pmdp_invalidate() already clears
> _SEGMENT_ENTRY_READ, which makes the entry pmd_protnone(), so GUP-fast
> does not follow it. Do that unconditionally. As noted in commit
> 7b4dde5e40ad6 ("s390/pgtable: Add s390 support for page table check"),
> an invalidated PMD that is pmd_protnone() is fine for the remaining
> pmd_protnone() users, which run under the page table lock or are
> lockless walkers that should not follow it anyway.
> 
> Assisted-by: LLM
> Signed-off-by: Zi Yan <ziy@nvidia.com>

Sashiko has reviewed this patch and found no issues. It looks great!

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20261015-anon_vma_uaf_fix-v1-0-7b977303b42b@nvidia.com?part=4