[PATCH RFC v4 0/12] mm: use per-VMA lock in __access_remote_vm for improved monitoring reliability

[PATCH RFC v4 0/12] mm: use per-VMA lock in __access_remote_vm for improved monitoring reliability

[Rik van Riel]

__access_remote_vm() holds mmap_read_lock() for the whole transfer. On
large machines, with large multi-threaded applications, the mmap_lock
is often contended, leading to things like reads of /proc/PID/cmdline
stalling.

This results in system monitoring tools getting stuck, right when
system information would be the most helpful.

That problem can be avoided by having __access_remote_vm() access
user memory under the per-VMA lock. The old code also looks up
the VMA for every page accessed, and walks the page tables for
every page inside a large folio. Cleaning that up speeds things
up nicely.

Using code that does not include its own lock dance, and does not
look up the VMA twice results in a modest speedup.

	read size   baseline    per-VMA     throughput
	8 B          201.7 ns    170.6 ns    ~15% faster
	4 KB         495 ns      442 ns      +12% (8266 -> 9265 MB/s)
	64 KB       4115 ns     3355 ns      +23% (15927 -> 19536 MB/s)
	1 MB       75902 ns    63641 ns      +19% (13815 -> 16477 MB/s)

The multi-threaded reader-versus-writer contention that motivates the lock
is not measured.

The rest of the series builds on this to batch per-VMA GUP. Patches 9-12
have follow_page_mask() report a contiguous PTE-mapped run of a large
folio, so the slow pin_user_pages() path walks it once instead of per page.
Other paths that walk a range benefit too.

That batching pays off on PTE-mapped mTHP. With mm/gup_test.c
(PIN_LONGTERM_BENCHMARK, the slow pin_user_pages() path) on a 256 MB
MADV_HUGEPAGE region in a 4 CPU VM, median get time over 16 iterations:

	gup_test -L -m 256 -n 65536 -r 16 -t
	                      before      after
	64 kB mTHP            3140 us      412 us   (7.6x)
	2 MB THP (control)      78 us       76 us
	4 kB base (control)   3010 us     3042 us

The PMD-mapped 2 MB THP already returns in one step, so only the PTE-mapped
large folio case improves.

Reaching a page under the per-VMA lock means looking up the VMA first,
which needs to untag the remote address without the mmap lock. The untag
masks (x86 LAM, riscv pointer masking) change only while the target is
single-threaded, so a remote reader races at most one such change.

Those masks are already read locklessly elsewhere, so patches 1 and 2 add
untagged_addr_remote_unlocked() helpers with READ_ONCE()/WRITE_ONCE()
annotations. A stale read sees the old or new mask, never a torn value, and
the remote untag is best-effort.

With the VMA in hand we need one page from it under the lock the caller
already holds. get_user_pages_remote() does not fit: it hard codes the mmap
lock and re-derives the VMA internally.

Rather than reimplement GUP in yet another caller, this version follows
David Hildenbrand's v2 guidance: start from a proper GUP interface that
takes a locked VMA, not an mm, and decide which faults can resolve under
the VMA lock.

Patch 3 renames get_user_page_vma_remote() to get_user_page_lookup_vma() to
free the name. Patch 5 adds get_user_page_vma(): a simplified
__get_user_pages() that walks the tables with follow_page_mask(), faults a
missing page in with faultin_page(), and returns it with a reference and
the caller's lock still held.

Like __get_user_pages() it runs check_vma_flags(), so callers need not
pre-check the VMA.

A VM_IO or VM_PFNMAP VMA is the exception: a COWed page with a struct page
is returned normally, while a raw PFN yields -EFAULT so the caller can
reach vma->vm_ops->access(). The caller sets FOLL_VMA_LOCK when it holds
the per-VMA lock, which reaches the fault code as FAULT_FLAG_VMA_LOCK.

Anything the per-VMA lock cannot finish releases the lock and returns
-EAGAIN, so the caller retries under the mmap lock: a dropped fault, a
userfaultfd VMA (which assumes current is the faulting task), a hard error,
or ->access() memory. faultin_page() reports the retry the same way for
both lock types.

__access_remote_vm() then uses the per-VMA lock when an access fits within
one VMA, and falls back to the mmap lock for multi-VMA accesses, stack
expansion, or an -EAGAIN from get_user_page_vma().

Walking page tables under only the per-VMA lock is safe against both page
table freeing and THP collapse. munmap() frees a VMA's page tables under
the VMA write lock that our read lock excludes, so they cannot be torn down
under the walk.

THP collapse needs neither lock: it retracts a PTE page under the page
table lock and frees it by RCU.

follow_page_pte() takes that same lock through pte_offset_map_lock() and
rechecks the pmd, so it either walks an intact table or sees the collapsed
pmd and faults the page in, while RCU keeps the retracted page valid.

A COWed page in a VM_PFNMAP mapping was previously unreachable through
/proc/pid/mem because generic_access_phys() rejects ioremaps of RAM. With
get_user_page_vma() returning the struct page directly, that read now
succeeds, which the new selftests cover along with the raw PFN path.

Thanks to Suren for pointing out the need, to David for pushing me
to just rework the whole area, to Usama for reviewing the arch patches,
and Lorenzo for the changelog and cover letter cleanups.

There is a lot more work that could be done in this area, but the
patch series is big enough. More changes can come in follow up work.
---
 arch/arm64/kernel/mte.c                        |   2 +-
 arch/riscv/include/asm/mmu_context.h           |   4 +-
 arch/riscv/include/asm/uaccess.h               |  10 +-
 arch/riscv/kernel/process.c                    |  12 +-
 arch/x86/include/asm/mmu_context.h             |   6 +-
 arch/x86/include/asm/uaccess_64.h              |  15 +-
 arch/x86/kernel/process_64.c                   |   4 +-
 arch/x86/kernel/uprobes.c                      |   2 +-
 include/linux/mm.h                             |  32 +--
 include/linux/uaccess.h                        |   7 +
 mm/gup.c                                       | 360 ++++++++++++++++++-----
 mm/internal.h                                  |   8 +-
 mm/memory.c                                    | 379 +++++++++++++++++--------
 mm/rmap.c                                      |   2 +-
 tools/testing/selftests/mm/Makefile            |   1 +
 tools/testing/selftests/mm/mthp_gup_cow_test.c | 213 ++++++++++++++
 tools/testing/selftests/mm/pfnmap.c            |  66 +++++
 tools/testing/selftests/mm/run_vmtests.sh      |   1 +
 18 files changed, 887 insertions(+), 237 deletions(-)

base-commit: 248951ddc14de84de3910f9b13f51491a8cd91df
---
To: Andrew Morton <akpm@linux-foundation.org>
To: linux-mm@kvack.org
To: linux-kernel@vger.kernel.org
Cc: Dave Hansen <dave.hansen@linux.intel.com>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Suren Baghdasaryan <surenb@google.com>
Cc: Lorenzo Stoakes <ljs@kernel.org>
Cc: Vlastimil Babka <vbabka@kernel.org>
Cc: David Hildenbrand <david@kernel.org>
Cc: "Liam R. Howlett" <liam@infradead.org>
Cc: Mike Rapoport <rppt@kernel.org>
Cc: Michal Hocko <mhocko@suse.com>
Cc: Jason Gunthorpe <jgg@ziepe.ca>
Cc: John Hubbard <jhubbard@nvidia.com>
Cc: Peter Xu <peterx@redhat.com>
Cc: Matthew Wilcox <willy@infradead.org>
Cc: Usama Arif <usamaarif642@gmail.com>
Cc: Rik van Riel <riel@surriel.com>

REVIEWERS NOTES:
Series based on the current Linus tree (v7.2-rc5, commit 248951ddc14d).
Patches 1-8 add per-VMA lock remote access; patches 9-12 batch per-VMA GUP
over PTE-mapped large folios.

Changes since v3 (RFC v3, 2026-07-17):
https://lore.kernel.org/all/20260717170036.743149-1-riel@surriel.com/

- Address Usama Arif's v3 review:
  - p1: correct the Acked-by address to usama.arif@linux.dev.
  - p2: rewrite the riscv changelog to describe only this patch's change,
    and replace the false "pmlen is stable" claim with the accurate story
    (pmlen may change until MM_CONTEXT_LOCK_PMLEN; READ_ONCE avoids a torn
    value, and the remote untag is best-effort).
  - p3: fix the "name space" typo and note "No functional change intended."
  - Split the check_vma_flags() ignore-flags change into its own prep patch
    (patch 4) so the PROT_NONE PFNMAP read hole fix is bisectable, keeping
    the VM_READ / FOLL_FORCE / arch-key checks for a PFNMAP VMA.
  - get_user_page_vma(): add the flush_anon_page()/flush_dcache_page() cache
    flushes __get_user_pages() does, and reword the faultin_page() kerneldoc
    to cover the per-VMA lock, not just the mmap lock.
- New patch 7: read remote strings (bpf_copy_from_user_task_str()) under the
  per-VMA lock, extending the approach to __copy_remote_vm_str().
- Factor __access_remote_vm() and the string reader onto a shared
  remote_vm_walk() helper, and fix a NUL-termination regression it exposed:
  the partial buffer is now terminated on a mid-walk error.
- Fold the mTHP GUP batching (patches 9-12), previously a separate follow-up,
  into this series; the cover letter now carries its gup_test numbers.
- Correct two changelog claims. The page table lock plus the pmd recheck, not
  the VMA write lock, guards the walk against concurrent THP collapse; RCU
  only keeps the retracted PTE page valid, and the VMA write lock excludes
  munmap(). The x86 untag mask changes while the target is single-threaded,
  so the lockless read is best-effort, not "set once and never changes."
- Rebase onto the current Linus tree (v7.2-rc5).

[PATCH RFC v4 01/12] x86/mm: add untagged_addr_remote_unlocked()

[Rik van Riel]

__access_remote_vm() reads another process's memory under the mmap lock. On
large machines running big multi-threaded applications, that lock is
contended between readers and writers: an mmap() or munmap() in one thread
stalls readers like /proc/PID/cmdline, /proc/PID/environ, or
process_vm_readv(), even though the memory they read is almost always
resident and reachable under the per-VMA lock instead.

Looking up the VMA first requires untagging the address.
untagged_addr_remote() asserts the mmap lock only because it reads
mm->context.untag_mask, which can race with the write in mm_enable_lam().

That mask changes only while the target is single-threaded, when it
enables LAM or resets it at exec. The read need not be stable: it is
already read locklessly elsewhere, and a remote untag is best-effort.

Add untagged_addr_remote_unlocked() for callers that have not taken the
mmap lock, and annotate access to mm->context.untag_mask with READ_ONCE()
and WRITE_ONCE() so the existing lockless reads are explicit and
KCSAN-clean. untagged_addr_remote() keeps its assertion and shares the same
code.

Assisted-by: Claude:claude-opus-4.8
Acked-by: Usama Arif <usama.arif@linux.dev>
Signed-off-by: Rik van Riel <riel@surriel.com>
---
 arch/x86/include/asm/mmu_context.h |  6 +++---
 arch/x86/include/asm/uaccess_64.h  | 15 ++++++++++++---
 arch/x86/kernel/process_64.c       |  4 ++--
 include/linux/uaccess.h            |  7 +++++++
 4 files changed, 24 insertions(+), 8 deletions(-)

diff --git a/arch/x86/include/asm/mmu_context.h b/arch/x86/include/asm/mmu_context.h
index ef5b507de34e..cee710f64658 100644
--- a/arch/x86/include/asm/mmu_context.h
+++ b/arch/x86/include/asm/mmu_context.h
@@ -100,18 +100,18 @@ static inline unsigned long mm_lam_cr3_mask(struct mm_struct *mm)
 static inline void dup_lam(struct mm_struct *oldmm, struct mm_struct *mm)
 {
 	mm->context.lam_cr3_mask = oldmm->context.lam_cr3_mask;
-	mm->context.untag_mask = oldmm->context.untag_mask;
+	WRITE_ONCE(mm->context.untag_mask, READ_ONCE(oldmm->context.untag_mask));
 }
 
 #define mm_untag_mask mm_untag_mask
 static inline unsigned long mm_untag_mask(struct mm_struct *mm)
 {
-	return mm->context.untag_mask;
+	return READ_ONCE(mm->context.untag_mask);
 }
 
 static inline void mm_reset_untag_mask(struct mm_struct *mm)
 {
-	mm->context.untag_mask = -1UL;
+	WRITE_ONCE(mm->context.untag_mask, -1UL);
 }
 
 #define arch_pgtable_dma_compat arch_pgtable_dma_compat
diff --git a/arch/x86/include/asm/uaccess_64.h b/arch/x86/include/asm/uaccess_64.h
index 20de34cc9aa6..a095165eb648 100644
--- a/arch/x86/include/asm/uaccess_64.h
+++ b/arch/x86/include/asm/uaccess_64.h
@@ -39,18 +39,27 @@ static inline unsigned long __untagged_addr(unsigned long addr)
 	(__force __typeof__(addr))__untagged_addr(__addr);		\
 })
 
+/*
+ * mm->context.untag_mask changes only when the target enables LAM or execs,
+ * always single-threaded. An unlocked remote reader can race that; READ_ONCE
+ * keeps the value whole and the untag is best-effort.
+ */
 static inline unsigned long __untagged_addr_remote(struct mm_struct *mm,
 						   unsigned long addr)
 {
-	mmap_assert_locked(mm);
-	return addr & (mm)->context.untag_mask;
+	return addr & READ_ONCE(mm->context.untag_mask);
 }
 
-#define untagged_addr_remote(mm, addr)	({				\
+#define untagged_addr_remote_unlocked(mm, addr)	({			\
 	unsigned long __addr = (__force unsigned long)(addr);		\
 	(__force __typeof__(addr))__untagged_addr_remote(mm, __addr);	\
 })
 
+#define untagged_addr_remote(mm, addr)	({				\
+	mmap_assert_locked(mm);						\
+	untagged_addr_remote_unlocked(mm, addr);			\
+})
+
 #endif
 
 #define valid_user_address(x) \
diff --git a/arch/x86/kernel/process_64.c b/arch/x86/kernel/process_64.c
index d44afbe005bb..9fa659117f38 100644
--- a/arch/x86/kernel/process_64.c
+++ b/arch/x86/kernel/process_64.c
@@ -814,7 +814,7 @@ static void enable_lam_func(void *__mm)
 static void mm_enable_lam(struct mm_struct *mm)
 {
 	mm->context.lam_cr3_mask = X86_CR3_LAM_U57;
-	mm->context.untag_mask =  ~GENMASK(62, 57);
+	WRITE_ONCE(mm->context.untag_mask, ~GENMASK(62, 57));
 
 	/*
 	 * Even though the process must still be single-threaded at this
@@ -952,7 +952,7 @@ long do_arch_prctl_64(struct task_struct *task, int option, unsigned long arg2)
 #endif
 #ifdef CONFIG_ADDRESS_MASKING
 	case ARCH_GET_UNTAG_MASK:
-		return put_user(task->mm->context.untag_mask,
+		return put_user(mm_untag_mask(task->mm),
 				(unsigned long __user *)arg2);
 	case ARCH_ENABLE_TAGGED_ADDR:
 		return prctl_enable_tagged_addr(task->mm, arg2);
diff --git a/include/linux/uaccess.h b/include/linux/uaccess.h
index eddbbb65ccc4..7e6e4c89184c 100644
--- a/include/linux/uaccess.h
+++ b/include/linux/uaccess.h
@@ -34,6 +34,13 @@
 })
 #endif
 
+#ifndef untagged_addr_remote_unlocked
+#define untagged_addr_remote_unlocked(mm, addr)	({	\
+	(void)(mm);					\
+	untagged_addr(addr);				\
+})
+#endif
+
 #ifdef masked_user_access_begin
  #define can_do_masked_user_access() 1
 # ifndef masked_user_write_access_begin

base-commit: 248951ddc14de84de3910f9b13f51491a8cd91df
-- 
2.53.0-Meta

[PATCH RFC v4 02/12] riscv/mm: add untagged_addr_remote_unlocked()

[Rik van Riel]

__access_remote_vm() reads another task's memory under the mmap lock. That
lock is contended on large multi-threaded workloads, where an mmap() or
munmap() stalls readers like /proc/PID/cmdline or process_vm_readv() even
though the target memory is resident and reachable under the per-VMA lock.

Reaching it under the per-VMA lock means looking up the VMA first, which
requires untagging the remote address without the mmap lock. riscv's
untagged_addr_remote() asserts that lock and has no unlocked variant; the
generic untagged_addr_remote_unlocked() falls back to untagged_addr(),
which masks against current->mm, not the target mm.

Add untagged_addr_remote_unlocked(), which masks @addr against the target
mm without asserting the lock, and redefine untagged_addr_remote() on top
of it so it keeps the assertion.

The mask width is mm->context.pmlen, written while a program is single
threaded, and frozen with MM_CONTEXT_LOCK_PMLEN once it spawns a second
thread.

That read need not be stable: pmlen is already read locklessly by
untagged_addr() and mm_untag_mask(), and a remote untag is best-effort.
Annotate the lockless reads with READ_ONCE() and the writes with
WRITE_ONCE(), so a reader sees one whole value and the accesses are
KCSAN-clean.

Assisted-by: Claude:claude-opus-4.8
Signed-off-by: Rik van Riel <riel@surriel.com>
---
 arch/riscv/include/asm/mmu_context.h |  4 ++--
 arch/riscv/include/asm/uaccess.h     | 10 +++++++---
 arch/riscv/kernel/process.c          | 12 +++++++-----
 3 files changed, 16 insertions(+), 10 deletions(-)

diff --git a/arch/riscv/include/asm/mmu_context.h b/arch/riscv/include/asm/mmu_context.h
index dbf27a78df6c..3ce16796e5a2 100644
--- a/arch/riscv/include/asm/mmu_context.h
+++ b/arch/riscv/include/asm/mmu_context.h
@@ -21,7 +21,7 @@ static inline void activate_mm(struct mm_struct *prev,
 			       struct mm_struct *next)
 {
 #ifdef CONFIG_RISCV_ISA_SUPM
-	next->context.pmlen = 0;
+	WRITE_ONCE(next->context.pmlen, 0);
 #endif
 	switch_mm(prev, next, NULL);
 }
@@ -44,7 +44,7 @@ DECLARE_STATIC_KEY_FALSE(use_asid_allocator);
 #define mm_untag_mask mm_untag_mask
 static inline unsigned long mm_untag_mask(struct mm_struct *mm)
 {
-	return -1UL >> mm->context.pmlen;
+	return -1UL >> READ_ONCE(mm->context.pmlen);
 }
 #endif
 
diff --git a/arch/riscv/include/asm/uaccess.h b/arch/riscv/include/asm/uaccess.h
index 5d4ec15584cf..53806e0f7dcf 100644
--- a/arch/riscv/include/asm/uaccess.h
+++ b/arch/riscv/include/asm/uaccess.h
@@ -16,7 +16,7 @@
 static inline unsigned long __untagged_addr_remote(struct mm_struct *mm, unsigned long addr)
 {
 	if (riscv_has_extension_unlikely(RISCV_ISA_EXT_SUPM)) {
-		u8 pmlen = mm->context.pmlen;
+		u8 pmlen = READ_ONCE(mm->context.pmlen);
 
 		/* Virtual addresses are sign-extended; physical addresses are zero-extended. */
 		if (IS_ENABLED(CONFIG_MMU))
@@ -33,12 +33,16 @@ static inline unsigned long __untagged_addr_remote(struct mm_struct *mm, unsigne
 	(__force __typeof__(addr))__untagged_addr_remote(current->mm, __addr);	\
 })
 
-#define untagged_addr_remote(mm, addr) ({					\
+#define untagged_addr_remote_unlocked(mm, addr) ({				\
 	unsigned long __addr = (__force unsigned long)(addr);			\
-	mmap_assert_locked(mm);							\
 	(__force __typeof__(addr))__untagged_addr_remote(mm, __addr);		\
 })
 
+#define untagged_addr_remote(mm, addr) ({					\
+	mmap_assert_locked(mm);							\
+	untagged_addr_remote_unlocked(mm, addr);				\
+})
+
 #define access_ok(addr, size) likely(__access_ok(untagged_addr(addr), size))
 #else
 #define untagged_addr(addr) (addr)
diff --git a/arch/riscv/kernel/process.c b/arch/riscv/kernel/process.c
index b2df7f72241a..6ae7552fed09 100644
--- a/arch/riscv/kernel/process.c
+++ b/arch/riscv/kernel/process.c
@@ -357,13 +357,15 @@ long set_tagged_addr_ctrl(struct task_struct *task, unsigned long arg)
 	if (mmap_write_lock_killable(mm))
 		return -EINTR;
 
-	if (test_bit(MM_CONTEXT_LOCK_PMLEN, &mm->context.flags) && mm->context.pmlen != pmlen) {
-		mmap_write_unlock(mm);
-		return -EBUSY;
+	if (test_bit(MM_CONTEXT_LOCK_PMLEN, &mm->context.flags)) {
+		if (READ_ONCE(mm->context.pmlen) != pmlen) {
+			mmap_write_unlock(mm);
+			return -EBUSY;
+		}
 	}
 
 	envcfg_update_bits(task, ENVCFG_PMM, pmm);
-	mm->context.pmlen = pmlen;
+	WRITE_ONCE(mm->context.pmlen, pmlen);
 
 	mmap_write_unlock(mm);
 
@@ -394,7 +396,7 @@ long get_tagged_addr_ctrl(struct task_struct *task)
 		break;
 	}
 
-	if (task->mm->context.pmlen)
+	if (READ_ONCE(task->mm->context.pmlen))
 		ret |= PR_TAGGED_ADDR_ENABLE;
 
 	return ret;
-- 
2.53.0-Meta

[PATCH RFC v4 03/12] mm: rename get_user_page_vma_remote() to get_user_page_lookup_vma()

[Rik van Riel]

get_user_page_vma_remote() faults in the page at @addr in a remote mm and
also looks up the VMA that covers it, handing both back to the caller.

This cleans up the namespace for adding a get_user_page_vma() variant where
the caller already has the vma.

No functional change intended.

Assisted-by: Claude:claude-opus-4.8
Acked-by: Usama Arif <usama.arif@linux.dev>
Signed-off-by: Rik van Riel <riel@surriel.com>
---
 arch/arm64/kernel/mte.c   | 2 +-
 arch/x86/kernel/uprobes.c | 2 +-
 include/linux/mm.h        | 2 +-
 mm/memory.c               | 4 ++--
 mm/rmap.c                 | 2 +-
 5 files changed, 6 insertions(+), 6 deletions(-)

diff --git a/arch/arm64/kernel/mte.c b/arch/arm64/kernel/mte.c
index 1a9aad6ef22a..7a6ecc3d9294 100644
--- a/arch/arm64/kernel/mte.c
+++ b/arch/arm64/kernel/mte.c
@@ -459,7 +459,7 @@ static int __access_remote_tags(struct mm_struct *mm, unsigned long addr,
 		struct vm_area_struct *vma;
 		unsigned long tags, offset;
 		void *maddr;
-		struct page *page = get_user_page_vma_remote(mm, addr,
+		struct page *page = get_user_page_lookup_vma(mm, addr,
 							     gup_flags, &vma);
 		struct folio *folio;
 
diff --git a/arch/x86/kernel/uprobes.c b/arch/x86/kernel/uprobes.c
index 3af979fb41d3..329efac0cfb3 100644
--- a/arch/x86/kernel/uprobes.c
+++ b/arch/x86/kernel/uprobes.c
@@ -1036,7 +1036,7 @@ static int copy_from_vaddr(struct mm_struct *mm, unsigned long vaddr, void *dst,
 	struct vm_area_struct *vma;
 	struct page *page;
 
-	page = get_user_page_vma_remote(mm, vaddr, gup_flags, &vma);
+	page = get_user_page_lookup_vma(mm, vaddr, gup_flags, &vma);
 	if (IS_ERR(page))
 		return PTR_ERR(page);
 	uprobe_copy_from_page(page, vaddr, dst, len);
diff --git a/include/linux/mm.h b/include/linux/mm.h
index 485df9c2dbdd..24ead14b4790 100644
--- a/include/linux/mm.h
+++ b/include/linux/mm.h
@@ -3238,7 +3238,7 @@ long pin_user_pages_remote(struct mm_struct *mm,
 /*
  * Retrieves a single page alongside its VMA. Does not support FOLL_NOWAIT.
  */
-static inline struct page *get_user_page_vma_remote(struct mm_struct *mm,
+static inline struct page *get_user_page_lookup_vma(struct mm_struct *mm,
 						    unsigned long addr,
 						    int gup_flags,
 						    struct vm_area_struct **vmap)
diff --git a/mm/memory.c b/mm/memory.c
index ff338c2abe92..3b86eeaf084f 100644
--- a/mm/memory.c
+++ b/mm/memory.c
@@ -7039,7 +7039,7 @@ static int __access_remote_vm(struct mm_struct *mm, unsigned long addr,
 		void *maddr;
 		struct folio *folio;
 		struct vm_area_struct *vma = NULL;
-		struct page *page = get_user_page_vma_remote(mm, addr,
+		struct page *page = get_user_page_lookup_vma(mm, addr,
 							     gup_flags, &vma);
 
 		if (IS_ERR(page)) {
@@ -7167,7 +7167,7 @@ static int __copy_remote_vm_str(struct mm_struct *mm, unsigned long addr,
 		struct page *page;
 		struct vm_area_struct *vma = NULL;
 
-		page = get_user_page_vma_remote(mm, addr, gup_flags, &vma);
+		page = get_user_page_lookup_vma(mm, addr, gup_flags, &vma);
 		if (IS_ERR(page)) {
 			/*
 			 * Treat as a total failure for now until we decide how
diff --git a/mm/rmap.c b/mm/rmap.c
index 1c77d5dc06e9..b36f2e219b8f 100644
--- a/mm/rmap.c
+++ b/mm/rmap.c
@@ -2838,7 +2838,7 @@ struct page *make_device_exclusive(struct mm_struct *mm, unsigned long addr,
 	 * (non-device-exclusive) PTE and issue a MMU_NOTIFY_EXCLUSIVE.
 	 */
 retry:
-	page = get_user_page_vma_remote(mm, addr,
+	page = get_user_page_lookup_vma(mm, addr,
 					FOLL_GET | FOLL_WRITE | FOLL_SPLIT_PMD,
 					&vma);
 	if (IS_ERR(page))
-- 
2.53.0-Meta

[PATCH RFC v4 04/12] mm/gup: let check_vma_flags() ignore selected VMA flags

[Rik van Riel]

check_vma_flags() rejects a VMA whose flags gup cannot handle. A caller
that wants to reach a COWed page in a VM_IO or VM_PFNMAP VMA, where that
page does have a struct page, cannot express the exception and has to
repeat the flag test itself.

Add an ignore_flags argument that check_vma_flags() clears from its local
copy of vma->vm_flags before the flag checks. A caller can drop a single
rejection this way while keeping every other check. All current callers
pass 0.

This prepares for get_user_page_vma(), which passes VM_IO | VM_PFNMAP to
reach those COWed pages.

No functional change intended.

Assisted-by: Claude:claude-opus-4.8
Signed-off-by: Rik van Riel <riel@surriel.com>
---
 mm/gup.c | 14 +++++++++++---
 1 file changed, 11 insertions(+), 3 deletions(-)

diff --git a/mm/gup.c b/mm/gup.c
index 0692119b7904..dcece7b5253c 100644
--- a/mm/gup.c
+++ b/mm/gup.c
@@ -1197,13 +1197,21 @@ static bool writable_file_mapping_allowed(struct vm_area_struct *vma,
 	return !vma_needs_dirty_tracking(vma);
 }
 
-static int check_vma_flags(struct vm_area_struct *vma, unsigned long gup_flags)
+static int check_vma_flags(struct vm_area_struct *vma, unsigned long gup_flags,
+			   vm_flags_t ignore_flags)
 {
 	vm_flags_t vm_flags = vma->vm_flags;
 	int write = (gup_flags & FOLL_WRITE);
 	int foreign = (gup_flags & FOLL_REMOTE);
 	bool vma_anon = vma_is_anonymous(vma);
 
+	/*
+	 * Opt out of the flag checks that read this local copy (the
+	 * VM_IO/VM_PFNMAP gate and the write/read/cow bits); checks that
+	 * re-read vma->vm_flags through helpers are unaffected.
+	 */
+	vm_flags &= ~ignore_flags;
+
 	if (vm_flags & (VM_IO | VM_PFNMAP))
 		return -EFAULT;
 
@@ -1387,7 +1395,7 @@ static long __get_user_pages(struct mm_struct *mm,
 					ret = -ENOMEM;
 					goto out;
 				}
-				if (check_vma_flags(vma, gup_flags)) {
+				if (check_vma_flags(vma, gup_flags, 0)) {
 					ret = -EINVAL;
 					goto out;
 				}
@@ -1408,7 +1416,7 @@ static long __get_user_pages(struct mm_struct *mm,
 				ret = -EFAULT;
 				goto out;
 			}
-			ret = check_vma_flags(vma, gup_flags);
+			ret = check_vma_flags(vma, gup_flags, 0);
 			if (ret)
 				goto out;
 		}
-- 
2.53.0-Meta

[PATCH RFC v4 05/12] mm/gup: add get_user_page_vma() to fault in a page under a held lock

[Rik van Riel]

__access_remote_vm() needs a single page from a VMA it has already
looked up and locked, faulting it in when necessary, under either the
mmap lock or the per-VMA lock. get_user_pages_remote() does not fit: it
hard codes the mmap lock and re-looks-up the VMA, neither of which is
wanted here.

Add get_user_page_vma(), a simplified __get_user_pages() that walks the
page tables with follow_page_mask(), faults a missing page in with
faultin_page(), and on success returns it with a reference and the
caller's lock still held. Like __get_user_pages() it runs
check_vma_flags(), so callers need not pre-check the VMA.

A VM_IO/VM_PFNMAP VMA is the exception to that check: it can still hold
COWed pages that have a struct page, so follow_page_mask() is allowed to
look for one. Memory with no struct page -- a raw PFN, or a present PFN
reported as -EEXIST -- is returned as -EFAULT, so the caller can reach it
through vma->vm_ops->access().

The caller sets FOLL_VMA_LOCK when it holds the per-VMA lock rather than
the mmap lock, which reaches the fault code as FAULT_FLAG_VMA_LOCK.

Anything that cannot complete under the per-VMA lock -- a dropped fault,
a userfaultfd VMA (uffd assumes current is the faulting task), a hard
error, or ->access() memory -- releases the lock and returns -EAGAIN,
so the caller retries under the mmap lock.

faultin_page() reports these retries as -EAGAIN for both lock types;
only the mmap caller records the dropped lock in *locked.

A VM_FAULT_ERROR decoding to no errno now returns -EFAULT instead of
BUG(), warning under CONFIG_DEBUG_VM. This relaxes the same case on the
existing __get_user_pages() path too.

Assisted-by: Claude:claude-opus-4.8
Signed-off-by: Rik van Riel <riel@surriel.com>
---
 mm/gup.c      | 176 +++++++++++++++++++++++++++++++++++++++++++-------
 mm/internal.h |   8 ++-
 2 files changed, 159 insertions(+), 25 deletions(-)

diff --git a/mm/gup.c b/mm/gup.c
index dcece7b5253c..cb7245b7542f 100644
--- a/mm/gup.c
+++ b/mm/gup.c
@@ -1080,9 +1080,16 @@ static int get_gate_page(struct mm_struct *mm, unsigned long address,
 }
 
 /*
- * mmap_lock must be held on entry.  If @flags has FOLL_UNLOCKABLE but not
- * FOLL_NOWAIT, the mmap_lock may be released.  If it is, *@locked will be set
- * to 0 and -EBUSY returned.
+ * The caller holds either the mmap lock, or the per-VMA lock (with
+ * FOLL_VMA_LOCK), on entry. If @flags has FOLL_UNLOCKABLE but not FOLL_NOWAIT,
+ * the mmap_lock may be released. If it is, *@locked will be set to 0 and
+ * -EAGAIN returned.
+ *
+ * The return value does not depend on the lock type: a fault that made
+ * progress but needs a retry (VM_FAULT_RETRY / VM_FAULT_COMPLETED) is reported
+ * as -EAGAIN for both the mmap lock and the per-VMA lock (FOLL_VMA_LOCK). Only
+ * the *@locked side effect is lock-type specific, as the per-VMA lock path has
+ * no unlockable mmap_lock to drop.
  */
 static int faultin_page(struct vm_area_struct *vma,
 		unsigned long address, unsigned int flags, bool unshare,
@@ -1097,6 +1104,8 @@ static int faultin_page(struct vm_area_struct *vma,
 		fault_flags |= FAULT_FLAG_WRITE;
 	if (flags & FOLL_REMOTE)
 		fault_flags |= FAULT_FLAG_REMOTE;
+	if (flags & FOLL_VMA_LOCK)
+		fault_flags |= FAULT_FLAG_VMA_LOCK;
 	if (flags & FOLL_UNLOCKABLE) {
 		fault_flags |= FAULT_FLAG_ALLOW_RETRY | FAULT_FLAG_KILLABLE;
 		/*
@@ -1125,41 +1134,160 @@ static int faultin_page(struct vm_area_struct *vma,
 
 	ret = handle_mm_fault(vma, address, fault_flags, NULL);
 
+	/*
+	 * A fully completed fault (VM_FAULT_COMPLETED) or one that needs a retry
+	 * (VM_FAULT_RETRY) has released the lock it was holding. Report both as
+	 * -EAGAIN so the caller retries: the mmap lock caller retakes it here,
+	 * the per-VMA lock caller (FOLL_VMA_LOCK) falls back to the mmap lock.
+	 *
+	 * Dropping the mmap lock is recorded in *@locked. There is no such lock
+	 * to drop under the per-VMA lock, where @locked is not used, so leave it
+	 * alone in that case.
+	 */
 	if (ret & VM_FAULT_COMPLETED) {
-		/*
-		 * With FAULT_FLAG_RETRY_NOWAIT we'll never release the
-		 * mmap lock in the page fault handler. Sanity check this.
-		 */
-		WARN_ON_ONCE(fault_flags & FAULT_FLAG_RETRY_NOWAIT);
-		*locked = 0;
-
-		/*
-		 * We should do the same as VM_FAULT_RETRY, but let's not
-		 * return -EBUSY since that's not reflecting the reality of
-		 * what has happened - we've just fully completed a page
-		 * fault, with the mmap lock released.  Use -EAGAIN to show
-		 * that we want to take the mmap lock _again_.
-		 */
+		if (!(flags & FOLL_VMA_LOCK)) {
+			/*
+			 * With FAULT_FLAG_RETRY_NOWAIT we'll never release the
+			 * mmap lock in the page fault handler. Sanity check this.
+			 */
+			WARN_ON_ONCE(fault_flags & FAULT_FLAG_RETRY_NOWAIT);
+			*locked = 0;
+		}
 		return -EAGAIN;
 	}
 
 	if (ret & VM_FAULT_ERROR) {
 		int err = vm_fault_to_errno(ret, flags);
 
-		if (err)
-			return err;
-		BUG();
+		/*
+		 * VM_FAULT_ERROR always decodes to an errno; a zero here would
+		 * mean handle_mm_fault() returned an unexpected combination.
+		 * Report -EFAULT rather than crash: under the per-VMA lock the
+		 * mmap lock retry produces the definitive result.
+		 */
+		VM_WARN_ON_ONCE(!err);
+		return err ? err : -EFAULT;
 	}
 
 	if (ret & VM_FAULT_RETRY) {
-		if (!(fault_flags & FAULT_FLAG_RETRY_NOWAIT))
+		if (!(flags & FOLL_VMA_LOCK) &&
+		    !(fault_flags & FAULT_FLAG_RETRY_NOWAIT))
 			*locked = 0;
-		return -EBUSY;
+		return -EAGAIN;
 	}
 
 	return 0;
 }
 
+/*
+ * get_user_page_vma - get one page from @vma, whose lock the caller already
+ * holds: the mmap lock, or (with FOLL_VMA_LOCK) the per-VMA lock. Walks the
+ * page tables, faulting the page in if needed, and on success returns it with
+ * a reference and the lock still held.
+ *
+ * Runs check_vma_flags() like __get_user_pages(), so callers need not pre-check
+ * the VMA; most rejections are returned as their error. A VM_IO/VM_PFNMAP VMA
+ * is the exception: a COWed page with a struct page is returned, while a raw
+ * PFN has none and yields -EFAULT, to be reached via vma->vm_ops->access().
+ *
+ * Under FOLL_VMA_LOCK, anything that cannot be finished under the per-VMA lock
+ * (a dropped fault, userfaultfd, a hard error, or ->access() memory) releases
+ * the lock and returns -EAGAIN, so the caller retries under the mmap lock.
+ */
+struct page *get_user_page_vma(struct vm_area_struct *vma, unsigned long addr,
+			       unsigned int gup_flags)
+{
+	bool vma_locked = gup_flags & FOLL_VMA_LOCK;
+	unsigned long page_mask;
+	struct page *page;
+	int locked = 1;
+	bool pfnmap;
+	int ret;
+
+	/*
+	 * Two lock modes are supported: the mmap lock, with neither flag, or
+	 * the per-VMA lock, with both FOLL_VMA_LOCK and FOLL_UNLOCKABLE. An
+	 * unlockable mmap fault would drop the lock and report it in *locked,
+	 * which this function does not relay, so reject a lone flag.
+	 */
+	VM_WARN_ON_ONCE(!(gup_flags & FOLL_VMA_LOCK) != !(gup_flags & FOLL_UNLOCKABLE));
+
+	/*
+	 * Validate the VMA up front, like __get_user_pages(). A VM_IO/VM_PFNMAP
+	 * VMA is not rejected outright: it can hold COWed pages that have a
+	 * struct page; let follow_page_mask() look for them, and treat only
+	 * its struct-page-less PFNs as unreachable. Other rejections
+	 * (secretmem, permissions, etc) result in immediate failure.
+	 */
+	ret = check_vma_flags(vma, gup_flags, VM_IO | VM_PFNMAP);
+	if (ret)
+		goto fail;
+	pfnmap = vma->vm_flags & (VM_IO | VM_PFNMAP);
+
+	for (;;) {
+		if (fatal_signal_pending(current)) {
+			ret = -EINTR;
+			goto fail;
+		}
+		cond_resched();
+
+		/* follow_page_mask() requires @page_mask; it is unused here. */
+		page = follow_page_mask(vma, addr,
+					gup_flags | FOLL_TOUCH | FOLL_GET,
+					&page_mask);
+		if (!IS_ERR_OR_NULL(page)) {
+			/* Match __get_user_pages(): flush for VIVT/aliasing caches. */
+			flush_anon_page(vma, page, addr);
+			flush_dcache_page(page);
+			return page;
+		}
+
+		/*
+		 * No struct page: a raw PFN of a VM_IO/VM_PFNMAP VMA, whether
+		 * seen by the up-front check (@pfnmap) or reported as -EEXIST
+		 * for a present PFN. Return -EFAULT so the caller reaches it
+		 * through vma->vm_ops->access().
+		 */
+		if (pfnmap || PTR_ERR(page) == -EEXIST) {
+			ret = -EFAULT;
+			goto fail;
+		}
+		/* A hard error from the walk itself. */
+		if (page && PTR_ERR(page) != -EMLINK) {
+			ret = PTR_ERR(page);
+			goto fail;
+		}
+
+		/*
+		 * The page is not present, or needs unsharing. A remote fault
+		 * under the per-VMA lock cannot deliver userfaultfd (which
+		 * assumes current is the faulting task), so fall back for those.
+		 */
+		if (vma_locked && userfaultfd_armed(vma)) {
+			ret = -EAGAIN;
+			goto fail;
+		}
+		ret = faultin_page(vma, addr, gup_flags | FOLL_REMOTE | FOLL_GET,
+				   PTR_ERR(page) == -EMLINK, &locked);
+		if (ret == -EAGAIN)
+			return ERR_PTR(-EAGAIN);	/* fault released the per-VMA lock */
+		if (ret)
+			goto fail;
+	}
+
+fail:
+	/*
+	 * Under the per-VMA lock the caller cannot reach ->access() or act on a
+	 * hard error (both need the mmap lock), so release the lock and have it
+	 * retry there; the mmap-lock pass produces the definitive error.
+	 */
+	if (vma_locked) {
+		vma_end_read(vma);
+		return ERR_PTR(-EAGAIN);
+	}
+	return ERR_PTR(ret);
+}
+
 /*
  * Writing to file-backed mappings which require folio dirty tracking using GUP
  * is a fundamentally broken operation, as kernel write access to GUP mappings
@@ -1197,8 +1325,8 @@ static bool writable_file_mapping_allowed(struct vm_area_struct *vma,
 	return !vma_needs_dirty_tracking(vma);
 }
 
-static int check_vma_flags(struct vm_area_struct *vma, unsigned long gup_flags,
-			   vm_flags_t ignore_flags)
+int check_vma_flags(struct vm_area_struct *vma, unsigned long gup_flags,
+		    vm_flags_t ignore_flags)
 {
 	vm_flags_t vm_flags = vma->vm_flags;
 	int write = (gup_flags & FOLL_WRITE);
diff --git a/mm/internal.h b/mm/internal.h
index 181e79f1d6a2..706f7f08fd18 100644
--- a/mm/internal.h
+++ b/mm/internal.h
@@ -1595,6 +1595,10 @@ struct vm_struct *__get_vm_area_node(unsigned long size,
  */
 int __must_check try_grab_folio(struct folio *folio, int refs,
 				unsigned int flags);
+int check_vma_flags(struct vm_area_struct *vma, unsigned long gup_flags,
+		    vm_flags_t ignore_flags);
+struct page *get_user_page_vma(struct vm_area_struct *vma, unsigned long addr,
+			       unsigned int gup_flags);
 
 /*
  * mm/huge_memory.c
@@ -1641,11 +1645,13 @@ enum {
 	FOLL_UNLOCKABLE = 1 << 21,
 	/* VMA lookup+checks compatible with MADV_POPULATE_(READ|WRITE) */
 	FOLL_MADV_POPULATE = 1 << 22,
+	/* caller holds the per-VMA lock, not the mmap lock */
+	FOLL_VMA_LOCK = 1 << 23,
 };
 
 #define INTERNAL_GUP_FLAGS (FOLL_TOUCH | FOLL_TRIED | FOLL_REMOTE | FOLL_PIN | \
 			    FOLL_FAST_ONLY | FOLL_UNLOCKABLE | \
-			    FOLL_MADV_POPULATE)
+			    FOLL_MADV_POPULATE | FOLL_VMA_LOCK)
 
 /*
  * Indicates for which pages that are write-protected in the page table,
-- 
2.53.0-Meta

[PATCH RFC v4 06/12] mm: use per-VMA lock in __access_remote_vm() for single-VMA accesses

[Rik van Riel]

__access_remote_vm() holds mmap_read_lock() for the whole transfer. On
large machines running big multi-threaded applications, that lock is
contended between readers and writers: an mmap() or munmap() stalls readers
like /proc/PID/cmdline, /proc/PID/environ, or /proc/PID/mem, even though the
memory they read is almost always resident.

Use the per-VMA lock to access a remote process's memory when the access
fits in one VMA. Fall back to the mmap_lock if the access crosses VMA
boundaries, or when get_user_page_vma() cannot finish the access under the
per-VMA lock.

Factor the walk into remote_vm_walk(): it selects the lock, faults in each
page with get_user_page_vma(), and hands the page to a per-page action.
__access_remote_vm() passes access_vm_page(), which copies it; a later
patch shares the walk for the remote string reader.

remote_access_lock() and remote_access_unlock() keep the lock selection out
of the walk. remote_access_lock() takes the per-VMA lock when the range
fits one VMA whose flags permit the access, and the mmap lock otherwise,
returning an ERR_PTR() when the mmap lock cannot be taken.

Looking up the VMA first untags the remote address with
untagged_addr_remote_unlocked(), added earlier in this series, so the untag
needs no mmap lock.

Walking the page tables under only the per-VMA lock is safe against both
page table freeing and THP collapse.

munmap() frees a VMA's page tables through free_pgtables(), which uses
neither the page table lock nor RCU. But it runs only after the VMA is
detached under the VMA write lock, which our per-VMA read lock excludes, so
no page table of this VMA is torn down under the walk.

THP collapse instead needs neither lock: file-backed collapse retracts a
page table under i_mmap_lock and the page table lock alone. It frees the
retracted PTE page by RCU, and pte_offset_map() holds the RCU read lock, so
the page stays valid for the walk.

follow_page_pte() takes that same page table lock through
pte_offset_map_lock() and rechecks the pmd, so it either walks an intact
table or sees the collapsed pmd and faults the THP in.

get_user_page_vma() returns -EFAULT for memory with no struct page: the raw
PFNs of a VM_IO/VM_PFNMAP VMA, ioremapped device memory reached through
ptrace and /proc/PID/mem.

access_vm_page() reaches that memory under the mmap lock through
vma->vm_ops->access(), via a new access_remote_vma_ops() helper, as
get_user_pages_remote() and the old ->access() fallback did before.

A COWed page in such a VMA now reads normally through get_user_page_vma();
previously it was routed to ->access(), whose generic_access_phys() rejects
the ioremap of a RAM page.

A single-threaded microbenchmark reading a remote process's memory through
/proc/PID/mem shows the per-VMA path costs less than the old
mmap_read_lock() plus get_user_pages_remote() route. The gain grows with
the read size as the per-page VMA re-lookup drops away. Median of three
pinned repetitions in a VM:

	read size   baseline    per-VMA     throughput
	8 B          201.7 ns    170.6 ns    ~15% faster
	4 KB         495 ns      442 ns      +12% (8266 -> 9265 MB/s)
	64 KB       4115 ns     3355 ns      +23% (15927 -> 19536 MB/s)
	1 MB       75902 ns    63641 ns      +19% (13815 -> 16477 MB/s)

This does not measure the multi-threaded reader-versus-writer contention
that motivates the per-VMA lock; that case remains to be quantified.

Assisted-by: Claude:claude-opus-4.8
Suggested-by: Suren Baghdasaryan <surenb@google.com>
Signed-off-by: Rik van Riel <riel@surriel.com>
---
 mm/memory.c | 268 +++++++++++++++++++++++++++++++++++++++++-----------
 1 file changed, 213 insertions(+), 55 deletions(-)

diff --git a/mm/memory.c b/mm/memory.c
index 3b86eeaf084f..273dfe12bc6d 100644
--- a/mm/memory.c
+++ b/mm/memory.c
@@ -7015,86 +7015,244 @@ EXPORT_SYMBOL_GPL(generic_access_phys);
 #endif
 
 /*
- * Access another process' address space as given in mm.
+ * VM_IO / VM_PFNMAP memory, such as an ioremapped device mapping, maps
+ * PFNs that have no struct page, so get_user_page_vma() cannot fetch it
+ * even though the page tables are populated. It can still be reached
+ * through vma->vm_ops->access().
+ *
+ * Returns the number of bytes transferred, or <= 0 if @vma cannot be
+ * accessed this way.
  */
-static int __access_remote_vm(struct mm_struct *mm, unsigned long addr,
-			      void *buf, int len, unsigned int gup_flags)
+static int access_remote_vma_ops(struct vm_area_struct *vma, unsigned long addr,
+				 void *buf, int len, int write)
+{
+#ifdef CONFIG_HAVE_IOREMAP_PROT
+	if (vma->vm_ops && vma->vm_ops->access)
+		return vma->vm_ops->access(vma, addr, buf, len, write);
+#endif
+	return 0;
+}
+
+/*
+ * Lock @mm to reach the remote range [@addr, @addr + @len).
+ *
+ * Take the per-VMA lock when the whole range fits in a single VMA whose
+ * flags permit the access. The RCU freed page tables then keep page table
+ * memory from being reused with unexpected contents while the lock is held.
+ * Otherwise fall back to the mmap lock, which also covers multi-VMA ranges,
+ * stack expansion, and ->access() memory.
+ *
+ * Return whether the mmap lock is held. The per-VMA locked VMA, when one is
+ * taken, is stored in *@vmap; it is NULL on the mmap lock path. *@vmap is
+ * set to an ERR_PTR() when the mmap lock could not be taken, so callers must
+ * check IS_ERR(*@vmap) before using either result.
+ */
+static bool remote_access_lock(struct mm_struct *mm, unsigned long addr,
+			       int len, unsigned int gup_flags,
+			       struct vm_area_struct **vmap)
+{
+	struct vm_area_struct *vma = NULL;
+
+#if defined(CONFIG_PER_VMA_LOCK) && defined(CONFIG_MMU_GATHER_RCU_TABLE_FREE)
+	vma = lock_vma_under_rcu(mm, addr);
+	if (vma) {
+		/* addr + len must not wrap, and must fit within the one VMA. */
+		if (addr + len < addr || addr + len > vma->vm_end ||
+		    check_vma_flags(vma, gup_flags, 0)) {
+			vma_end_read(vma);
+			vma = NULL;
+		}
+	}
+#endif
+
+	if (!vma) {
+		if (mmap_read_lock_killable(mm)) {
+			*vmap = ERR_PTR(-EINTR);
+			return false;
+		}
+		*vmap = NULL;
+		return true;
+	}
+
+	*vmap = vma;
+	return false;
+}
+
+/* Release the lock taken by remote_access_lock(). */
+static void remote_access_unlock(struct mm_struct *mm,
+				 struct vm_area_struct *vma, bool have_mmap_lock)
+{
+	if (have_mmap_lock)
+		mmap_read_unlock(mm);
+	else if (vma)
+		vma_end_read(vma);
+}
+
+/*
+ * Per-page action for a remote VM walk. Handle up to @len bytes at @addr on
+ * @page, advancing *@buf past the bytes read from or written to it. @page is
+ * NULL for struct-page-less memory (VM_IO / VM_PFNMAP) reached under the mmap
+ * lock.
+ *
+ * Return the number of source bytes handled at @addr, 0 to end the walk (a
+ * string reached its NUL, or ->access() memory could not be reached), or a
+ * negative errno to abort.
+ */
+typedef int (*remote_vm_action)(struct vm_area_struct *vma, struct page *page,
+				unsigned long addr, void **buf, int len,
+				int write);
+
+/*
+ * Walk the remote range [@addr, @addr + @len) of @mm, handing each page to
+ * @action. Use the per-VMA lock when the range fits one VMA, and fall back to
+ * the mmap lock for multi-VMA ranges, stack expansion (when @can_expand_stack
+ * is set), or when the per-VMA lock cannot finish a fault.
+ *
+ * Each page is faulted in with get_user_page_vma() under whichever lock is
+ * held. Return the number of bytes @action consumed; *@err is a negative
+ * errno when the walk aborted, else 0.
+ */
+static int remote_vm_walk(struct mm_struct *mm, unsigned long addr, void *buf,
+			  int len, unsigned int gup_flags, bool can_expand_stack,
+			  remote_vm_action action, int *err)
 {
 	void *old_buf = buf;
 	int write = gup_flags & FOLL_WRITE;
+	bool have_mmap_lock;
+	struct vm_area_struct *vma;
 
-	if (mmap_read_lock_killable(mm))
-		return 0;
+	*err = 0;
 
-	/* Untag the address before looking up the VMA */
-	addr = untagged_addr_remote(mm, addr);
+	/*
+	 * Set FOLL_REMOTE so check_vma_flags() applies the same protection key
+	 * rules as get_user_pages_remote() did: the current PKRU is not checked
+	 * against a VMA reached on @mm's behalf.
+	 */
+	gup_flags |= FOLL_REMOTE;
 
-	/* Avoid triggering the temporary warning in __get_user_pages */
-	if (!vma_lookup(mm, addr) && !expand_stack(mm, addr))
+	addr = untagged_addr_remote_unlocked(mm, addr);
+
+	have_mmap_lock = remote_access_lock(mm, addr, len, gup_flags, &vma);
+	if (IS_ERR(vma)) {
+		*err = -EFAULT;
 		return 0;
+	}
 
-	/* ignore errors, just check how much was successfully transferred */
 	while (len) {
-		int bytes, offset;
-		void *maddr;
-		struct folio *folio;
-		struct vm_area_struct *vma = NULL;
-		struct page *page = get_user_page_lookup_vma(mm, addr,
-							     gup_flags, &vma);
+		unsigned int foll_flags = gup_flags;
+		struct page *page;
+		int ret;
 
-		if (IS_ERR(page)) {
-			/* We might need to expand the stack to access it */
+		if (!vma || addr >= vma->vm_end) {
+			/* Any lookup here holds the mmap lock. */
+			VM_BUG_ON(!have_mmap_lock);
 			vma = vma_lookup(mm, addr);
-			if (!vma) {
+			if (!vma && can_expand_stack) {
+				/* expand_stack() drops the mmap lock if it fails */
 				vma = expand_stack(mm, addr);
-
-				/* mmap_lock was dropped on failure */
 				if (!vma)
-					return buf - old_buf;
-
-				/* Try again if stack expansion worked */
-				continue;
+					have_mmap_lock = false;
 			}
+			if (!vma) {
+				*err = -EFAULT;
+				break;
+			}
+		}
 
+		/*
+		 * FOLL_UNLOCKABLE lets the per-VMA fault retry, dropping the
+		 * lock, so the walk can fall back to the mmap lock.
+		 */
+		if (!have_mmap_lock)
+			foll_flags |= FOLL_VMA_LOCK | FOLL_UNLOCKABLE;
+
+		page = get_user_page_vma(vma, addr, foll_flags);
+		if (IS_ERR(page)) {
 			/*
-			 * Check if this is a VM_IO | VM_PFNMAP VMA, which
-			 * we can access using slightly different code.
+			 * get_user_page_vma() returns -EAGAIN, with the per-VMA
+			 * lock released, for anything it could not finish under
+			 * it; retake the mmap lock and retry. A different error
+			 * therefore only arrives under the mmap lock, where
+			 * struct-page-less memory can be reached via ->access().
 			 */
-			bytes = 0;
-#ifdef CONFIG_HAVE_IOREMAP_PROT
-			if (vma->vm_ops && vma->vm_ops->access)
-				bytes = vma->vm_ops->access(vma, addr, buf,
-							    len, write);
-#endif
-			if (bytes <= 0)
-				break;
-		} else {
-			folio = page_folio(page);
-			bytes = len;
-			offset = addr & (PAGE_SIZE-1);
-			if (bytes > PAGE_SIZE-offset)
-				bytes = PAGE_SIZE-offset;
-
-			maddr = kmap_local_folio(folio, folio_page_idx(folio, page) * PAGE_SIZE);
-			if (write) {
-				copy_to_user_page(vma, page, addr,
-						  maddr + offset, buf, bytes);
-				folio_mark_dirty_lock(folio);
-			} else {
-				copy_from_user_page(vma, page, addr,
-						    buf, maddr + offset, bytes);
+			if (PTR_ERR(page) == -EAGAIN) {
+				vma = NULL;
+				if (mmap_read_lock_killable(mm)) {
+					*err = -EFAULT;
+					break;
+				}
+				have_mmap_lock = true;
+				continue;
 			}
-			folio_release_kmap(folio, maddr);
+			if (WARN_ON_ONCE(!have_mmap_lock))
+				break;
+			page = NULL;
 		}
-		len -= bytes;
-		buf += bytes;
-		addr += bytes;
+
+		ret = action(vma, page, addr, &buf, len, write);
+		if (ret <= 0) {
+			if (ret < 0)
+				*err = ret;
+			break;
+		}
+		addr += ret;
+		len -= ret;
 	}
-	mmap_read_unlock(mm);
+
+	remote_access_unlock(mm, vma, have_mmap_lock);
 
 	return buf - old_buf;
 }
 
+/*
+ * Copy one page's worth of [@addr, @addr + @len) to or from *@buf. Reaches
+ * struct-page-less VM_IO / VM_PFNMAP memory through vma->vm_ops->access().
+ */
+static int access_vm_page(struct vm_area_struct *vma, struct page *page,
+			  unsigned long addr, void **buf, int len, int write)
+{
+	struct folio *folio;
+	int bytes, offset;
+	void *maddr;
+
+	if (!page) {
+		bytes = access_remote_vma_ops(vma, addr, *buf, len, write);
+		if (bytes > 0)
+			*buf += bytes;
+		return bytes;
+	}
+
+	bytes = len;
+	offset = addr & (PAGE_SIZE - 1);
+	if (bytes > PAGE_SIZE - offset)
+		bytes = PAGE_SIZE - offset;
+
+	folio = page_folio(page);
+	maddr = kmap_local_folio(folio, folio_page_idx(folio, page) * PAGE_SIZE);
+	if (write) {
+		copy_to_user_page(vma, page, addr, maddr + offset, *buf, bytes);
+		folio_mark_dirty_lock(folio);
+	} else {
+		copy_from_user_page(vma, page, addr, *buf, maddr + offset, bytes);
+	}
+	folio_release_kmap(folio, maddr);
+
+	*buf += bytes;
+	return bytes;
+}
+
+/*
+ * Access another process' address space as given in mm.
+ */
+static int __access_remote_vm(struct mm_struct *mm, unsigned long addr,
+			      void *buf, int len, unsigned int gup_flags)
+{
+	int err;
+
+	return remote_vm_walk(mm, addr, buf, len, gup_flags, true,
+			      access_vm_page, &err);
+}
+
 /**
  * access_remote_vm - access another process' address space
  * @mm:		the mm_struct of the target address space
-- 
2.53.0-Meta

[PATCH RFC v4 07/12] mm: read remote strings under the per-VMA lock

[Rik van Riel]

__copy_remote_vm_str() reads another process's memory under the mmap read
lock for the whole copy, and looks the VMA up again for every page through
get_user_page_lookup_vma().

This is the mmap-lock-only form that __access_remote_vm() used before it
moved to the per-VMA lock, and it shares the same contention: an mmap() or
munmap() stalls a bpf_copy_from_user_task_str() reader even though the
string is resident.

Read the string through remote_vm_walk(), the walk __access_remote_vm()
already uses, with a copy_vm_str() action that strscpy()s each page and
stops at the NUL. Bounding the copy by its maximum length keeps the per-VMA
path inside a single VMA, so no VMA is looked up again while that lock is
held.

A copy the per-VMA lock cannot finish falls back to the mmap lock: a fault
that dropped the lock, or a string running past the VMA. Memory with no
struct page (VM_IO/VM_PFNMAP) and stack expansion stay on the mmap lock
path and still return -EFAULT, as before; ->access() support is left for
later.

Assisted-by: Claude:claude-opus-4.8
Signed-off-by: Rik van Riel <riel@surriel.com>
---
 mm/memory.c | 111 ++++++++++++++++++++++------------------------------
 1 file changed, 47 insertions(+), 64 deletions(-)

diff --git a/mm/memory.c b/mm/memory.c
index 273dfe12bc6d..aaf620017f34 100644
--- a/mm/memory.c
+++ b/mm/memory.c
@@ -7296,84 +7296,67 @@ EXPORT_SYMBOL_GPL(access_process_vm);
 
 #ifdef CONFIG_BPF_SYSCALL
 /*
- * Copy a string from another process's address space as given in mm.
- * If there is any error return -EFAULT.
+ * Copy a NUL-terminated string from @addr into *@buf, up to @len bytes,
+ * stopping at the NUL. strscpy() always NUL terminates, so recopy the last
+ * byte of a page when more pages follow. A string is never read from
+ * struct-page-less VM_IO / VM_PFNMAP memory.
  */
-static int __copy_remote_vm_str(struct mm_struct *mm, unsigned long addr,
-				void *buf, int len, unsigned int gup_flags)
+static int copy_vm_str(struct vm_area_struct *vma, struct page *page,
+		       unsigned long addr, void **buf, int len, int write)
 {
-	void *old_buf = buf;
-	int err = 0;
-
-	*(char *)buf = '\0';
+	struct folio *folio;
+	int bytes, offset, retval;
+	void *maddr;
 
-	if (mmap_read_lock_killable(mm))
+	if (!page)
 		return -EFAULT;
 
-	addr = untagged_addr_remote(mm, addr);
+	bytes = len;
+	offset = addr & (PAGE_SIZE - 1);
+	if (bytes > PAGE_SIZE - offset)
+		bytes = PAGE_SIZE - offset;
 
-	/* Avoid triggering the temporary warning in __get_user_pages */
-	if (!vma_lookup(mm, addr)) {
-		err = -EFAULT;
-		goto out;
+	folio = page_folio(page);
+	maddr = kmap_local_folio(folio, folio_page_idx(folio, page) * PAGE_SIZE);
+	retval = strscpy(*buf, maddr + offset, bytes);
+	if (retval >= 0) {
+		/* Found the end of the string. */
+		*buf += retval;
+		folio_release_kmap(folio, maddr);
+		return 0;
 	}
 
-	while (len) {
-		int bytes, offset, retval;
-		void *maddr;
-		struct folio *folio;
-		struct page *page;
-		struct vm_area_struct *vma = NULL;
+	*buf += bytes - 1;
+	if (bytes != len) {
+		copy_from_user_page(vma, page, addr + bytes - 1, *buf,
+				    maddr + (PAGE_SIZE - 1), 1);
+		*buf += 1;
+	}
+	folio_release_kmap(folio, maddr);
 
-		page = get_user_page_lookup_vma(mm, addr, gup_flags, &vma);
-		if (IS_ERR(page)) {
-			/*
-			 * Treat as a total failure for now until we decide how
-			 * to handle the CONFIG_HAVE_IOREMAP_PROT case and
-			 * stack expansion.
-			 */
-			*(char *)buf = '\0';
-			err = -EFAULT;
-			goto out;
-		}
+	return bytes;
+}
 
-		folio = page_folio(page);
-		bytes = len;
-		offset = addr & (PAGE_SIZE - 1);
-		if (bytes > PAGE_SIZE - offset)
-			bytes = PAGE_SIZE - offset;
-
-		maddr = kmap_local_folio(folio, folio_page_idx(folio, page) * PAGE_SIZE);
-		retval = strscpy(buf, maddr + offset, bytes);
-		if (retval >= 0) {
-			/* Found the end of the string */
-			buf += retval;
-			folio_release_kmap(folio, maddr);
-			break;
-		}
+/*
+ * Copy a string from another process's address space as given in mm.
+ * If there is any error return -EFAULT.
+ */
+static int __copy_remote_vm_str(struct mm_struct *mm, unsigned long addr,
+				void *buf, int len, unsigned int gup_flags)
+{
+	int bytes, err;
 
-		buf += bytes - 1;
-		/*
-		 * Because strscpy always NUL terminates we need to
-		 * copy the last byte in the page if we are going to
-		 * load more pages
-		 */
-		if (bytes != len) {
-			addr += bytes - 1;
-			copy_from_user_page(vma, page, addr, buf, maddr + (PAGE_SIZE - 1), 1);
-			buf += 1;
-			addr += 1;
-		}
-		len -= bytes;
+	*(char *)buf = '\0';
 
-		folio_release_kmap(folio, maddr);
+	bytes = remote_vm_walk(mm, addr, buf, len, gup_flags, false,
+			       copy_vm_str, &err);
+	if (err) {
+		/* The contract guarantees a terminated buffer even on error. */
+		((char *)buf)[bytes] = '\0';
+		return err;
 	}
 
-out:
-	mmap_read_unlock(mm);
-	if (err)
-		return err;
-	return buf - old_buf;
+	return bytes;
 }
 
 /**
-- 
2.53.0-Meta

[PATCH RFC v4 08/12] selftests/mm: cover /proc/pid/mem access to VM_PFNMAP memory

[Rik van Riel]

Reading a VM_PFNMAP mapping through /proc/pid/mem exercises
__access_remote_vm() two ways: a COWed page has a struct page and is
returned by get_user_page_vma(), while a raw PFN has none and is reached
through vma->vm_ops->access().

Add two tests to pfnmap.c, both reading VM_PFNMAP memory through
/proc/self/mem.

procmem_cow_read maps the file MAP_PRIVATE and writable, writes to COW a
page, then reads it back. Without the struct-page path in
get_user_page_vma() this read is short: the access falls back to
generic_access_phys(), which ioremaps the PFN, and ioremap of a COWed RAM
page is rejected.

procmem_pfn_read reads a raw PFN back through ->access(). ioremap rejects
RAM, so it runs only for genuine device memory and is skipped for the
default /dev/mem System RAM target.

Assisted-by: Claude:claude-opus-4.8
Signed-off-by: Rik van Riel <riel@surriel.com>
---
 tools/testing/selftests/mm/pfnmap.c | 66 +++++++++++++++++++++++++++++
 1 file changed, 66 insertions(+)

diff --git a/tools/testing/selftests/mm/pfnmap.c b/tools/testing/selftests/mm/pfnmap.c
index 4f550822385a..6ff5d1029517 100644
--- a/tools/testing/selftests/mm/pfnmap.c
+++ b/tools/testing/selftests/mm/pfnmap.c
@@ -31,6 +31,7 @@ static sigjmp_buf sigjmp_buf_env;
 static char *file = "/dev/mem";
 static off_t file_offset;
 static int fd;
+static int target_is_ram;
 
 static void signal_handler(int sig)
 {
@@ -113,6 +114,7 @@ static void pfnmap_init(void)
 		if (err)
 			ksft_exit_skip("Cannot find ram target in '/proc/iomem': %s\n",
 				       strerror(-err));
+		target_is_ram = 1;
 	} else {
 		file_offset = 0;
 	}
@@ -271,6 +273,70 @@ TEST_F(pfnmap, fork)
 	ASSERT_EQ(ret, 0);
 }
 
+TEST_F(pfnmap, procmem_cow_read)
+{
+	char *priv, *buf;
+	ssize_t rc;
+	int mem_fd;
+
+	/*
+	 * A COWed page in a VM_PFNMAP mapping has a struct page, so reading it
+	 * through /proc/self/mem -- __access_remote_vm() -> get_user_page_vma()
+	 * -- returns it directly, instead of routing to vma->vm_ops->access(),
+	 * which ioremaps the PFN and cannot reach a COWed RAM page.
+	 *
+	 * Map the file MAP_PRIVATE and writable, write to COW a page into anon
+	 * memory, then read the page back through /proc/self/mem.
+	 */
+	self->size2 = self->pagesize;
+	self->addr2 = mmap(NULL, self->size2, PROT_READ | PROT_WRITE,
+			   MAP_PRIVATE, fd, file_offset);
+	if (self->addr2 == MAP_FAILED)
+		SKIP(return, "Cannot create a writable private pfnmap mapping");
+	priv = self->addr2;
+
+	/* COW the page and stamp known bytes into the anon copy. */
+	priv[0] = 0x42;
+	priv[self->pagesize - 1] = 0x24;
+
+	buf = malloc(self->pagesize);
+	ASSERT_NE(buf, NULL);
+
+	mem_fd = open("/proc/self/mem", O_RDONLY);
+	ASSERT_GE(mem_fd, 0);
+	rc = pread(mem_fd, buf, self->pagesize, (off_t)(uintptr_t)priv);
+	close(mem_fd);
+
+	ASSERT_EQ(rc, (ssize_t)self->pagesize);
+	EXPECT_EQ(buf[0], 0x42);
+	EXPECT_EQ(buf[self->pagesize - 1], 0x24);
+
+	free(buf);
+}
+
+TEST_F(pfnmap, procmem_pfn_read)
+{
+	char buf[64];
+	ssize_t rc;
+	int mem_fd;
+
+	/*
+	 * A raw PFN of a VM_IO/VM_PFNMAP mapping has no struct page, so
+	 * __access_remote_vm() reaches it through vma->vm_ops->access()
+	 * (generic_access_phys()). That ioremaps the PFN, which is rejected for
+	 * RAM, so this only applies to genuine device memory.
+	 */
+	if (target_is_ram)
+		SKIP(return, "Target is System RAM; ->access() cannot ioremap RAM");
+
+	mem_fd = open("/proc/self/mem", O_RDONLY);
+	ASSERT_GE(mem_fd, 0);
+	rc = pread(mem_fd, buf, sizeof(buf), (off_t)(uintptr_t)self->addr1);
+	close(mem_fd);
+
+	ASSERT_EQ(rc, (ssize_t)sizeof(buf));
+}
+
 int main(int argc, char **argv)
 {
 	for (int i = 1; i < argc; i++) {
-- 
2.53.0-Meta

[PATCH RFC v4 09/12] mm/gup: build get_user_page_lookup_vma() on get_user_page_vma()

[Rik van Riel]

get_user_page_lookup_vma() faults in one page of a remote mm and returns it
together with the VMA that covers it. It open-codes that with
get_user_pages_remote() followed by vma_lookup(), duplicating the
single-page walk that get_user_page_vma() now provides.

Every caller already holds the mmap lock, and the helper needs it anyway:
get_user_pages_remote(locked=NULL) and vma_lookup() both rely on it.

Look the VMA up under the held mmap lock and fault the page in through
get_user_page_vma() without FOLL_VMA_LOCK, so the lock is held for the
whole call and never dropped. Force FOLL_REMOTE and FOLL_TOUCH to
preserve the foreign-access permission check and accessed-bit behavior
that get_user_pages_remote() applied before.

Add mmap_assert_locked() so the lock requirement is documented and any
future caller that forgets it trips the assertion rather than walking page
tables unlocked.

Reject FOLL_UNLOCKABLE alongside the existing FOLL_NOWAIT check: both let
the fault handler drop the mmap lock, which would invalidate the VMA looked
up here. get_user_page_vma() passes neither, so the lock is held for the
whole walk and the returned VMA stays valid.

The one behavioral change is that the old path could drop and retake the
mmap lock across a fault, while get_user_page_vma() holds it throughout.
None of the callers rely on the lock being dropped.

Assisted-by: Claude:claude-opus-4.8
Signed-off-by: Rik van Riel <riel@surriel.com>
---
 include/linux/mm.h | 32 +++-----------------------------
 mm/gup.c           | 35 +++++++++++++++++++++++++++++++++++
 2 files changed, 38 insertions(+), 29 deletions(-)

diff --git a/include/linux/mm.h b/include/linux/mm.h
index 24ead14b4790..0f66d76e6ca7 100644
--- a/include/linux/mm.h
+++ b/include/linux/mm.h
@@ -3235,35 +3235,9 @@ long pin_user_pages_remote(struct mm_struct *mm,
 			   unsigned int gup_flags, struct page **pages,
 			   int *locked);
 
-/*
- * Retrieves a single page alongside its VMA. Does not support FOLL_NOWAIT.
- */
-static inline struct page *get_user_page_lookup_vma(struct mm_struct *mm,
-						    unsigned long addr,
-						    int gup_flags,
-						    struct vm_area_struct **vmap)
-{
-	struct page *page;
-	struct vm_area_struct *vma;
-	int got;
-
-	if (WARN_ON_ONCE(unlikely(gup_flags & FOLL_NOWAIT)))
-		return ERR_PTR(-EINVAL);
-
-	got = get_user_pages_remote(mm, addr, 1, gup_flags, &page, NULL);
-
-	if (got < 0)
-		return ERR_PTR(got);
-
-	vma = vma_lookup(mm, addr);
-	if (WARN_ON_ONCE(!vma)) {
-		put_page(page);
-		return ERR_PTR(-EINVAL);
-	}
-
-	*vmap = vma;
-	return page;
-}
+struct page *get_user_page_lookup_vma(struct mm_struct *mm, unsigned long addr,
+				      int gup_flags,
+				      struct vm_area_struct **vmap);
 
 long get_user_pages(unsigned long start, unsigned long nr_pages,
 		    unsigned int gup_flags, struct page **pages);
diff --git a/mm/gup.c b/mm/gup.c
index cb7245b7542f..7f4107f7ff10 100644
--- a/mm/gup.c
+++ b/mm/gup.c
@@ -1288,6 +1288,41 @@ struct page *get_user_page_vma(struct vm_area_struct *vma, unsigned long addr,
 	return ERR_PTR(ret);
 }
 
+/*
+ * get_user_page_lookup_vma - fault in one page of a remote mm and hand back the
+ * page along with the VMA that covers it. The caller must hold the mmap_lock.
+ * Returns with the mmap_lock still held.
+ *
+ * Looks up the VMA, and gets a reference to the page through
+ * get_user_page_vma(), faulting in the page if needed.
+ *
+ * FOLL_NOWAIT and FOLL_UNLOCKABLE are rejected: both let the fault handler
+ * drop the mmap lock, which could invalidate the looked-up VMA.
+ */
+struct page *get_user_page_lookup_vma(struct mm_struct *mm, unsigned long addr,
+				      int gup_flags,
+				      struct vm_area_struct **vmap)
+{
+	struct vm_area_struct *vma;
+	struct page *page;
+
+	if (WARN_ON_ONCE(unlikely(gup_flags & (FOLL_NOWAIT | FOLL_UNLOCKABLE))))
+		return ERR_PTR(-EINVAL);
+
+	mmap_assert_locked(mm);
+
+	vma = vma_lookup(mm, addr);
+	if (!vma)
+		return ERR_PTR(-EFAULT);
+
+	page = get_user_page_vma(vma, addr, gup_flags | FOLL_REMOTE | FOLL_TOUCH);
+	if (IS_ERR(page))
+		return page;
+
+	*vmap = vma;
+	return page;
+}
+
 /*
  * Writing to file-backed mappings which require folio dirty tracking using GUP
  * is a fundamentally broken operation, as kernel write access to GUP mappings
-- 
2.53.0-Meta

[PATCH RFC v4 10/12] mm/gup: pass an end address to follow_page_mask() and return a page count

[Rik van Riel]

follow_page_mask() reports how many pages the returned page covers with a
page_mask: a bitmask of the enclosing naturally-aligned huge page. Callers
turn that into a stride, which assumes the run is a power-of-two block
aligned to its own size.

That form cannot describe an arbitrary contiguous run, which a later patch
needs to batch PTE-mapped large folios.

Replace the page_mask output with nr_pages, the number of contiguous pages
the returned page covers starting at @address, and add an @end argument
bounding how far the walk looks, so a small request does not scan an entire
large folio.

The huge PMD and PUD paths report the same stride as before, now as a count
clamped to @end. __get_user_pages() uses the count directly.
get_user_page_vma() passes @address + PAGE_SIZE and still returns one page.

No functional change intended.

Assisted-by: Claude:claude-opus-4.8
Signed-off-by: Rik van Riel <riel@surriel.com>
---
 mm/gup.c | 102 ++++++++++++++++++++++++++++++-------------------------
 1 file changed, 56 insertions(+), 46 deletions(-)

diff --git a/mm/gup.c b/mm/gup.c
index 7f4107f7ff10..96b00ddf7e04 100644
--- a/mm/gup.c
+++ b/mm/gup.c
@@ -647,8 +647,9 @@ static inline bool can_follow_write_pud(pud_t pud, struct page *page,
 }
 
 static struct page *follow_huge_pud(struct vm_area_struct *vma,
-				    unsigned long addr, pud_t *pudp,
-				    int flags, unsigned long *page_mask)
+				    unsigned long addr, unsigned long end,
+				    pud_t *pudp, int flags,
+				    unsigned long *nr_pages)
 {
 	struct mm_struct *mm = vma->vm_mm;
 	struct page *page;
@@ -672,10 +673,13 @@ static struct page *follow_huge_pud(struct vm_area_struct *vma,
 		return ERR_PTR(-EMLINK);
 
 	ret = try_grab_folio(page_folio(page), 1, flags);
-	if (ret)
+	if (ret) {
 		page = ERR_PTR(ret);
-	else
-		*page_mask = HPAGE_PUD_NR - 1;
+	} else {
+		unsigned long off = (addr & ~PUD_MASK) >> PAGE_SHIFT;
+
+		*nr_pages = min(HPAGE_PUD_NR - off, (end - addr) >> PAGE_SHIFT);
+	}
 
 	return page;
 }
@@ -699,9 +703,9 @@ static inline bool can_follow_write_pmd(pmd_t pmd, struct page *page,
 }
 
 static struct page *follow_huge_pmd(struct vm_area_struct *vma,
-				    unsigned long addr, pmd_t *pmd,
-				    unsigned int flags,
-				    unsigned long *page_mask)
+				    unsigned long addr, unsigned long end,
+				    pmd_t *pmd, unsigned int flags,
+				    unsigned long *nr_pages)
 {
 	struct mm_struct *mm = vma->vm_mm;
 	pmd_t pmdval = *pmd;
@@ -738,23 +742,25 @@ static struct page *follow_huge_pmd(struct vm_area_struct *vma,
 #endif	/* CONFIG_TRANSPARENT_HUGEPAGE */
 
 	page += (addr & ~HPAGE_PMD_MASK) >> PAGE_SHIFT;
-	*page_mask = HPAGE_PMD_NR - 1;
+	*nr_pages = min(HPAGE_PMD_NR - ((addr & ~HPAGE_PMD_MASK) >> PAGE_SHIFT),
+			(end - addr) >> PAGE_SHIFT);
 
 	return page;
 }
 
 #else  /* CONFIG_PGTABLE_HAS_HUGE_LEAVES */
 static struct page *follow_huge_pud(struct vm_area_struct *vma,
-				    unsigned long addr, pud_t *pudp,
-				    int flags, unsigned long *page_mask)
+				    unsigned long addr, unsigned long end,
+				    pud_t *pudp, int flags,
+				    unsigned long *nr_pages)
 {
 	return NULL;
 }
 
 static struct page *follow_huge_pmd(struct vm_area_struct *vma,
-				    unsigned long addr, pmd_t *pmd,
-				    unsigned int flags,
-				    unsigned long *page_mask)
+				    unsigned long addr, unsigned long end,
+				    pmd_t *pmd, unsigned int flags,
+				    unsigned long *nr_pages)
 {
 	return NULL;
 }
@@ -800,7 +806,8 @@ static inline bool can_follow_write_pte(pte_t pte, struct page *page,
 }
 
 static struct page *follow_page_pte(struct vm_area_struct *vma,
-		unsigned long address, pmd_t *pmd, unsigned int flags)
+		unsigned long address, unsigned long end, pmd_t *pmd,
+		unsigned int flags, unsigned long *nr_pages)
 {
 	struct mm_struct *mm = vma->vm_mm;
 	struct folio *folio;
@@ -885,6 +892,7 @@ static struct page *follow_page_pte(struct vm_area_struct *vma,
 		 */
 		folio_mark_accessed(folio);
 	}
+
 out:
 	pte_unmap_unlock(ptep, ptl);
 	return page;
@@ -896,9 +904,9 @@ static struct page *follow_page_pte(struct vm_area_struct *vma,
 }
 
 static struct page *follow_pmd_mask(struct vm_area_struct *vma,
-				    unsigned long address, pud_t *pudp,
-				    unsigned int flags,
-				    unsigned long *page_mask)
+				    unsigned long address, unsigned long end,
+				    pud_t *pudp, unsigned int flags,
+				    unsigned long *nr_pages)
 {
 	pmd_t *pmd, pmdval;
 	spinlock_t *ptl;
@@ -912,7 +920,7 @@ static struct page *follow_pmd_mask(struct vm_area_struct *vma,
 	if (!pmd_present(pmdval))
 		return no_page_table(vma, flags, address);
 	if (likely(!pmd_leaf(pmdval)))
-		return follow_page_pte(vma, address, pmd, flags);
+		return follow_page_pte(vma, address, end, pmd, flags, nr_pages);
 
 	if (pmd_protnone(pmdval) && !gup_can_follow_protnone(vma, flags))
 		return no_page_table(vma, flags, address);
@@ -925,24 +933,24 @@ static struct page *follow_pmd_mask(struct vm_area_struct *vma,
 	}
 	if (unlikely(!pmd_leaf(pmdval))) {
 		spin_unlock(ptl);
-		return follow_page_pte(vma, address, pmd, flags);
+		return follow_page_pte(vma, address, end, pmd, flags, nr_pages);
 	}
 	if (pmd_trans_huge(pmdval) && (flags & FOLL_SPLIT_PMD)) {
 		spin_unlock(ptl);
 		split_huge_pmd(vma, pmd, address);
 		/* If pmd was left empty, stuff a page table in there quickly */
 		return pte_alloc(mm, pmd) ? ERR_PTR(-ENOMEM) :
-			follow_page_pte(vma, address, pmd, flags);
+			follow_page_pte(vma, address, end, pmd, flags, nr_pages);
 	}
-	page = follow_huge_pmd(vma, address, pmd, flags, page_mask);
+	page = follow_huge_pmd(vma, address, end, pmd, flags, nr_pages);
 	spin_unlock(ptl);
 	return page;
 }
 
 static struct page *follow_pud_mask(struct vm_area_struct *vma,
-				    unsigned long address, p4d_t *p4dp,
-				    unsigned int flags,
-				    unsigned long *page_mask)
+				    unsigned long address, unsigned long end,
+				    p4d_t *p4dp, unsigned int flags,
+				    unsigned long *nr_pages)
 {
 	pud_t *pudp, pud;
 	spinlock_t *ptl;
@@ -955,7 +963,7 @@ static struct page *follow_pud_mask(struct vm_area_struct *vma,
 		return no_page_table(vma, flags, address);
 	if (pud_leaf(pud)) {
 		ptl = pud_lock(mm, pudp);
-		page = follow_huge_pud(vma, address, pudp, flags, page_mask);
+		page = follow_huge_pud(vma, address, end, pudp, flags, nr_pages);
 		spin_unlock(ptl);
 		if (page)
 			return page;
@@ -964,13 +972,13 @@ static struct page *follow_pud_mask(struct vm_area_struct *vma,
 	if (unlikely(pud_bad(pud)))
 		return no_page_table(vma, flags, address);
 
-	return follow_pmd_mask(vma, address, pudp, flags, page_mask);
+	return follow_pmd_mask(vma, address, end, pudp, flags, nr_pages);
 }
 
 static struct page *follow_p4d_mask(struct vm_area_struct *vma,
-				    unsigned long address, pgd_t *pgdp,
-				    unsigned int flags,
-				    unsigned long *page_mask)
+				    unsigned long address, unsigned long end,
+				    pgd_t *pgdp, unsigned int flags,
+				    unsigned long *nr_pages)
 {
 	p4d_t *p4dp, p4d;
 
@@ -981,15 +989,16 @@ static struct page *follow_p4d_mask(struct vm_area_struct *vma,
 	if (!p4d_present(p4d) || p4d_bad(p4d))
 		return no_page_table(vma, flags, address);
 
-	return follow_pud_mask(vma, address, p4dp, flags, page_mask);
+	return follow_pud_mask(vma, address, end, p4dp, flags, nr_pages);
 }
 
 /**
  * follow_page_mask - look up a page descriptor from a user-virtual address
  * @vma: vm_area_struct mapping @address
  * @address: virtual address to look up
+ * @end: virtual address at which to stop batching contiguous pages
  * @flags: flags modifying lookup behaviour
- * @page_mask: a pointer to output page_mask
+ * @nr_pages: output; number of contiguous pages the caller can read
  *
  * @flags can have FOLL_ flags set, defined in <linux/mm.h>
  *
@@ -998,15 +1007,17 @@ static struct page *follow_p4d_mask(struct vm_area_struct *vma,
  * trigger a fault with FAULT_FLAG_UNSHARE set. Note that unsharing is only
  * relevant with FOLL_PIN and !FOLL_WRITE.
  *
- * On output, @page_mask is set according to the size of the page.
+ * On output, @nr_pages holds how many contiguous pages the folio that includes
+ * the returned page has mapped into this process, so the caller can advance
+ * over a large folio in one step.
  *
  * Return: the mapped (struct page *), %NULL if no mapping exists, or
  * an error pointer if there is a mapping to something not represented
  * by a page descriptor (see also vm_normal_page()).
  */
 static struct page *follow_page_mask(struct vm_area_struct *vma,
-			      unsigned long address, unsigned int flags,
-			      unsigned long *page_mask)
+			      unsigned long address, unsigned long end,
+			      unsigned int flags, unsigned long *nr_pages)
 {
 	pgd_t *pgd;
 	struct mm_struct *mm = vma->vm_mm;
@@ -1014,13 +1025,13 @@ static struct page *follow_page_mask(struct vm_area_struct *vma,
 
 	vma_pgtable_walk_begin(vma);
 
-	*page_mask = 0;
+	*nr_pages = 1;
 	pgd = pgd_offset(mm, address);
 
 	if (pgd_none(*pgd) || unlikely(pgd_bad(*pgd)))
 		page = no_page_table(vma, flags, address);
 	else
-		page = follow_p4d_mask(vma, address, pgd, flags, page_mask);
+		page = follow_p4d_mask(vma, address, end, pgd, flags, nr_pages);
 
 	vma_pgtable_walk_end(vma);
 
@@ -1198,7 +1209,7 @@ struct page *get_user_page_vma(struct vm_area_struct *vma, unsigned long addr,
 			       unsigned int gup_flags)
 {
 	bool vma_locked = gup_flags & FOLL_VMA_LOCK;
-	unsigned long page_mask;
+	unsigned long nr_pages;
 	struct page *page;
 	int locked = 1;
 	bool pfnmap;
@@ -1231,10 +1242,10 @@ struct page *get_user_page_vma(struct vm_area_struct *vma, unsigned long addr,
 		}
 		cond_resched();
 
-		/* follow_page_mask() requires @page_mask; it is unused here. */
-		page = follow_page_mask(vma, addr,
+		/* This helper hands back a single page; cap the batch at one. */
+		page = follow_page_mask(vma, addr, addr + PAGE_SIZE,
 					gup_flags | FOLL_TOUCH | FOLL_GET,
-					&page_mask);
+					&nr_pages);
 		if (!IS_ERR_OR_NULL(page)) {
 			/* Match __get_user_pages(): flush for VIVT/aliasing caches. */
 			flush_anon_page(vma, page, addr);
@@ -1529,7 +1540,6 @@ static long __get_user_pages(struct mm_struct *mm,
 {
 	long ret = 0, i = 0;
 	struct vm_area_struct *vma = NULL;
-	unsigned long page_mask = 0;
 
 	if (!nr_pages)
 		return 0;
@@ -1544,7 +1554,7 @@ static long __get_user_pages(struct mm_struct *mm,
 
 	do {
 		struct page *page;
-		unsigned int page_increm;
+		unsigned long page_increm;
 
 		/* first iteration or cross vma bound */
 		if (!vma || start >= vma->vm_end) {
@@ -1571,7 +1581,7 @@ static long __get_user_pages(struct mm_struct *mm,
 						pages ? &page : NULL);
 				if (ret)
 					goto out;
-				page_mask = 0;
+				page_increm = 1;
 				goto next_page;
 			}
 
@@ -1594,7 +1604,8 @@ static long __get_user_pages(struct mm_struct *mm,
 		}
 		cond_resched();
 
-		page = follow_page_mask(vma, start, gup_flags, &page_mask);
+		page = follow_page_mask(vma, start, start + nr_pages * PAGE_SIZE,
+					gup_flags, &page_increm);
 		if (!page || PTR_ERR(page) == -EMLINK) {
 			ret = faultin_page(vma, start, gup_flags,
 					   PTR_ERR(page) == -EMLINK, locked);
@@ -1627,7 +1638,6 @@ static long __get_user_pages(struct mm_struct *mm,
 			goto out;
 		}
 next_page:
-		page_increm = 1 + (~(start >> PAGE_SHIFT) & page_mask);
 		if (page_increm > nr_pages)
 			page_increm = nr_pages;
 
-- 
2.53.0-Meta

[PATCH RFC v4 11/12] mm/gup: batch contiguous PTE-mapped large folios in follow_page_mask()

[Rik van Riel]

follow_page_mask() returns one page per call for a PTE-mapped large folio,
so __get_user_pages() re-walks the page tables for every page of an mTHP
even though the folio maps a contiguous run. The huge PMD and PUD paths
already return the whole mapping in one step.

Report the contiguous run for the PTE case too. follow_pte_batch() uses
folio_pte_batch_flags() to count consecutive present PTEs that map
consecutive pages of the same folio with a uniform write bit, bounded by
the page table, @end, the VMA, and the folio itself.

Keep the per-PTE guarantees that follow_page_pte() makes for the head page.
folio_pte_batch_flags() with FPB_RESPECT_WRITE stops the run at a change in
the write bit, so the whole run matches the head.

A writable run is safe for any access: a writable anon page is exclusive,
so gup_must_unshare() cannot fire, and FOLL_WRITE is satisfied.

A read-only run is batched only for a plain read, since FOLL_WRITE would
need a COW fault per page and FOLL_PIN would need a per-page
gup_must_unshare() check.

Measured with mm/gup_test.c (PIN_LONGTERM_BENCHMARK, the slow
pin_user_pages() path) on a 256 MB MADV_HUGEPAGE anonymous region in a
4 CPU VM, median get time over 16 iterations. Each folio size was confirmed
through the per-size anon_fault_alloc counters (4096 folios for 64 kB, 128
for 2 MB):

  gup_test -L -m 256 -n 65536 -r 16 -t
                        before      after
  64 kB mTHP            3140 us      412 us   (7.6x)
  2 MB THP (control)      78 us       76 us
  4 kB base (control)   3010 us     3042 us

The PMD-mapped 2 MB THP already returns the whole mapping in one step, so
it stays fast and unchanged. The 4 kB baseline shows the per-page walk cost
that the 64 kB case paid before this change; only the PTE-mapped large
folio case improves.

Assisted-by: Claude:claude-opus-4.8
Signed-off-by: Rik van Riel <riel@surriel.com>
---
 mm/gup.c | 45 +++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 45 insertions(+)

diff --git a/mm/gup.c b/mm/gup.c
index 96b00ddf7e04..4910c93ba003 100644
--- a/mm/gup.c
+++ b/mm/gup.c
@@ -805,6 +805,43 @@ static inline bool can_follow_write_pte(pte_t pte, struct page *page,
 	return !userfaultfd_pte_wp(vma, pte);
 }
 
+/*
+ * Count the pages, starting at @address and bounded by @end, that a PTE-mapped
+ * large @folio maps contiguously and that can be returned together with the
+ * page at @address: consecutive present PTEs mapping consecutive pages of
+ * @folio with a uniform write bit, within this VMA and a single page table.
+ * Returns at least 1.
+ *
+ * gup_must_unshare() and the write-fault check are per PTE. A writable run is
+ * always safe: a writable anon page is exclusive, and FOLL_WRITE is satisfied.
+ * A read-only run is only safe for a plain read; FOLL_WRITE would need a COW
+ * fault per page and FOLL_PIN would need a per-page gup_must_unshare() check,
+ * so those fall back to a single page.
+ */
+static unsigned long follow_pte_batch(struct vm_area_struct *vma,
+		unsigned long address, unsigned long end, struct folio *folio,
+		struct page *page, pte_t *ptep, pte_t pte, unsigned int flags)
+{
+	pte_t batch_pte = pte;
+	unsigned long max;
+
+	if (!pte_write(pte) && (flags & (FOLL_WRITE | FOLL_PIN)))
+		return 1;
+
+	/*
+	 * folio_pte_batch_flags() scans forward from @ptep, so the run must
+	 * stay within this page table: bound it by the PMD as well as @end and
+	 * the VMA, since a large folio can be PTE-mapped across a PMD boundary.
+	 */
+	max = min((pmd_addr_end(address, end) - address) >> PAGE_SHIFT,
+		  (vma->vm_end - address) >> PAGE_SHIFT);
+	if (max <= 1)
+		return 1;
+
+	return folio_pte_batch_flags(folio, vma, ptep, &batch_pte, max,
+				     FPB_RESPECT_WRITE);
+}
+
 static struct page *follow_page_pte(struct vm_area_struct *vma,
 		unsigned long address, unsigned long end, pmd_t *pmd,
 		unsigned int flags, unsigned long *nr_pages)
@@ -893,6 +930,14 @@ static struct page *follow_page_pte(struct vm_area_struct *vma,
 		folio_mark_accessed(folio);
 	}
 
+	/*
+	 * A PTE-mapped large folio can be handed back as a contiguous batch,
+	 * so the caller advances over the whole run in one step instead of
+	 * walking the page tables for every page.
+	 */
+	if (folio_test_large(folio))
+		*nr_pages = follow_pte_batch(vma, address, end, folio, page,
+					     ptep, pte, flags);
 out:
 	pte_unmap_unlock(ptep, ptl);
 	return page;
-- 
2.53.0-Meta

[PATCH RFC v4 12/12] selftests/mm: add a slow-GUP content and COW test for mTHP

[Rik van Riel]

follow_page_mask() now batches a PTE-mapped large folio (mTHP) into one
contiguous run for the slow get_user_pages() path. A mis-batched run would
hand back the wrong pages or a stale COW copy, which the existing tests do
not catch: gup_test checks only pin/unpin integrity, and cow exercises COW
mostly at PMD size.

Add mthp_gup_cow_test. It forces 64kB-only mTHP, writes a per-page-distinct
pattern, then pins the region on the slow path (PIN_LONGTERM without
USE_FAST) and compares the bytes the kernel copies back from the pinned
pages against that pattern.

The test covers a read pin, a write pin, and COW after fork(): a child
write-pins to force per-page unshare and checks the copied contents, then
rewrites its copy while the parent verifies its own contents are intact.

It also reports how many pages sit in contiguous large-folio runs, so a
kernel without mTHP shows light coverage rather than passing vacuously.

Assisted-by: Claude:claude-opus-4.8
Signed-off-by: Rik van Riel <riel@surriel.com>
---
 tools/testing/selftests/mm/Makefile           |   1 +
 .../testing/selftests/mm/mthp_gup_cow_test.c  | 213 ++++++++++++++++++
 tools/testing/selftests/mm/run_vmtests.sh     |   1 +
 3 files changed, 215 insertions(+)
 create mode 100644 tools/testing/selftests/mm/mthp_gup_cow_test.c

diff --git a/tools/testing/selftests/mm/Makefile b/tools/testing/selftests/mm/Makefile
index e6df968f0971..6b917a4f73e0 100644
--- a/tools/testing/selftests/mm/Makefile
+++ b/tools/testing/selftests/mm/Makefile
@@ -83,6 +83,7 @@ TEST_GEN_FILES += mrelease_test
 TEST_GEN_FILES += mremap_dontunmap
 TEST_GEN_FILES += mremap_test
 TEST_GEN_FILES += mseal_test
+TEST_GEN_FILES += mthp_gup_cow_test
 TEST_GEN_FILES += on-fault-limit
 TEST_GEN_FILES += pagemap_ioctl
 TEST_GEN_FILES += pfnmap
diff --git a/tools/testing/selftests/mm/mthp_gup_cow_test.c b/tools/testing/selftests/mm/mthp_gup_cow_test.c
new file mode 100644
index 000000000000..52ee329d8641
--- /dev/null
+++ b/tools/testing/selftests/mm/mthp_gup_cow_test.c
@@ -0,0 +1,213 @@
+// SPDX-License-Identifier: GPL-2.0-only
+/*
+ * Verify that the slow GUP path (pin_user_pages -> follow_page_mask ->
+ * follow_pte_batch) returns the correct pages for a PTE-mapped large folio
+ * (mTHP), including that COW copies produce the right content.
+ *
+ * Uses the CONFIG_GUP_TEST PIN_LONGTERM interface: START pins a range on the
+ * slow path, READ copies the pinned pages' bytes back so we can compare them
+ * against the pattern we wrote.
+ */
+#define _GNU_SOURCE
+#include <stdio.h>
+#include <stdlib.h>
+#include <string.h>
+#include <stdint.h>
+#include <fcntl.h>
+#include <unistd.h>
+#include <errno.h>
+#include <sys/ioctl.h>
+#include <sys/mman.h>
+#include <sys/wait.h>
+#include <linux/types.h>
+
+#define ARRAY_SIZE(a)	(sizeof(a) / sizeof((a)[0]))
+
+#define GUP_DEV "/sys/kernel/debug/gup_test"
+
+#define PIN_LONGTERM_TEST_START	_IOW('g', 7, struct pin_longterm_test)
+#define PIN_LONGTERM_TEST_STOP	_IO('g', 8)
+#define PIN_LONGTERM_TEST_READ	_IOW('g', 9, __u64)
+#define USE_WRITE	1
+#define USE_FAST	2
+
+struct pin_longterm_test {
+	__u64 addr;
+	__u64 size;
+	__u32 flags;
+};
+
+#define ORDER_KB	64
+#define NR_FOLIOS	16
+#define REGION		((size_t)ORDER_KB * 1024 * NR_FOLIOS)
+
+static long PS;
+static int fails;
+static int tap;
+
+static void ok(int cond, const char *desc)
+{
+	printf("%s %d %s\n", cond ? "ok" : "not ok", ++tap, desc);
+	if (!cond)
+		fails++;
+}
+
+/* Deterministic, per-page-distinct pattern so any mis-order or leak shows. */
+static void fill(char *base, size_t sz, uint32_t salt)
+{
+	for (size_t off = 0; off < sz; off += PS) {
+		uint32_t k = off / PS;
+		uint64_t v = ((uint64_t)salt << 32) ^ (k * 0x9E3779B1u + 0x1234);
+
+		for (size_t i = 0; i < PS; i += sizeof(v))
+			memcpy(base + off + i, &v, sizeof(v));
+	}
+}
+
+static int wsysfs(const char *path, const char *val)
+{
+	int fd = open(path, O_WRONLY);
+
+	if (fd < 0)
+		return -1;
+	int r = write(fd, val, strlen(val));
+
+	close(fd);
+	return r < 0 ? -1 : 0;
+}
+
+/* Force sub-PMD 64kB mTHP only, so faults produce PTE-mapped large folios. */
+static void setup_mthp(void)
+{
+	const char *thp = "/sys/kernel/mm/transparent_hugepage";
+	char p[256];
+	static const int kb[] = { 16, 32, 64, 128, 256, 512, 1024, 2048 };
+
+	wsysfs("/sys/kernel/mm/transparent_hugepage/enabled", "never");
+	for (unsigned int i = 0; i < ARRAY_SIZE(kb); i++) {
+		snprintf(p, sizeof(p), "%s/hugepages-%dkB/enabled", thp, kb[i]);
+		wsysfs(p, kb[i] == ORDER_KB ? "always" : "never");
+	}
+}
+
+/* Count how many pages sit in a contiguous >=ORDER_KB PFN run (via pagemap). */
+static int count_large_pages(char *base, size_t sz)
+{
+	int pm = open("/proc/self/pagemap", O_RDONLY);
+	size_t n = sz / PS, large = 0;
+	uint64_t *pfn = calloc(n, sizeof(*pfn));
+
+	if (pm < 0)
+		return -1;
+	for (size_t k = 0; k < n; k++) {
+		uint64_t ent;
+		off_t idx = ((uintptr_t)base + k * PS) / PS * sizeof(ent);
+
+		if (pread(pm, &ent, sizeof(ent), idx) != sizeof(ent) ||
+		    !(ent & (1ULL << 63)))
+			pfn[k] = 0;
+		else
+			pfn[k] = ent & ((1ULL << 55) - 1);
+	}
+	close(pm);
+	for (size_t k = 0; k < n; k++)
+		if (k + 1 < n && pfn[k] && pfn[k + 1] == pfn[k] + 1)
+			large++;
+	free(pfn);
+	return large;
+}
+
+/* Pin @base..@sz on the slow path, read the pinned bytes back, compare to exp. */
+static int pin_verify(int fd, char *base, size_t sz, uint32_t wr, char *exp)
+{
+	struct pin_longterm_test a = {
+		.addr = (uintptr_t)base, .size = sz,
+		.flags = wr ? USE_WRITE : 0,		/* USE_FAST unset => slow */
+	};
+	char *got = mmap(NULL, sz, PROT_READ | PROT_WRITE,
+			 MAP_PRIVATE | MAP_ANONYMOUS, -1, 0);
+	__u64 ga = (uintptr_t)got;
+	int rc = -1;
+
+	if (got == MAP_FAILED)
+		return -1;
+	if (ioctl(fd, PIN_LONGTERM_TEST_START, &a)) {
+		fprintf(stderr, "START(%s) failed: %s\n",
+			wr ? "write" : "read", strerror(errno));
+		goto out;
+	}
+	if (ioctl(fd, PIN_LONGTERM_TEST_READ, &ga)) {
+		fprintf(stderr, "READ failed: %s\n", strerror(errno));
+		ioctl(fd, PIN_LONGTERM_TEST_STOP);
+		goto out;
+	}
+	ioctl(fd, PIN_LONGTERM_TEST_STOP);
+	rc = memcmp(got, exp, sz) ? 1 : 0;
+out:
+	munmap(got, sz);
+	return rc;
+}
+
+int main(void)
+{
+	PS = sysconf(_SC_PAGESIZE);
+	setup_mthp();
+
+	int fd = open(GUP_DEV, O_RDWR);
+
+	if (fd < 0) {
+		fprintf(stderr, "open %s: %s (CONFIG_GUP_TEST?)\n",
+			GUP_DEV, strerror(errno));
+		return 2;
+	}
+
+	char *r = mmap(NULL, REGION, PROT_READ | PROT_WRITE,
+		       MAP_PRIVATE | MAP_ANONYMOUS, -1, 0);
+	if (r == MAP_FAILED) {
+		perror("mmap");
+		return 2;
+	}
+	fill(r, REGION, 0xA1);				/* pattern P */
+	char *expP = malloc(REGION);
+
+	memcpy(expP, r, REGION);
+
+	int large = count_large_pages(r, REGION);
+
+	printf("# %d/%zu pages in contiguous large-folio runs\n",
+	       large, REGION / PS);
+	if (large < (int)(REGION / PS) / 4)
+		printf("# WARN: little mTHP backing; batch path lightly covered\n");
+
+	/* A: read pin over writable mTHP -> batches -> content must equal P. */
+	ok(pin_verify(fd, r, REGION, 0, expP) == 0,
+	   "slow read-pin of mTHP returns correct contents");
+
+	/* B: write pin -> FOLL_WRITE batch path -> content must equal P. */
+	ok(pin_verify(fd, r, REGION, 1, expP) == 0,
+	   "slow write-pin of mTHP returns correct contents");
+
+	/* C: COW isolation. Child write-pins (unshares) then rewrites; parent P. */
+	pid_t pid = fork();
+
+	if (pid == 0) {
+		int cfd = open(GUP_DEV, O_RDWR);
+		int a = pin_verify(cfd, r, REGION, 1, expP);	/* COW copy == P */
+
+		fill(r, REGION, 0xB2);			/* child writes Q */
+		_exit(a == 0 ? 0 : 1);
+	}
+	int st = 0;
+
+	waitpid(pid, &st, 0);
+	ok(WIFEXITED(st) && WEXITSTATUS(st) == 0,
+	   "child write-pin after COW returns correct (copied) contents");
+	ok(memcmp(r, expP, REGION) == 0,
+	   "parent contents intact after child COW writes");
+	/* D: parent read-pin again after the COW split still correct. */
+	ok(pin_verify(fd, r, REGION, 0, expP) == 0,
+	   "parent slow read-pin after COW still correct");
+
+	printf("# totals: pass:%d fail:%d\n", tap - fails, fails);
+	return fails ? 1 : 0;
+}
diff --git a/tools/testing/selftests/mm/run_vmtests.sh b/tools/testing/selftests/mm/run_vmtests.sh
index 8c296dedf047..b49a3eb0c205 100755
--- a/tools/testing/selftests/mm/run_vmtests.sh
+++ b/tools/testing/selftests/mm/run_vmtests.sh
@@ -289,6 +289,7 @@ fi
 # Dump pages 0, 19, and 4096, using pin_user_pages:
 CATEGORY="gup_test" run_test ./gup_test -ct -F 0x1 0 19 0x1000
 CATEGORY="gup_test" run_test ./gup_longterm
+CATEGORY="gup_test" run_test ./mthp_gup_cow_test
 
 CATEGORY="userfaultfd" run_test ./uffd-unit-tests
 uffd_stress_bin=./uffd-stress
-- 
2.53.0-Meta