[PATCH RFC v4 0/12] mm: use per-VMA lock in __access_remote_vm for improved monitoring reliability

[Rik van Riel <riel@surriel.com>]

__access_remote_vm() holds mmap_read_lock() for the whole transfer. On
large machines, with large multi-threaded applications, the mmap_lock
is often contended, leading to things like reads of /proc/PID/cmdline
stalling.

This results in system monitoring tools getting stuck, right when
system information would be the most helpful.

That problem can be avoided by having __access_remote_vm() access
user memory under the per-VMA lock. The old code also looks up
the VMA for every page accessed, and walks the page tables for
every page inside a large folio. Cleaning that up speeds things
up nicely.

Using code that does not include its own lock dance, and does not
look up the VMA twice results in a modest speedup.

	read size   baseline    per-VMA     throughput
	8 B          201.7 ns    170.6 ns    ~15% faster
	4 KB         495 ns      442 ns      +12% (8266 -> 9265 MB/s)
	64 KB       4115 ns     3355 ns      +23% (15927 -> 19536 MB/s)
	1 MB       75902 ns    63641 ns      +19% (13815 -> 16477 MB/s)

The multi-threaded reader-versus-writer contention that motivates the lock
is not measured.

The rest of the series builds on this to batch per-VMA GUP. Patches 9-12
have follow_page_mask() report a contiguous PTE-mapped run of a large
folio, so the slow pin_user_pages() path walks it once instead of per page.
Other paths that walk a range benefit too.

That batching pays off on PTE-mapped mTHP. With mm/gup_test.c
(PIN_LONGTERM_BENCHMARK, the slow pin_user_pages() path) on a 256 MB
MADV_HUGEPAGE region in a 4 CPU VM, median get time over 16 iterations:

	gup_test -L -m 256 -n 65536 -r 16 -t
	                      before      after
	64 kB mTHP            3140 us      412 us   (7.6x)
	2 MB THP (control)      78 us       76 us
	4 kB base (control)   3010 us     3042 us

The PMD-mapped 2 MB THP already returns in one step, so only the PTE-mapped
large folio case improves.

Reaching a page under the per-VMA lock means looking up the VMA first,
which needs to untag the remote address without the mmap lock. The untag
masks (x86 LAM, riscv pointer masking) change only while the target is
single-threaded, so a remote reader races at most one such change.

Those masks are already read locklessly elsewhere, so patches 1 and 2 add
untagged_addr_remote_unlocked() helpers with READ_ONCE()/WRITE_ONCE()
annotations. A stale read sees the old or new mask, never a torn value, and
the remote untag is best-effort.

With the VMA in hand we need one page from it under the lock the caller
already holds. get_user_pages_remote() does not fit: it hard codes the mmap
lock and re-derives the VMA internally.

Rather than reimplement GUP in yet another caller, this version follows
David Hildenbrand's v2 guidance: start from a proper GUP interface that
takes a locked VMA, not an mm, and decide which faults can resolve under
the VMA lock.

Patch 3 renames get_user_page_vma_remote() to get_user_page_lookup_vma() to
free the name. Patch 5 adds get_user_page_vma(): a simplified
__get_user_pages() that walks the tables with follow_page_mask(), faults a
missing page in with faultin_page(), and returns it with a reference and
the caller's lock still held.

Like __get_user_pages() it runs check_vma_flags(), so callers need not
pre-check the VMA.

A VM_IO or VM_PFNMAP VMA is the exception: a COWed page with a struct page
is returned normally, while a raw PFN yields -EFAULT so the caller can
reach vma->vm_ops->access(). The caller sets FOLL_VMA_LOCK when it holds
the per-VMA lock, which reaches the fault code as FAULT_FLAG_VMA_LOCK.

Anything the per-VMA lock cannot finish releases the lock and returns
-EAGAIN, so the caller retries under the mmap lock: a dropped fault, a
userfaultfd VMA (which assumes current is the faulting task), a hard error,
or ->access() memory. faultin_page() reports the retry the same way for
both lock types.

__access_remote_vm() then uses the per-VMA lock when an access fits within
one VMA, and falls back to the mmap lock for multi-VMA accesses, stack
expansion, or an -EAGAIN from get_user_page_vma().

Walking page tables under only the per-VMA lock is safe against both page
table freeing and THP collapse. munmap() frees a VMA's page tables under
the VMA write lock that our read lock excludes, so they cannot be torn down
under the walk.

THP collapse needs neither lock: it retracts a PTE page under the page
table lock and frees it by RCU.

follow_page_pte() takes that same lock through pte_offset_map_lock() and
rechecks the pmd, so it either walks an intact table or sees the collapsed
pmd and faults the page in, while RCU keeps the retracted page valid.

A COWed page in a VM_PFNMAP mapping was previously unreachable through
/proc/pid/mem because generic_access_phys() rejects ioremaps of RAM. With
get_user_page_vma() returning the struct page directly, that read now
succeeds, which the new selftests cover along with the raw PFN path.

Thanks to Suren for pointing out the need, to David for pushing me
to just rework the whole area, to Usama for reviewing the arch patches,
and Lorenzo for the changelog and cover letter cleanups.

There is a lot more work that could be done in this area, but the
patch series is big enough. More changes can come in follow up work.
---
 arch/arm64/kernel/mte.c                        |   2 +-
 arch/riscv/include/asm/mmu_context.h           |   4 +-
 arch/riscv/include/asm/uaccess.h               |  10 +-
 arch/riscv/kernel/process.c                    |  12 +-
 arch/x86/include/asm/mmu_context.h             |   6 +-
 arch/x86/include/asm/uaccess_64.h              |  15 +-
 arch/x86/kernel/process_64.c                   |   4 +-
 arch/x86/kernel/uprobes.c                      |   2 +-
 include/linux/mm.h                             |  32 +--
 include/linux/uaccess.h                        |   7 +
 mm/gup.c                                       | 360 ++++++++++++++++++-----
 mm/internal.h                                  |   8 +-
 mm/memory.c                                    | 379 +++++++++++++++++--------
 mm/rmap.c                                      |   2 +-
 tools/testing/selftests/mm/Makefile            |   1 +
 tools/testing/selftests/mm/mthp_gup_cow_test.c | 213 ++++++++++++++
 tools/testing/selftests/mm/pfnmap.c            |  66 +++++
 tools/testing/selftests/mm/run_vmtests.sh      |   1 +
 18 files changed, 887 insertions(+), 237 deletions(-)

base-commit: 248951ddc14de84de3910f9b13f51491a8cd91df
---
To: Andrew Morton <akpm@linux-foundation.org>
To: linux-mm@kvack.org
To: linux-kernel@vger.kernel.org
Cc: Dave Hansen <dave.hansen@linux.intel.com>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Suren Baghdasaryan <surenb@google.com>
Cc: Lorenzo Stoakes <ljs@kernel.org>
Cc: Vlastimil Babka <vbabka@kernel.org>
Cc: David Hildenbrand <david@kernel.org>
Cc: "Liam R. Howlett" <liam@infradead.org>
Cc: Mike Rapoport <rppt@kernel.org>
Cc: Michal Hocko <mhocko@suse.com>
Cc: Jason Gunthorpe <jgg@ziepe.ca>
Cc: John Hubbard <jhubbard@nvidia.com>
Cc: Peter Xu <peterx@redhat.com>
Cc: Matthew Wilcox <willy@infradead.org>
Cc: Usama Arif <usamaarif642@gmail.com>
Cc: Rik van Riel <riel@surriel.com>

REVIEWERS NOTES:
Series based on the current Linus tree (v7.2-rc5, commit 248951ddc14d).
Patches 1-8 add per-VMA lock remote access; patches 9-12 batch per-VMA GUP
over PTE-mapped large folios.

Changes since v3 (RFC v3, 2026-07-17):
https://lore.kernel.org/all/20260717170036.743149-1-riel@surriel.com/

- Address Usama Arif's v3 review:
  - p1: correct the Acked-by address to usama.arif@linux.dev.
  - p2: rewrite the riscv changelog to describe only this patch's change,
    and replace the false "pmlen is stable" claim with the accurate story
    (pmlen may change until MM_CONTEXT_LOCK_PMLEN; READ_ONCE avoids a torn
    value, and the remote untag is best-effort).
  - p3: fix the "name space" typo and note "No functional change intended."
  - Split the check_vma_flags() ignore-flags change into its own prep patch
    (patch 4) so the PROT_NONE PFNMAP read hole fix is bisectable, keeping
    the VM_READ / FOLL_FORCE / arch-key checks for a PFNMAP VMA.
  - get_user_page_vma(): add the flush_anon_page()/flush_dcache_page() cache
    flushes __get_user_pages() does, and reword the faultin_page() kerneldoc
    to cover the per-VMA lock, not just the mmap lock.
- New patch 7: read remote strings (bpf_copy_from_user_task_str()) under the
  per-VMA lock, extending the approach to __copy_remote_vm_str().
- Factor __access_remote_vm() and the string reader onto a shared
  remote_vm_walk() helper, and fix a NUL-termination regression it exposed:
  the partial buffer is now terminated on a mid-walk error.
- Fold the mTHP GUP batching (patches 9-12), previously a separate follow-up,
  into this series; the cover letter now carries its gup_test numbers.
- Correct two changelog claims. The page table lock plus the pmd recheck, not
  the VMA write lock, guards the walk against concurrent THP collapse; RCU
  only keeps the retracted PTE page valid, and the VMA write lock excludes
  munmap(). The x86 untag mask changes while the target is single-threaded,
  so the lockless read is best-effort, not "set once and never changes."
- Rebase onto the current Linus tree (v7.2-rc5).