[PATCH v3 0/6] mm: gup: batch PTE-mapped large folios in gup_fast_pte_range()

[PATCH v3 0/6] mm: gup: batch PTE-mapped large folios in gup_fast_pte_range()

[Yeoreum Yun]

GUP-fast grabs a PTE-mapped large folio one page at a time.  Every PTE
costs a try_grab_folio_fast() (a refcount cmpxchg, plus the pincount
atomic, a full barrier and a node stat update for FOLL_PIN), a
gup_fast_folio_allowed() and a folio_set_referenced(), so a 64 kB mTHP
pays sixteen of each and a PTE-mapped 2 MB THP pays 512.

The users are the hot ones: iov_iter_extract_user_pages() pins every
O_DIRECT buffer through pin_user_pages_fast(), io_uring registers its
buffers through it, and so does RDMA memory registration.
With mTHP those buffers are PTE-mapped large folios.

After the head page has been grabbed and verified, extend the run to the
following PTEs that map the next pages of the same folio with the
same protection bits, take the remaining references for the
run with try_grab_folio(), and fill pages[] from the run.
The PTEs of the run are read while the reference on the folio is held,
so the folio can neither be collapsed, freed nor split underneath the scan.

A read-only run of an anonymous folio stops at the first page that
gup_must_unshare() rejects, so the per-page exclusivity guarantee for FOLL_PIN
is kept.

folio_pte_batch_flags() does the scan.  It reads the entries with
ptep_get(), which is not safe without the page table lock on
CONFIG_GUP_GET_PXX_LOW_HIGH (x86 PAE, mips32, sh) or with arm64 contpte,
and GUP-fast reads every PTE with ptep_get_lockless() for that reason.
Add FPB_LOCKLESS to make the helper do the same; it is inlined per call
site, so the other callers do not change.

The order-0 path keeps its instruction sequence.  The folio flags are
tested before folio_set_referenced() on purpose: a load of the flags
right after that locked instruction cost about 25% on order-0 pages in
the gup_test benchmark below, and reading them first brings that path
back to the baseline.

gup_test ioctl over a 256 MB anonymous region, FOLL_WRITE, 65536 pages
per call, median of 15 runs, two baseline boots / one patched boot, on
an x86-64 VM:

                       get_user_pages_fast()   pin_user_pages_fast()
                       before       after      before       after
  4 kB pages           849/789 us   783 us     1174/1138 us 1094 us
  64 kB mTHP           812/811 us   126 us     1215/1113 us  188 us
  1 MB mTHP            799/774 us    59 us     1185/1077 us   64 us
  2 MB THP, PTE-mapped 803/786 us    57 us     1159/1120 us   58 us
  2 MB THP, PMD-mapped  42/40 us     40 us       46/41 us     42 us

fio O_DIRECT randread of a null_blk device (CPU bound) with the I/O
buffer in 64 kB mTHP, median of 15 five-second runs:

  bs=1M psync                 126.2/125.6 GB/s  ->  220.6 GB/s
  bs=1M io_uring, iodepth 16   71.2/71.6 GB/s   ->  112.1 GB/s
  bs=64k io_uring, iodepth 16  50.2/50.7 GB/s   ->   60.1 GB/s

IORING_REGISTER_BUFFERS of 1 GB, median of 15:

64 kB mTHP           6179/5998 us  ->  1599 us
  1 MB mTHP            5790/5801 us  ->   893 us
  4 kB pages           7173/6741 us  ->  6763 us

---
Changes in v3:
- move gup_must_unshare() and INTERNAL_GUP_FLAGS from internal.h to gup.c
- drop the folio_pte_batch_flags() patch for shadow-stack.
- ignore VM_HUGEPAGE for shadow-stack via madvise().
- add r-b / a-b tags.
- Link to v2: https://lore.kernel.org/r/20261102-refactor_gup_fast-v2-0-bc1ac018c834@arm.com

Changes in v2:
- fix exceptional case for shstk on x86.
- remove redendant check for pmd change in each pte barch.
- Link to v1: https://lore.kernel.org/all/20260917083747.786-1-aa9736195201@gmail.com/

---
Yeoreum Yun (5):
      mm: khugepaged: ignore MADV_HUGEPAGE for shadow stacks
      mm: gup: clarify a reason to check pte/pmd in gup_fast_pte_range()
      mm: gup: move gup_must_unshare() and INTERNAL_GUP_FLAGS to gup.c
      mm: gup: introduce gup_unshared_pages()
      mm: gup: batch PTE-mapped large folios in gup_fast_pte_range()

Yuan-Hao Hsu (1):
      mm: internal: introduce FPB_LOCKLESS for folio_pte_batch_flags()

 mm/gup.c        | 185 +++++++++++++++++++++++++++++++++++++++++++++-----------
 mm/internal.h   |  78 +++---------------------
 mm/khugepaged.c |   3 +
 3 files changed, 159 insertions(+), 107 deletions(-)
---
base-commit: b3710a1d0f145a4f6f1d44fdaff56a6af8cffe89
change-id: 20261015-refactor_gup_fast-b55e3f593dc2

Best regards,
-- 
Sincerely,
Yeoreum Yun

[PATCH v3 1/6] mm: khugepaged: ignore MADV_HUGEPAGE for shadow stacks

[Yeoreum Yun]

commit 3efb9800557b ("mm: do not map the shadow stack as THP") makes
shadow stack mappings default to VM_NOHUGEPAGE. However, userspace can
still clear this flag and set VM_HUGEPAGE through madvise().

Allowing large folios in x86 shadow stacks exposes a limitation in
folio_pte_batch_flags(). For example:

  1. create a shadow stack mapping.
  2. apply MADV_HUGEPAGE and populate the mapping with a PTE-mapped
     large folio.
  3. apply MADV_DONTFORK to the first base page of that folio.
  4. call fork().

In the parent, the first page remains writable and anonymous-exclusive,
while the remaining pages shared with the child become write-protected
and lose PageAnonExclusive(). These pages can still belong to the same
large folio.

On x86, writable shadow-stack PTEs have _PAGE_RW clear and _PAGE_DIRTY
set. Write-protected PTEs have both bits clear, with the dirty state
preserved in _PAGE_SAVED_DIRTY.

When FPB_RESPECT_WRITE is set without FPB_RESPECT_DIRTY,
folio_pte_batch_flags() normalizes PTEs using pte_mkclean(), which clears
both dirty bits. This loses the distinction between writable
shadow-stack PTEs and write-protected PTEs, allowing them to be grouped
into the same batch. GUP-fast can encounter such a batch when walking
across the VMA boundary created by MADV_DONTFORK.

Since there is no practical use-case shadow stack with large folio,
ignore MADV_HUGEPAGE for shadow stack mappings, preserving their
VM_NOHUGEPAGE setting and preventing large-folio allocation.

Fixes: 3efb9800557b ("mm: do not map the shadow stack as THP")
Suggested-by: David Hildenbrand (Arm) <david@kernel.org>
Signed-off-by: Yeoreum Yun <yeoreum.yun@arm.com>
Cc: <stable@vger.kernel.org>
---
 mm/khugepaged.c | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/mm/khugepaged.c b/mm/khugepaged.c
index 40fad28c52f6f..51ec043fa26c3 100644
--- a/mm/khugepaged.c
+++ b/mm/khugepaged.c
@@ -367,6 +367,9 @@ int hugepage_madvise(struct vm_area_struct *vma,
 {
 	switch (advice) {
 	case MADV_HUGEPAGE:
+		/* PTE batching doesn't work reliably. */
+		if (vma_test_single_mask(vma, VMA_SHADOW_STACK))
+			break;
 		*vm_flags &= ~VM_NOHUGEPAGE;
 		*vm_flags |= VM_HUGEPAGE;
 		break;

-- 
2.43.0

Re: [PATCH v3 1/6] mm: khugepaged: ignore MADV_HUGEPAGE for shadow stacks

[David Hildenbrand (Arm)]

On 11/9/26 13:32, Yeoreum Yun wrote:
> commit 3efb9800557b ("mm: do not map the shadow stack as THP") makes
> shadow stack mappings default to VM_NOHUGEPAGE. However, userspace can
> still clear this flag and set VM_HUGEPAGE through madvise().
> 
> Allowing large folios in x86 shadow stacks exposes a limitation in
> folio_pte_batch_flags(). For example:
> 
>   1. create a shadow stack mapping.
>   2. apply MADV_HUGEPAGE and populate the mapping with a PTE-mapped
>      large folio.
>   3. apply MADV_DONTFORK to the first base page of that folio.
>   4. call fork().
> 
> In the parent, the first page remains writable and anonymous-exclusive,
> while the remaining pages shared with the child become write-protected
> and lose PageAnonExclusive(). These pages can still belong to the same
> large folio.
> 
> On x86, writable shadow-stack PTEs have _PAGE_RW clear and _PAGE_DIRTY
> set. Write-protected PTEs have both bits clear, with the dirty state
> preserved in _PAGE_SAVED_DIRTY.
> 
> When FPB_RESPECT_WRITE is set without FPB_RESPECT_DIRTY,
> folio_pte_batch_flags() normalizes PTEs using pte_mkclean(), which clears
> both dirty bits. This loses the distinction between writable
> shadow-stack PTEs and write-protected PTEs, allowing them to be grouped
> into the same batch. GUP-fast can encounter such a batch when walking
> across the VMA boundary created by MADV_DONTFORK.
> 
> Since there is no practical use-case shadow stack with large folio,
> ignore MADV_HUGEPAGE for shadow stack mappings, preserving their
> VM_NOHUGEPAGE setting and preventing large-folio allocation.
> 
> Fixes: 3efb9800557b ("mm: do not map the shadow stack as THP")
> Suggested-by: David Hildenbrand (Arm) <david@kernel.org>
> Signed-off-by: Yeoreum Yun <yeoreum.yun@arm.com>
> Cc: <stable@vger.kernel.org>
> ---
>  mm/khugepaged.c | 3 +++
>  1 file changed, 3 insertions(+)
> 
> diff --git a/mm/khugepaged.c b/mm/khugepaged.c
> index 40fad28c52f6f..51ec043fa26c3 100644
> --- a/mm/khugepaged.c
> +++ b/mm/khugepaged.c
> @@ -367,6 +367,9 @@ int hugepage_madvise(struct vm_area_struct *vma,
>  {
>  	switch (advice) {
>  	case MADV_HUGEPAGE:
> +		/* PTE batching doesn't work reliably. */
> +		if (vma_test_single_mask(vma, VMA_SHADOW_STACK))
> +			break;
>  		*vm_flags &= ~VM_NOHUGEPAGE;
>  		*vm_flags |= VM_HUGEPAGE;
>  		break;
> 

Ack, I prefer this solution for now until we know how to make pte_mkclean() on
x86 shadow stack PTEs not result in an implicit pte_wrprotect().

Acked-by: David Hildenbrand (Arm) <david@kernel.org>

-- 
Cheers,

David

Re: [PATCH v3 1/6] mm: khugepaged: ignore MADV_HUGEPAGE for shadow stacks

[Yeoreum Yun]

> On 11/9/26 13:32, Yeoreum Yun wrote:
> > commit 3efb9800557b ("mm: do not map the shadow stack as THP") makes
> > shadow stack mappings default to VM_NOHUGEPAGE. However, userspace can
> > still clear this flag and set VM_HUGEPAGE through madvise().
> > 
> > Allowing large folios in x86 shadow stacks exposes a limitation in
> > folio_pte_batch_flags(). For example:
> > 
> >   1. create a shadow stack mapping.
> >   2. apply MADV_HUGEPAGE and populate the mapping with a PTE-mapped
> >      large folio.
> >   3. apply MADV_DONTFORK to the first base page of that folio.
> >   4. call fork().
> > 
> > In the parent, the first page remains writable and anonymous-exclusive,
> > while the remaining pages shared with the child become write-protected
> > and lose PageAnonExclusive(). These pages can still belong to the same
> > large folio.
> > 
> > On x86, writable shadow-stack PTEs have _PAGE_RW clear and _PAGE_DIRTY
> > set. Write-protected PTEs have both bits clear, with the dirty state
> > preserved in _PAGE_SAVED_DIRTY.
> > 
> > When FPB_RESPECT_WRITE is set without FPB_RESPECT_DIRTY,
> > folio_pte_batch_flags() normalizes PTEs using pte_mkclean(), which clears
> > both dirty bits. This loses the distinction between writable
> > shadow-stack PTEs and write-protected PTEs, allowing them to be grouped
> > into the same batch. GUP-fast can encounter such a batch when walking
> > across the VMA boundary created by MADV_DONTFORK.
> > 
> > Since there is no practical use-case shadow stack with large folio,
> > ignore MADV_HUGEPAGE for shadow stack mappings, preserving their
> > VM_NOHUGEPAGE setting and preventing large-folio allocation.
> > 
> > Fixes: 3efb9800557b ("mm: do not map the shadow stack as THP")
> > Suggested-by: David Hildenbrand (Arm) <david@kernel.org>
> > Signed-off-by: Yeoreum Yun <yeoreum.yun@arm.com>
> > Cc: <stable@vger.kernel.org>
> > ---
> >  mm/khugepaged.c | 3 +++
> >  1 file changed, 3 insertions(+)
> > 
> > diff --git a/mm/khugepaged.c b/mm/khugepaged.c
> > index 40fad28c52f6f..51ec043fa26c3 100644
> > --- a/mm/khugepaged.c
> > +++ b/mm/khugepaged.c
> > @@ -367,6 +367,9 @@ int hugepage_madvise(struct vm_area_struct *vma,
> >  {
> >  	switch (advice) {
> >  	case MADV_HUGEPAGE:
> > +		/* PTE batching doesn't work reliably. */
> > +		if (vma_test_single_mask(vma, VMA_SHADOW_STACK))
> > +			break;
> >  		*vm_flags &= ~VM_NOHUGEPAGE;
> >  		*vm_flags |= VM_HUGEPAGE;
> >  		break;
> > 
> 
> Ack, I prefer this solution for now until we know how to make pte_mkclean() on
> x86 shadow stack PTEs not result in an implicit pte_wrprotect().

Yeap. As we discussed in offline, just skip clearing dirty bit for
shstk in pte_mkclear() wouldn't work since it would make a another
regression -- wouldn't be page-outed.

Might it would be good to use another soft-bit for shstk in x86.
But, for now, it seems enough as you said.

> 
> Acked-by: David Hildenbrand (Arm) <david@kernel.org>

Thanks!

-- 
Sincerely,
Yeoreum Yun