From: Rik van Riel <riel@surriel.com>
To: linux-kernel@vger.kernel.org
Cc: linux-mm@kvack.org, David Hildenbrand <david@kernel.org>,
Andrew Morton <akpm@linux-foundation.org>,
Suren Baghdasaryan <surenb@google.com>,
Lorenzo Stoakes <ljs@kernel.org>,
Christoph Hellwig <hch@infradead.org>,
Yuan-Hao Hsu <aa9736195201@gmail.com>,
John Hubbard <jhubbard@nvidia.com>,
Jason Gunthorpe <jgg@ziepe.ca>, Peter Xu <peterx@redhat.com>,
Rik van Riel <riel@surriel.com>
Subject: [PATCH v5 4/8] mm/gup: split out folio grab-and-store helper
Date: Mon, 16 Nov 2026 16:01:23 -0500 [thread overview]
Message-ID: <20261116210127.3837898-5-riel@surriel.com> (raw)
In-Reply-To: <20261116210127.3837898-1-riel@surriel.com>
follow_page_pte() does two things once it has resolved a present PTE
to a page: run the per-PTE safety checks (write-fault, unshare), then
grab a folio reference and hand the page back: fault it in if pinning,
mark it dirty/accessed, and store the caller's array slot.
Split the second part into gup_grab_fill_folio_pages(), which stores the
pages with gup_fill_pages(). The cache flushes stay with the caller in
__get_user_pages(), which runs gup_flush_pages() without any page
table lock held. Storing one slot under the lock is cheap; flushing is
not, and needs no lock once the folio reference is held.
No functional changes intended.
Suggested-by: David Hildenbrand <david@kernel.org>
Signed-off-by: Rik van Riel <riel@surriel.com>
---
mm/gup.c | 76 +++++++++++++++++++++++++++++++++++---------------------
1 file changed, 47 insertions(+), 29 deletions(-)
diff --git a/mm/gup.c b/mm/gup.c
index c59d436be7ad..efb2ad64be80 100644
--- a/mm/gup.c
+++ b/mm/gup.c
@@ -921,6 +921,50 @@ static inline bool can_follow_write_pte(pte_t pte, struct page *page,
return !userfaultfd_pte_wp(vma, pte);
}
+/*
+ * Grab a folio reference for @nr pages starting at @page, mark the folio
+ * accessed (and dirty for FOLL_WRITE), and store the pages for the caller.
+ * The caller (__get_user_pages()) flushes the caches with gup_flush_pages().
+ */
+static long gup_grab_fill_folio_pages(struct folio *folio,
+ struct page *page, unsigned long nr, bool dirty,
+ unsigned int flags, struct page **pages)
+{
+ long ret;
+
+ /* try_grab_folio() does nothing unless FOLL_GET or FOLL_PIN is set. */
+ ret = try_grab_folio(folio, nr, flags);
+ if (unlikely(ret))
+ return ret;
+
+ /*
+ * We need to make the page accessible if and only if we are going
+ * to access its content (the FOLL_PIN case). Please see
+ * Documentation/core-api/pin_user_pages.rst for details.
+ */
+ if (flags & FOLL_PIN) {
+ ret = arch_make_folio_accessible(folio);
+ if (ret) {
+ gup_put_folio(folio, nr, flags);
+ return ret;
+ }
+ }
+ if (flags & FOLL_TOUCH) {
+ if ((flags & FOLL_WRITE) && !dirty && !folio_test_dirty(folio))
+ folio_mark_dirty(folio);
+ /*
+ * pte_mkyoung() would be more correct here, but atomic care
+ * is needed to avoid losing the dirty bit: it is easier to use
+ * folio_mark_accessed().
+ */
+ folio_mark_accessed(folio);
+ }
+
+ gup_fill_pages(pages, page, nr);
+
+ return 0;
+}
+
static long follow_page_pte(struct vm_area_struct *vma,
unsigned long address, pmd_t *pmd, unsigned int flags,
struct page **pages)
@@ -976,36 +1020,10 @@ static long follow_page_pte(struct vm_area_struct *vma,
VM_WARN_ON_ONCE_PAGE((flags & FOLL_PIN) && PageAnon(page) &&
!PageAnonExclusive(page), page);
- /* try_grab_folio() does nothing unless FOLL_GET or FOLL_PIN is set. */
- ret = try_grab_folio(folio, 1, flags);
- if (unlikely(ret))
+ ret = gup_grab_fill_folio_pages(folio, page, 1,
+ pte_dirty(pte), flags, pages);
+ if (ret)
goto out;
-
- /*
- * We need to make the page accessible if and only if we are going
- * to access its content (the FOLL_PIN case). Please see
- * Documentation/core-api/pin_user_pages.rst for details.
- */
- if (flags & FOLL_PIN) {
- ret = arch_make_folio_accessible(folio);
- if (ret) {
- gup_put_folio(folio, 1, flags);
- goto out;
- }
- }
- if (flags & FOLL_TOUCH) {
- if ((flags & FOLL_WRITE) &&
- !pte_dirty(pte) && !folio_test_dirty(folio))
- folio_mark_dirty(folio);
- /*
- * pte_mkyoung() would be more correct here, but atomic care
- * is needed to avoid losing the dirty bit: it is easier to use
- * folio_mark_accessed().
- */
- folio_mark_accessed(folio);
- }
-
- gup_fill_pages(pages, page, 1);
ret = 1;
out:
pte_unmap_unlock(ptep, ptl);
--
2.56.0
next prev parent reply other threads:[~2026-11-16 21:01 UTC|newest]
Thread overview: 8+ messages / expand[flat|nested] mbox.gz Atom feed top
2026-11-16 21:01 [PATCH v5 0/8] mm/gup: walk and batch whole PTE tables in slow GUP Rik van Riel
2026-11-16 21:01 ` [PATCH v5 1/8] mm/gup: break out gup_fill_pages() helper Rik van Riel
2026-11-16 21:01 ` [PATCH v5 2/8] mm/gup: break out gup_flush_pages() helper Rik van Riel
2026-11-16 21:01 ` [PATCH v5 3/8] mm/gup: convert follow_page_mask() to return a long Rik van Riel
2026-11-16 21:01 ` Rik van Riel [this message]
2026-11-16 21:01 ` [PATCH v5 5/8] mm/gup: break out gup_lookup_pte() helper Rik van Riel
2026-11-16 21:01 ` [PATCH v5 7/8] mm/gup: walk multiple PTEs per follow_page_pte() call Rik van Riel
2026-11-16 21:01 ` [PATCH v5 8/8] mm/gup: batch contiguous same-folio PTEs into one refcount grab Rik van Riel
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20261116210127.3837898-5-riel@surriel.com \
--to=riel@surriel.com \
--cc=aa9736195201@gmail.com \
--cc=akpm@linux-foundation.org \
--cc=david@kernel.org \
--cc=hch@infradead.org \
--cc=jgg@ziepe.ca \
--cc=jhubbard@nvidia.com \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-mm@kvack.org \
--cc=ljs@kernel.org \
--cc=peterx@redhat.com \
--cc=surenb@google.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox