From: Rik van Riel <riel@surriel.com>
To: linux-kernel@vger.kernel.org
Cc: linux-mm@kvack.org, David Hildenbrand <david@kernel.org>,
Andrew Morton <akpm@linux-foundation.org>,
Suren Baghdasaryan <surenb@google.com>,
Lorenzo Stoakes <ljs@kernel.org>,
Christoph Hellwig <hch@infradead.org>,
Yuan-Hao Hsu <aa9736195201@gmail.com>,
John Hubbard <jhubbard@nvidia.com>,
Jason Gunthorpe <jgg@ziepe.ca>, Peter Xu <peterx@redhat.com>,
Rik van Riel <riel@surriel.com>
Subject: [PATCH v5 1/8] mm/gup: break out gup_fill_pages() helper
Date: Mon, 16 Nov 2026 16:01:20 -0500 [thread overview]
Message-ID: <20261116210127.3837898-2-riel@surriel.com> (raw)
In-Reply-To: <20261116210127.3837898-1-riel@surriel.com>
__get_user_pages() fills pages[] in an open-coded loop, and the
GUP-fast PTE, PMD and PUD leaf paths each have their own copy of
the same loop.
Move the store into a gup_fill_pages() helper, shared by the slow
and the fast paths. The helper keeps a pages==NULL check, so the
fault-in users that pass no pages array (faultin_page_range(),
populate_vma_page_range()) need no conditionals at their call
sites. Cache flushing stays open-coded for now; it moves into its
own helper in the next patch.
No functional changes intended.
Suggested-by: David Hildenbrand <david@kernel.org>
Signed-off-by: Rik van Riel <riel@surriel.com>
---
mm/gup.c | 35 +++++++++++++++++++++++++----------
1 file changed, 25 insertions(+), 10 deletions(-)
diff --git a/mm/gup.c b/mm/gup.c
index b0a0019de80a..e0698d2075b4 100644
--- a/mm/gup.c
+++ b/mm/gup.c
@@ -710,6 +710,25 @@ static inline bool gup_must_unshare(struct vm_area_struct *vma,
return gup_pages_not_requiring_unshare(vma, flags, folio, page, 1) != 1;
}
+/*
+ * Store @nr_pages consecutive pages starting at @page into @pages.
+ * The caller must hold at least one reference on the folio, so it
+ * cannot go away, or get split while the array is filled.
+ *
+ * Some GUP users only fault pages in and pass no pages array.
+ */
+static void gup_fill_pages(struct page **pages, struct page *page,
+ unsigned long nr_pages)
+{
+ unsigned long i;
+
+ if (!pages)
+ return;
+
+ for (i = 0; i < nr_pages; i++)
+ pages[i] = page + i;
+}
+
#ifdef CONFIG_PGTABLE_HAS_HUGE_LEAVES
/* FOLL_FORCE can write to even unwritable PUDs in COW mappings. */
static inline bool can_follow_write_pud(pud_t pud, struct page *page,
@@ -1579,9 +1598,9 @@ static long __get_user_pages(struct mm_struct *mm,
}
}
+ gup_fill_pages(pages + i, page, page_increm);
for (j = 0; j < page_increm; j++) {
subpage = page + j;
- pages[i + j] = subpage;
flush_anon_page(vma, subpage, start + j * PAGE_SIZE);
flush_dcache_page(subpage);
}
@@ -2946,7 +2965,6 @@ static unsigned long gup_fast_pte_range(pmd_t pmd, pmd_t *pmdp,
do {
pte_t pte = ptep_get_lockless(ptep);
struct page *page;
- unsigned long i;
nr_batch = 1;
@@ -3017,8 +3035,7 @@ static unsigned long gup_fast_pte_range(pmd_t pmd, pmd_t *pmdp,
nr_batch = 1;
folio_set_referenced(folio);
- for (i = 0; i < nr_batch; i++)
- pages[nr_pages + i] = page + i;
+ gup_fill_pages(pages + nr_pages, page, nr_batch);
nr_pages += nr_batch;
folio = NULL;
} while (ptep += nr_batch, addr += nr_batch * PAGE_SIZE, addr != end);
@@ -3054,7 +3071,7 @@ static unsigned long gup_fast_pmd_leaf(pmd_t orig, pmd_t *pmdp,
{
struct page *mapped_page, *page;
struct folio *folio;
- unsigned long nr_pages, i;
+ unsigned long nr_pages;
/* See gup_fast_pte_range() */
if (pmd_protnone(orig))
@@ -3089,8 +3106,7 @@ static unsigned long gup_fast_pmd_leaf(pmd_t orig, pmd_t *pmdp,
return 0;
}
- for (i = 0; i < nr_pages; i++)
- *(pages++) = page++;
+ gup_fill_pages(pages, page, nr_pages);
folio_set_referenced(folio);
return nr_pages;
}
@@ -3101,7 +3117,7 @@ static unsigned long gup_fast_pud_leaf(pud_t orig, pud_t *pudp,
{
struct page *mapped_page, *page;
struct folio *folio;
- unsigned long nr_pages, i;
+ unsigned long nr_pages;
if (!pud_access_permitted(orig, flags & FOLL_WRITE))
return 0;
@@ -3133,8 +3149,7 @@ static unsigned long gup_fast_pud_leaf(pud_t orig, pud_t *pudp,
return 0;
}
- for (i = 0; i < nr_pages; i++)
- *(pages++) = page++;
+ gup_fill_pages(pages, page, nr_pages);
folio_set_referenced(folio);
return nr_pages;
}
--
2.56.0
next prev parent reply other threads:[~2026-11-16 21:02 UTC|newest]
Thread overview: 11+ messages / expand[flat|nested] mbox.gz Atom feed top
2026-11-16 21:01 [PATCH v5 0/8] mm/gup: walk and batch whole PTE tables in slow GUP Rik van Riel
2026-11-16 21:01 ` Rik van Riel [this message]
2026-11-16 21:01 ` [PATCH v5 2/8] mm/gup: break out gup_flush_pages() helper Rik van Riel
2026-11-16 21:01 ` [PATCH v5 3/8] mm/gup: convert follow_page_mask() to return a long Rik van Riel
2026-11-16 21:01 ` [PATCH v5 4/8] mm/gup: split out folio grab-and-store helper Rik van Riel
2026-11-16 21:01 ` [PATCH v5 5/8] mm/gup: break out gup_lookup_pte() helper Rik van Riel
2026-11-17 13:48 ` Lorenzo Stoakes (ARM)
2026-11-16 21:01 ` [PATCH v5 6/8] mm/gup: return a huge page's full count from follow_page_mask() Rik van Riel
2026-11-16 21:01 ` [PATCH v5 7/8] mm/gup: walk multiple PTEs per follow_page_pte() call Rik van Riel
2026-11-16 21:01 ` [PATCH v5 8/8] mm/gup: batch contiguous same-folio PTEs into one refcount grab Rik van Riel
2026-11-17 13:30 ` [PATCH v5 0/8] mm/gup: walk and batch whole PTE tables in slow GUP Lorenzo Stoakes (ARM)
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20261116210127.3837898-2-riel@surriel.com \
--to=riel@surriel.com \
--cc=aa9736195201@gmail.com \
--cc=akpm@linux-foundation.org \
--cc=david@kernel.org \
--cc=hch@infradead.org \
--cc=jgg@ziepe.ca \
--cc=jhubbard@nvidia.com \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-mm@kvack.org \
--cc=ljs@kernel.org \
--cc=peterx@redhat.com \
--cc=surenb@google.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.