* [RFC PATCH 0/5] gup: introduce folio-range interface for pinning user memory
@ 2026-11-18 21:34 Swarna Prabhu
2026-11-18 21:34 ` [PATCH 1/5] mm/gup: add struct gup_out output descriptor Swarna Prabhu
` (5 more replies)
0 siblings, 6 replies; 7+ messages in thread
From: Swarna Prabhu @ 2026-11-18 21:34 UTC (permalink / raw)
To: akpm, david, jgg, jhubbard, viro
Cc: linux-mm, linux-block, linux-kernel, dave, riel, yeoreum.yun, hch,
Swarna Prabhu
From: Swarna Prabhu <s.prabhu@samsung.com>
This series is an RFC to gather design feedback to introduce a folio-range
interface for pinning user memory in the GUP fast path. Pinned memory is
returned as folio ranges instead of a page array: a run of consecutive PTEs
mapping consecutive pages of one folio is pinned once and handed back as a single
range. Block direct I/O converts each range into a single bvec, hence reducing the
per-page pin and merge work.
Base and dependencies
=====================
This series is rebased on the following two series applied:
- "mm: gup: batch PTE-mapped large folios in gup_fast_pte_range()"
v4 (Yeoreum Yun; originally posted by Yuan-Hao Hsu [3]), applied to
mm/core.git gup-7.5.ptebatch [4]. It batches the reference grabs for
PTE-mapped large folios in GUP-fast.
- "batch lookups in follow_page_mask()" v5 (Rik van Riel) [5]. It makes
follow_page_mask() return a page count and record pages through
gup_fill_pages().
Note: "[PATCH v5 1/8] mm/gup: break out gup_fill_pages() helper" was
posted just as testing for this RFC was done. To avoid delaying
feedback on the overall design, this RFC retains the tested base
linked in [5] and will be rebased onto the latest revision for next
version.
This series builds on both: the run detection in gup_fast_pte_range() and
the batched slow-path lookup are reused, and this series changes how the
result of a run is recorded and returned.
Background
==========
Kundan's LPC '24 talk "Transitioning get_user_pages to folio based mapping"
[1] proposed a struct folio_vec (based on bcachefs) passed in place of a
pages array. Building on that and on suggestions from the community [2],
this series exposes folio + index + nr_pages, where each range spans a
single folio:
struct folio_range {
struct folio *folio;
unsigned int idx;
unsigned int nr_pages;
};
Design
======
A "struct gup_out" descriptor in mm/gup.c holds either the page array or
the folio-range array. The fast and slow page-table walkers append to the
descriptor instead of advancing a pages[] pointer, so a single walk
serves both interfaces without duplicating the walkers.
The PTE batching in gup_fast_pte_range() is provided by Yeoreum Yun's
series [4], which this RFC builds on. This series reuses that batching
and its page-table revalidation logic. For the folio-range interface,
each pinned run is recorded directly as a folio range instead of being
expanded into individual page pointers in the output array.
pin_user_folio_ranges_fast() stops once the caller's range array is
full, so callers size the array in ranges, independently of the number
of pages requested. A single GUP-fast walk is additionally capped at
EXTRACT_MAX_PAGES (1024 pages) to bound the time spent with interrupts
disabled when the buffer is backed by small pages.
unpin_user_folio_ranges() walks the range array and calls gup_put_folio()
once per range, releasing the references pin_user_folio_ranges_fast()
took for it.
In iov_iter_extract_bvecs(), the pinned folio ranges are stored into the
free slot of the bvec array and converted in place, one bvec per range.
Block direct I/O picks this up through bio_iov_iter_get_pages() without
changes to the block layer.
Non-folio mappings
==================
Memory that is not backed by folios, such as pages a driver inserted
into a VM_MIXEDMAP VMA, makes pin_user_folio_ranges_fast() fail with
-EOPNOTSUPP when no pages were pinned; such iterators fall back to the
page-array path.
For folio-range output, gup_fast_folio_allowed() additionally requires
a non-NULL folio->mapping, except for hugetlb folios. If the mapping is
NULL, it falls back to the slow path, where the VMA can be checked.
The existing file-backed and secretmem restrictions remain unchanged.
FOLL_LONGTERM is not supported yet and is rejected.
Testing
=======
Measured the time spent in bio_iov_iter_get_pages() per I/O with
function_graph, on the baseline and the baseline plus this series.
Both kernels ran with the performance governor, boost disabled, all
C-states except POLL disabled, and fio pinned to one CPU:
trace-cmd record -p function_graph -g bio_iov_iter_get_pages \
--max-graph-depth 8 -O funcgraph-overhead -O nofuncgraph-irqs -F -c \
fio --name=io_uring_test --filename=/dev/nvme1n1 --ioengine=io_uring \
--rw=randwrite --direct=1 --bs=<bs> --iodepth=8 --iomem=mmap \
--iomem_align=2M --numjobs=1 --size=64M --cpus_allowed=23
Median of calls not interrupted during the walk:
bs buffer baseline patched result
128K 64 kB mTHP 13.08 us 13.09 us neutral
1M 64 kB mTHP 63.0 us 63.6 us neutral
4M 2 MB THP (PMD) 25.9 us 8.65 us ~3.0x faster
8M 2 MB THP (PMD) 49.4 us ~17.0 us ~2.9x faster
Note: the baseline is the upstream kernel with Yeoreum Yun's [4] and
Rik van Riel's [5] changes applied.
For larger I/O, the baseline's bio extraction path requests at max
256 pages per GUP call, limited by the available bvec slots. This
series instead uses those slots for folio ranges, each of which can
describe many pages, thereby allowing the series to request up to 1024
pages per call which accounts for the speed up observed in
'bio_iov_iter_get_pages' for 4MB and 8MB DIRECT I/Os.
Testing: Ran selftests gup_test, gup_longterm and cow on the patched kernel
and no regressions were observed.
Link [1]: https://lpc.events/event/18/contributions/1773/
Link [2]: https://lore.kernel.org/all/f2b5189f-10de-4685-97f3-6ee08d159743@kernel.org/
Link [3]: https://lore.kernel.org/all/20260917083747.786-1-aa9736195201@gmail.com/
Link [4]: https://lore.kernel.org/all/20261110-refactor_gup_fast-v4-0-e779a7674274@arm.com/
Link [5]: https://lore.kernel.org/all/20261102193026.3860700-1-riel@surriel.com/
Swarna Prabhu (5):
mm/gup: add struct gup_out output descriptor
mm/gup: make the slow path walkers record into struct gup_out
mm/gup: make the GUP-fast walkers record into struct gup_out
mm/gup: return pinned memory as folio ranges
iov_iter: extract bvecs of user-backed iters from folio ranges
include/linux/mm.h | 37 +++
include/linux/mm_types.h | 23 ++
lib/iov_iter.c | 100 ++++++
mm/gup.c | 695 +++++++++++++++++++++++++++++++--------
4 files changed, 717 insertions(+), 138 deletions(-)
--
2.43.0
^ permalink raw reply [flat|nested] 7+ messages in thread
* [PATCH 1/5] mm/gup: add struct gup_out output descriptor
2026-11-18 21:34 [RFC PATCH 0/5] gup: introduce folio-range interface for pinning user memory Swarna Prabhu
@ 2026-11-18 21:34 ` Swarna Prabhu
2026-11-18 21:34 ` [PATCH 2/5] mm/gup: make the slow path walkers record into struct gup_out Swarna Prabhu
` (4 subsequent siblings)
5 siblings, 0 replies; 7+ messages in thread
From: Swarna Prabhu @ 2026-11-18 21:34 UTC (permalink / raw)
To: akpm, david, jgg, jhubbard, viro
Cc: linux-mm, linux-block, linux-kernel, dave, riel, yeoreum.yun, hch,
Swarna Prabhu
From: Swarna Prabhu <s.prabhu@samsung.com>
The GUP page table walkers hand the pages they grabbed back through a
pages[] pointer that every level bumps on its own, which ties them to
reporting memory as an array of pages.
Add 'struct gup_out', a descriptor the walkers can append to instead,
along with 'gup_out_add_range()' to record consecutive pages of a single
folio. The descriptor keeps count of the pages recorded so far, so a
walker no longer has to care where in the array it is, no matter how
often it is called for a single GUP request.
The @folio argument of 'gup_out_add_range()' is not used yet, it is
there for describing memory by its folio later in the series.
No functional change.
Signed-off-by: Swarna Prabhu <s.prabhu@samsung.com>
---
mm/gup.c | 39 +++++++++++++++++++++++++++++++++++++++
1 file changed, 39 insertions(+)
diff --git a/mm/gup.c b/mm/gup.c
index 3b3888578a7e..0433579ed618 100644
--- a/mm/gup.c
+++ b/mm/gup.c
@@ -482,6 +482,45 @@ static inline void mm_set_has_pinned_flag(struct mm_struct *mm)
mm_flags_set(MMF_HAS_PINNED, mm);
}
+/*
+ * Output descriptor of the GUP page table walkers.
+ *
+ * GUP hands the memory it grabbed back to the caller as an array of pages
+ * (@pages).
+ *
+ * @nr_pages is the number of pages recorded so far: the walkers simply keep
+ * appending to the descriptor, no matter how often they are called for a
+ * single GUP request.
+ */
+struct gup_out {
+ struct page **pages;
+ unsigned long nr_pages;
+};
+
+#define GUP_OUT_PAGES(_pages) ((struct gup_out){ .pages = (_pages) })
+
+/**
+ * gup_out_add_range() - record consecutive pages of a single folio
+ * @out: the output descriptor
+ * @folio: the folio @page belongs to
+ * @page: the first page to record
+ * @nr_pages: the number of consecutive pages to record
+ *
+ * The caller must hold a reference (FOLL_GET) or pin (FOLL_PIN) for each of
+ * the @nr_pages pages.
+ */
+static inline void gup_out_add_range(struct gup_out *out, struct folio *folio,
+ struct page *page, unsigned long nr_pages)
+{
+ struct page **pages = out->pages + out->nr_pages;
+ unsigned long i;
+
+ for (i = 0; i < nr_pages; i++)
+ pages[i] = page + i;
+
+ out->nr_pages += nr_pages;
+}
+
#ifdef CONFIG_MMU
#ifdef CONFIG_HAVE_GUP_FAST
--
2.53.0
^ permalink raw reply related [flat|nested] 7+ messages in thread
* [PATCH 2/5] mm/gup: make the slow path walkers record into struct gup_out
2026-11-18 21:34 [RFC PATCH 0/5] gup: introduce folio-range interface for pinning user memory Swarna Prabhu
2026-11-18 21:34 ` [PATCH 1/5] mm/gup: add struct gup_out output descriptor Swarna Prabhu
@ 2026-11-18 21:34 ` Swarna Prabhu
2026-11-18 21:34 ` [PATCH 3/5] mm/gup: make the GUP-fast " Swarna Prabhu
` (3 subsequent siblings)
5 siblings, 0 replies; 7+ messages in thread
From: Swarna Prabhu @ 2026-11-18 21:34 UTC (permalink / raw)
To: akpm, david, jgg, jhubbard, viro
Cc: linux-mm, linux-block, linux-kernel, dave, riel, yeoreum.yun, hch,
Swarna Prabhu
From: Swarna Prabhu <s.prabhu@samsung.com>
Pass 'struct gup_out' down the slow path in place of the pages array.
For PMD/PUD leaves, receive the first page in a local variable and
append the run through 'gup_fill_pages' after unlocking.
'gup_grab_folio_pages()' appends PTE runs under the page table lock
and 'gup_flush_pages()' flushes the newly recorded pages once
the lock has been dropped.
Descriptor based appending removes the pages[] pointer bumping in
'__get_user_pages_locked()'. !CONFIG_MMU walker records through the
descriptor as well.
Retain the existing array-based interface as wrappers around the new
*_out variants. With FOLL_LONGTERM a retry restores the descriptor
to where it was before, discarding what the failed attempt recorded.
No functional change.
Signed-off-by: Swarna Prabhu <s.prabhu@samsung.com>
Assisted-by: LLM
---
mm/gup.c | 214 ++++++++++++++++++++++++++++++++++---------------------
1 file changed, 131 insertions(+), 83 deletions(-)
diff --git a/mm/gup.c b/mm/gup.c
index 0433579ed618..0b8dd3fde06b 100644
--- a/mm/gup.c
+++ b/mm/gup.c
@@ -749,36 +749,40 @@ static inline bool gup_must_unshare(struct vm_area_struct *vma,
return gup_pages_not_requiring_unshare(vma, flags, folio, page, 1) != 1;
}
+/* Record @nr pages starting at @page in @out, and flush their caches. */
static void gup_fill_pages(struct vm_area_struct *vma, unsigned long address,
- struct page *page, unsigned long nr, struct page **pages)
+ struct page *page, unsigned long nr, struct gup_out *out)
{
unsigned long i;
- if (!pages)
+ if (!out)
return;
+ gup_out_add_range(out, page_folio(page), page, nr);
+
for (i = 0; i < nr; i++) {
struct page *subpage = page + i;
- pages[i] = subpage;
flush_anon_page(vma, subpage, address + i * PAGE_SIZE);
flush_dcache_page(subpage);
}
}
/*
- * Flush the caches for @nr pages stored in @pages. The caller already
- * holds folio references, so the pages cannot go away; this runs
- * without any page table lock held.
+ * Flush the caches for the @nr pages most recently recorded in @out, the first
+ * of which is mapped at @address. The caller already holds folio references,
+ * so the pages cannot go away; this runs without any page table lock held.
*/
static void gup_flush_pages(struct vm_area_struct *vma, unsigned long address,
- struct page **pages, unsigned long nr)
+ struct gup_out *out, unsigned long nr)
{
+ struct page **pages;
unsigned long i;
- if (!pages)
+ if (!out)
return;
+ pages = out->pages + out->nr_pages - nr;
for (i = 0; i < nr; i++) {
flush_anon_page(vma, pages[i], address + i * PAGE_SIZE);
flush_dcache_page(pages[i]);
@@ -972,9 +976,8 @@ static inline bool can_follow_write_pte(pte_t pte, struct page *page,
*/
static long gup_grab_folio_pages(struct folio *folio,
struct page *page, unsigned long nr, bool dirty,
- unsigned int flags, struct page **pages)
+ unsigned int flags, struct gup_out *out)
{
- unsigned long i;
long ret;
/* try_grab_folio() does nothing unless FOLL_GET or FOLL_PIN is set. */
@@ -1005,10 +1008,8 @@ static long gup_grab_folio_pages(struct folio *folio,
folio_mark_accessed(folio);
}
- if (pages) {
- for (i = 0; i < nr; i++)
- pages[i] = page + i;
- }
+ if (out)
+ gup_out_add_range(out, folio, page, nr);
return 0;
}
@@ -1100,7 +1101,7 @@ static unsigned long gup_pte_batch_len(struct vm_area_struct *vma,
*/
static long follow_page_pte(struct vm_area_struct *vma,
unsigned long address, unsigned long end, pmd_t *pmd,
- unsigned int flags, struct page **pages)
+ unsigned int flags, struct gup_out *out)
{
struct mm_struct *mm = vma->vm_mm;
bool need_no_page_table = false;
@@ -1143,8 +1144,7 @@ static long follow_page_pte(struct vm_area_struct *vma,
batch = gup_pte_batch_len(vma, address, walk_end, folio,
ptep, &batch_pte, flags);
ret = gup_grab_folio_pages(folio, page, batch,
- pte_dirty(batch_pte), flags,
- pages ? pages + nr : NULL);
+ pte_dirty(batch_pte), flags, out);
if (ret)
break;
nr += batch;
@@ -1158,7 +1158,7 @@ static long follow_page_pte(struct vm_area_struct *vma,
pte_unmap_unlock(orig_ptep, ptl);
if (nr) {
- gup_flush_pages(vma, base, pages, nr);
+ gup_flush_pages(vma, base, out, nr);
return nr;
}
if (need_no_page_table)
@@ -1168,9 +1168,10 @@ static long follow_page_pte(struct vm_area_struct *vma,
static long follow_pmd_mask(struct vm_area_struct *vma,
unsigned long address, unsigned long end, pud_t *pudp,
- unsigned int flags, struct page **pages)
+ unsigned int flags, struct gup_out *out)
{
pmd_t *pmd, pmdval;
+ struct page *page;
spinlock_t *ptl;
long ret;
struct mm_struct *mm = vma->vm_mm;
@@ -1182,7 +1183,7 @@ static long follow_pmd_mask(struct vm_area_struct *vma,
if (!pmd_present(pmdval))
return no_page_table(vma, flags, address);
if (likely(!pmd_leaf(pmdval)))
- return follow_page_pte(vma, address, end, pmd, flags, pages);
+ return follow_page_pte(vma, address, end, pmd, flags, out);
if (pmd_protnone(pmdval) && !gup_can_follow_protnone(vma, flags))
return no_page_table(vma, flags, address);
@@ -1195,33 +1196,34 @@ static long follow_pmd_mask(struct vm_area_struct *vma,
}
if (unlikely(!pmd_leaf(pmdval))) {
spin_unlock(ptl);
- return follow_page_pte(vma, address, end, pmd, flags, pages);
+ return follow_page_pte(vma, address, end, pmd, flags, out);
}
if (pmd_trans_huge(pmdval) && (flags & FOLL_SPLIT_PMD)) {
spin_unlock(ptl);
split_huge_pmd(vma, pmd, address);
/* If pmd was left empty, stuff a page table in there quickly */
return pte_alloc(mm, pmd) ? -ENOMEM :
- follow_page_pte(vma, address, end, pmd, flags, pages);
+ follow_page_pte(vma, address, end, pmd, flags, out);
}
- ret = follow_huge_pmd(vma, address, end, pmd, flags, pages);
+ ret = follow_huge_pmd(vma, address, end, pmd, flags, &page);
spin_unlock(ptl);
/*
- * The ref is already held, so the page cannot go away: fill the
- * array and flush caches without the pmd lock.
+ * The ref is already held, so the page cannot go away: record the
+ * pages and flush caches without the pmd lock.
*/
- if (ret > 0 && pages)
- gup_fill_pages(vma, address, pages[0], ret, pages);
+ if (ret > 0)
+ gup_fill_pages(vma, address, page, ret, out);
return ret;
}
static long follow_pud_mask(struct vm_area_struct *vma,
unsigned long address, unsigned long end, p4d_t *p4dp,
- unsigned int flags, struct page **pages)
+ unsigned int flags, struct gup_out *out)
{
pud_t *pudp, pud;
+ struct page *page;
spinlock_t *ptl;
long ret;
struct mm_struct *mm = vma->vm_mm;
@@ -1232,16 +1234,16 @@ static long follow_pud_mask(struct vm_area_struct *vma,
return no_page_table(vma, flags, address);
if (pud_leaf(pud)) {
ptl = pud_lock(mm, pudp);
- ret = follow_huge_pud(vma, address, end, pudp, flags, pages);
+ ret = follow_huge_pud(vma, address, end, pudp, flags, &page);
spin_unlock(ptl);
/*
- * The ref is already held, so the page cannot go away: fill
- * the array and flush caches without the lock. A 1 GB folio
+ * The ref is already held, so the page cannot go away: record
+ * the pages and flush caches without the lock. A 1 GB folio
* can be up to HPAGE_PUD_NR pages, too long to flush under a
* spinlock.
*/
- if (ret > 0 && pages)
- gup_fill_pages(vma, address, pages[0], ret, pages);
+ if (ret > 0)
+ gup_fill_pages(vma, address, page, ret, out);
if (ret)
return ret;
return no_page_table(vma, flags, address);
@@ -1249,12 +1251,12 @@ static long follow_pud_mask(struct vm_area_struct *vma,
if (unlikely(pud_bad(pud)))
return no_page_table(vma, flags, address);
- return follow_pmd_mask(vma, address, end, pudp, flags, pages);
+ return follow_pmd_mask(vma, address, end, pudp, flags, out);
}
static long follow_p4d_mask(struct vm_area_struct *vma,
unsigned long address, unsigned long end, pgd_t *pgdp,
- unsigned int flags, struct page **pages)
+ unsigned int flags, struct gup_out *out)
{
p4d_t *p4dp, p4d;
@@ -1265,7 +1267,7 @@ static long follow_p4d_mask(struct vm_area_struct *vma,
if (!p4d_present(p4d) || p4d_bad(p4d))
return no_page_table(vma, flags, address);
- return follow_pud_mask(vma, address, end, p4dp, flags, pages);
+ return follow_pud_mask(vma, address, end, p4dp, flags, out);
}
/**
@@ -1274,9 +1276,9 @@ static long follow_p4d_mask(struct vm_area_struct *vma,
* @address: virtual address to look up
* @end: virtual address at which to stop batching contiguous pages
* @flags: flags modifying lookup behaviour
- * @pages: array to receive the pages, refcounted per @flags, or NULL to
- * walk the page tables (e.g. to fault pages in) without collecting
- * or refcounting them
+ * @out: output descriptor to receive the pages, refcounted per @flags, or
+ * NULL to walk the page tables (e.g. to fault pages in) without
+ * collecting or refcounting them
*
* @flags can have FOLL_ flags set, defined in <linux/mm.h>
*
@@ -1286,14 +1288,14 @@ static long follow_p4d_mask(struct vm_area_struct *vma,
* relevant with FOLL_PIN and !FOLL_WRITE.
*
* Return: the number of contiguous pages starting at @address that were
- * placed into @pages (if non-NULL), which may be fewer than the pages
+ * recorded in @out (if non-NULL), which may be fewer than the pages
* requested via @end; 0 if no mapping exists at @address; or a negative
* errno for a mapping to something not represented by a page descriptor
* (see also vm_normal_page()).
*/
static long follow_page_mask(struct vm_area_struct *vma,
unsigned long address, unsigned long end,
- unsigned int flags, struct page **pages)
+ unsigned int flags, struct gup_out *out)
{
pgd_t *pgd;
struct mm_struct *mm = vma->vm_mm;
@@ -1306,7 +1308,7 @@ static long follow_page_mask(struct vm_area_struct *vma,
if (pgd_none(*pgd) || unlikely(pgd_bad(*pgd)))
ret = no_page_table(vma, flags, address);
else
- ret = follow_p4d_mask(vma, address, end, pgd, flags, pages);
+ ret = follow_p4d_mask(vma, address, end, pgd, flags, out);
vma_pgtable_walk_end(vma);
@@ -1589,9 +1591,8 @@ static struct vm_area_struct *gup_vma_lookup(struct mm_struct *mm,
* @start: starting user address
* @nr_pages: number of pages from start to pin
* @gup_flags: flags modifying pin behaviour
- * @pages: array that receives pointers to the pages pinned.
- * Should be at least nr_pages long. Or NULL, if caller
- * only intends to ensure the pages are faulted in.
+ * @out: output descriptor receiving the pinned pages, or NULL if the
+ * caller only intends to ensure the pages are faulted in.
* @locked: whether we're still with the mmap_lock held
*
* Returns either number of pages pinned (which may be less than the
@@ -1639,7 +1640,7 @@ static struct vm_area_struct *gup_vma_lookup(struct mm_struct *mm,
*/
static long __get_user_pages(struct mm_struct *mm,
unsigned long start, unsigned long nr_pages,
- unsigned int gup_flags, struct page **pages,
+ unsigned int gup_flags, struct gup_out *out,
int *locked)
{
long ret = 0, i = 0;
@@ -1650,7 +1651,7 @@ static long __get_user_pages(struct mm_struct *mm,
start = untagged_addr_remote(mm, start);
- VM_WARN_ON_ONCE(!!pages != !!(gup_flags & (FOLL_GET | FOLL_PIN)));
+ VM_WARN_ON_ONCE(!!out != !!(gup_flags & (FOLL_GET | FOLL_PIN)));
/* FOLL_GET and FOLL_PIN are mutually exclusive. */
VM_WARN_ON_ONCE((gup_flags & (FOLL_PIN | FOLL_GET)) ==
@@ -1682,11 +1683,10 @@ static long __get_user_pages(struct mm_struct *mm,
if (!vma && in_gate_area(mm, start)) {
ret = get_gate_page(mm, start & PAGE_MASK,
gup_flags, &vma,
- pages ? &page : NULL);
+ out ? &page : NULL);
if (ret)
goto out;
- gup_fill_pages(vma, start, page, 1,
- pages ? pages + i : NULL);
+ gup_fill_pages(vma, start, page, 1, out);
i++;
start += PAGE_SIZE;
nr_pages--;
@@ -1713,7 +1713,7 @@ static long __get_user_pages(struct mm_struct *mm,
cond_resched();
nr = follow_page_mask(vma, start, start + nr_pages * PAGE_SIZE,
- gup_flags, pages ? &pages[i] : NULL);
+ gup_flags, out);
if (!nr || nr == -EMLINK) {
ret = faultin_page(vma, start, gup_flags,
nr == -EMLINK, locked);
@@ -1733,12 +1733,12 @@ static long __get_user_pages(struct mm_struct *mm,
} else if (nr == -EEXIST) {
/*
* Proper page table entry exists, but no corresponding
- * struct page. If the caller expects **pages to be
- * filled in, bail out now, because that can't be done
+ * struct page. If the caller expects the pages to be
+ * reported, bail out now, because that can't be done
* for this page. Otherwise advance by the one page
* follow_page_mask() looked at.
*/
- if (pages) {
+ if (out) {
ret = nr;
goto out;
}
@@ -1896,10 +1896,10 @@ static bool gup_signal_pending(unsigned int flags)
* Please note that this function, unlike __get_user_pages(), will not return 0
* for nr_pages > 0, unless FOLL_NOWAIT is used.
*/
-static __always_inline long __get_user_pages_locked(struct mm_struct *mm,
+static __always_inline long __get_user_pages_locked_out(struct mm_struct *mm,
unsigned long start,
unsigned long nr_pages,
- struct page **pages,
+ struct gup_out *out,
int *locked,
unsigned int flags)
{
@@ -1934,12 +1934,12 @@ static __always_inline long __get_user_pages_locked(struct mm_struct *mm,
* FOLL_PIN always expects pages to be non-null, but no need to assert
* that here, as any failures will be obvious enough.
*/
- if (pages && !(flags & FOLL_PIN))
+ if (out && !(flags & FOLL_PIN))
flags |= FOLL_GET;
pages_done = 0;
for (;;) {
- ret = __get_user_pages(mm, start, nr_pages, flags, pages,
+ ret = __get_user_pages(mm, start, nr_pages, flags, out,
locked);
if (!(flags & FOLL_UNLOCKABLE)) {
/* VM_FAULT_RETRY couldn't trigger, bypass */
@@ -1965,12 +1965,7 @@ static __always_inline long __get_user_pages_locked(struct mm_struct *mm,
pages_done = ret;
break;
}
- /*
- * VM_FAULT_RETRY triggered, so seek to the faulting offset.
- * For the prefault case (!pages) we only update counts.
- */
- if (likely(pages))
- pages += ret;
+ /* VM_FAULT_RETRY triggered, so seek to the faulting offset. */
start += ret << PAGE_SHIFT;
/* The lock was temporarily dropped, so we must unlock later */
@@ -2000,7 +1995,7 @@ static __always_inline long __get_user_pages_locked(struct mm_struct *mm,
*locked = 1;
ret = __get_user_pages(mm, start, 1, flags | FOLL_TRIED,
- pages, locked);
+ out, locked);
if (!*locked) {
/* Continue to retry until we succeeded */
VM_WARN_ON_ONCE(ret != 0);
@@ -2016,8 +2011,6 @@ static __always_inline long __get_user_pages_locked(struct mm_struct *mm,
pages_done++;
if (!nr_pages)
break;
- if (likely(pages))
- pages++;
start += PAGE_SIZE;
}
if (must_unlock && *locked) {
@@ -2040,6 +2033,19 @@ static __always_inline long __get_user_pages_locked(struct mm_struct *mm,
return pages_done;
}
+static __always_inline long __get_user_pages_locked(struct mm_struct *mm,
+ unsigned long start,
+ unsigned long nr_pages,
+ struct page **pages,
+ int *locked,
+ unsigned int flags)
+{
+ struct gup_out out = GUP_OUT_PAGES(pages);
+
+ return __get_user_pages_locked_out(mm, start, nr_pages,
+ pages ? &out : NULL, locked, flags);
+}
+
/**
* populate_vma_page_range() - populate a range of pages in the vma.
* @vma: target vma
@@ -2230,8 +2236,8 @@ int __mm_populate(unsigned long start, unsigned long len, int ignore_errors)
return ret; /* 0 or negative error code */
}
#else /* CONFIG_MMU */
-static long __get_user_pages_locked(struct mm_struct *mm, unsigned long start,
- unsigned long nr_pages, struct page **pages,
+static long __get_user_pages_locked_out(struct mm_struct *mm, unsigned long start,
+ unsigned long nr_pages, struct gup_out *out,
int *locked, unsigned int foll_flags)
{
struct vm_area_struct *vma;
@@ -2255,7 +2261,7 @@ static long __get_user_pages_locked(struct mm_struct *mm, unsigned long start,
}
/* See the MMU variant: support the traditional behavior. */
- if (pages && !(foll_flags & FOLL_PIN))
+ if (out && !(foll_flags & FOLL_PIN))
foll_flags |= FOLL_GET;
/* calculate required read or write permissions.
@@ -2276,16 +2282,19 @@ static long __get_user_pages_locked(struct mm_struct *mm, unsigned long start,
!(vm_flags & vma->vm_flags))
break;
- if (pages) {
- pages[i] = virt_to_page((void *)start);
- if (!pages[i])
+ if (out) {
+ struct page *page = virt_to_page((void *)start);
+ struct folio *folio;
+
+ if (!page)
break;
- ret = try_grab_folio(page_folio(pages[i]), 1, foll_flags);
+ folio = page_folio(page);
+ ret = try_grab_folio(folio, 1, foll_flags);
if (unlikely(ret)) {
- pages[i] = NULL;
err = ret;
break;
}
+ gup_out_add_range(out, folio, page, 1);
}
start = (start + PAGE_SIZE) & PAGE_MASK;
@@ -2298,6 +2307,17 @@ static long __get_user_pages_locked(struct mm_struct *mm, unsigned long start,
return i ? : err;
}
+
+static long __get_user_pages_locked(struct mm_struct *mm, unsigned long start,
+ unsigned long nr_pages, struct page **pages,
+ int *locked, unsigned int foll_flags)
+{
+ struct gup_out out = GUP_OUT_PAGES(pages);
+
+ return __get_user_pages_locked_out(mm, start, nr_pages,
+ pages ? &out : NULL, locked,
+ foll_flags);
+}
#endif /* !CONFIG_MMU */
/**
@@ -2722,37 +2742,65 @@ static long check_and_migrate_movable_folios(unsigned long nr_folios,
* __gup_longterm_locked() is a wrapper for __get_user_pages_locked which
* allows us to process the FOLL_LONGTERM flag.
*/
-static long __gup_longterm_locked(struct mm_struct *mm,
+static long __gup_longterm_locked_out(struct mm_struct *mm,
unsigned long start,
unsigned long nr_pages,
- struct page **pages,
+ struct gup_out *out,
int *locked,
unsigned int gup_flags)
{
+ unsigned long nr_pages_done;
unsigned int flags;
long rc, nr_pinned_pages;
if (!(gup_flags & FOLL_LONGTERM))
- return __get_user_pages_locked(mm, start, nr_pages, pages,
- locked, gup_flags);
+ return __get_user_pages_locked_out(mm, start, nr_pages, out,
+ locked, gup_flags);
+
+ if (WARN_ON_ONCE(!out))
+ return -EINVAL;
+
+ /*
+ * We might have to start over, in which case everything recorded by
+ * this call has to be discarded again.
+ */
+ nr_pages_done = out->nr_pages;
flags = memalloc_pin_save();
do {
- nr_pinned_pages = __get_user_pages_locked(mm, start, nr_pages,
- pages, locked,
- gup_flags);
+ out->nr_pages = nr_pages_done;
+
+ nr_pinned_pages = __get_user_pages_locked_out(mm, start,
+ nr_pages, out,
+ locked,
+ gup_flags);
if (nr_pinned_pages <= 0) {
rc = nr_pinned_pages;
break;
}
/* FOLL_LONGTERM implies FOLL_PIN */
- rc = check_and_migrate_movable_pages(nr_pinned_pages, pages);
+ rc = check_and_migrate_movable_pages(nr_pinned_pages,
+ out->pages + nr_pages_done);
} while (rc == -EAGAIN);
memalloc_pin_restore(flags);
return rc ? rc : nr_pinned_pages;
}
+static long __gup_longterm_locked(struct mm_struct *mm,
+ unsigned long start,
+ unsigned long nr_pages,
+ struct page **pages,
+ int *locked,
+ unsigned int gup_flags)
+{
+ struct gup_out out = GUP_OUT_PAGES(pages);
+
+ return __gup_longterm_locked_out(mm, start, nr_pages,
+ pages ? &out : NULL, locked,
+ gup_flags);
+}
+
#define INTERNAL_GUP_FLAGS (FOLL_TOUCH | FOLL_TRIED | FOLL_REMOTE | FOLL_PIN | \
FOLL_FAST_ONLY | FOLL_UNLOCKABLE | \
FOLL_MADV_POPULATE)
--
2.53.0
^ permalink raw reply related [flat|nested] 7+ messages in thread
* [PATCH 3/5] mm/gup: make the GUP-fast walkers record into struct gup_out
2026-11-18 21:34 [RFC PATCH 0/5] gup: introduce folio-range interface for pinning user memory Swarna Prabhu
2026-11-18 21:34 ` [PATCH 1/5] mm/gup: add struct gup_out output descriptor Swarna Prabhu
2026-11-18 21:34 ` [PATCH 2/5] mm/gup: make the slow path walkers record into struct gup_out Swarna Prabhu
@ 2026-11-18 21:34 ` Swarna Prabhu
2026-11-18 21:34 ` [PATCH 4/5] mm/gup: return pinned memory as folio ranges Swarna Prabhu
` (2 subsequent siblings)
5 siblings, 0 replies; 7+ messages in thread
From: Swarna Prabhu @ 2026-11-18 21:34 UTC (permalink / raw)
To: akpm, david, jgg, jhubbard, viro
Cc: linux-mm, linux-block, linux-kernel, dave, riel, yeoreum.yun, hch,
Swarna Prabhu
From: Swarna Prabhu <s.prabhu@samsung.com>
Pass 'struct gup_out' down the GUP-fast walkers and record acquired
pages through 'gup_out_add_range()' replacing pages array indexing and
pointer advancement. Record each existing PTE batch with one append.
Retain the walker's page-count return value so that callers can detect
short walks. Have gup_fast() obtain its result from the descriptor and
reset the count after releasing pins when a concurrent fork is detected.
Move fallback handling into gup_fast_fallback_out(), sharing the
descriptor with the slow path so recording continues after the pages
acquired by GUP-fast. Keep gup_fast_fallback() as an array-based wrapper
that initializes an empty descriptor.
No functional change.
Signed-off-by: Swarna Prabhu <s.prabhu@samsung.com>
Assisted-by: LLM
---
mm/gup.c | 94 +++++++++++++++++++++++++++++++++-----------------------
1 file changed, 56 insertions(+), 38 deletions(-)
diff --git a/mm/gup.c b/mm/gup.c
index 0b8dd3fde06b..43b92058a213 100644
--- a/mm/gup.c
+++ b/mm/gup.c
@@ -3146,7 +3146,7 @@ static bool gup_fast_folio_allowed(struct folio *folio, unsigned int flags)
*/
static unsigned long gup_fast_pte_range(pmd_t pmd, pmd_t *pmdp,
unsigned long addr, unsigned long end,
- unsigned int flags, struct page **pages)
+ unsigned int flags, struct gup_out *out)
{
unsigned long nr_pages = 0, nr_batch;
struct folio *folio = NULL;
@@ -3158,7 +3158,6 @@ static unsigned long gup_fast_pte_range(pmd_t pmd, pmd_t *pmdp,
do {
pte_t pte = ptep_get_lockless(ptep);
struct page *page;
- unsigned long i;
nr_batch = 1;
@@ -3229,8 +3228,7 @@ static unsigned long gup_fast_pte_range(pmd_t pmd, pmd_t *pmdp,
nr_batch = 1;
folio_set_referenced(folio);
- for (i = 0; i < nr_batch; i++)
- pages[nr_pages + i] = page + i;
+ gup_out_add_range(out, folio, page, nr_batch);
nr_pages += nr_batch;
folio = NULL;
} while (ptep += nr_batch, addr += nr_batch * PAGE_SIZE, addr != end);
@@ -3254,7 +3252,7 @@ static unsigned long gup_fast_pte_range(pmd_t pmd, pmd_t *pmdp,
*/
static unsigned long gup_fast_pte_range(pmd_t pmd, pmd_t *pmdp,
unsigned long addr, unsigned long end,
- unsigned int flags, struct page **pages)
+ unsigned int flags, struct gup_out *out)
{
return 0;
}
@@ -3262,11 +3260,11 @@ static unsigned long gup_fast_pte_range(pmd_t pmd, pmd_t *pmdp,
static unsigned long gup_fast_pmd_leaf(pmd_t orig, pmd_t *pmdp,
unsigned long addr, unsigned long end,
- unsigned int flags, struct page **pages)
+ unsigned int flags, struct gup_out *out)
{
struct page *mapped_page, *page;
struct folio *folio;
- unsigned long nr_pages, i;
+ unsigned long nr_pages;
/* See gup_fast_pte_range() */
if (pmd_protnone(orig))
@@ -3301,19 +3299,18 @@ static unsigned long gup_fast_pmd_leaf(pmd_t orig, pmd_t *pmdp,
return 0;
}
- for (i = 0; i < nr_pages; i++)
- *(pages++) = page++;
+ gup_out_add_range(out, folio, page, nr_pages);
folio_set_referenced(folio);
return nr_pages;
}
static unsigned long gup_fast_pud_leaf(pud_t orig, pud_t *pudp,
unsigned long addr, unsigned long end,
- unsigned int flags, struct page **pages)
+ unsigned int flags, struct gup_out *out)
{
struct page *mapped_page, *page;
struct folio *folio;
- unsigned long nr_pages, i;
+ unsigned long nr_pages;
if (!pud_access_permitted(orig, flags & FOLL_WRITE))
return 0;
@@ -3345,15 +3342,14 @@ static unsigned long gup_fast_pud_leaf(pud_t orig, pud_t *pudp,
return 0;
}
- for (i = 0; i < nr_pages; i++)
- *(pages++) = page++;
+ gup_out_add_range(out, folio, page, nr_pages);
folio_set_referenced(folio);
return nr_pages;
}
static unsigned long gup_fast_pmd_range(pud_t *pudp, pud_t pud,
unsigned long addr, unsigned long end,
- unsigned int flags, struct page **pages)
+ unsigned int flags, struct gup_out *out)
{
unsigned long next, nr_pages = 0, chunk_nr_pages;
pmd_t *pmdp;
@@ -3369,12 +3365,12 @@ static unsigned long gup_fast_pmd_range(pud_t *pudp, pud_t pud,
if (unlikely(pmd_leaf(pmd))) {
chunk_nr_pages = gup_fast_pmd_leaf(pmd, pmdp, addr,
next, flags,
- &pages[nr_pages]);
+ out);
} else
chunk_nr_pages = gup_fast_pte_range(pmd, pmdp, addr,
next, flags,
- &pages[nr_pages]);
+ out);
nr_pages += chunk_nr_pages;
if (chunk_nr_pages != (next - addr) >> PAGE_SHIFT)
break;
@@ -3385,7 +3381,7 @@ static unsigned long gup_fast_pmd_range(pud_t *pudp, pud_t pud,
static unsigned long gup_fast_pud_range(p4d_t *p4dp, p4d_t p4d,
unsigned long addr, unsigned long end,
- unsigned int flags, struct page **pages)
+ unsigned int flags, struct gup_out *out)
{
unsigned long next, nr_pages = 0, chunk_nr_pages;
pud_t *pudp;
@@ -3400,11 +3396,11 @@ static unsigned long gup_fast_pud_range(p4d_t *p4dp, p4d_t p4d,
if (unlikely(pud_leaf(pud)))
chunk_nr_pages = gup_fast_pud_leaf(pud, pudp, addr,
next, flags,
- &pages[nr_pages]);
+ out);
else
chunk_nr_pages = gup_fast_pmd_range(pudp, pud, addr,
next, flags,
- &pages[nr_pages]);
+ out);
nr_pages += chunk_nr_pages;
if (chunk_nr_pages != (next - addr) >> PAGE_SHIFT)
break;
@@ -3414,7 +3410,7 @@ static unsigned long gup_fast_pud_range(p4d_t *p4dp, p4d_t p4d,
}
static unsigned long gup_fast_p4d_range(pgd_t *pgdp, pgd_t pgd, unsigned long addr,
- unsigned long end, unsigned int flags, struct page **pages)
+ unsigned long end, unsigned int flags, struct gup_out *out)
{
unsigned long next, nr_pages = 0, chunk_nr_pages;
p4d_t *p4dp;
@@ -3428,7 +3424,7 @@ static unsigned long gup_fast_p4d_range(pgd_t *pgdp, pgd_t pgd, unsigned long ad
break;
BUILD_BUG_ON(p4d_leaf(p4d));
chunk_nr_pages = gup_fast_pud_range(p4dp, p4d, addr, next,
- flags, &pages[nr_pages]);
+ flags, out);
nr_pages += chunk_nr_pages;
if (chunk_nr_pages != (next - addr) >> PAGE_SHIFT)
break;
@@ -3439,7 +3435,7 @@ static unsigned long gup_fast_p4d_range(pgd_t *pgdp, pgd_t pgd, unsigned long ad
static unsigned long gup_fast_pgd_range(unsigned long addr,
unsigned long end, unsigned int flags,
- struct page **pages)
+ struct gup_out *out)
{
unsigned long next, nr_pages = 0, chunk_nr_pages;
pgd_t *pgdp;
@@ -3453,7 +3449,7 @@ static unsigned long gup_fast_pgd_range(unsigned long addr,
break;
BUILD_BUG_ON(pgd_leaf(pgd));
chunk_nr_pages = gup_fast_p4d_range(pgdp, pgd, addr, next,
- flags, &pages[nr_pages]);
+ flags, out);
nr_pages += chunk_nr_pages;
if (chunk_nr_pages != (next - addr) >> PAGE_SHIFT)
break;
@@ -3464,7 +3460,7 @@ static unsigned long gup_fast_pgd_range(unsigned long addr,
#else
static inline unsigned long gup_fast_pgd_range(unsigned long addr,
unsigned long end, unsigned int flags,
- struct page **pages)
+ struct gup_out *out)
{
return 0;
}
@@ -3481,12 +3477,27 @@ static bool gup_fast_permitted(unsigned long start, unsigned long end)
}
#endif
+/*
+ * Undo what GUP-fast recorded in @out.
+ *
+ * Don't perform any sanity checks because we might have raced with fork() and
+ * some anonymous pages might now actually be shared -- which is why we're
+ * unpinning after all.
+ */
+static void gup_fast_unpin(struct gup_out *out)
+{
+ gup_fast_unpin_user_pages(out->pages, out->nr_pages);
+ out->nr_pages = 0;
+}
+
static unsigned long gup_fast(unsigned long start, unsigned long end,
- unsigned int gup_flags, struct page **pages)
+ unsigned int gup_flags, struct gup_out *out)
{
- unsigned long flags, nr_pinned;
+ unsigned long flags;
unsigned seq;
+ VM_WARN_ON_ONCE(out->nr_pages);
+
if (!IS_ENABLED(CONFIG_HAVE_GUP_FAST) ||
!gup_fast_permitted(start, end))
return 0;
@@ -3508,7 +3519,7 @@ static unsigned long gup_fast(unsigned long start, unsigned long end,
* that come from callers of tlb_remove_table_sync_one().
*/
local_irq_save(flags);
- nr_pinned = gup_fast_pgd_range(start, end, gup_flags, pages);
+ gup_fast_pgd_range(start, end, gup_flags, out);
local_irq_restore(flags);
/*
@@ -3517,22 +3528,22 @@ static unsigned long gup_fast(unsigned long start, unsigned long end,
*/
if (gup_flags & FOLL_PIN) {
if (read_seqcount_retry(¤t->mm->write_protect_seq, seq)) {
- gup_fast_unpin_user_pages(pages, nr_pinned);
+ gup_fast_unpin(out);
return 0;
} else {
- sanity_check_pinned_pages(pages, nr_pinned);
+ sanity_check_pinned_pages(out->pages, out->nr_pages);
}
}
- return nr_pinned;
+ return out->nr_pages;
}
-static int gup_fast_fallback(unsigned long start, unsigned long nr_pages,
- unsigned int gup_flags, struct page **pages)
+static long gup_fast_fallback_out(unsigned long start, unsigned long nr_pages,
+ unsigned int gup_flags, struct gup_out *out)
{
unsigned long len, end;
unsigned long nr_pinned;
int locked = 0;
- int ret;
+ long ret;
if (WARN_ON_ONCE(gup_flags & ~(FOLL_WRITE | FOLL_LONGTERM |
FOLL_FORCE | FOLL_PIN | FOLL_GET |
@@ -3553,16 +3564,15 @@ static int gup_fast_fallback(unsigned long start, unsigned long nr_pages,
if (end > TASK_SIZE_MAX)
return -EFAULT;
- nr_pinned = gup_fast(start, end, gup_flags, pages);
+ nr_pinned = gup_fast(start, end, gup_flags, out);
if (nr_pinned == nr_pages || gup_flags & FOLL_FAST_ONLY)
return nr_pinned;
/* Slow path: try to get the remaining pages with get_user_pages */
start += nr_pinned << PAGE_SHIFT;
- pages += nr_pinned;
- ret = __gup_longterm_locked(current->mm, start, nr_pages - nr_pinned,
- pages, &locked,
- gup_flags | FOLL_TOUCH | FOLL_UNLOCKABLE);
+ ret = __gup_longterm_locked_out(current->mm, start,
+ nr_pages - nr_pinned, out, &locked,
+ gup_flags | FOLL_TOUCH | FOLL_UNLOCKABLE);
if (ret < 0) {
/*
* The caller has to unpin the pages we already pinned so
@@ -3575,6 +3585,14 @@ static int gup_fast_fallback(unsigned long start, unsigned long nr_pages,
return ret + nr_pinned;
}
+static int gup_fast_fallback(unsigned long start, unsigned long nr_pages,
+ unsigned int gup_flags, struct page **pages)
+{
+ struct gup_out out = GUP_OUT_PAGES(pages);
+
+ return gup_fast_fallback_out(start, nr_pages, gup_flags, &out);
+}
+
/**
* get_user_pages_fast_only() - pin user pages in memory
* @start: starting user address
--
2.53.0
^ permalink raw reply related [flat|nested] 7+ messages in thread
* [PATCH 4/5] mm/gup: return pinned memory as folio ranges
2026-11-18 21:34 [RFC PATCH 0/5] gup: introduce folio-range interface for pinning user memory Swarna Prabhu
` (2 preceding siblings ...)
2026-11-18 21:34 ` [PATCH 3/5] mm/gup: make the GUP-fast " Swarna Prabhu
@ 2026-11-18 21:34 ` Swarna Prabhu
2026-11-18 21:34 ` [PATCH 5/5] iov_iter: extract bvecs of user-backed iters from " Swarna Prabhu
2026-11-18 23:28 ` [RFC PATCH 0/5] gup: introduce folio-range interface for pinning user memory David Hildenbrand (Arm)
5 siblings, 0 replies; 7+ messages in thread
From: Swarna Prabhu @ 2026-11-18 21:34 UTC (permalink / raw)
To: akpm, david, jgg, jhubbard, viro
Cc: linux-mm, linux-block, linux-kernel, dave, riel, yeoreum.yun, hch,
Swarna Prabhu
From: Swarna Prabhu <s.prabhu@samsung.com>
Direct I/O maps user space addresses into the kernel by pinning user
memory and reporting it as an array of pages. With mTHP, consecutive
pages commonly belong to a single large folio, so everything the caller
does afterwards is repeated PAGE_SIZE at a time for memory that is
contiguous.
Kundan's LPC 2024 talk 'Transitioning get_user_pages to folio' proposed
a struct folio_vec (modeled on bcachefs) that would pass and retrieve
folio_vec_array instead of pages array. Based of that series and
subsequent community discussions, exposing folio ranges through GUP
using struct folio_range:
struct folio_range {
struct folio *folio;
unsigned int idx;
unsigned int nr_pages;
};
A range never spans more than one folio, so callers that need pages can
always get them via 'folio_range_page()', while callers that work on folios
can use the folio and 'folio_range_offset()' directly. For folio-range
output, GUP-fast conservatively defers pages with a NULL mapping to the
slow path, except for hugetlb folios. The slow path rejects VM_MIXEDMAP
VMAs with -EOPNOTSUPP if no pages were pinned; otherwise it returns pages
that are pinned successfully. Requests for an array of pages retain their
existing GUP-fast eligibility.
To avoid duplicating the page walkers, use 'struct gup_out' descriptor that
holds either page array or the folio range array, so a single walk serves
both the interfaces. Recording appends to the descriptor, removing the
explicit pages[] pointer previously used in '__get_user_pages_locked()' and
'gup_fast_fallback()'.
Recording can now fail: when the folio range array is full and the run
requires a new folio-range entry. The references or pins acquired is
released and the slow-path helper returns -ENOSPC. '__get_user_pages()'
treats this as a request to stop and returns the number of pages recorded
so far.
'pin_user_folio_ranges_fast()' stops once the caller's range array is
full, so callers can size it independently of the number of pages
requested. FOLL_LONGTERM is not supported as of now.
Signed-off-by: Swarna Prabhu <s.prabhu@samsung.com>
Assisted-by: LLM
---
include/linux/mm.h | 37 ++++
include/linux/mm_types.h | 23 +++
mm/gup.c | 413 ++++++++++++++++++++++++++++++++++-----
3 files changed, 424 insertions(+), 49 deletions(-)
diff --git a/include/linux/mm.h b/include/linux/mm.h
index 9f167950926a..d13863ef27f0 100644
--- a/include/linux/mm.h
+++ b/include/linux/mm.h
@@ -235,6 +235,39 @@ static inline unsigned long folio_page_idx(const struct folio *folio,
return page - &folio->page;
}
+/**
+ * folio_range_page - Return the first page of a folio range.
+ * @fr: The folio range.
+ *
+ * Return: The first page described by @fr.
+ */
+static inline struct page *folio_range_page(const struct folio_range *fr)
+{
+ return folio_page(fr->folio, fr->idx);
+}
+
+/**
+ * folio_range_offset - Return the byte offset of a range within its folio.
+ * @fr: The folio range.
+ *
+ * Return: The byte offset into @fr.
+ */
+static inline size_t folio_range_offset(const struct folio_range *fr)
+{
+ return (size_t)fr->idx << PAGE_SHIFT;
+}
+
+/**
+ * folio_range_size - Return the size of a folio range in bytes.
+ * @fr: The folio range.
+ *
+ * Return: The size of the folio range.
+ */
+static inline size_t folio_range_size(const struct folio_range *fr)
+{
+ return (size_t)fr->nr_pages << PAGE_SHIFT;
+}
+
static inline struct folio *lru_to_folio(struct list_head *head)
{
return list_entry((head)->prev, struct folio, lru);
@@ -3541,6 +3574,10 @@ int get_user_pages_fast(unsigned long start, int nr_pages,
unsigned int gup_flags, struct page **pages);
int pin_user_pages_fast(unsigned long start, int nr_pages,
unsigned int gup_flags, struct page **pages);
+long pin_user_folio_ranges_fast(unsigned long start, unsigned long nr_pages,
+ unsigned int gup_flags, struct folio_range *frs,
+ unsigned int max_frs, unsigned int *nr_frs);
+void unpin_user_folio_ranges(struct folio_range *frs, unsigned int nr_frs);
void folio_add_pin(struct folio *folio);
int account_locked_vm(struct mm_struct *mm, unsigned long pages, bool inc);
diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h
index 1ed35d258d2e..1a3b99ac8bbe 100644
--- a/include/linux/mm_types.h
+++ b/include/linux/mm_types.h
@@ -550,6 +550,29 @@ FOLIO_MATCH(flags, _flags_3);
FOLIO_MATCH(compound_info, _head_3);
#undef FOLIO_MATCH
+/**
+ * struct folio_range - A range of consecutive pages within a single folio.
+ * @folio: The folio the pages belong to.
+ * @idx: Index of the first page of the range within @folio.
+ * @nr_pages: Number of consecutive pages in the range.
+ *
+ * Describes @nr_pages consecutive pages of @folio, starting at page @idx of
+ * that folio. All pages of a range are guaranteed to belong to the same
+ * folio, so a range can always be described by a folio + offset + length,
+ * without having to look up the folio of individual pages: something that
+ * will no longer be possible for arbitrary pages once memory descriptors
+ * are split up.
+ *
+ * A range never spans more than one folio: consumers that want the underlying
+ * pages can obtain them with folio_range_page(), consumers that operate on
+ * folios can use @folio together with folio_range_offset() directly.
+ */
+struct folio_range {
+ struct folio *folio;
+ unsigned int idx;
+ unsigned int nr_pages;
+};
+
/**
* struct ptdesc - Memory descriptor for page tables.
* @pt_flags: enum pt_flags plus zone/node/section.
diff --git a/mm/gup.c b/mm/gup.c
index 43b92058a213..8424cec41719 100644
--- a/mm/gup.c
+++ b/mm/gup.c
@@ -67,6 +67,39 @@ static inline void sanity_check_pinned_pages(struct page **pages,
}
}
+/* See sanity_check_pinned_pages(). */
+static inline void sanity_check_pinned_folio_ranges(struct folio_range *frs,
+ unsigned int nr_frs)
+{
+ unsigned int i, j;
+
+ if (!IS_ENABLED(CONFIG_DEBUG_VM))
+ return;
+
+ for (i = 0; i < nr_frs; i++) {
+ struct folio *folio = frs[i].folio;
+
+ if (is_zero_folio(folio) || !folio_test_anon(folio))
+ continue;
+
+ if (!folio_test_large(folio) || folio_test_hugetlb(folio)) {
+ VM_WARN_ON_ONCE_FOLIO(!PageAnonExclusive(&folio->page),
+ folio);
+ continue;
+ }
+
+ if (PageAnonExclusive(&folio->page))
+ continue;
+
+ /* A PTE-mapped THP: each page has to be exclusive. */
+ for (j = 0; j < frs[i].nr_pages; j++) {
+ struct page *page = folio_page(folio, frs[i].idx + j);
+
+ VM_WARN_ON_ONCE_PAGE(!PageAnonExclusive(page), page);
+ }
+ }
+}
+
/*
* Return the folio with ref appropriately incremented,
* or NULL if that failed.
@@ -437,6 +470,23 @@ void unpin_user_folio(struct folio *folio, unsigned long npages)
}
EXPORT_SYMBOL(unpin_user_folio);
+/**
+ * unpin_user_folio_ranges() - release an array of gup-pinned folio ranges
+ * @frs: array of folio ranges to be released
+ * @nr_frs: number of folio ranges in the @frs array
+ *
+ * For each range in the @frs array, release the pins that
+ * pin_user_folio_ranges_fast() took for its pages.
+ */
+void unpin_user_folio_ranges(struct folio_range *frs, unsigned int nr_frs)
+{
+ unsigned int i;
+
+ for (i = 0; i < nr_frs; i++)
+ gup_put_folio(frs[i].folio, frs[i].nr_pages, FOLL_PIN);
+}
+EXPORT_SYMBOL_GPL(unpin_user_folio_ranges);
+
/**
* unpin_folios() - release an array of gup-pinned folios.
* @folios: array of folios to be marked dirty and released.
@@ -485,20 +535,65 @@ static inline void mm_set_has_pinned_flag(struct mm_struct *mm)
/*
* Output descriptor of the GUP page table walkers.
*
- * GUP hands the memory it grabbed back to the caller as an array of pages
- * (@pages).
+ * GUP hands the memory it grabbed back to the caller either as an array of
+ * pages (@pages), or as an array of folio ranges (@frs), where each range
+ * describes consecutive pages of a single folio. Exactly one of the two is
+ * set.
*
- * @nr_pages is the number of pages recorded so far: the walkers simply keep
- * appending to the descriptor, no matter how often they are called for a
- * single GUP request.
+ * @nr_pages is the number of pages recorded so far and is maintained in both
+ * cases: the walkers simply keep appending to the descriptor, no matter how
+ * often they are called for a single GUP request.
*/
struct gup_out {
struct page **pages;
+ struct folio_range *frs;
unsigned long nr_pages;
+ unsigned int nr_frs;
+ unsigned int max_frs;
};
#define GUP_OUT_PAGES(_pages) ((struct gup_out){ .pages = (_pages) })
+/*
+ * Whether no further folio range can be recorded. The @pages array is sized
+ * by the caller to hold all requested pages, so it can never run full.
+ */
+static inline bool gup_out_is_full(struct gup_out *out)
+{
+ return out->frs && out->nr_frs == out->max_frs;
+}
+
+static inline bool gup_out_add_folio_range(struct gup_out *out,
+ struct folio *folio,
+ struct page *page,
+ unsigned long nr_pages)
+{
+ unsigned long idx = folio_page_idx(folio, page);
+ struct folio_range *fr;
+
+ /*
+ * Ranges are only ever grown by walkers that hold a reference on the
+ * folio for each page they add, so merging with the previous range
+ * keeps the reference count of that range consistent.
+ */
+ if (out->nr_frs) {
+ fr = &out->frs[out->nr_frs - 1];
+ if (fr->folio == folio && fr->idx + fr->nr_pages == idx) {
+ fr->nr_pages += nr_pages;
+ return true;
+ }
+ }
+
+ if (out->nr_frs == out->max_frs)
+ return false;
+
+ fr = &out->frs[out->nr_frs++];
+ fr->folio = folio;
+ fr->idx = idx;
+ fr->nr_pages = nr_pages;
+ return true;
+}
+
/**
* gup_out_add_range() - record consecutive pages of a single folio
* @out: the output descriptor
@@ -508,17 +603,27 @@ struct gup_out {
*
* The caller must hold a reference (FOLL_GET) or pin (FOLL_PIN) for each of
* the @nr_pages pages.
+ *
+ * Return: %true if the pages were recorded. %false if the folio range array
+ * is full, in which case the caller has to release the references it grabbed
+ * for these pages and stop the walk.
*/
-static inline void gup_out_add_range(struct gup_out *out, struct folio *folio,
+static inline bool gup_out_add_range(struct gup_out *out, struct folio *folio,
struct page *page, unsigned long nr_pages)
{
- struct page **pages = out->pages + out->nr_pages;
- unsigned long i;
+ if (out->frs) {
+ if (!gup_out_add_folio_range(out, folio, page, nr_pages))
+ return false;
+ } else {
+ struct page **pages = out->pages + out->nr_pages;
+ unsigned long i;
- for (i = 0; i < nr_pages; i++)
- pages[i] = page + i;
+ for (i = 0; i < nr_pages; i++)
+ pages[i] = page + i;
+ }
out->nr_pages += nr_pages;
+ return true;
}
#ifdef CONFIG_MMU
@@ -749,16 +854,26 @@ static inline bool gup_must_unshare(struct vm_area_struct *vma,
return gup_pages_not_requiring_unshare(vma, flags, folio, page, 1) != 1;
}
-/* Record @nr pages starting at @page in @out, and flush their caches. */
-static void gup_fill_pages(struct vm_area_struct *vma, unsigned long address,
- struct page *page, unsigned long nr, struct gup_out *out)
+/*
+ * Record @nr pages starting at @page, for which the caller holds references
+ * per @flags, in @out, and flush their caches. If @out has no room for
+ * another folio range, the references are dropped and -ENOSPC is returned.
+ */
+static int gup_fill_pages(struct vm_area_struct *vma, unsigned long address,
+ struct page *page, unsigned long nr, unsigned int flags,
+ struct gup_out *out)
{
+ struct folio *folio;
unsigned long i;
if (!out)
- return;
+ return 0;
- gup_out_add_range(out, page_folio(page), page, nr);
+ folio = page_folio(page);
+ if (!gup_out_add_range(out, folio, page, nr)) {
+ gup_put_folio(folio, nr, flags);
+ return -ENOSPC;
+ }
for (i = 0; i < nr; i++) {
struct page *subpage = page + i;
@@ -766,6 +881,8 @@ static void gup_fill_pages(struct vm_area_struct *vma, unsigned long address,
flush_anon_page(vma, subpage, address + i * PAGE_SIZE);
flush_dcache_page(subpage);
}
+
+ return 0;
}
/*
@@ -782,6 +899,28 @@ static void gup_flush_pages(struct vm_area_struct *vma, unsigned long address,
if (!out)
return;
+ if (out->frs) {
+ const struct folio_range *fr = &out->frs[out->nr_frs];
+
+ /*
+ * The first pages may have been merged into a range that was
+ * recorded earlier, so walk back from the end of the array.
+ */
+ while (nr) {
+ unsigned long run = min((--fr)->nr_pages, nr);
+ struct page *page = folio_range_page(fr) +
+ fr->nr_pages - run;
+
+ nr -= run;
+ for (i = 0; i < run; i++) {
+ flush_anon_page(vma, page + i,
+ address + (nr + i) * PAGE_SIZE);
+ flush_dcache_page(page + i);
+ }
+ }
+ return;
+ }
+
pages = out->pages + out->nr_pages - nr;
for (i = 0; i < nr; i++) {
flush_anon_page(vma, pages[i], address + i * PAGE_SIZE);
@@ -997,6 +1136,12 @@ static long gup_grab_folio_pages(struct folio *folio,
return ret;
}
}
+
+ if (out && !gup_out_add_range(out, folio, page, nr)) {
+ gup_put_folio(folio, nr, flags);
+ return -ENOSPC;
+ }
+
if (flags & FOLL_TOUCH) {
if ((flags & FOLL_WRITE) && !dirty && !folio_test_dirty(folio))
folio_mark_dirty(folio);
@@ -1008,9 +1153,6 @@ static long gup_grab_folio_pages(struct folio *folio,
folio_mark_accessed(folio);
}
- if (out)
- gup_out_add_range(out, folio, page, nr);
-
return 0;
}
@@ -1212,8 +1354,12 @@ static long follow_pmd_mask(struct vm_area_struct *vma,
* The ref is already held, so the page cannot go away: record the
* pages and flush caches without the pmd lock.
*/
- if (ret > 0)
- gup_fill_pages(vma, address, page, ret, out);
+ if (ret > 0) {
+ int err = gup_fill_pages(vma, address, page, ret, flags, out);
+
+ if (err)
+ return err;
+ }
return ret;
}
@@ -1242,8 +1388,13 @@ static long follow_pud_mask(struct vm_area_struct *vma,
* can be up to HPAGE_PUD_NR pages, too long to flush under a
* spinlock.
*/
- if (ret > 0)
- gup_fill_pages(vma, address, page, ret, out);
+ if (ret > 0) {
+ int err = gup_fill_pages(vma, address, page, ret,
+ flags, out);
+
+ if (err)
+ return err;
+ }
if (ret)
return ret;
return no_page_table(vma, flags, address);
@@ -1289,9 +1440,9 @@ static long follow_p4d_mask(struct vm_area_struct *vma,
*
* Return: the number of contiguous pages starting at @address that were
* recorded in @out (if non-NULL), which may be fewer than the pages
- * requested via @end; 0 if no mapping exists at @address; or a negative
- * errno for a mapping to something not represented by a page descriptor
- * (see also vm_normal_page()).
+ * requested via @end; 0 if no mapping exists at @address; -ENOSPC if @out
+ * has no room for another folio range; or a negative errno for a mapping to
+ * something not represented by a page descriptor (see also vm_normal_page()).
*/
static long follow_page_mask(struct vm_area_struct *vma,
unsigned long address, unsigned long end,
@@ -1686,7 +1837,12 @@ static long __get_user_pages(struct mm_struct *mm,
out ? &page : NULL);
if (ret)
goto out;
- gup_fill_pages(vma, start, page, 1, out);
+ if (gup_fill_pages(vma, start, page, 1,
+ gup_flags, out)) {
+ /* The folio range array is full. */
+ ret = 0;
+ goto out;
+ }
i++;
start += PAGE_SIZE;
nr_pages--;
@@ -1700,6 +1856,10 @@ static long __get_user_pages(struct mm_struct *mm,
ret = check_vma_flags(vma, gup_flags);
if (ret)
goto out;
+ if (out && out->frs && (vma->vm_flags & VM_MIXEDMAP)) {
+ ret = i ? 0 : -EOPNOTSUPP;
+ goto out;
+ }
}
retry:
/*
@@ -1743,6 +1903,13 @@ static long __get_user_pages(struct mm_struct *mm,
goto out;
}
nr = 1;
+ } else if (nr == -ENOSPC && out && gup_out_is_full(out)) {
+ /*
+ * The output can only run out of space when returning
+ * folio ranges: stop and report what we got so far.
+ */
+ ret = 0;
+ goto out;
} else if (nr < 0) {
ret = nr;
goto out;
@@ -2294,7 +2461,10 @@ static long __get_user_pages_locked_out(struct mm_struct *mm, unsigned long star
err = ret;
break;
}
- gup_out_add_range(out, folio, page, 1);
+ if (!gup_out_add_range(out, folio, page, 1)) {
+ gup_put_folio(folio, 1, foll_flags);
+ break;
+ }
}
start = (start + PAGE_SIZE) & PAGE_MASK;
@@ -2739,8 +2909,8 @@ static long check_and_migrate_movable_folios(unsigned long nr_folios,
#endif /* CONFIG_MIGRATION */
/*
- * __gup_longterm_locked() is a wrapper for __get_user_pages_locked which
- * allows us to process the FOLL_LONGTERM flag.
+ * __gup_longterm_locked_out() is a wrapper for __get_user_pages_locked_out
+ * which allows us to process the FOLL_LONGTERM flag.
*/
static long __gup_longterm_locked_out(struct mm_struct *mm,
unsigned long start,
@@ -2749,7 +2919,7 @@ static long __gup_longterm_locked_out(struct mm_struct *mm,
int *locked,
unsigned int gup_flags)
{
- unsigned long nr_pages_done;
+ unsigned long nr_pages_done, nr_frs_done;
unsigned int flags;
long rc, nr_pinned_pages;
@@ -2757,7 +2927,12 @@ static long __gup_longterm_locked_out(struct mm_struct *mm,
return __get_user_pages_locked_out(mm, start, nr_pages, out,
locked, gup_flags);
- if (WARN_ON_ONCE(!out))
+ /*
+ * check_and_migrate_movable_pages() cannot deal with folio ranges
+ * yet, which is why pin_user_folio_ranges_fast() rejects
+ * FOLL_LONGTERM.
+ */
+ if (WARN_ON_ONCE(!out || out->frs))
return -EINVAL;
/*
@@ -2765,10 +2940,12 @@ static long __gup_longterm_locked_out(struct mm_struct *mm,
* this call has to be discarded again.
*/
nr_pages_done = out->nr_pages;
+ nr_frs_done = out->nr_frs;
flags = memalloc_pin_save();
do {
out->nr_pages = nr_pages_done;
+ out->nr_frs = nr_frs_done;
nr_pinned_pages = __get_user_pages_locked_out(mm, start,
nr_pages, out,
@@ -2809,8 +2986,8 @@ static long __gup_longterm_locked(struct mm_struct *mm,
* Check that the given flags are valid for the exported gup/pup interface, and
* update them with the required flags that the caller must have set.
*/
-static bool is_valid_gup_args(struct page **pages, int *locked,
- unsigned int *gup_flags_p, unsigned int to_set)
+static __always_inline bool __is_valid_gup_args(bool has_output, int *locked,
+ unsigned int *gup_flags_p, unsigned int to_set)
{
unsigned int gup_flags = *gup_flags_p;
@@ -2842,8 +3019,8 @@ static bool is_valid_gup_args(struct page **pages, int *locked,
if (WARN_ON_ONCE(!(gup_flags & FOLL_PIN) && (gup_flags & FOLL_LONGTERM)))
return false;
- /* Pages input must be given if using GET/PIN */
- if (WARN_ON_ONCE((gup_flags & (FOLL_GET | FOLL_PIN)) && !pages))
+ /* An output array must be given if using GET/PIN */
+ if (WARN_ON_ONCE((gup_flags & (FOLL_GET | FOLL_PIN)) && !has_output))
return false;
/* We want to allow the pgmap to be hot-unplugged at all times */
@@ -2855,6 +3032,12 @@ static bool is_valid_gup_args(struct page **pages, int *locked,
return true;
}
+static bool is_valid_gup_args(struct page **pages, int *locked,
+ unsigned int *gup_flags_p, unsigned int to_set)
+{
+ return __is_valid_gup_args(pages != NULL, locked, gup_flags_p, to_set);
+}
+
#ifdef CONFIG_MMU
/**
* get_user_pages_remote() - pin user pages in memory
@@ -3048,7 +3231,15 @@ EXPORT_SYMBOL(get_user_pages_unlocked);
* in the fast path, so instead we whitelist known good cases and if in doubt,
* fall back to the slow path.
*/
-static bool gup_fast_folio_allowed(struct folio *folio, unsigned int flags)
+/*
+ * @folio_ranges is set when the caller requests folio-range output. The NULL
+ * mapping check is conservative and defers to slow path which gets checked
+ * against the VMA information. A page without any mapping might not be a
+ * folio at all (e.g. a bare page a driver installed via vm_insert_page()), so
+ * such pages are left to the slow path, which can consult the vma.
+ */
+static bool gup_fast_folio_allowed(struct folio *folio, unsigned int flags,
+ bool folio_ranges)
{
bool reject_file_backed = false;
struct address_space *mapping;
@@ -3069,7 +3260,7 @@ static bool gup_fast_folio_allowed(struct folio *folio, unsigned int flags)
if (IS_ENABLED(CONFIG_SECRETMEM) && !folio_test_large(folio))
check_secretmem = true;
- if (!reject_file_backed && !check_secretmem)
+ if (!reject_file_backed && !check_secretmem && !folio_ranges)
return true;
if (WARN_ON_ONCE(folio_test_slab(folio)))
@@ -3104,10 +3295,10 @@ static bool gup_fast_folio_allowed(struct folio *folio, unsigned int flags)
* Otherwise, a NULL mapping proves this is not a secretmem folio
* (secretmem folios always have a valid mapping to the secretmem
* inode's address_space), so in that case, we can continue with the
- * fast path.
+ * fast path, unless the caller needs it to be a folio.
*/
if (!mapping)
- return !reject_file_backed;
+ return !reject_file_backed && !folio_ranges;
/* Anonymous folios pose no problem. */
mapping_flags = (unsigned long)mapping & FOLIO_MAPPING_FLAGS;
@@ -3181,6 +3372,21 @@ static unsigned long gup_fast_pte_range(pmd_t pmd, pmd_t *pmdp,
VM_WARN_ON_ONCE(!pfn_valid(pte_pfn(pte)));
page = pte_page(pte);
+ if (gup_out_is_full(out)) {
+ struct folio_range *last;
+ unsigned long next_idx;
+
+ if (!out->nr_frs)
+ goto out;
+
+ last = &out->frs[out->nr_frs - 1];
+ next_idx = (unsigned long)last->idx + last->nr_pages;
+
+ if (next_idx >= folio_nr_pages(last->folio) ||
+ page != folio_page(last->folio, next_idx))
+ goto out;
+ }
+
folio = try_grab_folio_fast(page, 1, flags);
if (!folio)
goto out;
@@ -3197,7 +3403,7 @@ static unsigned long gup_fast_pte_range(pmd_t pmd, pmd_t *pmdp,
unlikely(pte_val(pte) != pte_val(ptep_get_lockless(ptep))))
goto out;
- if (!gup_fast_folio_allowed(folio, flags))
+ if (!gup_fast_folio_allowed(folio, flags, out->frs))
goto out;
if (folio_test_large(folio))
@@ -3228,7 +3434,13 @@ static unsigned long gup_fast_pte_range(pmd_t pmd, pmd_t *pmdp,
nr_batch = 1;
folio_set_referenced(folio);
- gup_out_add_range(out, folio, page, nr_batch);
+
+ /* Only fails if the caller's output array is full. */
+ if (!gup_out_add_range(out, folio, page, nr_batch)) {
+ gup_put_folio(folio, nr_batch, flags);
+ folio = NULL;
+ goto out;
+ }
nr_pages += nr_batch;
folio = NULL;
} while (ptep += nr_batch, addr += nr_batch * PAGE_SIZE, addr != end);
@@ -3289,7 +3501,7 @@ static unsigned long gup_fast_pmd_leaf(pmd_t orig, pmd_t *pmdp,
return 0;
}
- if (!gup_fast_folio_allowed(folio, flags)) {
+ if (!gup_fast_folio_allowed(folio, flags, out->frs)) {
gup_put_folio(folio, nr_pages, flags);
return 0;
}
@@ -3299,7 +3511,11 @@ static unsigned long gup_fast_pmd_leaf(pmd_t orig, pmd_t *pmdp,
return 0;
}
- gup_out_add_range(out, folio, page, nr_pages);
+ if (!gup_out_add_range(out, folio, page, nr_pages)) {
+ gup_put_folio(folio, nr_pages, flags);
+ return 0;
+ }
+
folio_set_referenced(folio);
return nr_pages;
}
@@ -3331,7 +3547,7 @@ static unsigned long gup_fast_pud_leaf(pud_t orig, pud_t *pudp,
return 0;
}
- if (!gup_fast_folio_allowed(folio, flags)) {
+ if (!gup_fast_folio_allowed(folio, flags, out->frs)) {
gup_put_folio(folio, nr_pages, flags);
return 0;
}
@@ -3342,7 +3558,11 @@ static unsigned long gup_fast_pud_leaf(pud_t orig, pud_t *pudp,
return 0;
}
- gup_out_add_range(out, folio, page, nr_pages);
+ if (!gup_out_add_range(out, folio, page, nr_pages)) {
+ gup_put_folio(folio, nr_pages, flags);
+ return 0;
+ }
+
folio_set_referenced(folio);
return nr_pages;
}
@@ -3486,10 +3706,27 @@ static bool gup_fast_permitted(unsigned long start, unsigned long end)
*/
static void gup_fast_unpin(struct gup_out *out)
{
- gup_fast_unpin_user_pages(out->pages, out->nr_pages);
+ if (out->frs) {
+ unsigned int i;
+
+ for (i = 0; i < out->nr_frs; i++)
+ gup_put_folio(out->frs[i].folio, out->frs[i].nr_pages,
+ FOLL_PIN);
+ out->nr_frs = 0;
+ } else {
+ gup_fast_unpin_user_pages(out->pages, out->nr_pages);
+ }
out->nr_pages = 0;
}
+static void gup_fast_sanity_check(struct gup_out *out)
+{
+ if (out->frs)
+ sanity_check_pinned_folio_ranges(out->frs, out->nr_frs);
+ else
+ sanity_check_pinned_pages(out->pages, out->nr_pages);
+}
+
static unsigned long gup_fast(unsigned long start, unsigned long end,
unsigned int gup_flags, struct gup_out *out)
{
@@ -3531,7 +3768,7 @@ static unsigned long gup_fast(unsigned long start, unsigned long end,
gup_fast_unpin(out);
return 0;
} else {
- sanity_check_pinned_pages(out->pages, out->nr_pages);
+ gup_fast_sanity_check(out);
}
}
return out->nr_pages;
@@ -3558,8 +3795,8 @@ static long gup_fast_fallback_out(unsigned long start, unsigned long nr_pages,
might_lock_read(¤t->mm->mmap_lock);
start = untagged_addr(start) & PAGE_MASK;
- len = nr_pages << PAGE_SHIFT;
- if (check_add_overflow(start, len, &end))
+ if (check_shl_overflow(nr_pages, PAGE_SHIFT, &len) ||
+ check_add_overflow(start, len, &end))
return -EOVERFLOW;
if (end > TASK_SIZE_MAX)
return -EFAULT;
@@ -3568,6 +3805,10 @@ static long gup_fast_fallback_out(unsigned long start, unsigned long nr_pages,
if (nr_pinned == nr_pages || gup_flags & FOLL_FAST_ONLY)
return nr_pinned;
+ /* No room left to report anything the slow path could get us. */
+ if (gup_out_is_full(out))
+ return nr_pinned;
+
/* Slow path: try to get the remaining pages with get_user_pages */
start += nr_pinned << PAGE_SHIFT;
ret = __gup_longterm_locked_out(current->mm, start,
@@ -3688,6 +3929,80 @@ int pin_user_pages_fast(unsigned long start, int nr_pages,
}
EXPORT_SYMBOL_GPL(pin_user_pages_fast);
+/**
+ * pin_user_folio_ranges_fast() - pin user pages in memory without taking
+ * locks, reporting them as folio ranges
+ *
+ * @start: starting user address
+ * @nr_pages: number of pages from start to pin
+ * @gup_flags: flags modifying pin behaviour
+ * @frs: array that receives the pinned folio ranges
+ * @max_frs: number of entries in the @frs array
+ * @nr_frs: receives the number of folio ranges filled into @frs
+ *
+ * Like pin_user_pages_fast(), but instead of reporting each page separately,
+ * consecutive pages of the same folio are reported as a single folio range.
+ * A range never spans more than one folio, so a caller that has to fall back
+ * to per-page processing can always do so via folio_range_page().
+ *
+ * Pinning stops early once @max_frs ranges have been filled, so that the
+ * number of pages pinned can be smaller than @nr_pages even though more pages
+ * are present. Callers that cannot deal with short pins should size @frs to
+ * hold @nr_pages ranges, which is the worst case.
+ *
+ * FOLL_LONGTERM is not supported yet.
+ *
+ * The pins have to be released with unpin_user_folio_ranges(), or per range
+ * with unpin_user_folio(fr->folio, fr->nr_pages).
+ *
+ * Note that if the zero page is amongst the returned ranges, it will not have
+ * pins in it and unpinning will not remove pins from it.
+ *
+ * Memory that is not necessarily backed by folios, such as pages a driver
+ * inserted into a VM_MIXEDMAP vma, cannot be reported as folio ranges: pinning
+ * stops in front of it and -EOPNOTSUPP is returned if it comes first. Callers
+ * have to fall back to pin_user_pages_fast() for it.
+ *
+ * Return: number of pages pinned, which can be less than @nr_pages. For a zero
+ * page request, returns 0 and sets @nr_frs to 0. If @nr_pages is non-zero and
+ * no pages were pinned, a negative errno is returned. @nr_frs is only valid if
+ * the return value is positive.
+ */
+long pin_user_folio_ranges_fast(unsigned long start, unsigned long nr_pages,
+ unsigned int gup_flags, struct folio_range *frs,
+ unsigned int max_frs, unsigned int *nr_frs)
+{
+ struct gup_out out = {
+ .frs = frs,
+ .max_frs = max_frs,
+ };
+ long ret;
+
+ if (WARN_ON_ONCE(!frs || !max_frs))
+ return -EINVAL;
+
+ /*
+ * Migrating away from unpinnable memory in the FOLL_LONGTERM case
+ * still requires an array of pages or folios; until that is sorted
+ * out, don't pretend we can handle it.
+ */
+ if (WARN_ON_ONCE(gup_flags & FOLL_LONGTERM))
+ return -EINVAL;
+
+ if (!nr_pages) {
+ *nr_frs = 0;
+ return 0;
+ }
+
+ if (!__is_valid_gup_args(true, NULL, &gup_flags, FOLL_PIN))
+ return -EINVAL;
+
+ ret = gup_fast_fallback_out(start, nr_pages, gup_flags, &out);
+ *nr_frs = out.nr_frs;
+ return ret;
+}
+EXPORT_SYMBOL_GPL(pin_user_folio_ranges_fast);
+
/**
* pin_user_pages_remote() - pin pages of a remote process
*
--
2.53.0
^ permalink raw reply related [flat|nested] 7+ messages in thread
* [PATCH 5/5] iov_iter: extract bvecs of user-backed iters from folio ranges
2026-11-18 21:34 [RFC PATCH 0/5] gup: introduce folio-range interface for pinning user memory Swarna Prabhu
` (3 preceding siblings ...)
2026-11-18 21:34 ` [PATCH 4/5] mm/gup: return pinned memory as folio ranges Swarna Prabhu
@ 2026-11-18 21:34 ` Swarna Prabhu
2026-11-18 23:28 ` [RFC PATCH 0/5] gup: introduce folio-range interface for pinning user memory David Hildenbrand (Arm)
5 siblings, 0 replies; 7+ messages in thread
From: Swarna Prabhu @ 2026-11-18 21:34 UTC (permalink / raw)
To: akpm, david, jgg, jhubbard, viro
Cc: linux-mm, linux-block, linux-kernel, dave, riel, yeoreum.yun, hch,
Swarna Prabhu
From: Swarna Prabhu <s.prabhu@samsung.com>
iov_iter_extract_bvecs() pins a page array placed in the bvec array and
then walks it with get_contig_folio_len() to merge pages that turn out to
belong to the same folio back into one bvec.
For user-backed iterators, have GUP fill folio ranges straight into the
free part of the bvec array instead. A folio range has the same size as
a bvec and becomes exactly one bvec, so each range is converted in its own
slot and trimmed to the requested byte range on the way. This removes the
page array and the merging pass. Ranges that are not converted are still
folio ranges and are unpinned with their exact page count.
Since one range covers many pages, the number of pages pinned per call is
no longer limited to one page per bvec entry; it is bounded by the bvec
array, plus a cap on how many pages a single GUP-fast walk looks at.
Memory that is not backed by folios, such as pages a driver inserted
into a VM_MIXEDMAP vma, makes pin_user_folio_ranges_fast() fail with
-EOPNOTSUPP when no pages were pinned; such iterators fall back to the
page array path.
Each extraction requests at most 1024 pages, bounding the amount of work
performed by a single GUP call.
Block direct I/O picks this up through bio_iov_iter_get_pages() without
any change to the block layer.
Signed-off-by: Swarna Prabhu <s.prabhu@samsung.com>
---
lib/iov_iter.c | 100 +++++++++++++++++++++++++++++++++++++++++++++++++
1 file changed, 100 insertions(+)
diff --git a/lib/iov_iter.c b/lib/iov_iter.c
index fa4fe2e92383..457e226c0251 100644
--- a/lib/iov_iter.c
+++ b/lib/iov_iter.c
@@ -1919,6 +1919,95 @@ static unsigned int get_contig_folio_len(struct page **pages,
#define PAGE_PTRS_PER_BVEC (sizeof(struct bio_vec) / sizeof(struct page *))
+/*
+ * Upper limit on the number of pages a single GUP-fast walk is asked to look
+ * at to cap spending too much time in the GUP-fast path.
+ */
+#define EXTRACT_MAX_PAGES 1024
+
+/*
+ * Extract bvecs from a user-backed iterator. GUP fills folio ranges straight
+ * into the free part of the bvec array, one range per run of consecutive pages
+ * of a folio. A folio range has the same size as a bvec and turns into
+ * exactly one bvec, so each range is converted in its own slot.
+ *
+ * Returns -EOPNOTSUPP, with @iter left untouched, if the memory at the start of
+ * @iter is not backed by folios, so the caller can extract it as pages instead.
+ */
+static ssize_t iov_iter_extract_user_bvecs(struct iov_iter *iter,
+ struct bio_vec *bv, size_t max_size, unsigned short *nr_vecs,
+ unsigned short max_vecs, iov_iter_extraction_t extraction_flags)
+{
+ unsigned short entries_left = max_vecs - *nr_vecs;
+ unsigned int gup_flags = 0, nr_frs = 0, i;
+ unsigned long addr, nr_pages;
+ struct folio_range *frs;
+ size_t offset, left, len;
+ long res;
+
+ max_size = min_t(size_t, min_t(size_t, max_size, iter->count),
+ MAX_RW_COUNT);
+ if (unlikely(!max_size))
+ return -EFAULT;
+ if (WARN_ON_ONCE(!entries_left))
+ return -ENOMEM;
+
+ if (iter->data_source == ITER_DEST)
+ gup_flags |= FOLL_WRITE;
+ if (extraction_flags & ITER_ALLOW_P2PDMA)
+ gup_flags |= FOLL_PCI_P2PDMA;
+ if (iter->nofault)
+ gup_flags |= FOLL_NOFAULT;
+
+ addr = first_iovec_segment(iter, &max_size);
+ offset = addr % PAGE_SIZE;
+ addr &= PAGE_MASK;
+ nr_pages = min_t(unsigned long,
+ DIV_ROUND_UP(max_size + offset, PAGE_SIZE),
+ EXTRACT_MAX_PAGES);
+
+ /*
+ * Range i is converted into slot *nr_vecs + i, which is the memory it
+ * lives in, so converting cannot overwrite a range that has not been
+ * converted yet.
+ */
+ BUILD_BUG_ON(sizeof(struct folio_range) != sizeof(struct bio_vec));
+ frs = (struct folio_range *)(bv + *nr_vecs);
+
+ res = pin_user_folio_ranges_fast(addr, nr_pages, gup_flags, frs,
+ entries_left, &nr_frs);
+ if (unlikely(res <= 0))
+ return res ? res : -EFAULT;
+
+ /* We may have pinned fewer pages than asked for. */
+ max_size = min_t(size_t, max_size, res * PAGE_SIZE - offset);
+ iov_iter_advance(iter, max_size);
+
+ for (i = 0, left = max_size; left && i < nr_frs; i++) {
+ /* Copy it out first: bv[*nr_vecs] is the same memory as frs[i]. */
+ struct folio_range fr = frs[i];
+ struct page *page = folio_range_page(&fr);
+
+ if (*nr_vecs > 0 &&
+ !zone_device_pages_have_same_pgmap(bv[*nr_vecs - 1].bv_page,
+ page))
+ break;
+
+ len = min(folio_range_size(&fr) - offset, left);
+ bvec_set_folio(&bv[(*nr_vecs)++], fr.folio, len,
+ folio_range_offset(&fr) + offset);
+ left -= len;
+ offset = 0;
+ }
+
+ /* Not overwritten, so still folio ranges; no-op once i == nr_frs. */
+ if (unlikely(i < nr_frs))
+ unpin_user_folio_ranges(&frs[i], nr_frs - i);
+
+ iov_iter_revert(iter, left);
+ return max_size - left;
+}
+
/**
* iov_iter_extract_bvecs - Extract bvecs from an iterator
* @iter: the iterator to extract from
@@ -1966,6 +2055,17 @@ ssize_t iov_iter_extract_bvecs(struct iov_iter *iter, struct bio_vec *bv,
return -EINVAL;
}
+ if (likely(user_backed_iter(iter))) {
+ size = iov_iter_extract_user_bvecs(iter, bv, max_size, nr_vecs,
+ max_vecs, extraction_flags);
+ /*
+ * Memory that is not backed by folios cannot be pinned as folio
+ * ranges; fall back to pinning it as an array of pages.
+ */
+ if (size != -EOPNOTSUPP)
+ return size;
+ }
+
/*
* Move page array up in the allocated memory for the bio vecs as far as
* possible so that we can start filling biovecs from the beginning
--
2.53.0
^ permalink raw reply related [flat|nested] 7+ messages in thread
* Re: [RFC PATCH 0/5] gup: introduce folio-range interface for pinning user memory
2026-11-18 21:34 [RFC PATCH 0/5] gup: introduce folio-range interface for pinning user memory Swarna Prabhu
` (4 preceding siblings ...)
2026-11-18 21:34 ` [PATCH 5/5] iov_iter: extract bvecs of user-backed iters from " Swarna Prabhu
@ 2026-11-18 23:28 ` David Hildenbrand (Arm)
5 siblings, 0 replies; 7+ messages in thread
From: David Hildenbrand (Arm) @ 2026-11-18 23:28 UTC (permalink / raw)
To: Swarna Prabhu, akpm, jgg, jhubbard, viro
Cc: linux-mm, linux-block, linux-kernel, dave, riel, yeoreum.yun, hch,
Swarna Prabhu
On 11/18/26 22:34, Swarna Prabhu wrote:
> From: Swarna Prabhu <s.prabhu@samsung.com>
>
> This series is an RFC to gather design feedback to introduce a folio-range
> interface for pinning user memory in the GUP fast path. Pinned memory is
> returned as folio ranges instead of a page array: a run of consecutive PTEs
> mapping consecutive pages of one folio is pinned once and handed back as a single
> range. Block direct I/O converts each range into a single bvec, hence reducing the
> per-page pin and merge work.
>
> Base and dependencies
> =====================
>
> This series is rebased on the following two series applied:
>
> - "mm: gup: batch PTE-mapped large folios in gup_fast_pte_range()"
> v4 (Yeoreum Yun; originally posted by Yuan-Hao Hsu [3]), applied to
> mm/core.git gup-7.5.ptebatch [4]. It batches the reference grabs for
> PTE-mapped large folios in GUP-fast.
>
> - "batch lookups in follow_page_mask()" v5 (Rik van Riel) [5]. It makes
> follow_page_mask() return a page count and record pages through
> gup_fill_pages().
>
> Note: "[PATCH v5 1/8] mm/gup: break out gup_fill_pages() helper" was
> posted just as testing for this RFC was done. To avoid delaying
> feedback on the overall design, this RFC retains the tested base
> linked in [5] and will be rebased onto the latest revision for next
> version.
>
> This series builds on both: the run detection in gup_fast_pte_range() and
> the batched slow-path lookup are reused, and this series changes how the
> result of a run is recorded and returned.
>
> Background
> ==========
>
> Kundan's LPC '24 talk "Transitioning get_user_pages to folio based mapping"
> [1] proposed a struct folio_vec (based on bcachefs) passed in place of a
> pages array. Building on that and on suggestions from the community [2],
> this series exposes folio + index + nr_pages, where each range spans a
> single folio:
>
> struct folio_range {
> struct folio *folio;
> unsigned int idx;
> unsigned int nr_pages;
> };
>
> Design
> ======
>
> A "struct gup_out" descriptor in mm/gup.c holds either the page array or
> the folio-range array. The fast and slow page-table walkers append to the
> descriptor instead of advancing a pages[] pointer, so a single walk
> serves both interfaces without duplicating the walkers.
>
> The PTE batching in gup_fast_pte_range() is provided by Yeoreum Yun's
> series [4], which this RFC builds on. This series reuses that batching
> and its page-table revalidation logic. For the folio-range interface,
> each pinned run is recorded directly as a folio range instead of being
> expanded into individual page pointers in the output array.
>
> pin_user_folio_ranges_fast() stops once the caller's range array is
> full, so callers size the array in ranges, independently of the number
> of pages requested. A single GUP-fast walk is additionally capped at
> EXTRACT_MAX_PAGES (1024 pages) to bound the time spent with interrupts
> disabled when the buffer is backed by small pages.
>
> unpin_user_folio_ranges() walks the range array and calls gup_put_folio()
> once per range, releasing the references pin_user_folio_ranges_fast()
> took for it.
>
> In iov_iter_extract_bvecs(), the pinned folio ranges are stored into the
> free slot of the bvec array and converted in place, one bvec per range.
> Block direct I/O picks this up through bio_iov_iter_get_pages() without
> changes to the block layer.
>
> Non-folio mappings
> ==================
>
> Memory that is not backed by folios, such as pages a driver inserted
> into a VM_MIXEDMAP VMA, makes pin_user_folio_ranges_fast() fail with
> -EOPNOTSUPP when no pages were pinned; such iterators fall back to the
> page-array path.
>
> For folio-range output, gup_fast_folio_allowed() additionally requires
> a non-NULL folio->mapping, except for hugetlb folios. If the mapping is
> NULL, it falls back to the slow path, where the VMA can be checked.
> The existing file-backed and secretmem restrictions remain unchanged.
>
> FOLL_LONGTERM is not supported yet and is rejected.
>
> Testing
> =======
>
> Measured the time spent in bio_iov_iter_get_pages() per I/O with
> function_graph, on the baseline and the baseline plus this series.
> Both kernels ran with the performance governor, boost disabled, all
> C-states except POLL disabled, and fio pinned to one CPU:
>
> trace-cmd record -p function_graph -g bio_iov_iter_get_pages \
> --max-graph-depth 8 -O funcgraph-overhead -O nofuncgraph-irqs -F -c \
> fio --name=io_uring_test --filename=/dev/nvme1n1 --ioengine=io_uring \
> --rw=randwrite --direct=1 --bs=<bs> --iodepth=8 --iomem=mmap \
> --iomem_align=2M --numjobs=1 --size=64M --cpus_allowed=23
>
> Median of calls not interrupted during the walk:
>
> bs buffer baseline patched result
> 128K 64 kB mTHP 13.08 us 13.09 us neutral
> 1M 64 kB mTHP 63.0 us 63.6 us neutral
> 4M 2 MB THP (PMD) 25.9 us 8.65 us ~3.0x faster
> 8M 2 MB THP (PMD) 49.4 us ~17.0 us ~2.9x faster
>
> Note: the baseline is the upstream kernel with Yeoreum Yun's [4] and
> Rik van Riel's [5] changes applied.
>
> For larger I/O, the baseline's bio extraction path requests at max
> 256 pages per GUP call, limited by the available bvec slots. This
> series instead uses those slots for folio ranges, each of which can
> describe many pages, thereby allowing the series to request up to 1024
> pages per call which accounts for the speed up observed in
> 'bio_iov_iter_get_pages' for 4MB and 8MB DIRECT I/Os.
>
> Testing: Ran selftests gup_test, gup_longterm and cow on the patched kernel
> and no regressions were observed.
>
> Link [1]: https://lpc.events/event/18/contributions/1773/
> Link [2]: https://lore.kernel.org/all/f2b5189f-10de-4685-97f3-6ee08d159743@kernel.org/
> Link [3]: https://lore.kernel.org/all/20260917083747.786-1-aa9736195201@gmail.com/
> Link [4]: https://lore.kernel.org/all/20261110-refactor_gup_fast-v4-0-e779a7674274@arm.com/
> Link [5]: https://lore.kernel.org/all/20261102193026.3860700-1-riel@surriel.com/
>
> Swarna Prabhu (5):
> mm/gup: add struct gup_out output descriptor
> mm/gup: make the slow path walkers record into struct gup_out
> mm/gup: make the GUP-fast walkers record into struct gup_out
> mm/gup: return pinned memory as folio ranges
> iov_iter: extract bvecs of user-backed iters from folio ranges
>
> include/linux/mm.h | 37 +++
> include/linux/mm_types.h | 23 ++
> lib/iov_iter.c | 100 ++++++
> mm/gup.c | 695 +++++++++++++++++++++++++++++++--------
That's *a lot* of churn from someone new to MM. :)
This will need a lot of guidance from MM experts given that this is your first
MM submission.
Sorry, but we don't have capacity for this:
https://lore.kernel.org/linux-mm/a2f584f6-e382-4876-aa6d-d5a34de6ee59@kernel.org/
--
Cheers,
David
^ permalink raw reply [flat|nested] 7+ messages in thread
end of thread, other threads:[~2026-11-18 23:28 UTC | newest]
Thread overview: 7+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2026-11-18 21:34 [RFC PATCH 0/5] gup: introduce folio-range interface for pinning user memory Swarna Prabhu
2026-11-18 21:34 ` [PATCH 1/5] mm/gup: add struct gup_out output descriptor Swarna Prabhu
2026-11-18 21:34 ` [PATCH 2/5] mm/gup: make the slow path walkers record into struct gup_out Swarna Prabhu
2026-11-18 21:34 ` [PATCH 3/5] mm/gup: make the GUP-fast " Swarna Prabhu
2026-11-18 21:34 ` [PATCH 4/5] mm/gup: return pinned memory as folio ranges Swarna Prabhu
2026-11-18 21:34 ` [PATCH 5/5] iov_iter: extract bvecs of user-backed iters from " Swarna Prabhu
2026-11-18 23:28 ` [RFC PATCH 0/5] gup: introduce folio-range interface for pinning user memory David Hildenbrand (Arm)
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox