* [PATCH 0/3] mm/migrate: batch PTE restoration for large folios
@ 2026-11-20 3:29 Wandun Chen
2026-11-20 3:29 ` [PATCH 1/3] mm: factor out common softleaf offset PTE batch scanning Wandun Chen
` (2 more replies)
0 siblings, 3 replies; 6+ messages in thread
From: Wandun Chen @ 2026-11-20 3:29 UTC (permalink / raw)
To: david, akpm, linux-mm, linux-kernel
Cc: ljs, liam, vbabka, rppt, surenb, mhocko, ziy, matthew.brost,
joshua.hahnjy, byungchul, gourry, ying.huang, apopple
From: Wandun Chen <chenwandun@lixiang.com>
remove_migration_pte() restores PTE-mapped large folios one PTE at a
time, constructing each new PTE and updating the folio references and
rmap separately.
Following David Hildenbrand's suggestion [1], this series batches
adjacent migration entries with consecutive PFNs and matching type
and flags. When a batch restores the whole folio in an mlocked VMA,
the rmap helpers automatically mark the folio as mlocked. It also
extends pte-batching to device-private folios.
The series applies on top of the mlock tracking fix for PTE-mapped
large-folio migration [2].
[1] https://lore.kernel.org/lkml/5f9c449e-43b3-4732-9c14-8199418cace0@kernel.org/
[2] https://lore.kernel.org/lkml/20261113083927.2727599-1-chenwandun1@gmail.com/
Wandun Chen (3):
mm: factor out common softleaf offset PTE batch scanning
mm/migrate: batch PTE restoration in remove_migration_pte()
mm/migrate: batch PTE restoration for device-private folios
mm/internal.h | 28 ++++++++++++++++++++--------
mm/migrate.c | 51 +++++++++++++++++++++++++++++++++++++++++++++------
2 files changed, 65 insertions(+), 14 deletions(-)
--
2.43.0
^ permalink raw reply [flat|nested] 6+ messages in thread* [PATCH 1/3] mm: factor out common softleaf offset PTE batch scanning 2026-11-20 3:29 [PATCH 0/3] mm/migrate: batch PTE restoration for large folios Wandun Chen @ 2026-11-20 3:29 ` Wandun Chen 2026-11-20 15:34 ` Lorenzo Stoakes (ARM) 2026-11-20 3:29 ` [PATCH 2/3] mm/migrate: batch PTE restoration in remove_migration_pte() Wandun Chen 2026-11-20 3:29 ` [PATCH 3/3] mm/migrate: batch PTE restoration for device-private folios Wandun Chen 2 siblings, 1 reply; 6+ messages in thread From: Wandun Chen @ 2026-11-20 3:29 UTC (permalink / raw) To: david, akpm, linux-mm, linux-kernel Cc: ljs, liam, vbabka, rppt, surenb, mhocko, ziy, matthew.brost, joshua.hahnjy, byungchul, gourry, ying.huang, apopple From: Wandun Chen <chenwandun@lixiang.com> Extract the consecutive-offset scan from swap_pte_batch() into __softleaf_pte_batch() to prepare for migration PTE batching in a subsequent patch. No functional change. Signed-off-by: Wandun Chen <chenwandun@lixiang.com> --- mm/internal.h | 22 ++++++++++++++-------- 1 file changed, 14 insertions(+), 8 deletions(-) diff --git a/mm/internal.h b/mm/internal.h index 3a1643afedc0..9f78c12882cb 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -512,29 +512,26 @@ static inline pte_t pte_next_swp_offset(pte_t pte) } /** - * swap_pte_batch - detect a PTE batch for a set of contiguous swap entries + * __softleaf_pte_batch - detect a PTE batch for contiguous softleaf entries * @start_ptep: Page table pointer for the first entry. * @max_nr: The maximum number of table entries to consider. * @pte: Page table entry for the first entry. * - * Detect a batch of contiguous swap entries: consecutive (non-present) PTEs - * containing swap entries all with consecutive offsets and targeting the same - * swap type, all with matching swp pte bits. + * Detect a batch of contiguous swap or migration entries: consecutive + * (non-present) PTEs with consecutive offsets/PFNs and the same entry type, + * all with matching swp pte bits. * * max_nr must be at least one and must be limited by the caller so scanning * cannot exceed a single page table. * * Return: the number of table entries in the batch. */ -static inline int swap_pte_batch(pte_t *start_ptep, int max_nr, pte_t pte) +static inline int __softleaf_pte_batch(pte_t *start_ptep, int max_nr, pte_t pte) { pte_t expected_pte = pte_next_swp_offset(pte); const pte_t *end_ptep = start_ptep + max_nr; pte_t *ptep = start_ptep + 1; - VM_WARN_ON(max_nr < 1); - VM_WARN_ON(!softleaf_is_swap(softleaf_from_pte(pte))); - while (ptep < end_ptep) { pte = ptep_get(ptep); @@ -546,6 +543,15 @@ static inline int swap_pte_batch(pte_t *start_ptep, int max_nr, pte_t pte) return ptep - start_ptep; } + +static inline int swap_pte_batch(pte_t *start_ptep, int max_nr, pte_t pte) +{ + VM_WARN_ON(max_nr < 1); + VM_WARN_ON(!softleaf_is_swap(softleaf_from_pte(pte))); + + return __softleaf_pte_batch(start_ptep, max_nr, pte); +} + #endif /* CONFIG_MMU */ void __acct_reclaim_writeback(pg_data_t *pgdat, struct folio *folio, -- 2.43.0 ^ permalink raw reply related [flat|nested] 6+ messages in thread
* Re: [PATCH 1/3] mm: factor out common softleaf offset PTE batch scanning 2026-11-20 3:29 ` [PATCH 1/3] mm: factor out common softleaf offset PTE batch scanning Wandun Chen @ 2026-11-20 15:34 ` Lorenzo Stoakes (ARM) 0 siblings, 0 replies; 6+ messages in thread From: Lorenzo Stoakes (ARM) @ 2026-11-20 15:34 UTC (permalink / raw) To: Wandun Chen Cc: david, akpm, linux-mm, linux-kernel, liam, vbabka, rppt, surenb, mhocko, ziy, matthew.brost, joshua.hahnjy, byungchul, gourry, ying.huang, apopple On Fri, Nov 20, 2026 at 11:29:47AM +0800, Wandun Chen wrote: > From: Wandun Chen <chenwandun@lixiang.com> > > Extract the consecutive-offset scan from swap_pte_batch() into > __softleaf_pte_batch() to prepare for migration PTE batching in a > subsequent patch. > > No functional change. > > Signed-off-by: Wandun Chen <chenwandun@lixiang.com> > --- > mm/internal.h | 22 ++++++++++++++-------- > 1 file changed, 14 insertions(+), 8 deletions(-) > > diff --git a/mm/internal.h b/mm/internal.h > index 3a1643afedc0..9f78c12882cb 100644 > --- a/mm/internal.h > +++ b/mm/internal.h > @@ -512,29 +512,26 @@ static inline pte_t pte_next_swp_offset(pte_t pte) > } > > /** > - * swap_pte_batch - detect a PTE batch for a set of contiguous swap entries > + * __softleaf_pte_batch - detect a PTE batch for contiguous softleaf entries > * @start_ptep: Page table pointer for the first entry. > * @max_nr: The maximum number of table entries to consider. > * @pte: Page table entry for the first entry. > * > - * Detect a batch of contiguous swap entries: consecutive (non-present) PTEs > - * containing swap entries all with consecutive offsets and targeting the same > - * swap type, all with matching swp pte bits. > + * Detect a batch of contiguous swap or migration entries: consecutive You mean softleaf entries? > + * (non-present) PTEs with consecutive offsets/PFNs and the same entry type, > + * all with matching swp pte bits. Maybe 'softleaf bits'? Ah I wish I finished the softleaf conversion properly :P > * > * max_nr must be at least one and must be limited by the caller so scanning > * cannot exceed a single page table. > * > * Return: the number of table entries in the batch. > */ > -static inline int swap_pte_batch(pte_t *start_ptep, int max_nr, pte_t pte) > +static inline int __softleaf_pte_batch(pte_t *start_ptep, int max_nr, pte_t pte) > { > pte_t expected_pte = pte_next_swp_offset(pte); > const pte_t *end_ptep = start_ptep + max_nr; > pte_t *ptep = start_ptep + 1; > > - VM_WARN_ON(max_nr < 1); > - VM_WARN_ON(!softleaf_is_swap(softleaf_from_pte(pte))); > - > while (ptep < end_ptep) { > pte = ptep_get(ptep); > > @@ -546,6 +543,15 @@ static inline int swap_pte_batch(pte_t *start_ptep, int max_nr, pte_t pte) > > return ptep - start_ptep; > } > + > +static inline int swap_pte_batch(pte_t *start_ptep, int max_nr, pte_t pte) > +{ > + VM_WARN_ON(max_nr < 1); I wonder if max_nr should be nr. Or is it a maximum range to examine I guess? > + VM_WARN_ON(!softleaf_is_swap(softleaf_from_pte(pte))); Since you're getting a softleaf_t here... > + > + return __softleaf_pte_batch(start_ptep, max_nr, pte); ...Does the batching need the full PTE or can you just pass the entry? > +} > + > #endif /* CONFIG_MMU */ > > void __acct_reclaim_writeback(pg_data_t *pgdat, struct folio *folio, > -- > 2.43.0 > -- Cheers, Lorenzo ^ permalink raw reply [flat|nested] 6+ messages in thread
* [PATCH 2/3] mm/migrate: batch PTE restoration in remove_migration_pte() 2026-11-20 3:29 [PATCH 0/3] mm/migrate: batch PTE restoration for large folios Wandun Chen 2026-11-20 3:29 ` [PATCH 1/3] mm: factor out common softleaf offset PTE batch scanning Wandun Chen @ 2026-11-20 3:29 ` Wandun Chen 2026-11-20 15:22 ` Lorenzo Stoakes (ARM) 2026-11-20 3:29 ` [PATCH 3/3] mm/migrate: batch PTE restoration for device-private folios Wandun Chen 2 siblings, 1 reply; 6+ messages in thread From: Wandun Chen @ 2026-11-20 3:29 UTC (permalink / raw) To: david, akpm, linux-mm, linux-kernel Cc: ljs, liam, vbabka, rppt, surenb, mhocko, ziy, matthew.brost, joshua.hahnjy, byungchul, gourry, ying.huang, apopple From: Wandun Chen <chenwandun@lixiang.com> remove_migration_pte() restores PTE-mapped large folios one PTE at a time, constructing each new PTE and updating the folio references and rmap separately. Add migration_pte_batch() to find adjacent migration entries with consecutive PFNs and matching type and flags. When a batch covers the whole folio, the rmap helpers already call mlock_vma_folio(). Skip the explicit call to avoid double mlock accounting. Device-private PTEs encode PFNs as swap offsets. set_ptes() does not handle this encoding when constructing subsequent PTEs, such as on arm64 with a 16KB page granule, leaving device-private destinations unbatched. The next patch enables batching for these entries by advancing their softleaf offsets. Suggested-by: David Hildenbrand (Arm) <david@kernel.org> Link: https://lore.kernel.org/lkml/5f9c449e-43b3-4732-9c14-8199418cace0@kernel.org/ Signed-off-by: Wandun Chen <chenwandun@lixiang.com> --- mm/internal.h | 6 ++++++ mm/migrate.c | 35 +++++++++++++++++++++++++++++------ 2 files changed, 35 insertions(+), 6 deletions(-) diff --git a/mm/internal.h b/mm/internal.h index 9f78c12882cb..e6ecf307d532 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -552,6 +552,12 @@ static inline int swap_pte_batch(pte_t *start_ptep, int max_nr, pte_t pte) return __softleaf_pte_batch(start_ptep, max_nr, pte); } +static inline int migration_pte_batch(pte_t *start_ptep, int max_nr, pte_t pte) +{ + VM_WARN_ON(!softleaf_is_migration(softleaf_from_pte(pte))); + + return __softleaf_pte_batch(start_ptep, max_nr, pte); +} #endif /* CONFIG_MMU */ void __acct_reclaim_writeback(pg_data_t *pgdat, struct folio *folio, diff --git a/mm/migrate.c b/mm/migrate.c index 3c39d309e6bd..e000ef77ea87 100644 --- a/mm/migrate.c +++ b/mm/migrate.c @@ -358,6 +358,7 @@ static bool remove_migration_pte(struct folio *folio, while (page_vma_mapped_walk(&pvmw)) { rmap_t rmap_flags = RMAP_NONE; unsigned long idx = 0; + int nr = 1; softleaf_t entry; struct page *new; pte_t old_pte; @@ -441,17 +442,34 @@ static bool remove_migration_pte(struct folio *folio, } else #endif { + if (folio_test_large(rmap_walk_arg->folio) && + !is_device_private_page(new)) { + unsigned long end_addr = pmd_addr_end(pvmw.address, vma->vm_end); + unsigned long max_nr = (end_addr - pvmw.address) >> PAGE_SHIFT; + + max_nr = min_t(unsigned long, max_nr, + PTRS_PER_PTE - pte_index(pvmw.address)); + + max_nr = min_t(unsigned long, max_nr, + folio_nr_pages(rmap_walk_arg->folio) - idx); + + nr = migration_pte_batch(pvmw.pte, max_nr, old_pte); + } + if (nr > 1) + folio_ref_add(folio, nr - 1); + if (folio_test_anon(folio)) - folio_add_anon_rmap_pte(folio, new, vma, - pvmw.address, rmap_flags); + folio_add_anon_rmap_ptes(folio, new, nr, vma, + pvmw.address, rmap_flags); else - folio_add_file_rmap_pte(folio, new, vma); - set_pte_at(vma->vm_mm, pvmw.address, pvmw.pte, pte); + folio_add_file_rmap_ptes(folio, new, nr, vma); + set_ptes(vma->vm_mm, pvmw.address, pvmw.pte, pte, nr); - ptes++; + ptes += nr; } if (READ_ONCE(vma->vm_flags) & VM_LOCKED) { if (folio_test_large(folio) && + nr < folio_nr_pages(folio) && ptes == folio_nr_pages(folio) && !(pvmw.flags & PVMW_PGTABLE_CROSSED)) mlock_vma_folio(folio, vma); @@ -462,7 +480,12 @@ static bool remove_migration_pte(struct folio *folio, compound_order(new)); /* No need to invalidate - it was non-present before */ - update_mmu_cache(vma, pvmw.address, pvmw.pte); + update_mmu_cache_range(NULL, vma, pvmw.address, pvmw.pte, nr); + + if (nr == folio_nr_pages(folio)) { + page_vma_mapped_walk_done(&pvmw); + break; + } } return true; -- 2.43.0 ^ permalink raw reply related [flat|nested] 6+ messages in thread
* Re: [PATCH 2/3] mm/migrate: batch PTE restoration in remove_migration_pte() 2026-11-20 3:29 ` [PATCH 2/3] mm/migrate: batch PTE restoration in remove_migration_pte() Wandun Chen @ 2026-11-20 15:22 ` Lorenzo Stoakes (ARM) 0 siblings, 0 replies; 6+ messages in thread From: Lorenzo Stoakes (ARM) @ 2026-11-20 15:22 UTC (permalink / raw) To: Wandun Chen Cc: david, akpm, linux-mm, linux-kernel, liam, vbabka, rppt, surenb, mhocko, ziy, matthew.brost, joshua.hahnjy, byungchul, gourry, ying.huang, apopple On Fri, Nov 20, 2026 at 11:29:48AM +0800, Wandun Chen wrote: > From: Wandun Chen <chenwandun@lixiang.com> > > remove_migration_pte() restores PTE-mapped large folios one PTE at a > time, constructing each new PTE and updating the folio references and > rmap separately. > > Add migration_pte_batch() to find adjacent migration entries with > consecutive PFNs and matching type and flags. > > When a batch covers the whole folio, the rmap helpers already call > mlock_vma_folio(). Skip the explicit call to avoid double mlock > accounting. > > Device-private PTEs encode PFNs as swap offsets. set_ptes() does not > handle this encoding when constructing subsequent PTEs, such as on > arm64 with a 16KB page granule, leaving device-private destinations > unbatched. The next patch enables batching for these entries by > advancing their softleaf offsets. > > Suggested-by: David Hildenbrand (Arm) <david@kernel.org> > Link: https://lore.kernel.org/lkml/5f9c449e-43b3-4732-9c14-8199418cace0@kernel.org/ > Signed-off-by: Wandun Chen <chenwandun@lixiang.com> > --- > mm/internal.h | 6 ++++++ > mm/migrate.c | 35 +++++++++++++++++++++++++++++------ > 2 files changed, 35 insertions(+), 6 deletions(-) > > diff --git a/mm/internal.h b/mm/internal.h > index 9f78c12882cb..e6ecf307d532 100644 > --- a/mm/internal.h > +++ b/mm/internal.h > @@ -552,6 +552,12 @@ static inline int swap_pte_batch(pte_t *start_ptep, int max_nr, pte_t pte) > return __softleaf_pte_batch(start_ptep, max_nr, pte); > } > > +static inline int migration_pte_batch(pte_t *start_ptep, int max_nr, pte_t pte) > +{ > + VM_WARN_ON(!softleaf_is_migration(softleaf_from_pte(pte))); > + > + return __softleaf_pte_batch(start_ptep, max_nr, pte); > +} > #endif /* CONFIG_MMU */ > > void __acct_reclaim_writeback(pg_data_t *pgdat, struct folio *folio, > diff --git a/mm/migrate.c b/mm/migrate.c > index 3c39d309e6bd..e000ef77ea87 100644 > --- a/mm/migrate.c > +++ b/mm/migrate.c > @@ -358,6 +358,7 @@ static bool remove_migration_pte(struct folio *folio, > while (page_vma_mapped_walk(&pvmw)) { > rmap_t rmap_flags = RMAP_NONE; > unsigned long idx = 0; > + int nr = 1; > softleaf_t entry; > struct page *new; > pte_t old_pte; > @@ -441,17 +442,34 @@ static bool remove_migration_pte(struct folio *folio, > } else > #endif > { > + if (folio_test_large(rmap_walk_arg->folio) && > + !is_device_private_page(new)) { > + unsigned long end_addr = pmd_addr_end(pvmw.address, vma->vm_end); > + unsigned long max_nr = (end_addr - pvmw.address) >> PAGE_SHIFT; const please > + > + max_nr = min_t(unsigned long, max_nr, > + PTRS_PER_PTE - pte_index(pvmw.address)); > + > + max_nr = min_t(unsigned long, max_nr, > + folio_nr_pages(rmap_walk_arg->folio) - idx); > + > + nr = migration_pte_batch(pvmw.pte, max_nr, old_pte); > + } > + if (nr > 1) > + folio_ref_add(folio, nr - 1); > + Please let's not insert a bunch more open-coded bits in an already way-too-big function :) If you need to add functionality to a function like this, please take the time to pay down technical debt and refactor it first. > if (folio_test_anon(folio)) > - folio_add_anon_rmap_pte(folio, new, vma, > - pvmw.address, rmap_flags); > + folio_add_anon_rmap_ptes(folio, new, nr, vma, > + pvmw.address, rmap_flags); > else > - folio_add_file_rmap_pte(folio, new, vma); > - set_pte_at(vma->vm_mm, pvmw.address, pvmw.pte, pte); > + folio_add_file_rmap_ptes(folio, new, nr, vma); > + set_ptes(vma->vm_mm, pvmw.address, pvmw.pte, pte, nr); > > - ptes++; > + ptes += nr; > } > if (READ_ONCE(vma->vm_flags) & VM_LOCKED) { > if (folio_test_large(folio) && > + nr < folio_nr_pages(folio) && > ptes == folio_nr_pages(folio) && > !(pvmw.flags & PVMW_PGTABLE_CROSSED)) > mlock_vma_folio(folio, vma); > @@ -462,7 +480,12 @@ static bool remove_migration_pte(struct folio *folio, > compound_order(new)); > > /* No need to invalidate - it was non-present before */ > - update_mmu_cache(vma, pvmw.address, pvmw.pte); > + update_mmu_cache_range(NULL, vma, pvmw.address, pvmw.pte, nr); > + > + if (nr == folio_nr_pages(folio)) { > + page_vma_mapped_walk_done(&pvmw); > + break; > + } > } > > return true; > -- > 2.43.0 > -- Cheers, Lorenzo ^ permalink raw reply [flat|nested] 6+ messages in thread
* [PATCH 3/3] mm/migrate: batch PTE restoration for device-private folios 2026-11-20 3:29 [PATCH 0/3] mm/migrate: batch PTE restoration for large folios Wandun Chen 2026-11-20 3:29 ` [PATCH 1/3] mm: factor out common softleaf offset PTE batch scanning Wandun Chen 2026-11-20 3:29 ` [PATCH 2/3] mm/migrate: batch PTE restoration in remove_migration_pte() Wandun Chen @ 2026-11-20 3:29 ` Wandun Chen 2 siblings, 0 replies; 6+ messages in thread From: Wandun Chen @ 2026-11-20 3:29 UTC (permalink / raw) To: david, akpm, linux-mm, linux-kernel Cc: ljs, liam, vbabka, rppt, surenb, mhocko, ziy, matthew.brost, joshua.hahnjy, byungchul, gourry, ying.huang, apopple From: Wandun Chen <chenwandun@lixiang.com> Device-private PTEs encode PFN as softleaf offset. set_ptes() does not handle this encoding when constructing subsequent PTEs, such as on arm64 with a 16KB page granule, leaving device-private destinations folio unbatched. Add set_device_private_ptes() to construct subsequent entries with pte_next_swp_offset(). This enables remove_migration_pte() to batch PTE restoration for device-private folios. Signed-off-by: Wandun Chen <chenwandun@lixiang.com> --- mm/migrate.c | 22 +++++++++++++++++++--- 1 file changed, 19 insertions(+), 3 deletions(-) diff --git a/mm/migrate.c b/mm/migrate.c index e000ef77ea87..3d3d91fd61c7 100644 --- a/mm/migrate.c +++ b/mm/migrate.c @@ -345,6 +345,19 @@ struct rmap_walk_arg { bool map_unused_to_zeropage; }; +static void set_device_private_ptes(struct mm_struct *mm, unsigned long addr, + pte_t *ptep, pte_t pte, unsigned int nr) +{ + for (;;) { + set_pte_at(mm, addr, ptep, pte); + if (--nr == 0) + break; + addr += PAGE_SIZE; + ptep++; + pte = pte_next_swp_offset(pte); + } +} + /* * Restore a potential migration pte to a working pte entry */ @@ -442,8 +455,7 @@ static bool remove_migration_pte(struct folio *folio, } else #endif { - if (folio_test_large(rmap_walk_arg->folio) && - !is_device_private_page(new)) { + if (folio_test_large(rmap_walk_arg->folio)) { unsigned long end_addr = pmd_addr_end(pvmw.address, vma->vm_end); unsigned long max_nr = (end_addr - pvmw.address) >> PAGE_SHIFT; @@ -463,7 +475,11 @@ static bool remove_migration_pte(struct folio *folio, pvmw.address, rmap_flags); else folio_add_file_rmap_ptes(folio, new, nr, vma); - set_ptes(vma->vm_mm, pvmw.address, pvmw.pte, pte, nr); + if (unlikely(is_device_private_page(new))) + set_device_private_ptes(vma->vm_mm, pvmw.address, + pvmw.pte, pte, nr); + else + set_ptes(vma->vm_mm, pvmw.address, pvmw.pte, pte, nr); ptes += nr; } -- 2.43.0 ^ permalink raw reply related [flat|nested] 6+ messages in thread
end of thread, other threads:[~2026-11-20 15:35 UTC | newest] Thread overview: 6+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2026-11-20 3:29 [PATCH 0/3] mm/migrate: batch PTE restoration for large folios Wandun Chen 2026-11-20 3:29 ` [PATCH 1/3] mm: factor out common softleaf offset PTE batch scanning Wandun Chen 2026-11-20 15:34 ` Lorenzo Stoakes (ARM) 2026-11-20 3:29 ` [PATCH 2/3] mm/migrate: batch PTE restoration in remove_migration_pte() Wandun Chen 2026-11-20 15:22 ` Lorenzo Stoakes (ARM) 2026-11-20 3:29 ` [PATCH 3/3] mm/migrate: batch PTE restoration for device-private folios Wandun Chen
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.