* [PATCH 0/3] mm/migrate: batch PTE restoration for large folios
@ 2026-11-20 3:29 Wandun Chen
2026-11-20 3:29 ` [PATCH 1/3] mm: factor out common softleaf offset PTE batch scanning Wandun Chen
` (2 more replies)
0 siblings, 3 replies; 6+ messages in thread
From: Wandun Chen @ 2026-11-20 3:29 UTC (permalink / raw)
To: david, akpm, linux-mm, linux-kernel
Cc: ljs, liam, vbabka, rppt, surenb, mhocko, ziy, matthew.brost,
joshua.hahnjy, byungchul, gourry, ying.huang, apopple
From: Wandun Chen <chenwandun@lixiang.com>
remove_migration_pte() restores PTE-mapped large folios one PTE at a
time, constructing each new PTE and updating the folio references and
rmap separately.
Following David Hildenbrand's suggestion [1], this series batches
adjacent migration entries with consecutive PFNs and matching type
and flags. When a batch restores the whole folio in an mlocked VMA,
the rmap helpers automatically mark the folio as mlocked. It also
extends pte-batching to device-private folios.
The series applies on top of the mlock tracking fix for PTE-mapped
large-folio migration [2].
[1] https://lore.kernel.org/lkml/5f9c449e-43b3-4732-9c14-8199418cace0@kernel.org/
[2] https://lore.kernel.org/lkml/20261113083927.2727599-1-chenwandun1@gmail.com/
Wandun Chen (3):
mm: factor out common softleaf offset PTE batch scanning
mm/migrate: batch PTE restoration in remove_migration_pte()
mm/migrate: batch PTE restoration for device-private folios
mm/internal.h | 28 ++++++++++++++++++++--------
mm/migrate.c | 51 +++++++++++++++++++++++++++++++++++++++++++++------
2 files changed, 65 insertions(+), 14 deletions(-)
--
2.43.0
^ permalink raw reply [flat|nested] 6+ messages in thread* [PATCH 1/3] mm: factor out common softleaf offset PTE batch scanning 2026-11-20 3:29 [PATCH 0/3] mm/migrate: batch PTE restoration for large folios Wandun Chen @ 2026-11-20 3:29 ` Wandun Chen 2026-11-20 15:34 ` Lorenzo Stoakes (ARM) 2026-11-20 3:29 ` [PATCH 2/3] mm/migrate: batch PTE restoration in remove_migration_pte() Wandun Chen 2026-11-20 3:29 ` [PATCH 3/3] mm/migrate: batch PTE restoration for device-private folios Wandun Chen 2 siblings, 1 reply; 6+ messages in thread From: Wandun Chen @ 2026-11-20 3:29 UTC (permalink / raw) To: david, akpm, linux-mm, linux-kernel Cc: ljs, liam, vbabka, rppt, surenb, mhocko, ziy, matthew.brost, joshua.hahnjy, byungchul, gourry, ying.huang, apopple From: Wandun Chen <chenwandun@lixiang.com> Extract the consecutive-offset scan from swap_pte_batch() into __softleaf_pte_batch() to prepare for migration PTE batching in a subsequent patch. No functional change. Signed-off-by: Wandun Chen <chenwandun@lixiang.com> --- mm/internal.h | 22 ++++++++++++++-------- 1 file changed, 14 insertions(+), 8 deletions(-) diff --git a/mm/internal.h b/mm/internal.h index 3a1643afedc0..9f78c12882cb 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -512,29 +512,26 @@ static inline pte_t pte_next_swp_offset(pte_t pte) } /** - * swap_pte_batch - detect a PTE batch for a set of contiguous swap entries + * __softleaf_pte_batch - detect a PTE batch for contiguous softleaf entries * @start_ptep: Page table pointer for the first entry. * @max_nr: The maximum number of table entries to consider. * @pte: Page table entry for the first entry. * - * Detect a batch of contiguous swap entries: consecutive (non-present) PTEs - * containing swap entries all with consecutive offsets and targeting the same - * swap type, all with matching swp pte bits. + * Detect a batch of contiguous swap or migration entries: consecutive + * (non-present) PTEs with consecutive offsets/PFNs and the same entry type, + * all with matching swp pte bits. * * max_nr must be at least one and must be limited by the caller so scanning * cannot exceed a single page table. * * Return: the number of table entries in the batch. */ -static inline int swap_pte_batch(pte_t *start_ptep, int max_nr, pte_t pte) +static inline int __softleaf_pte_batch(pte_t *start_ptep, int max_nr, pte_t pte) { pte_t expected_pte = pte_next_swp_offset(pte); const pte_t *end_ptep = start_ptep + max_nr; pte_t *ptep = start_ptep + 1; - VM_WARN_ON(max_nr < 1); - VM_WARN_ON(!softleaf_is_swap(softleaf_from_pte(pte))); - while (ptep < end_ptep) { pte = ptep_get(ptep); @@ -546,6 +543,15 @@ static inline int swap_pte_batch(pte_t *start_ptep, int max_nr, pte_t pte) return ptep - start_ptep; } + +static inline int swap_pte_batch(pte_t *start_ptep, int max_nr, pte_t pte) +{ + VM_WARN_ON(max_nr < 1); + VM_WARN_ON(!softleaf_is_swap(softleaf_from_pte(pte))); + + return __softleaf_pte_batch(start_ptep, max_nr, pte); +} + #endif /* CONFIG_MMU */ void __acct_reclaim_writeback(pg_data_t *pgdat, struct folio *folio, -- 2.43.0 ^ permalink raw reply related [flat|nested] 6+ messages in thread
* Re: [PATCH 1/3] mm: factor out common softleaf offset PTE batch scanning 2026-11-20 3:29 ` [PATCH 1/3] mm: factor out common softleaf offset PTE batch scanning Wandun Chen @ 2026-11-20 15:34 ` Lorenzo Stoakes (ARM) 0 siblings, 0 replies; 6+ messages in thread From: Lorenzo Stoakes (ARM) @ 2026-11-20 15:34 UTC (permalink / raw) To: Wandun Chen Cc: david, akpm, linux-mm, linux-kernel, liam, vbabka, rppt, surenb, mhocko, ziy, matthew.brost, joshua.hahnjy, byungchul, gourry, ying.huang, apopple On Fri, Nov 20, 2026 at 11:29:47AM +0800, Wandun Chen wrote: > From: Wandun Chen <chenwandun@lixiang.com> > > Extract the consecutive-offset scan from swap_pte_batch() into > __softleaf_pte_batch() to prepare for migration PTE batching in a > subsequent patch. > > No functional change. > > Signed-off-by: Wandun Chen <chenwandun@lixiang.com> > --- > mm/internal.h | 22 ++++++++++++++-------- > 1 file changed, 14 insertions(+), 8 deletions(-) > > diff --git a/mm/internal.h b/mm/internal.h > index 3a1643afedc0..9f78c12882cb 100644 > --- a/mm/internal.h > +++ b/mm/internal.h > @@ -512,29 +512,26 @@ static inline pte_t pte_next_swp_offset(pte_t pte) > } > > /** > - * swap_pte_batch - detect a PTE batch for a set of contiguous swap entries > + * __softleaf_pte_batch - detect a PTE batch for contiguous softleaf entries > * @start_ptep: Page table pointer for the first entry. > * @max_nr: The maximum number of table entries to consider. > * @pte: Page table entry for the first entry. > * > - * Detect a batch of contiguous swap entries: consecutive (non-present) PTEs > - * containing swap entries all with consecutive offsets and targeting the same > - * swap type, all with matching swp pte bits. > + * Detect a batch of contiguous swap or migration entries: consecutive You mean softleaf entries? > + * (non-present) PTEs with consecutive offsets/PFNs and the same entry type, > + * all with matching swp pte bits. Maybe 'softleaf bits'? Ah I wish I finished the softleaf conversion properly :P > * > * max_nr must be at least one and must be limited by the caller so scanning > * cannot exceed a single page table. > * > * Return: the number of table entries in the batch. > */ > -static inline int swap_pte_batch(pte_t *start_ptep, int max_nr, pte_t pte) > +static inline int __softleaf_pte_batch(pte_t *start_ptep, int max_nr, pte_t pte) > { > pte_t expected_pte = pte_next_swp_offset(pte); > const pte_t *end_ptep = start_ptep + max_nr; > pte_t *ptep = start_ptep + 1; > > - VM_WARN_ON(max_nr < 1); > - VM_WARN_ON(!softleaf_is_swap(softleaf_from_pte(pte))); > - > while (ptep < end_ptep) { > pte = ptep_get(ptep); > > @@ -546,6 +543,15 @@ static inline int swap_pte_batch(pte_t *start_ptep, int max_nr, pte_t pte) > > return ptep - start_ptep; > } > + > +static inline int swap_pte_batch(pte_t *start_ptep, int max_nr, pte_t pte) > +{ > + VM_WARN_ON(max_nr < 1); I wonder if max_nr should be nr. Or is it a maximum range to examine I guess? > + VM_WARN_ON(!softleaf_is_swap(softleaf_from_pte(pte))); Since you're getting a softleaf_t here... > + > + return __softleaf_pte_batch(start_ptep, max_nr, pte); ...Does the batching need the full PTE or can you just pass the entry? > +} > + > #endif /* CONFIG_MMU */ > > void __acct_reclaim_writeback(pg_data_t *pgdat, struct folio *folio, > -- > 2.43.0 > -- Cheers, Lorenzo ^ permalink raw reply [flat|nested] 6+ messages in thread
* [PATCH 2/3] mm/migrate: batch PTE restoration in remove_migration_pte() 2026-11-20 3:29 [PATCH 0/3] mm/migrate: batch PTE restoration for large folios Wandun Chen 2026-11-20 3:29 ` [PATCH 1/3] mm: factor out common softleaf offset PTE batch scanning Wandun Chen @ 2026-11-20 3:29 ` Wandun Chen 2026-11-20 15:22 ` Lorenzo Stoakes (ARM) 2026-11-20 3:29 ` [PATCH 3/3] mm/migrate: batch PTE restoration for device-private folios Wandun Chen 2 siblings, 1 reply; 6+ messages in thread From: Wandun Chen @ 2026-11-20 3:29 UTC (permalink / raw) To: david, akpm, linux-mm, linux-kernel Cc: ljs, liam, vbabka, rppt, surenb, mhocko, ziy, matthew.brost, joshua.hahnjy, byungchul, gourry, ying.huang, apopple From: Wandun Chen <chenwandun@lixiang.com> remove_migration_pte() restores PTE-mapped large folios one PTE at a time, constructing each new PTE and updating the folio references and rmap separately. Add migration_pte_batch() to find adjacent migration entries with consecutive PFNs and matching type and flags. When a batch covers the whole folio, the rmap helpers already call mlock_vma_folio(). Skip the explicit call to avoid double mlock accounting. Device-private PTEs encode PFNs as swap offsets. set_ptes() does not handle this encoding when constructing subsequent PTEs, such as on arm64 with a 16KB page granule, leaving device-private destinations unbatched. The next patch enables batching for these entries by advancing their softleaf offsets. Suggested-by: David Hildenbrand (Arm) <david@kernel.org> Link: https://lore.kernel.org/lkml/5f9c449e-43b3-4732-9c14-8199418cace0@kernel.org/ Signed-off-by: Wandun Chen <chenwandun@lixiang.com> --- mm/internal.h | 6 ++++++ mm/migrate.c | 35 +++++++++++++++++++++++++++++------ 2 files changed, 35 insertions(+), 6 deletions(-) diff --git a/mm/internal.h b/mm/internal.h index 9f78c12882cb..e6ecf307d532 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -552,6 +552,12 @@ static inline int swap_pte_batch(pte_t *start_ptep, int max_nr, pte_t pte) return __softleaf_pte_batch(start_ptep, max_nr, pte); } +static inline int migration_pte_batch(pte_t *start_ptep, int max_nr, pte_t pte) +{ + VM_WARN_ON(!softleaf_is_migration(softleaf_from_pte(pte))); + + return __softleaf_pte_batch(start_ptep, max_nr, pte); +} #endif /* CONFIG_MMU */ void __acct_reclaim_writeback(pg_data_t *pgdat, struct folio *folio, diff --git a/mm/migrate.c b/mm/migrate.c index 3c39d309e6bd..e000ef77ea87 100644 --- a/mm/migrate.c +++ b/mm/migrate.c @@ -358,6 +358,7 @@ static bool remove_migration_pte(struct folio *folio, while (page_vma_mapped_walk(&pvmw)) { rmap_t rmap_flags = RMAP_NONE; unsigned long idx = 0; + int nr = 1; softleaf_t entry; struct page *new; pte_t old_pte; @@ -441,17 +442,34 @@ static bool remove_migration_pte(struct folio *folio, } else #endif { + if (folio_test_large(rmap_walk_arg->folio) && + !is_device_private_page(new)) { + unsigned long end_addr = pmd_addr_end(pvmw.address, vma->vm_end); + unsigned long max_nr = (end_addr - pvmw.address) >> PAGE_SHIFT; + + max_nr = min_t(unsigned long, max_nr, + PTRS_PER_PTE - pte_index(pvmw.address)); + + max_nr = min_t(unsigned long, max_nr, + folio_nr_pages(rmap_walk_arg->folio) - idx); + + nr = migration_pte_batch(pvmw.pte, max_nr, old_pte); + } + if (nr > 1) + folio_ref_add(folio, nr - 1); + if (folio_test_anon(folio)) - folio_add_anon_rmap_pte(folio, new, vma, - pvmw.address, rmap_flags); + folio_add_anon_rmap_ptes(folio, new, nr, vma, + pvmw.address, rmap_flags); else - folio_add_file_rmap_pte(folio, new, vma); - set_pte_at(vma->vm_mm, pvmw.address, pvmw.pte, pte); + folio_add_file_rmap_ptes(folio, new, nr, vma); + set_ptes(vma->vm_mm, pvmw.address, pvmw.pte, pte, nr); - ptes++; + ptes += nr; } if (READ_ONCE(vma->vm_flags) & VM_LOCKED) { if (folio_test_large(folio) && + nr < folio_nr_pages(folio) && ptes == folio_nr_pages(folio) && !(pvmw.flags & PVMW_PGTABLE_CROSSED)) mlock_vma_folio(folio, vma); @@ -462,7 +480,12 @@ static bool remove_migration_pte(struct folio *folio, compound_order(new)); /* No need to invalidate - it was non-present before */ - update_mmu_cache(vma, pvmw.address, pvmw.pte); + update_mmu_cache_range(NULL, vma, pvmw.address, pvmw.pte, nr); + + if (nr == folio_nr_pages(folio)) { + page_vma_mapped_walk_done(&pvmw); + break; + } } return true; -- 2.43.0 ^ permalink raw reply related [flat|nested] 6+ messages in thread
* Re: [PATCH 2/3] mm/migrate: batch PTE restoration in remove_migration_pte() 2026-11-20 3:29 ` [PATCH 2/3] mm/migrate: batch PTE restoration in remove_migration_pte() Wandun Chen @ 2026-11-20 15:22 ` Lorenzo Stoakes (ARM) 0 siblings, 0 replies; 6+ messages in thread From: Lorenzo Stoakes (ARM) @ 2026-11-20 15:22 UTC (permalink / raw) To: Wandun Chen Cc: david, akpm, linux-mm, linux-kernel, liam, vbabka, rppt, surenb, mhocko, ziy, matthew.brost, joshua.hahnjy, byungchul, gourry, ying.huang, apopple On Fri, Nov 20, 2026 at 11:29:48AM +0800, Wandun Chen wrote: > From: Wandun Chen <chenwandun@lixiang.com> > > remove_migration_pte() restores PTE-mapped large folios one PTE at a > time, constructing each new PTE and updating the folio references and > rmap separately. > > Add migration_pte_batch() to find adjacent migration entries with > consecutive PFNs and matching type and flags. > > When a batch covers the whole folio, the rmap helpers already call > mlock_vma_folio(). Skip the explicit call to avoid double mlock > accounting. > > Device-private PTEs encode PFNs as swap offsets. set_ptes() does not > handle this encoding when constructing subsequent PTEs, such as on > arm64 with a 16KB page granule, leaving device-private destinations > unbatched. The next patch enables batching for these entries by > advancing their softleaf offsets. > > Suggested-by: David Hildenbrand (Arm) <david@kernel.org> > Link: https://lore.kernel.org/lkml/5f9c449e-43b3-4732-9c14-8199418cace0@kernel.org/ > Signed-off-by: Wandun Chen <chenwandun@lixiang.com> > --- > mm/internal.h | 6 ++++++ > mm/migrate.c | 35 +++++++++++++++++++++++++++++------ > 2 files changed, 35 insertions(+), 6 deletions(-) > > diff --git a/mm/internal.h b/mm/internal.h > index 9f78c12882cb..e6ecf307d532 100644 > --- a/mm/internal.h > +++ b/mm/internal.h > @@ -552,6 +552,12 @@ static inline int swap_pte_batch(pte_t *start_ptep, int max_nr, pte_t pte) > return __softleaf_pte_batch(start_ptep, max_nr, pte); > } > > +static inline int migration_pte_batch(pte_t *start_ptep, int max_nr, pte_t pte) > +{ > + VM_WARN_ON(!softleaf_is_migration(softleaf_from_pte(pte))); > + > + return __softleaf_pte_batch(start_ptep, max_nr, pte); > +} > #endif /* CONFIG_MMU */ > > void __acct_reclaim_writeback(pg_data_t *pgdat, struct folio *folio, > diff --git a/mm/migrate.c b/mm/migrate.c > index 3c39d309e6bd..e000ef77ea87 100644 > --- a/mm/migrate.c > +++ b/mm/migrate.c > @@ -358,6 +358,7 @@ static bool remove_migration_pte(struct folio *folio, > while (page_vma_mapped_walk(&pvmw)) { > rmap_t rmap_flags = RMAP_NONE; > unsigned long idx = 0; > + int nr = 1; > softleaf_t entry; > struct page *new; > pte_t old_pte; > @@ -441,17 +442,34 @@ static bool remove_migration_pte(struct folio *folio, > } else > #endif > { > + if (folio_test_large(rmap_walk_arg->folio) && > + !is_device_private_page(new)) { > + unsigned long end_addr = pmd_addr_end(pvmw.address, vma->vm_end); > + unsigned long max_nr = (end_addr - pvmw.address) >> PAGE_SHIFT; const please > + > + max_nr = min_t(unsigned long, max_nr, > + PTRS_PER_PTE - pte_index(pvmw.address)); > + > + max_nr = min_t(unsigned long, max_nr, > + folio_nr_pages(rmap_walk_arg->folio) - idx); > + > + nr = migration_pte_batch(pvmw.pte, max_nr, old_pte); > + } > + if (nr > 1) > + folio_ref_add(folio, nr - 1); > + Please let's not insert a bunch more open-coded bits in an already way-too-big function :) If you need to add functionality to a function like this, please take the time to pay down technical debt and refactor it first. > if (folio_test_anon(folio)) > - folio_add_anon_rmap_pte(folio, new, vma, > - pvmw.address, rmap_flags); > + folio_add_anon_rmap_ptes(folio, new, nr, vma, > + pvmw.address, rmap_flags); > else > - folio_add_file_rmap_pte(folio, new, vma); > - set_pte_at(vma->vm_mm, pvmw.address, pvmw.pte, pte); > + folio_add_file_rmap_ptes(folio, new, nr, vma); > + set_ptes(vma->vm_mm, pvmw.address, pvmw.pte, pte, nr); > > - ptes++; > + ptes += nr; > } > if (READ_ONCE(vma->vm_flags) & VM_LOCKED) { > if (folio_test_large(folio) && > + nr < folio_nr_pages(folio) && > ptes == folio_nr_pages(folio) && > !(pvmw.flags & PVMW_PGTABLE_CROSSED)) > mlock_vma_folio(folio, vma); > @@ -462,7 +480,12 @@ static bool remove_migration_pte(struct folio *folio, > compound_order(new)); > > /* No need to invalidate - it was non-present before */ > - update_mmu_cache(vma, pvmw.address, pvmw.pte); > + update_mmu_cache_range(NULL, vma, pvmw.address, pvmw.pte, nr); > + > + if (nr == folio_nr_pages(folio)) { > + page_vma_mapped_walk_done(&pvmw); > + break; > + } > } > > return true; > -- > 2.43.0 > -- Cheers, Lorenzo ^ permalink raw reply [flat|nested] 6+ messages in thread
* [PATCH 3/3] mm/migrate: batch PTE restoration for device-private folios 2026-11-20 3:29 [PATCH 0/3] mm/migrate: batch PTE restoration for large folios Wandun Chen 2026-11-20 3:29 ` [PATCH 1/3] mm: factor out common softleaf offset PTE batch scanning Wandun Chen 2026-11-20 3:29 ` [PATCH 2/3] mm/migrate: batch PTE restoration in remove_migration_pte() Wandun Chen @ 2026-11-20 3:29 ` Wandun Chen 2 siblings, 0 replies; 6+ messages in thread From: Wandun Chen @ 2026-11-20 3:29 UTC (permalink / raw) To: david, akpm, linux-mm, linux-kernel Cc: ljs, liam, vbabka, rppt, surenb, mhocko, ziy, matthew.brost, joshua.hahnjy, byungchul, gourry, ying.huang, apopple From: Wandun Chen <chenwandun@lixiang.com> Device-private PTEs encode PFN as softleaf offset. set_ptes() does not handle this encoding when constructing subsequent PTEs, such as on arm64 with a 16KB page granule, leaving device-private destinations folio unbatched. Add set_device_private_ptes() to construct subsequent entries with pte_next_swp_offset(). This enables remove_migration_pte() to batch PTE restoration for device-private folios. Signed-off-by: Wandun Chen <chenwandun@lixiang.com> --- mm/migrate.c | 22 +++++++++++++++++++--- 1 file changed, 19 insertions(+), 3 deletions(-) diff --git a/mm/migrate.c b/mm/migrate.c index e000ef77ea87..3d3d91fd61c7 100644 --- a/mm/migrate.c +++ b/mm/migrate.c @@ -345,6 +345,19 @@ struct rmap_walk_arg { bool map_unused_to_zeropage; }; +static void set_device_private_ptes(struct mm_struct *mm, unsigned long addr, + pte_t *ptep, pte_t pte, unsigned int nr) +{ + for (;;) { + set_pte_at(mm, addr, ptep, pte); + if (--nr == 0) + break; + addr += PAGE_SIZE; + ptep++; + pte = pte_next_swp_offset(pte); + } +} + /* * Restore a potential migration pte to a working pte entry */ @@ -442,8 +455,7 @@ static bool remove_migration_pte(struct folio *folio, } else #endif { - if (folio_test_large(rmap_walk_arg->folio) && - !is_device_private_page(new)) { + if (folio_test_large(rmap_walk_arg->folio)) { unsigned long end_addr = pmd_addr_end(pvmw.address, vma->vm_end); unsigned long max_nr = (end_addr - pvmw.address) >> PAGE_SHIFT; @@ -463,7 +475,11 @@ static bool remove_migration_pte(struct folio *folio, pvmw.address, rmap_flags); else folio_add_file_rmap_ptes(folio, new, nr, vma); - set_ptes(vma->vm_mm, pvmw.address, pvmw.pte, pte, nr); + if (unlikely(is_device_private_page(new))) + set_device_private_ptes(vma->vm_mm, pvmw.address, + pvmw.pte, pte, nr); + else + set_ptes(vma->vm_mm, pvmw.address, pvmw.pte, pte, nr); ptes += nr; } -- 2.43.0 ^ permalink raw reply related [flat|nested] 6+ messages in thread
end of thread, other threads:[~2026-11-20 15:35 UTC | newest] Thread overview: 6+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2026-11-20 3:29 [PATCH 0/3] mm/migrate: batch PTE restoration for large folios Wandun Chen 2026-11-20 3:29 ` [PATCH 1/3] mm: factor out common softleaf offset PTE batch scanning Wandun Chen 2026-11-20 15:34 ` Lorenzo Stoakes (ARM) 2026-11-20 3:29 ` [PATCH 2/3] mm/migrate: batch PTE restoration in remove_migration_pte() Wandun Chen 2026-11-20 15:22 ` Lorenzo Stoakes (ARM) 2026-11-20 3:29 ` [PATCH 3/3] mm/migrate: batch PTE restoration for device-private folios Wandun Chen
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox