[PATCH v2 0/7] mm: batch rmap walks during large folio migration

[Shivank Garg <shivankg@amd.com>]

Migrating a PTE-mapped large folio currently walks the folio's rmap
one page at a time. It does so twice per migration:
1. try_to_migrate_one(): Replace present PTEs with migration entries
2. remove_migration_pte(): Restore migration entries to working PTEs.

For a folio with N contiguous PTEs, each walk repeats page_vma_mapped_walk()
and the associated operation per subpage of the folio. (256 iterations
for 1M folio).

This series enables both walks to process a batch of consecutive PTEs
that map the same large folio and do some refactoring and clean up on
the way.

Performance:

AMD EPYC ZEN 3, 2-socket system (1 node per socket).
move_pages() migrating 1 GB from node 0 -> node 1, throughput
in GB/s:
                       batch-         batch-restore
folio     vanilla       restore-only   + unmap         restore%   restore+unmap%
4K        2.96±0.15     3.02±0.14      2.99±0.17         +2.2%    +1.0%
64K       5.99±0.10     6.12±0.08      7.65±0.16         +2.2%   +27.7%
256K      6.48±0.09     6.67±0.08      9.23±0.14         +2.9%   +42.4%
1M        6.76±0.10     6.98±0.10      9.74±0.19         +3.1%   +44.0%
2M       10.50±0.20    10.44±0.16     10.67±0.21         -0.5%    +1.6%

In plain migration, the folio copy itself dominates the cost, so these rmap walk
savings are only partly visible end-to-end.

When the folio copy is batched-copy and offloaded to DMA engine [1], the copy is
no longer the bottleneck and rmap-walk batching translates into much larger gains:

This series + [1], DMA Offload on PTDMA (DCBM), 16 channels:

		    batch-copy                      dcbm+restore
folio    vanilla    -offload(dcbm)  dcbm+restore    +unmap          (rmap batch impact)
4K       3.05±0.13   2.76±0.11       2.77±0.12        2.74±0.09
64K      5.85±0.11  10.66±0.27      11.56±1.45       17.55±0.56
256K     6.26±0.08  13.64±2.73      13.68±0.16       34.76±0.57       (2.54x)
1M       6.58±0.08  14.25±0.07      15.27±0.77       44.28±0.27       (3.11x)
2M      10.61±0.24  65.26±0.96      65.25±1.65       65.66±0.87

Applies cleanly on mm-new+[2].

[1] https://lore.kernel.org/linux-mm/20260630-shivank-batch-migrate-offload-v6-0-da95d7e8b8a2@amd.com
[2] https://lore.kernel.org/linux-mm/20260723070905.3422276-1-dev.jain@arm.com

Signed-off-by: Shivank Garg <shivankg@amd.com>
---
Changes in v2:

- Rebase onto mm-new, drop the borrowed folio_try_share_anon_rmap_ptes() patch
  and add it as dependency on Dev's series and adapt to recent migrate/rmap changes.
- Switch to recently added batched helpers - finish_folio_unmap, set_softleaf_ptes,
  and page_anon_exclusive_batch.
- Rename migration_softleaf_entry_to_pte() to migration_entry_to_pte(). (David)
- Split PTE construction from hugetlb handling into separate patch. (David)
  Mirror the split in mm/rmap.c patch as well.
- Rename make_migration_swp_pte() to make_migration_pte().
- Early exit the single-entry hugetlb page_vma_mapped_walk() paths - removes
  unnecessary nesting.
- Add the missing page_vma_mapped_walk_done() in remove_migration_pte_hugetlb(),
  which leaked the huge-PTE lock. (syzbot ci, Sashiko)
- Fix handling of hwpoisoned subpages in the batched unmap. Read PageHWPoison()
  once and skip batching when mapped page is poisoned, and use same result for
  hwpoison handling. (Sashiko)
- Remove the pte_unused(pteval) handling from try_to_migrate_hugetlb_one() (Sashiko).
- Link to v1: https://lore.kernel.org/r/20260712-migrate-rmap-batch-v1-0-872a734431d1@amd.com

---
Shivank Garg (7):
      mm: factor out generic PTE batch detection from swap_pte_batch()
      mm/migrate: factor out migration PTE construction
      mm/migrate: split remove_migration_pte_hugetlb() out of remove_migration_pte()
      mm/migrate: batch the restore-side migration rmap walk
      mm/rmap: factor out migration PTE construction
      mm/rmap: split try_to_migrate_hugetlb_one() out of try_to_migrate_one()
      mm/rmap: batch the unmap of large folios in try_to_migrate_one()

 mm/internal.h |  36 ++---
 mm/migrate.c  | 191 ++++++++++++++++++--------
 mm/rmap.c     | 419 ++++++++++++++++++++++++++++++++++++----------------------
 3 files changed, 416 insertions(+), 230 deletions(-)
---
base-commit: 480a31230b426efb005b6e71a14ef80f405f18b6
change-id: 20260701-migrate-rmap-batch-8ecc04de1905
prerequisite-message-id: 20260723070905.3422276-1-dev.jain@arm.com
prerequisite-patch-id: 5fdb03c3ea4a037f66fbd1c9b5d04e24e86406eb
prerequisite-patch-id: 033980575a1ef89d70e43a67c5530e2cf6cf97aa
prerequisite-patch-id: 6f7508334fc2bfc0df104ac50d36ed88072487e4
prerequisite-patch-id: 3f1aaae36e02d8b9343f7e719f732f10c1767572
prerequisite-patch-id: 98bc99c6bc274594946d022b3998f5f1dc70012f
prerequisite-patch-id: dd35ec00b82bf235f2e41e286d16eab6a3fd1e9e
prerequisite-patch-id: 552ee939dc6a7b3cce51373f4e75fa01a9d2b979
prerequisite-patch-id: 485cdbbe4e65b13494dc5104d4b8b7d0d23911a4

Best regards,
-- 
Shivank Garg <shivankg@amd.com>