[PATCH RFC v6 0/5] Accelerate page migration with batch copying and hardware offload

[PATCH RFC v6 0/5] Accelerate page migration with batch copying and hardware offload

[Shivank Garg]

This is the sixth RFC of the patchset to enhance page migration by
batching folio-copy operations and enabling acceleration via DMA offload.
I had intended to split the batch-copy path into a non-RFC series this
round. However, given some major changes, it does not hurt to take
one more pass with the RFC tag.

Single-threaded, folio-by-folio copying bottlenecks page migration in
modern systems with deep memory hierarchies, especially for large folios
where copy overhead dominates, leaving significant hardware potential
untapped.

By batching the copy phase, we create an opportunity for hardware
acceleration. This series builds the framework and provides a DMA
offload driver (dcbm) as a reference implementation, targeting bulk
migration workloads where offloading the copy improves throughput
and latency while freeing CPU cycles.

See the RFC V3 cover letter [3] for motivation.

Changelog since V5:
-------------------
1. Few cleanups and preparatory patches have been split out into a separate
   series [12].
2. Rename FOLIO_ALREADY_COPIED to FOLIO_CONTENT_COPIED. (David Hildenbrand)
3. Mark the copy-done state per folio (FOLIO_CONTENT_COPIED) right after
   each successful copy, instead of threading an already_copied bool
   through migrate_folios_move() -> migrate_folio_move() ->
   move_to_new_folio(). This looks clean and lets a partially failed batch
   fall back per folio rather than re-copying the whole
   batch. (David)
4. Moved folios_mc_copy() into mm/migrate.c as migrate_folios_mc_copy()
   to have per-folio copy marker (David). And add cond_resched() (Sashiko)
5. FOLIO_CONTENT_COPIED packs into the unused low bits of the anon_vma
   pointer in migrate_info, but 32-bit cannot guarantee bit-2 is free.
   The flag is BIT(2) on CONFIG_64BIT and 0 otherwise,
   and MIGRATION_COPY_OFFLOAD now depends on 64BIT. (Sashiko, Zi Yan)
6. Move copy-offload interface out of migrate.c into
   migrate_copy_offload.c. Restructure patches (migrate_offload_do_batch) (David)
7. Add MAINTAINERS entry for drivers/migrate_offload (David)
8. Avoid DMA-pinned folios on the batch path to avoid stale data race.
   (Sashiko). So, could not rename folio_can_batch_copy) to
   page-specific name, as it's no longer specific to movable_ops.
9. Add configurable migration reason callback. (David Rientjes)
10. Renamed the dispatch static_call to migrate_offload_batch_copy_fn and
    the hot-path predicate to migrate_should_offload().
11. Fix the DCBM crash when build with =y. Expose DCBM runtime
    knobs as module parameters under  /sys/module/dcbm/parameters instead
    of earlier /sys/kernel.
12. Rebased on v7.2-rc1.

DESIGN:
-------

New migration flow:

[ migrate_pages_batch() ]
    |
    |--> offload = migrate_should_offload(reason)  // core filters by migration reason
    |
    |--> for each folio:
    |      migrate_folio_unmap()	// unmap the folio
    |      +--> (success):
    |           if offload && folio_can_batch_copy():
    |               -> unmap_batch / dst_batch	// batch list for copy offloading
    |           else:
    |               -> unmap_single / dst_single // single list for per-folio CPU copy
    |
    |--> try_to_unmap_flush()                  // single batched TLB flush
    |
    |--> Batch copy (if unmap_batch not empty):
    |    - Migrator is configurable at runtime via module parameters.
    |      static_call(migrate_offload_batch_copy_fn) // Pluggable Migrators
    |              /            |             \
    |             v             v             v
    |     [ default ]  [ DMA (dcbm) ]   [ ... ]
    |      Each copied dst is marked FOLIO_CONTENT_COPIED; folios left
    |      unmarked (driver error / copy failure) fall back to per-folio
    |      CPU copy in the move phase.
    |
    +--> migrate_folios_move()                 // metadata, PTEs, finalize

Offload registration:

    A driver fills struct migrator { .name, .offload_copy, .owner } and
    calls migrate_offload_register(). This:
       - pins the module
       - patches the migrate_offload_batch_copy_fn static_call target
       - enables the migrate_offload_enabled static branch.

    migrate_offload_unregister() disables the branch, reverts the
    static_call, then synchronize_srcu() waits for in-flight migrations
    before module_put().

PERFORMANCE RESULTS:
--------------------

AMD EPYC 7713 (Zen 3), 2 sockets, 32 cores, SMT on, 
1 NUMA node per socket, 256 GB/node, v7.2-rc1, DVFS=Performance, PTDMA
(16 DMA channels).

Benchmark: move_pages() syscall to move pages between two NUMA nodes.

1). Moving different sized folios such that total transfer size is constant
(1GB), with different number of DMA channels. Throughput in GB/s.

a. Baseline (vanilla kernel, single-threaded, serial folio_copy):
================================================================================
4K          | 16K        | 64K        | 256K       | 1M          | 2M          |
================================================================================
3.28±0.14   | 4.98±0.18  | 6.19±0.08  | 6.77±0.08  | 7.02±0.11   | 10.80±0.13  |

b. DMA offload (Patched Kernel, dcbm driver, N DMA channels):
============================================================================================
N channel| 4K        | 16K         | 64K         | 256K        | 1M          | 2M          |
============================================================================================
1      | 2.38±0.17   | 2.77±0.03   | 3.21±0.03   | 5.00±0.02   | 5.09±0.64   | 12.62±0.07  |
2      | 2.87±0.11   | 4.06±0.05   | 5.09±0.04   | 6.97±0.08   | 8.43±0.06   | 14.32±0.10  |
4      | 3.32±0.07   | 5.30±0.06   | 7.21±0.09   | 9.69±0.15   | 11.36±0.13  | 26.98±0.19  |
8      | 3.68±0.09   | 6.28±0.10   | 9.16±0.13   | 12.05±0.16  | 15.33±2.80  | 46.06±0.55  |
12     | 3.83±0.05   | 6.65±0.17   | 10.00±0.16  | 12.98±0.18  | 15.87±0.19  | 61.31±1.28  |
16     | 3.94±0.09   | 6.78±0.10   | 10.48±0.13  | 13.48±0.20  | 16.90±0.24  | 65.06±2.46  |

2). First-folio latency: custom tracepoints (in migrate_pages_batch enter/exit,
    migrate_folio_done) measure latency per migrate_pages_batch() call.

    Throughput (GB/s) and first-folio latency (us), median of 10 runs.

a. Vanilla Kernel:

NR_MAX_BATCHED_MIGRATION upstream default value is 512.
--- Order 0 (4K folios) ---       --- Order 9 (2M folios) ---
     n      vanilla/cpu                n      vanilla/cpu
(folios)    GB/s | first(us)      (folios)    GB/s | first(us)
--------------------------        --------------------------
     1       0.03 |     24             1       6.86 |    204
     4       0.13 |     30             4       8.68 |    191
     8       0.27 |     27             8       7.92 |    207
    16       0.43 |     34            16       6.77 |    234
    64       1.12 |     51            64      10.44 |    179
   256       1.67 |    166           256      10.43 |    181
   512       1.98 |    255           512      10.55 |    179
  2048       2.38 |    233
  4096       2.42 |    168
 16384       2.72 |    167
 65536       3.00 |    156
262144       3.10 |    151

b. Patched kernel:
N = NR_MAX_BATCHED_MIGRATION (in pages), Total migrated data fixed at
1 GB. Change N with knob (just for testing) to measure impact of
different max batched size.

--- ORDER 0 (4K folios) ---

     N         offload/dma1          offload/dma4          offload/dma16
               GB/s | first(us)      GB/s | first(us)      GB/s | first(us)
------------------------------------------------------------------------
   512         2.21 |    628         3.29 |    275         3.25 |    245
  1024         2.06 |   1271         3.21 |    601         3.36 |    518
  2048         2.02 |   2646         3.00 |   1388         3.20 |   1110
  4096         2.08 |   4832         3.17 |   2514         3.41 |   2175
  8192         2.16 |   9253         3.14 |   4839         3.62 |   3592
 16384         2.24 |  17543         3.23 |   9680         3.58 |   7144
 32768         2.22 |  36408         3.26 |  19301         3.67 |  14524
 65536         2.12 |  82572         3.24 |  38091         3.62 |  29835
131072         2.08 | 153669         3.17 |  79744         3.48 |  62157
262144         2.05 | 332297         2.97 | 175315         3.33 | 134774

--- ORDER 9 (2M folios) ---

     N         offload/dma1          offload/dma4          offload/dma16
               GB/s | first(us)      GB/s | first(us)      GB/s | first(us)
------------------------------------------------------------------------
   512        11.74 |    160        11.71 |    160        11.75 |    159
  1024        12.18 |    310        13.82 |    274        13.76 |    275
  2048        12.39 |    612        25.55 |    290        25.69 |    289
  4096        12.54 |   1211        26.25 |    564        42.36 |    334
  8192        12.54 |   2421        26.82 |   1111        51.85 |    485
 16384        12.61 |   4824        26.91 |   2209        54.26 |    925
 32768        12.62 |   9652        27.04 |   4404        54.72 |   1942
 65536        12.64 |  19287        26.95 |   8835        57.30 |   3535
131072        12.64 |  38824        26.95 |  17900        58.58 |   7747
262144        12.66 |  77610        26.95 |  35743        66.31 |  13801

OPEN QUESTION:
--------------

The best batch size depends on the hardware, and bigger isn't always better.
NR_MAX_BATCHED_MIGRATION decides how many pages we move at once.
Higher batch size can help amortize the setup cost of migrator but
increases the first-folio latency (the folio is inaccessible for this
window).

Goals could be workload dependent, e.g. higher throughput versus
same throughput under a bounded latency.

Should this be tunable to accommodate different hardware and goals?

FOLLOW-UPS:
-----------
1. dmaengine_prep_dma_memcpy_sg() in DCBM (Vinod Koul); needs the ptdma/sdxi
   SG hook - device_prep_dma_memcpy_sg [10]. will post separately. This will
   address two concerns:
   - IOMMU SG merging in DCBM: dma_map_sgtable() may merge PFNs
   unevenly so src.nents != dst.nents. (Gregory)
   - Descriptor Chaining: out-of-order completion of DMA descriptors on some
   dma engine eg. Intel DSA can yield incorrect results. (Karim Manaouil)
2. SDXI as a second migrator [11]. SDXI is a generic memcpy engine
   without DMA_PRIVATE, so channel acquisition uses dma_find_channel()
   rather than dma_request_chan_by_mask(); I have a local DCBM variant
   working and will post once SDXI settles.
3. Revisit Multi-threaded CPU copy migrator once the infra is settled or
   follow it separately as it brings other discussion, like whom to
   charge? [13]. 
4. Batching the migration rmap walks: try_to_migrate_one() and
   remove_migration_pte() overheads dominate for PTE-mapped large
   folios. I have this working locally, and will post it separately.

EARLIER POSTINGS:
-----------------
[1] RFC V5: https://lore.kernel.org/all/20260428155043.39251-2-shivankg@amd.com
[2] RFC V4: https://lore.kernel.org/all/20260309120725.308854-3-shivankg@amd.com
[3] RFC V3: https://lore.kernel.org/all/20250923174752.35701-1-shivankg@amd.com
[4] RFC V2: https://lore.kernel.org/all/20250319192211.10092-1-shivankg@amd.com
[5] RFC V1: https://lore.kernel.org/all/20240614221525.19170-1-shivankg@amd.com
[6] RFC from Zi Yan: https://lore.kernel.org/all/20250103172419.4148674-1-ziy@nvidia.com

RELATED DISCUSSIONS:
--------------------
[7] MM-alignment Session [Nov 12, 2025]:
    https://lore.kernel.org/linux-mm/bd6a3c75-b9f0-cbcf-f7c4-1ef5dff06d24@google.com
[8] Linux Memory Hotness and Promotion call [Nov 6, 2025]:
    https://lore.kernel.org/linux-mm/8ff2fd10-c9ac-4912-cf56-7ecd4afd2770@google.com
[9] LSFMM 2025:
    https://lore.kernel.org/all/cf6fc05d-c0b0-4de3-985e-5403977aa3aa@amd.com
[10] DMA_MEMCPY_SG comparison:
     https://lore.kernel.org/linux-mm/3e73addb-ac01-4a05-bc75-c6c1c56072df@amd.com
[11] SDXI V3:
     https://lore.kernel.org/all/20260605-sdxi-base-v3-0-4d38ca2bdffe@amd.com
[12] migrate cleanups/prep:
     https://lore.kernel.org/all/20260626-migrate-cleanups-prep-v1-0-a95933af7619@amd.com
[13] Charge calling thread for multi-threaded copy:
     https://lore.kernel.org/all/633F4EFC-13A9-40DF-A27D-DBBDD0AF44F3@nvidia.com/
[14] OSS India:
     https://ossindia2025.sched.com/event/23Jk1
   
Thanks to everyone who reviewed, tested, or participated in discussions
around this series.

Signed-off-by: Shivank Garg <shivankg@amd.com>
---
Shivank Garg (4):
      mm/migrate: skip data copy for already-copied folios
      mm/migrate: add batch-copy path in migrate_pages_batch
      mm/migrate: add copy offload registration infrastructure
      drivers/migrate_offload: add DMA batch copy driver (dcbm)

Zi Yan (1):
      mm/migrate: adjust NR_MAX_BATCHED_MIGRATION for testing

 MAINTAINERS                           |   2 +
 drivers/Kconfig                       |   2 +
 drivers/Makefile                      |   2 +
 drivers/migrate_offload/Kconfig       |   9 +
 drivers/migrate_offload/Makefile      |   1 +
 drivers/migrate_offload/dcbm/Makefile |   1 +
 drivers/migrate_offload/dcbm/dcbm.c   | 481 ++++++++++++++++++++++++++++++++++
 include/linux/migrate.h               |  28 ++
 include/linux/migrate_copy_offload.h  |  69 +++++
 mm/Kconfig                            |   6 +
 mm/Makefile                           |   1 +
 mm/migrate.c                          | 133 +++++++---
 mm/migrate_copy_offload.c             | 249 ++++++++++++++++++
 13 files changed, 954 insertions(+), 30 deletions(-)
---
base-commit: 8b84e29dc92dbcada913c9bab976aa6f761b04e6
change-id: 20260616-shivank-batch-migrate-offload-deac05cbeaa5
prerequisite-change-id: 20260626-migrate-cleanups-prep-77a0ad340cc6:v1
prerequisite-patch-id: 816a6f6957cc9c0f3903db7b7f462a4ade2a7519
prerequisite-patch-id: d8521dbc801fb6e7cbaa5be8cecad491f7e2f809
prerequisite-patch-id: 3fcedadd87cd30b3fe8833e0ed33890f15c0e950

Best regards,
-- 
Shivank Garg <shivankg@amd.com>

[PATCH RFC v6 1/5] mm/migrate: skip data copy for already-copied folios

[Shivank Garg]

Add FOLIO_CONTENT_COPIED marker to the dst->migrate_info migration
state. When set, it tells __migrate_folio() to skip folio_mc_copy()
and perform metadata-only migration. This bit is not set yet, the
batch-copy path enables it later in a subsequent patch.

Update __migrate_folio_extract() to preserve FOLIO_CONTENT_COPIED
while it consumes old states of src folio.

Move the dst->migrate_info state enum to migrate.h header file so
offload drivers can see FOLIO_CONTENT_COPIED.

On 32Bit, we do not have spare bit to store FOLIO_CONTENT_COPIED
info, migration copy offload is disabled for them.

Signed-off-by: Shivank Garg <shivankg@amd.com>
---
 include/linux/migrate.h | 25 +++++++++++++++++++++++++
 mm/migrate.c            | 29 +++++++++++++----------------
 2 files changed, 38 insertions(+), 16 deletions(-)

diff --git a/include/linux/migrate.h b/include/linux/migrate.h
index d5af2b7f577b..876035df2fdc 100644
--- a/include/linux/migrate.h
+++ b/include/linux/migrate.h
@@ -72,6 +72,31 @@ int folio_migrate_mapping(struct address_space *mapping,
 		struct folio *newfolio, struct folio *folio, int extra_count);
 int set_movable_ops(const struct movable_operations *ops, enum pagetype type);
 
+/*
+ * To record some information during migration, we use the migrate_info
+ * field of struct folio of the newly allocated destination folio,
+ * together with the anon_vma pointer. The state is encoded in the
+ * unused low bits of the pointer.
+ * This is safe because nobody is using it except us.
+ *
+ * FOLIO_WAS_MAPPED and FOLIO_WAS_MLOCKED record the src folios's state
+ * at unmap time and are consumed during the move/undo phase.
+ * FOLIO_CONTENT_COPIED tells __migrate_folio() that the folio contents
+ * have already been copied, so the per-folio copy can be skipped. A
+ * driver must set this bit on each dst folio it copied.
+ */
+enum {
+	FOLIO_WAS_MAPPED	= BIT(0),
+	FOLIO_WAS_MLOCKED	= BIT(1),
+	FOLIO_OLD_STATES	= FOLIO_WAS_MAPPED | FOLIO_WAS_MLOCKED,
+#ifdef CONFIG_64BIT
+	FOLIO_CONTENT_COPIED	= BIT(2),
+#else
+	/* On 32bit we do not have a spare bit, migration-copy offload is disabled. */
+	FOLIO_CONTENT_COPIED	= 0,
+#endif
+};
+
 #else
 
 static inline void putback_movable_pages(struct list_head *l) {}
diff --git a/mm/migrate.c b/mm/migrate.c
index 7a83032a14b5..b3f632575c82 100644
--- a/mm/migrate.c
+++ b/mm/migrate.c
@@ -859,14 +859,21 @@ static int __migrate_folio(struct address_space *mapping, struct folio *dst,
 			   enum migrate_mode mode)
 {
 	int rc, expected_count = folio_expected_ref_count(src) + 1;
+	const bool already_copied = dst->migrate_info & FOLIO_CONTENT_COPIED;
+
+	/* Consume the content copied marker */
+	if (already_copied)
+		dst->migrate_info &= ~FOLIO_CONTENT_COPIED;
 
 	/* Check whether src does not have extra refs before we do more work */
 	if (folio_ref_count(src) != expected_count)
 		return -EAGAIN;
 
-	rc = folio_mc_copy(dst, src);
-	if (unlikely(rc))
-		return rc;
+	if (!already_copied) {
+		rc = folio_mc_copy(dst, src);
+		if (unlikely(rc))
+			return rc;
+	}
 
 	rc = __folio_migrate_mapping(mapping, dst, src, expected_count);
 	if (rc)
@@ -1129,17 +1136,6 @@ static int move_to_new_folio(struct folio *dst, struct folio *src,
 	return rc;
 }
 
-/*
- * To record some information during migration, we use the migrate_info
- * field of struct folio of the newly allocated destination folio.
- * This is safe because nobody is using it except us.
- */
-enum {
-	FOLIO_WAS_MAPPED = BIT(0),
-	FOLIO_WAS_MLOCKED = BIT(1),
-	FOLIO_OLD_STATES = FOLIO_WAS_MAPPED | FOLIO_WAS_MLOCKED,
-};
-
 static void __migrate_folio_record(struct folio *dst,
 		int old_folio_state, struct anon_vma *anon_vma)
 {
@@ -1151,9 +1147,10 @@ static void __migrate_folio_extract(struct folio *dst,
 {
 	unsigned long info = dst->migrate_info;
 
-	*anon_vmap = (struct anon_vma *)(info & ~FOLIO_OLD_STATES);
+	*anon_vmap = (struct anon_vma *)(info & ~(FOLIO_OLD_STATES |
+						  FOLIO_CONTENT_COPIED));
 	*old_folio_state = info & FOLIO_OLD_STATES;
-	dst->migrate_info = 0;
+	dst->migrate_info &= FOLIO_CONTENT_COPIED;
 }
 
 /* Restore the source folio to the original state upon failure */

-- 
2.43.0

[PATCH RFC v6 2/5] mm/migrate: add batch-copy path in migrate_pages_batch

[Shivank Garg]

Add migrate_folios_mc_copy() which walks src/dst folio lists in
lockstep, copies folio content via folio_mc_copy() and marks the dst
with FOLIO_CONTENT_COPIED so the move phase in __migrate_folio() can
skip the per-folio copy. The folios_cnt parameter is unused here,
present for the offload_copy callback signature compatibility used by
later patches in the series.

Split unmapped folios into batch-eligible (unmap_batch/dst_batch) and
standard (unmap_single/dst_single) lists. A folio is batch-eligible
when the migration reason is allowed by the driver, refcount equals
expected and it's not a movable_ops page.

After TLB flush, batch copy the eligible folios via
migrate_folios_mc_copy(), then run the move phase. The
FOLIO_CONTENT_COPIED marker is consumed on first read in the
__migrate_folio, so a -EAGAIN retry falls back to fresh per-folio copy.

Signed-off-by: Shivank Garg <shivankg@amd.com>
---
 include/linux/migrate.h |   3 ++
 mm/migrate.c            | 107 ++++++++++++++++++++++++++++++++++++++++++------
 2 files changed, 97 insertions(+), 13 deletions(-)

diff --git a/include/linux/migrate.h b/include/linux/migrate.h
index 876035df2fdc..e62c974b81d5 100644
--- a/include/linux/migrate.h
+++ b/include/linux/migrate.h
@@ -71,6 +71,9 @@ void folio_migrate_flags(struct folio *newfolio, struct folio *folio);
 int folio_migrate_mapping(struct address_space *mapping,
 		struct folio *newfolio, struct folio *folio, int extra_count);
 int set_movable_ops(const struct movable_operations *ops, enum pagetype type);
+int migrate_folios_mc_copy(struct list_head *dst_list,
+		struct list_head *src_list,
+		unsigned int __always_unused folios_cnt);
 
 /*
  * To record some information during migration, we use the migrate_info
diff --git a/mm/migrate.c b/mm/migrate.c
index b3f632575c82..41b732e78a67 100644
--- a/mm/migrate.c
+++ b/mm/migrate.c
@@ -51,6 +51,12 @@
 #include "internal.h"
 #include "swap.h"
 
+/* For now, never offload. Wired up in later patch. */
+static bool migrate_should_offload(int reason)
+{
+	return false;
+}
+
 static const struct movable_operations *offline_movable_ops;
 static const struct movable_operations *zsmalloc_movable_ops;
 
@@ -1707,6 +1713,60 @@ static int migrate_hugetlbs(struct list_head *from, new_folio_t get_new_folio,
 	return nr_failed;
 }
 
+static bool folio_can_batch_copy(struct folio *folio)
+{
+	/* movable_ops pages have a separate migration path */
+	if (unlikely(page_has_movable_ops(&folio->page)))
+		return false;
+
+	/*
+	 * Only batch when refcount matches the expected refcount.
+	 * Otherwise a GUP holder could modify src between the
+	 * batch copy and the move-phase refcount check, leaving the
+	 * dst folio with a stale copy.
+	 */
+	return folio_ref_count(folio) == folio_expected_ref_count(folio) + 1;
+}
+
+/**
+ * migrate_folios_mc_copy - Copy the contents of a list of folios.
+ * @dst_list: destination folio list.
+ * @src_list: source folio list.
+ * @folios_cnt: unused here, present for callback signature compatibility.
+ *
+ * Walk src and dst folio lists in lockstep, copy each folio via
+ * folio_mc_copy(), and mark the dst with FOLIO_CONTENT_COPIED so the
+ * move phase in __migrate_folio() skips the per-folio copy. The caller
+ * must ensure both lists have the same number of entries.
+ *
+ * On error, dst folios already copied keep their FOLIO_CONTENT_COPIED
+ * marker, while the remaining folios fallback to per-folio CPU copy in
+ * the move phase.
+ *
+ * This function may sleep.
+ *
+ * Return: 0 on success, negative errno on failure.
+ */
+int migrate_folios_mc_copy(struct list_head *dst_list,
+		struct list_head *src_list,
+		unsigned int __always_unused folios_cnt)
+{
+	struct folio *src, *dst;
+	int ret;
+
+	dst = list_first_entry(dst_list, struct folio, lru);
+	list_for_each_entry(src, src_list, lru) {
+		ret = folio_mc_copy(dst, src);
+		if (ret)
+			return ret;
+		dst->migrate_info |= FOLIO_CONTENT_COPIED;
+		dst = list_next_entry(dst, lru);
+		cond_resched();
+	}
+
+	return 0;
+}
+
 static void migrate_folios_move(struct list_head *src_folios,
 		struct list_head *dst_folios,
 		free_folio_t put_new_folio, unsigned long private,
@@ -1806,9 +1866,13 @@ static int migrate_pages_batch(struct list_head *from,
 	bool is_large = false;
 	struct folio *folio, *folio2, *dst = NULL;
 	int rc, rc_saved = 0, nr_pages;
-	LIST_HEAD(unmap_folios);
-	LIST_HEAD(dst_folios);
+	unsigned int nr_batch = 0;
+	LIST_HEAD(unmap_batch);
+	LIST_HEAD(dst_batch);
+	LIST_HEAD(unmap_single);
+	LIST_HEAD(dst_single);
 	bool nosplit = (reason == MR_NUMA_MISPLACED);
+	bool offload = migrate_should_offload(reason);
 
 	VM_WARN_ON_ONCE(mode != MIGRATE_ASYNC &&
 			!list_empty(from) && !list_is_singular(from));
@@ -1902,8 +1966,8 @@ static int migrate_pages_batch(struct list_head *from,
 					private, folio, &dst, mode, ret_folios);
 			/*
 			 * The rules are:
-			 *	0: folio will be put on unmap_folios list,
-			 *	   dst folio put on dst_folios list
+			 *	0: folio put on unmap_batch or unmap_single,
+			 *	   dst folio put on dst_batch or dst_single
 			 *	-EAGAIN: stay on the from list
 			 *	-ENOMEM: stay on the from list
 			 *	Other errno: put on ret_folios list
@@ -1944,7 +2008,7 @@ static int migrate_pages_batch(struct list_head *from,
 				/* nr_failed isn't updated for not used */
 				stats->nr_thp_failed += thp_retry;
 				rc_saved = rc;
-				if (list_empty(&unmap_folios))
+				if (list_empty(&unmap_batch) && list_empty(&unmap_single))
 					goto out;
 				else
 					goto move;
@@ -1954,8 +2018,14 @@ static int migrate_pages_batch(struct list_head *from,
 				nr_retry_pages += nr_pages;
 				break;
 			case 0:
-				list_move_tail(&folio->lru, &unmap_folios);
-				list_add_tail(&dst->lru, &dst_folios);
+				if (offload && folio_can_batch_copy(folio)) {
+					list_move_tail(&folio->lru, &unmap_batch);
+					list_add_tail(&dst->lru, &dst_batch);
+					nr_batch++;
+				} else {
+					list_move_tail(&folio->lru, &unmap_single);
+					list_add_tail(&dst->lru, &dst_single);
+				}
 				break;
 			default:
 				/*
@@ -1978,17 +2048,26 @@ static int migrate_pages_batch(struct list_head *from,
 	/* Flush TLBs for all unmapped folios */
 	try_to_unmap_flush();
 
+	/* Batch-copy eligible folios before the move phase */
+	if (!list_empty(&unmap_batch))
+		migrate_folios_mc_copy(&dst_batch, &unmap_batch, nr_batch);
+
 	retry = 1;
 	for (pass = 0; pass < nr_pass && retry; pass++) {
 		retry = 0;
 		thp_retry = 0;
 		nr_retry_pages = 0;
 
-		/* Move the unmapped folios */
-		migrate_folios_move(&unmap_folios, &dst_folios,
-				put_new_folio, private, mode, reason,
-				ret_folios, stats, &retry, &thp_retry,
-				&nr_failed, &nr_retry_pages);
+		if (!list_empty(&unmap_batch))
+			migrate_folios_move(&unmap_batch, &dst_batch, put_new_folio,
+					private, mode, reason, ret_folios, stats,
+					&retry, &thp_retry, &nr_failed,
+					&nr_retry_pages);
+		if (!list_empty(&unmap_single))
+			migrate_folios_move(&unmap_single, &dst_single, put_new_folio,
+					private, mode, reason, ret_folios, stats,
+					&retry, &thp_retry, &nr_failed,
+					&nr_retry_pages);
 	}
 	nr_failed += retry;
 	stats->nr_thp_failed += thp_retry;
@@ -1997,7 +2076,9 @@ static int migrate_pages_batch(struct list_head *from,
 	rc = rc_saved ? : nr_failed;
 out:
 	/* Cleanup remaining folios */
-	migrate_folios_undo(&unmap_folios, &dst_folios,
+	migrate_folios_undo(&unmap_batch, &dst_batch,
+			put_new_folio, private, ret_folios);
+	migrate_folios_undo(&unmap_single, &dst_single,
 			put_new_folio, private, ret_folios);
 
 	return rc;

-- 
2.43.0

[PATCH RFC v6 3/5] mm/migrate: add copy offload registration infrastructure

[Shivank Garg]

Add a registration interface that lets a single offload provider
(DMA, multi-threaded CPU copy, etc) take over the batch folio copy
performed by migrate_pages_batch().

The provider fills in a struct migrator with an offload_copy()
callback and calls migrate_offload_register(), passing the set of
migration reasons it wants to handle. Registration updates the
migrate_offload_batch_copy_fn() static_call and enables the
migrate_offload_enabled static branch. migrate_offload_unregister()
reverts both.

The active_reason_mask is the single source of truth for which migration
reasons are eligible for batched copy and offload. This is checked with
the migrate_should_offload(). The active reason mask can be changed at
runtime via migrate_offload_set_reason_mask(). Also, add
migrate_offload_reason_mask_parse() and
migrate_offload_reason_mask_format() so a provider can expose its reason
mask using migrate_reason_names[] (e.g. "compaction,demotion", "all",
"none").

Only one migrator can be active at a time. A second registration
gets -EBUSY, and only the active migrator can unregister itself.
The static_call dispatch is protected by SRCU so that the
synchronize_srcu() in unregister waits for all in-flight copy before
the module reference is dropped.

Callbacks must set FOLIO_CONTENT_COPIED on each successfully copied
dst folio so __migrate_folio() skips the per-folio copy; folios
without the marker fall back to CPU copy in the move phase.

Co-developed-by: Mike Day <michael.day@amd.com>
Signed-off-by: Mike Day <michael.day@amd.com>
Signed-off-by: Shivank Garg <shivankg@amd.com>
---
 MAINTAINERS                          |   2 +
 include/linux/migrate_copy_offload.h |  69 ++++++++++
 mm/Kconfig                           |   6 +
 mm/Makefile                          |   1 +
 mm/migrate.c                         |   9 +-
 mm/migrate_copy_offload.c            | 249 +++++++++++++++++++++++++++++++++++
 6 files changed, 329 insertions(+), 7 deletions(-)

diff --git a/MAINTAINERS b/MAINTAINERS
index 762355b62d54..fa32b447f3e5 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -17065,9 +17065,11 @@ F:	Documentation/ABI/testing/sysfs-kernel-mm-mempolicy-weighted-interleave
 F:	include/linux/mempolicy.h
 F:	include/uapi/linux/mempolicy.h
 F:	include/linux/migrate.h
+F:	include/linux/migrate_copy_offload.h
 F:	include/linux/migrate_mode.h
 F:	mm/mempolicy.c
 F:	mm/migrate.c
+F:	mm/migrate_copy_offload.c
 F:	mm/migrate_device.c
 
 MEMORY MANAGEMENT - MGLRU (MULTI-GEN LRU)
diff --git a/include/linux/migrate_copy_offload.h b/include/linux/migrate_copy_offload.h
new file mode 100644
index 000000000000..97941ebd4979
--- /dev/null
+++ b/include/linux/migrate_copy_offload.h
@@ -0,0 +1,69 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+#ifndef _LINUX_MIGRATE_COPY_OFFLOAD_H
+#define _LINUX_MIGRATE_COPY_OFFLOAD_H
+
+#include <linux/bits.h>
+#include <linux/errno.h>
+#include <linux/migrate_mode.h>
+
+struct list_head;
+struct module;
+
+#define MIGRATOR_NAME_LEN 32
+
+/*
+ * Reasons that may ever be offloaded. A driver's mask is clamped to this.
+ * Allow all reasons by default.
+ */
+#define MIGRATE_OFFLOAD_REASONS_ALLOWED		(BIT(MR_TYPES) - 1)
+
+/**
+ * struct migrator - batch-copy provider for page migration.
+ * @name: name of the provider.
+ * @offload_copy: copy @folio_cnt folios from @src_list to @dst_list.
+ *
+ *	The migrator may inspect @folio_cnt to decide whether the batch
+ *	is worth offloading, e.g. skip when the batch is too small to
+ *	amortize setup cost.
+ *	The callback must set FOLIO_CONTENT_COPIED in dst->migrate_info
+ *	for each successfully copied destination folio.
+ *	Folios without this marker are copied via per-folio CPU copy in
+ *	the move phase.
+ *
+ * @owner: module providing the migrator. NULL for built-in (=y) drivers.
+ */
+struct migrator {
+	char name[MIGRATOR_NAME_LEN];
+	int (*offload_copy)(struct list_head *dst_list, struct list_head *src_list,
+			unsigned int folio_cnt);
+	struct module *owner;
+};
+
+#ifdef CONFIG_MIGRATION_COPY_OFFLOAD
+int migrate_offload_register(const struct migrator *m, unsigned long reason_mask);
+int migrate_offload_unregister(const struct migrator *m);
+int migrate_offload_set_reason_mask(const struct migrator *m, unsigned long mask);
+int migrate_offload_reason_mask_parse(const char *buf, unsigned long *maskp);
+int migrate_offload_reason_mask_format(char *buf, unsigned long mask);
+bool migrate_should_offload(int reason);
+int migrate_offload_batch_copy(struct list_head *dst_batch,
+		struct list_head *src_batch, unsigned int nr_batch);
+#else
+static inline int migrate_offload_register(const struct migrator *m,
+		unsigned long reason_mask) { return -EOPNOTSUPP; }
+static inline int migrate_offload_unregister(const struct migrator *m) { return -EOPNOTSUPP; }
+static inline int migrate_offload_set_reason_mask(const struct migrator *m,
+		unsigned long mask) { return -EOPNOTSUPP; }
+static inline int migrate_offload_reason_mask_parse(const char *buf,
+		unsigned long *maskp) { return -EOPNOTSUPP; }
+static inline int migrate_offload_reason_mask_format(char *buf,
+		unsigned long mask) { return -EOPNOTSUPP; }
+static inline bool migrate_should_offload(int reason) { return false; }
+static inline int migrate_offload_batch_copy(struct list_head *dst_batch,
+		struct list_head *src_batch, unsigned int nr_batch)
+{
+	return -EOPNOTSUPP;
+}
+#endif
+
+#endif /* _LINUX_MIGRATE_COPY_OFFLOAD_H */
diff --git a/mm/Kconfig b/mm/Kconfig
index 9e0ca4824905..c65639bae268 100644
--- a/mm/Kconfig
+++ b/mm/Kconfig
@@ -700,6 +700,12 @@ config MIGRATION
 config DEVICE_MIGRATION
 	def_bool MIGRATION && ZONE_DEVICE
 
+# Page-migration batch-copy offload infrastructure.
+# Selected by migrator drivers (e.g. CONFIG_DCBM_DMA).
+config MIGRATION_COPY_OFFLOAD
+	bool
+	depends on MIGRATION && 64BIT
+
 config ARCH_ENABLE_HUGEPAGE_MIGRATION
 	bool
 
diff --git a/mm/Makefile b/mm/Makefile
index eff9f9e7e061..256fb532b672 100644
--- a/mm/Makefile
+++ b/mm/Makefile
@@ -96,6 +96,7 @@ obj-$(CONFIG_FAILSLAB) += failslab.o
 obj-$(CONFIG_FAIL_PAGE_ALLOC) += fail_page_alloc.o
 obj-$(CONFIG_MEMTEST)		+= memtest.o
 obj-$(CONFIG_MIGRATION) += migrate.o
+obj-$(CONFIG_MIGRATION_COPY_OFFLOAD) += migrate_copy_offload.o
 obj-$(CONFIG_NUMA) += memory-tiers.o
 obj-$(CONFIG_DEVICE_MIGRATION) += migrate_device.o
 obj-$(CONFIG_TRANSPARENT_HUGEPAGE) += huge_memory.o khugepaged.o
diff --git a/mm/migrate.c b/mm/migrate.c
index 41b732e78a67..4fed3110ca0a 100644
--- a/mm/migrate.c
+++ b/mm/migrate.c
@@ -43,6 +43,7 @@
 #include <linux/sched/sysctl.h>
 #include <linux/memory-tiers.h>
 #include <linux/pagewalk.h>
+#include <linux/migrate_copy_offload.h>
 
 #include <asm/tlbflush.h>
 
@@ -51,12 +52,6 @@
 #include "internal.h"
 #include "swap.h"
 
-/* For now, never offload. Wired up in later patch. */
-static bool migrate_should_offload(int reason)
-{
-	return false;
-}
-
 static const struct movable_operations *offline_movable_ops;
 static const struct movable_operations *zsmalloc_movable_ops;
 
@@ -2050,7 +2045,7 @@ static int migrate_pages_batch(struct list_head *from,
 
 	/* Batch-copy eligible folios before the move phase */
 	if (!list_empty(&unmap_batch))
-		migrate_folios_mc_copy(&dst_batch, &unmap_batch, nr_batch);
+		migrate_offload_batch_copy(&dst_batch, &unmap_batch, nr_batch);
 
 	retry = 1;
 	for (pass = 0; pass < nr_pass && retry; pass++) {
diff --git a/mm/migrate_copy_offload.c b/mm/migrate_copy_offload.c
new file mode 100644
index 000000000000..335db8e80556
--- /dev/null
+++ b/mm/migrate_copy_offload.c
@@ -0,0 +1,249 @@
+// SPDX-License-Identifier: GPL-2.0
+#include <linux/bitops.h>
+#include <linux/jump_label.h>
+#include <linux/kstrtox.h>
+#include <linux/migrate.h>
+#include <linux/migrate_copy_offload.h>
+#include <linux/mm.h>
+#include <linux/module.h>
+#include <linux/slab.h>
+#include <linux/srcu.h>
+#include <linux/static_call.h>
+#include <linux/string.h>
+#include <linux/sysfs.h>
+
+DEFINE_STATIC_KEY_FALSE(migrate_offload_enabled);
+DEFINE_SRCU(migrate_offload_srcu);
+DEFINE_STATIC_CALL(migrate_offload_batch_copy_fn, migrate_folios_mc_copy);
+
+static DEFINE_MUTEX(migrator_mutex);
+static const struct migrator *active_migrator;
+
+/*
+ * The active migrator's reason mask. This is the single source of truth
+ * for which reasons are offloaded.
+ */
+static unsigned long active_reason_mask;
+
+bool migrate_should_offload(int reason)
+{
+	if (!static_branch_unlikely(&migrate_offload_enabled))
+		return false;
+
+	return READ_ONCE(active_reason_mask) & BIT(reason);
+}
+
+/**
+ * migrate_offload_reason_mask_parse - parse migration reason mask.
+ * @buf: input string, either a number (hex/decimal, e.g. "0x101") or a
+ *	comma-separated list of reason names (see migrate_reason_names[]),
+ *	e.g. "compaction,demotion". The tokens "all" and "none" select
+ *	every or no reason.
+ * @maskp: parsed mask, clamped to MIGRATE_OFFLOAD_REASONS_ALLOWED.
+ *
+ * Return: 0 on success, -EINVAL on an unknown name, -ENOMEM on OOM.
+ */
+int migrate_offload_reason_mask_parse(const char *buf, unsigned long *maskp)
+{
+	unsigned long mask;
+	char *copy, *p, *tok;
+	int ret = 0;
+
+	/* A plain number is treated as a raw mask. */
+	if (!kstrtoul(buf, 0, &mask))
+		goto done;
+
+	copy = kstrdup(buf, GFP_KERNEL);
+	if (!copy)
+		return -ENOMEM;
+	mask = 0;
+	p = strim(copy);
+
+	while ((tok = strsep(&p, ",")) != NULL) {
+		int i;
+
+		tok = strim(tok);
+		if (!*tok)
+			continue;
+		if (!strcmp(tok, "none")) {
+			mask = 0;
+			continue;
+		}
+		if (!strcmp(tok, "all")) {
+			mask = MIGRATE_OFFLOAD_REASONS_ALLOWED;
+			continue;
+		}
+		i = match_string(migrate_reason_names, MR_TYPES, tok);
+		if (i < 0) {
+			ret = -EINVAL;
+			break;
+		}
+		mask |= BIT(i);
+	}
+	kfree(copy);
+	if (ret)
+		return ret;
+done:
+	*maskp = mask & MIGRATE_OFFLOAD_REASONS_ALLOWED;
+	return 0;
+}
+EXPORT_SYMBOL_GPL(migrate_offload_reason_mask_parse);
+
+/**
+ * migrate_offload_reason_mask_format - render a reason mask as names.
+ * @buf: output buffer from kernel_param_ops get.
+ * @mask: reason mask to render.
+ *
+ * Emits a comma-separated list of reason names, or "none".
+ *
+ * Return: number of bytes written to @buf.
+ */
+int migrate_offload_reason_mask_format(char *buf, unsigned long mask)
+{
+	int len = 0, i;
+
+	for_each_set_bit(i, &mask, MR_TYPES)
+		len += sysfs_emit_at(buf, len, "%s%s",
+				     len ? "," : "", migrate_reason_names[i]);
+	if (!len)
+		return sysfs_emit(buf, "none\n");
+
+	len += sysfs_emit_at(buf, len, "\n");
+	return len;
+}
+EXPORT_SYMBOL_GPL(migrate_offload_reason_mask_format);
+
+/*
+ * Hand the batch to the registered migrator. The migrator may decline
+ * (typically based on batch size), in which case the move phase falls
+ * back to per-folio CPU copy.
+ */
+int migrate_offload_batch_copy(struct list_head *dst_batch,
+		struct list_head *src_batch, unsigned int nr_batch)
+{
+	int idx, rc;
+
+	idx = srcu_read_lock(&migrate_offload_srcu);
+	rc = static_call(migrate_offload_batch_copy_fn)(dst_batch, src_batch, nr_batch);
+	srcu_read_unlock(&migrate_offload_srcu, idx);
+	return rc;
+}
+
+/**
+ * migrate_offload_set_reason_mask - update the active migrator's reason mask.
+ * @m: migrator (must be the currently active one).
+ * @mask: new reason mask.
+ *
+ * Return: 0 on success, -EINVAL if @m is NULL or not the active migrator.
+ */
+int migrate_offload_set_reason_mask(const struct migrator *m, unsigned long mask)
+{
+	if (!m)
+		return -EINVAL;
+
+	mask &= MIGRATE_OFFLOAD_REASONS_ALLOWED;
+
+	mutex_lock(&migrator_mutex);
+	if (active_migrator != m) {
+		mutex_unlock(&migrator_mutex);
+		return -EINVAL;
+	}
+	WRITE_ONCE(active_reason_mask, mask);
+	mutex_unlock(&migrator_mutex);
+	return 0;
+}
+EXPORT_SYMBOL_GPL(migrate_offload_set_reason_mask);
+
+/**
+ * migrate_offload_register - register a batch-copy provider for page migration.
+ * @m: migrator to install.
+ * @reason_mask: initial set of BIT(MR_*) reasons to offload, can be changed
+ *	later via migrate_offload_set_reason_mask().
+ *
+ * Only one provider can be active at a time, returns -EBUSY if another migrator
+ * is already registered.
+ *
+ * Return: 0 on success, negative errno on failure.
+ */
+int migrate_offload_register(const struct migrator *m, unsigned long reason_mask)
+{
+	unsigned long mask;
+	int ret = 0;
+
+	if (!m || !m->offload_copy)
+		return -EINVAL;
+
+	mask = reason_mask & MIGRATE_OFFLOAD_REASONS_ALLOWED;
+
+	mutex_lock(&migrator_mutex);
+	if (active_migrator) {
+		ret = -EBUSY;
+		goto unlock;
+	}
+
+	/* @owner is NULL for built-in (=y) drivers; nothing to ref. */
+	if (m->owner && !try_module_get(m->owner)) {
+		ret = -ENODEV;
+		goto unlock;
+	}
+
+	WRITE_ONCE(active_reason_mask, mask);
+	static_call_update(migrate_offload_batch_copy_fn, m->offload_copy);
+	active_migrator = m;
+	static_branch_enable(&migrate_offload_enabled);
+
+unlock:
+	mutex_unlock(&migrator_mutex);
+
+	if (ret)
+		pr_err("migrate_offload: %s: failed to register (%d)\n", m->name, ret);
+	else
+		pr_info("migrate_offload: enabled by %s (reason_mask=0x%lx)\n",
+			m->name, mask);
+	return ret;
+}
+EXPORT_SYMBOL_GPL(migrate_offload_register);
+
+/**
+ * migrate_offload_unregister - unregister the active batch-copy provider.
+ * @m: migrator to remove (must be the currently active one).
+ *
+ * Reverts static_call targets and waits for SRCU grace period so that
+ * no in-flight migration is still calling the driver functions before
+ * releasing the module.
+ *
+ * Return: 0 on success, negative errno on failure.
+ */
+int migrate_offload_unregister(const struct migrator *m)
+{
+	struct module *owner;
+
+	if (!m)
+		return -EINVAL;
+
+	mutex_lock(&migrator_mutex);
+	if (active_migrator != m) {
+		mutex_unlock(&migrator_mutex);
+		return -EINVAL;
+	}
+
+	/*
+	 * Disable the static branch first so new migrate_pages_batch() calls
+	 * cannot enter the batch path.
+	 */
+	static_branch_disable(&migrate_offload_enabled);
+	WRITE_ONCE(active_reason_mask, 0);
+	static_call_update(migrate_offload_batch_copy_fn, migrate_folios_mc_copy);
+	owner = active_migrator->owner;
+	active_migrator = NULL;
+	mutex_unlock(&migrator_mutex);
+
+	/* Wait for all in-flight callers to finish before module_put(). */
+	synchronize_srcu(&migrate_offload_srcu);
+	if (owner)
+		module_put(owner);
+
+	pr_info("migrate_offload: disabled by %s\n", m->name);
+	return 0;
+}
+EXPORT_SYMBOL_GPL(migrate_offload_unregister);

-- 
2.43.0

[PATCH RFC v6 4/5] drivers/migrate_offload: add DMA batch copy driver (dcbm)

[Shivank Garg]

Add a simple DMAEngine-based migrator that plugs into the page
migration copy-offload infrastructure and batch-copies folios via
DMA memcpy channels. It is intended for testing the offload plumbing
and as a template for future migrators (SDXI, multi-threaded CPU copy,
etc.).

On success, the dst folios are marked with FOLIO_CONTENT_COPIED so
__migrate_folio skips the per-folio copy. On any DMA error,
FOLIO_CONTENT_COPIED market is not set and the migration falls back
to per-folio CPU copy.

dcbm registers with reason mask with reason mask
MIGRATE_OFFLOAD_REASONS_ALLOWED, this set can be narrowed at runtime
via the reason_mask parameter.

Runtime controls are under /sys/module/dcbm/parameters/:

  offloading      - enable/disable DMA offload (0/1)
  nr_dma_chan     - max DMA channels to use (1..N)
  reason_mask     - reasons to offload: names (e.g. "compaction,demotion"),
		    "all"/"none" or raw numerical mask.
  folios_migrated - folios DMA-copied (write to reset)
  folios_failures - fallback count (write to reset)

CONFIG_DCBM_DMA selects MIGRATION_COPY_OFFLOAD so enabling the
driver pulls in the infrastructure automatically.

Channel acquisition uses dma_request_chan_by_mask(DMA_MEMCPY), which
works for providers that set DMA_PRIVATE (e.g. AMD PTDMA). Generic
mem-to-mem engines that do not set DMA_PRIVATE (e.g. SDXI) should
acquire channels via dma_find_channel(DMA_MEMCPY) or the async_tx
APIs, which can be added in a follow-up.

[Correctness Note:
 Karim: Descriptor can complete out of order for some dmaengines.
 so descriptor chaining may yield incorrect results. For correctness
 we need to add callback to every descriptor
 https://lore.kernel.org/all/20260619160725.lfcxrbj5go67qy6u@wrangler
 Shivank: device_prep_dma_memcpy_sg() would solve this.
]

Reviewed-by: Karim Manaouil <kmanaouil.dev@gmail.com>
Signed-off-by: Shivank Garg <shivankg@amd.com>
---
 drivers/Kconfig                       |   2 +
 drivers/Makefile                      |   2 +
 drivers/migrate_offload/Kconfig       |   9 +
 drivers/migrate_offload/Makefile      |   1 +
 drivers/migrate_offload/dcbm/Makefile |   1 +
 drivers/migrate_offload/dcbm/dcbm.c   | 481 ++++++++++++++++++++++++++++++++++
 6 files changed, 496 insertions(+)

diff --git a/drivers/Kconfig b/drivers/Kconfig
index f2bed2ddeb66..3e83a1475cbc 100644
--- a/drivers/Kconfig
+++ b/drivers/Kconfig
@@ -253,4 +253,6 @@ source "drivers/cdx/Kconfig"
 
 source "drivers/resctrl/Kconfig"
 
+source "drivers/migrate_offload/Kconfig"
+
 endmenu
diff --git a/drivers/Makefile b/drivers/Makefile
index 0841ea851847..88cb8e3e88df 100644
--- a/drivers/Makefile
+++ b/drivers/Makefile
@@ -42,6 +42,8 @@ obj-y				+= clk/
 # really early.
 obj-$(CONFIG_DMADEVICES)	+= dma/
 
+obj-$(CONFIG_MIGRATION_COPY_OFFLOAD)	+= migrate_offload/
+
 # SOC specific infrastructure drivers.
 obj-y				+= soc/
 obj-$(CONFIG_PM_GENERIC_DOMAINS)	+= pmdomain/
diff --git a/drivers/migrate_offload/Kconfig b/drivers/migrate_offload/Kconfig
new file mode 100644
index 000000000000..c9f2e21a95cf
--- /dev/null
+++ b/drivers/migrate_offload/Kconfig
@@ -0,0 +1,9 @@
+config DCBM_DMA
+	tristate "DMA Core Batch Migrator"
+	depends on MIGRATION && DMA_ENGINE && 64BIT
+	select MIGRATION_COPY_OFFLOAD
+	help
+	  DMA-based batch copy engine for page migration. Uses
+	  DMAEngine memcpy channels to offload folio data copies
+	  during migration. Primarily intended for testing the copy
+	  offload infrastructure.
diff --git a/drivers/migrate_offload/Makefile b/drivers/migrate_offload/Makefile
new file mode 100644
index 000000000000..9e16018beb15
--- /dev/null
+++ b/drivers/migrate_offload/Makefile
@@ -0,0 +1 @@
+obj-$(CONFIG_DCBM_DMA)		+= dcbm/
diff --git a/drivers/migrate_offload/dcbm/Makefile b/drivers/migrate_offload/dcbm/Makefile
new file mode 100644
index 000000000000..56ba47cce0f1
--- /dev/null
+++ b/drivers/migrate_offload/dcbm/Makefile
@@ -0,0 +1 @@
+obj-$(CONFIG_DCBM_DMA) += dcbm.o
diff --git a/drivers/migrate_offload/dcbm/dcbm.c b/drivers/migrate_offload/dcbm/dcbm.c
new file mode 100644
index 000000000000..bacbcc689171
--- /dev/null
+++ b/drivers/migrate_offload/dcbm/dcbm.c
@@ -0,0 +1,481 @@
+// SPDX-License-Identifier: GPL-2.0-only
+/*
+ * DMA Core Batch Migrator (DCBM)
+ *
+ * Uses DMAEngine memcpy channels to offload batch folio copies during
+ * page migration. Reference driver meant for testing the offload
+ * infrastructure.
+ *
+ * Copyright (C) 2024-26 Advanced Micro Devices, Inc.
+ */
+
+#include <linux/module.h>
+#include <linux/dma-mapping.h>
+#include <linux/dmaengine.h>
+#include <linux/migrate.h>
+#include <linux/migrate_copy_offload.h>
+
+#define MAX_DMA_CHANNELS	16
+
+static atomic_long_t folios_migrated;
+static atomic_long_t folios_failures;
+
+static bool offloading_enabled;
+static unsigned int nr_dma_channels = 1;
+static DEFINE_MUTEX(dcbm_mutex);
+
+struct dma_work {
+	struct dma_chan *chan;
+	struct completion done;
+	atomic_t pending;
+	struct sg_table *src_sgt;
+	struct sg_table *dst_sgt;
+	bool mapped;
+};
+
+static void dma_completion_callback(void *data)
+{
+	struct dma_work *work = data;
+
+	if (atomic_dec_and_test(&work->pending))
+		complete(&work->done);
+}
+
+static int setup_sg_tables(struct dma_work *work, struct list_head **src_pos,
+			   struct list_head **dst_pos, int nr)
+{
+	struct scatterlist *sg_src, *sg_dst;
+	struct device *dev;
+	int i, ret;
+
+	work->src_sgt = kmalloc_obj(*work->src_sgt, GFP_KERNEL);
+	if (!work->src_sgt)
+		return -ENOMEM;
+	work->dst_sgt = kmalloc_obj(*work->dst_sgt, GFP_KERNEL);
+	if (!work->dst_sgt) {
+		ret = -ENOMEM;
+		goto err_free_src;
+	}
+
+	ret = sg_alloc_table(work->src_sgt, nr, GFP_KERNEL);
+	if (ret)
+		goto err_free_dst;
+	ret = sg_alloc_table(work->dst_sgt, nr, GFP_KERNEL);
+	if (ret)
+		goto err_free_src_table;
+
+	sg_src = work->src_sgt->sgl;
+	sg_dst = work->dst_sgt->sgl;
+	for (i = 0; i < nr; i++) {
+		struct folio *src = list_entry(*src_pos, struct folio, lru);
+		struct folio *dst = list_entry(*dst_pos, struct folio, lru);
+
+		sg_set_folio(sg_src, src, folio_size(src), 0);
+		sg_set_folio(sg_dst, dst, folio_size(dst), 0);
+
+		*src_pos = (*src_pos)->next;
+		*dst_pos = (*dst_pos)->next;
+
+		if (i < nr - 1) {
+			sg_src = sg_next(sg_src);
+			sg_dst = sg_next(sg_dst);
+		}
+	}
+
+	dev = dmaengine_get_dma_device(work->chan);
+	if (!dev) {
+		ret = -ENODEV;
+		goto err_free_dst_table;
+	}
+	ret = dma_map_sgtable(dev, work->src_sgt, DMA_TO_DEVICE,
+			      DMA_ATTR_SKIP_CPU_SYNC | DMA_ATTR_NO_KERNEL_MAPPING);
+	if (ret)
+		goto err_free_dst_table;
+	ret = dma_map_sgtable(dev, work->dst_sgt, DMA_FROM_DEVICE,
+			      DMA_ATTR_SKIP_CPU_SYNC | DMA_ATTR_NO_KERNEL_MAPPING);
+	if (ret)
+		goto err_unmap_src;
+
+	/*
+	 * TODO: IOMMU may merge segments unevenly on the two sides, fall back
+	 * bail to CPU copy. In practice, I have not observed merging in tests.
+	 * Handling unequal nents is left for follow-up.
+	 */
+	if (work->src_sgt->nents != work->dst_sgt->nents) {
+		ret = -EINVAL;
+		goto err_unmap_dst;
+	}
+	work->mapped = true;
+	return 0;
+
+err_unmap_dst:
+	dma_unmap_sgtable(dev, work->dst_sgt, DMA_FROM_DEVICE,
+			  DMA_ATTR_SKIP_CPU_SYNC | DMA_ATTR_NO_KERNEL_MAPPING);
+err_unmap_src:
+	dma_unmap_sgtable(dev, work->src_sgt, DMA_TO_DEVICE,
+			  DMA_ATTR_SKIP_CPU_SYNC | DMA_ATTR_NO_KERNEL_MAPPING);
+err_free_dst_table:
+	sg_free_table(work->dst_sgt);
+err_free_src_table:
+	sg_free_table(work->src_sgt);
+err_free_dst:
+	kfree(work->dst_sgt);
+	work->dst_sgt = NULL;
+err_free_src:
+	kfree(work->src_sgt);
+	work->src_sgt = NULL;
+	return ret;
+}
+
+static void cleanup_dma_work(struct dma_work *works, int actual_channels)
+{
+	struct device *dev;
+	int i;
+
+	if (!works)
+		return;
+
+	for (i = 0; i < actual_channels; i++) {
+		if (!works[i].chan)
+			continue;
+
+		dev = dmaengine_get_dma_device(works[i].chan);
+
+		if (works[i].mapped)
+			dmaengine_terminate_sync(works[i].chan);
+
+		if (dev && works[i].mapped) {
+			if (works[i].src_sgt) {
+				dma_unmap_sgtable(dev, works[i].src_sgt,
+						  DMA_TO_DEVICE,
+						  DMA_ATTR_SKIP_CPU_SYNC |
+						  DMA_ATTR_NO_KERNEL_MAPPING);
+				sg_free_table(works[i].src_sgt);
+				kfree(works[i].src_sgt);
+			}
+			if (works[i].dst_sgt) {
+				dma_unmap_sgtable(dev, works[i].dst_sgt,
+						  DMA_FROM_DEVICE,
+						  DMA_ATTR_SKIP_CPU_SYNC |
+						  DMA_ATTR_NO_KERNEL_MAPPING);
+				sg_free_table(works[i].dst_sgt);
+				kfree(works[i].dst_sgt);
+			}
+		}
+		dma_release_channel(works[i].chan);
+	}
+	kfree(works);
+}
+
+static int submit_dma_transfers(struct dma_work *work)
+{
+	struct scatterlist *sg_src, *sg_dst;
+	struct dma_async_tx_descriptor *tx;
+	unsigned long flags = DMA_CTRL_ACK;
+	dma_cookie_t cookie;
+	int i;
+
+	atomic_set(&work->pending, 1);
+
+	sg_src = work->src_sgt->sgl;
+	sg_dst = work->dst_sgt->sgl;
+	for_each_sgtable_dma_sg(work->src_sgt, sg_src, i) {
+		if (i == work->src_sgt->nents - 1)
+			flags |= DMA_PREP_INTERRUPT;
+
+		tx = dmaengine_prep_dma_memcpy(work->chan,
+					       sg_dma_address(sg_dst),
+					       sg_dma_address(sg_src),
+					       sg_dma_len(sg_src), flags);
+		if (!tx) {
+			atomic_set(&work->pending, 0);
+			return -EIO;
+		}
+
+		if (i == work->src_sgt->nents - 1) {
+			tx->callback = dma_completion_callback;
+			tx->callback_param = work;
+		}
+
+		cookie = dmaengine_submit(tx);
+		if (dma_submit_error(cookie)) {
+			atomic_set(&work->pending, 0);
+			return -EIO;
+		}
+		sg_dst = sg_next(sg_dst);
+	}
+	return 0;
+}
+
+/**
+ * folios_copy_dma - copy a batch of folios via DMA memcpy
+ * @dst_list: destination folio list
+ * @src_list: source folio list
+ * @nr_folios: number of folios in each list
+ *
+ * Return: 0 on success, negative errno on failure.
+ */
+static int folios_copy_dma(struct list_head *dst_list,
+			   struct list_head *src_list, unsigned int nr_folios)
+{
+	struct folio *dst;
+	struct dma_work *works;
+	struct list_head *src_pos = src_list->next;
+	struct list_head *dst_pos = dst_list->next;
+	int i, folios_per_chan, ret;
+	dma_cap_mask_t mask;
+	int actual_channels = 0;
+	unsigned int max_channels;
+
+	max_channels = min3(READ_ONCE(nr_dma_channels), nr_folios,
+			    (unsigned int)MAX_DMA_CHANNELS);
+
+	works = kcalloc(max_channels, sizeof(*works), GFP_KERNEL);
+	if (!works)
+		return -ENOMEM;
+
+	dma_cap_zero(mask);
+	dma_cap_set(DMA_MEMCPY, mask);
+
+	for (i = 0; i < max_channels; i++) {
+		works[actual_channels].chan = dma_request_chan_by_mask(&mask);
+		if (IS_ERR(works[actual_channels].chan))
+			break;
+		init_completion(&works[actual_channels].done);
+		actual_channels++;
+	}
+
+	if (actual_channels == 0) {
+		kfree(works);
+		return -ENODEV;
+	}
+
+	for (i = 0; i < actual_channels; i++) {
+		folios_per_chan = nr_folios * (i + 1) / actual_channels -
+				(nr_folios * i) / actual_channels;
+		if (folios_per_chan == 0)
+			continue;
+
+		ret = setup_sg_tables(&works[i], &src_pos, &dst_pos,
+				      folios_per_chan);
+		if (ret)
+			goto err_cleanup;
+	}
+
+	for (i = 0; i < actual_channels; i++) {
+		if (!works[i].mapped)
+			continue;
+		ret = submit_dma_transfers(&works[i]);
+		if (ret)
+			goto err_cleanup;
+	}
+
+	for (i = 0; i < actual_channels; i++) {
+		if (atomic_read(&works[i].pending) > 0)
+			dma_async_issue_pending(works[i].chan);
+	}
+
+	for (i = 0; i < actual_channels; i++) {
+		if (atomic_read(&works[i].pending) == 0)
+			continue;
+		if (!wait_for_completion_timeout(&works[i].done,
+						 msecs_to_jiffies(10000))) {
+			ret = -ETIMEDOUT;
+			goto err_cleanup;
+		}
+	}
+
+	/*
+	 * All folios copied; mark each dst with FOLIO_CONTENT_COPIED so
+	 * __migrate_folio() skips the per-folio copy in the move phase.
+	 */
+	list_for_each_entry(dst, dst_list, lru)
+		dst->migrate_info |= FOLIO_CONTENT_COPIED;
+
+	cleanup_dma_work(works, actual_channels);
+
+	atomic_long_add(nr_folios, &folios_migrated);
+	return 0;
+
+err_cleanup:
+	pr_warn_ratelimited("dcbm: DMA copy failed (%d), falling back to CPU\n",
+			    ret);
+	cleanup_dma_work(works, actual_channels);
+
+	atomic_long_add(nr_folios, &folios_failures);
+	return ret;
+}
+
+static const struct migrator dma_migrator = {
+	.name = "DCBM",
+	.offload_copy = folios_copy_dma,
+	.owner = THIS_MODULE,
+};
+
+static unsigned long dcbm_reason_mask = MIGRATE_OFFLOAD_REASONS_ALLOWED;
+
+/* offloading: enable/disable DMA migration offload */
+static int offloading_param_set(const char *val, const struct kernel_param *kp)
+{
+	bool enable;
+	int ret;
+
+	ret = kstrtobool(val, &enable);
+	if (ret)
+		return ret;
+
+	mutex_lock(&dcbm_mutex);
+	if (enable == offloading_enabled) {
+		mutex_unlock(&dcbm_mutex);
+		return 0;
+	}
+	if (enable) {
+		ret = migrate_offload_register(&dma_migrator,
+					       READ_ONCE(dcbm_reason_mask));
+		if (ret) {
+			mutex_unlock(&dcbm_mutex);
+			return ret;
+		}
+		WRITE_ONCE(offloading_enabled, true);
+	} else {
+		migrate_offload_unregister(&dma_migrator);
+		WRITE_ONCE(offloading_enabled, false);
+	}
+	mutex_unlock(&dcbm_mutex);
+	return 0;
+}
+
+static int offloading_param_get(char *buffer, const struct kernel_param *kp)
+{
+	return sysfs_emit(buffer, "%d\n", READ_ONCE(offloading_enabled));
+}
+
+static const struct kernel_param_ops offloading_param_ops = {
+	.set = offloading_param_set,
+	.get = offloading_param_get,
+};
+module_param_cb(offloading, &offloading_param_ops, NULL, 0644);
+MODULE_PARM_DESC(offloading, "Enable DMA migration offload (0/1)");
+
+/* nr_dma_chan: max DMA channels to use per batch */
+static int nr_dma_chan_param_set(const char *val, const struct kernel_param *kp)
+{
+	unsigned int new_val;
+	int ret;
+
+	ret = kstrtouint(val, 0, &new_val);
+	if (ret)
+		return ret;
+	if (new_val < 1 || new_val > MAX_DMA_CHANNELS)
+		return -EINVAL;
+
+	mutex_lock(&dcbm_mutex);
+	WRITE_ONCE(nr_dma_channels, new_val);
+	mutex_unlock(&dcbm_mutex);
+	return 0;
+}
+
+static int nr_dma_chan_param_get(char *buffer, const struct kernel_param *kp)
+{
+	return sysfs_emit(buffer, "%u\n", READ_ONCE(nr_dma_channels));
+}
+
+static const struct kernel_param_ops nr_dma_chan_param_ops = {
+	.set = nr_dma_chan_param_set,
+	.get = nr_dma_chan_param_get,
+};
+module_param_cb(nr_dma_chan, &nr_dma_chan_param_ops, NULL, 0644);
+MODULE_PARM_DESC(nr_dma_chan, "Max DMA channels to use (1..16)");
+
+/* reason_mask: set of MR_* reasons this migrator handles */
+static int reason_mask_param_set(const char *val, const struct kernel_param *kp)
+{
+	unsigned long mask;
+	int ret;
+
+	ret = migrate_offload_reason_mask_parse(val, &mask);
+	if (ret)
+		return ret;
+
+	mutex_lock(&dcbm_mutex);
+	WRITE_ONCE(dcbm_reason_mask, mask);
+	if (offloading_enabled)
+		migrate_offload_set_reason_mask(&dma_migrator, mask);
+	mutex_unlock(&dcbm_mutex);
+	return 0;
+}
+
+static int reason_mask_param_get(char *buffer, const struct kernel_param *kp)
+{
+	return migrate_offload_reason_mask_format(buffer, READ_ONCE(dcbm_reason_mask));
+}
+
+static const struct kernel_param_ops reason_mask_param_ops = {
+	.set = reason_mask_param_set,
+	.get = reason_mask_param_get,
+};
+module_param_cb(reason_mask, &reason_mask_param_ops, NULL, 0644);
+MODULE_PARM_DESC(reason_mask,
+		 "Reasons to offload: comma-separated names (e.g. compaction,demotion), 'all', 'none', or a raw hex mask");
+
+/* folios_migrated / folios_failures: counters; any write resets to 0 */
+static int folios_migrated_param_set(const char *val, const struct kernel_param *kp)
+{
+	atomic_long_set(&folios_migrated, 0);
+	return 0;
+}
+
+static int folios_migrated_param_get(char *buffer, const struct kernel_param *kp)
+{
+	return sysfs_emit(buffer, "%ld\n", atomic_long_read(&folios_migrated));
+}
+
+static const struct kernel_param_ops folios_migrated_param_ops = {
+	.set = folios_migrated_param_set,
+	.get = folios_migrated_param_get,
+};
+module_param_cb(folios_migrated, &folios_migrated_param_ops, NULL, 0644);
+MODULE_PARM_DESC(folios_migrated, "Folios DMA-copied (write to reset)");
+
+static int folios_failures_param_set(const char *val, const struct kernel_param *kp)
+{
+	atomic_long_set(&folios_failures, 0);
+	return 0;
+}
+
+static int folios_failures_param_get(char *buffer, const struct kernel_param *kp)
+{
+	return sysfs_emit(buffer, "%ld\n", atomic_long_read(&folios_failures));
+}
+
+static const struct kernel_param_ops folios_failures_param_ops = {
+	.set = folios_failures_param_set,
+	.get = folios_failures_param_get,
+};
+module_param_cb(folios_failures, &folios_failures_param_ops, NULL, 0644);
+MODULE_PARM_DESC(folios_failures, "DMA-copy failure count (write to reset)");
+
+static int __init dcbm_init(void)
+{
+	pr_info("dcbm: DMA Core Batch Migrator initialized\n");
+	return 0;
+}
+
+static void __exit dcbm_exit(void)
+{
+	mutex_lock(&dcbm_mutex);
+	if (offloading_enabled) {
+		migrate_offload_unregister(&dma_migrator);
+		offloading_enabled = false;
+	}
+	mutex_unlock(&dcbm_mutex);
+
+	pr_info("dcbm: DMA Core Batch Migrator unloaded\n");
+}
+
+module_init(dcbm_init);
+module_exit(dcbm_exit);
+
+MODULE_LICENSE("GPL");
+MODULE_AUTHOR("Shivank Garg");
+MODULE_DESCRIPTION("DMA Core Batch Migrator");

-- 
2.43.0

[PATCH RFC v6 5/5] mm/migrate: adjust NR_MAX_BATCHED_MIGRATION for testing

[Shivank Garg]

From: Zi Yan <ziy@nvidia.com>

Change NR_MAX_BATCHED_MIGRATION to HPAGE_PUD_NR to allow batching THP
copies.

These are for testing purpose only.

Signed-off-by: Zi Yan <ziy@nvidia.com>
Signed-off-by: Shivank Garg <shivankg@amd.com>
---
 mm/migrate.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/mm/migrate.c b/mm/migrate.c
index 4fed3110ca0a..4be1f3dc1603 100644
--- a/mm/migrate.c
+++ b/mm/migrate.c
@@ -1592,7 +1592,7 @@ static inline int try_split_folio(struct folio *folio, struct list_head *split_f
 }
 
 #ifdef CONFIG_TRANSPARENT_HUGEPAGE
-#define NR_MAX_BATCHED_MIGRATION	HPAGE_PMD_NR
+#define NR_MAX_BATCHED_MIGRATION	HPAGE_PUD_NR
 #else
 #define NR_MAX_BATCHED_MIGRATION	512
 #endif

-- 
2.43.0

Re: [PATCH RFC v6 0/5] Accelerate page migration with batch copying and hardware offload

[Huang, Ying]

Hi, Garg,

Thanks for updated patch.

Shivank Garg <shivankg@amd.com> writes:

[snip]

>
> PERFORMANCE RESULTS:
> --------------------
>
> AMD EPYC 7713 (Zen 3), 2 sockets, 32 cores, SMT on, 
> 1 NUMA node per socket, 256 GB/node, v7.2-rc1, DVFS=Performance, PTDMA
> (16 DMA channels).
>
> Benchmark: move_pages() syscall to move pages between two NUMA nodes.
>
> 1). Moving different sized folios such that total transfer size is constant
> (1GB), with different number of DMA channels. Throughput in GB/s.
>
> a. Baseline (vanilla kernel, single-threaded, serial folio_copy):
> ================================================================================
> 4K          | 16K        | 64K        | 256K       | 1M          | 2M          |
> ================================================================================
> 3.28±0.14   | 4.98±0.18  | 6.19±0.08  | 6.77±0.08  | 7.02±0.11   | 10.80±0.13  |
>
> b. DMA offload (Patched Kernel, dcbm driver, N DMA channels):
> ============================================================================================
> N channel| 4K        | 16K         | 64K         | 256K        | 1M          | 2M          |
> ============================================================================================
> 1      | 2.38±0.17   | 2.77±0.03   | 3.21±0.03   | 5.00±0.02   | 5.09±0.64   | 12.62±0.07  |
> 2      | 2.87±0.11   | 4.06±0.05   | 5.09±0.04   | 6.97±0.08   | 8.43±0.06   | 14.32±0.10  |
> 4      | 3.32±0.07   | 5.30±0.06   | 7.21±0.09   | 9.69±0.15   | 11.36±0.13  | 26.98±0.19  |
> 8      | 3.68±0.09   | 6.28±0.10   | 9.16±0.13   | 12.05±0.16  | 15.33±2.80  | 46.06±0.55  |
> 12     | 3.83±0.05   | 6.65±0.17   | 10.00±0.16  | 12.98±0.18  | 15.87±0.19  | 61.31±1.28  |
> 16     | 3.94±0.09   | 6.78±0.10   | 10.48±0.13  | 13.48±0.20  | 16.90±0.24  | 65.06±2.46  |
>
> 2). First-folio latency: custom tracepoints (in migrate_pages_batch enter/exit,
>     migrate_folio_done) measure latency per migrate_pages_batch() call.
>
>     Throughput (GB/s) and first-folio latency (us), median of 10 runs.
>
> a. Vanilla Kernel:
>
> NR_MAX_BATCHED_MIGRATION upstream default value is 512.
> --- Order 0 (4K folios) ---       --- Order 9 (2M folios) ---
>      n      vanilla/cpu                n      vanilla/cpu
> (folios)    GB/s | first(us)      (folios)    GB/s | first(us)
> --------------------------        --------------------------
>      1       0.03 |     24             1       6.86 |    204
>      4       0.13 |     30             4       8.68 |    191
>      8       0.27 |     27             8       7.92 |    207
>     16       0.43 |     34            16       6.77 |    234
>     64       1.12 |     51            64      10.44 |    179
>    256       1.67 |    166           256      10.43 |    181
>    512       1.98 |    255           512      10.55 |    179
>   2048       2.38 |    233
>   4096       2.42 |    168
>  16384       2.72 |    167
>  65536       3.00 |    156
> 262144       3.10 |    151
>
> b. Patched kernel:
> N = NR_MAX_BATCHED_MIGRATION (in pages), Total migrated data fixed at
> 1 GB. Change N with knob (just for testing) to measure impact of
> different max batched size.
>
> --- ORDER 0 (4K folios) ---
>
>      N         offload/dma1          offload/dma4          offload/dma16
>                GB/s | first(us)      GB/s | first(us)      GB/s | first(us)
> ------------------------------------------------------------------------
>    512         2.21 |    628         3.29 |    275         3.25 |    245
>   1024         2.06 |   1271         3.21 |    601         3.36 |    518
>   2048         2.02 |   2646         3.00 |   1388         3.20 |   1110
>   4096         2.08 |   4832         3.17 |   2514         3.41 |   2175
>   8192         2.16 |   9253         3.14 |   4839         3.62 |   3592
>  16384         2.24 |  17543         3.23 |   9680         3.58 |   7144
>  32768         2.22 |  36408         3.26 |  19301         3.67 |  14524
>  65536         2.12 |  82572         3.24 |  38091         3.62 |  29835
> 131072         2.08 | 153669         3.17 |  79744         3.48 |  62157
> 262144         2.05 | 332297         2.97 | 175315         3.33 | 134774
>
> --- ORDER 9 (2M folios) ---
>
>      N         offload/dma1          offload/dma4          offload/dma16
>                GB/s | first(us)      GB/s | first(us)      GB/s | first(us)
> ------------------------------------------------------------------------
>    512        11.74 |    160        11.71 |    160        11.75 |    159
>   1024        12.18 |    310        13.82 |    274        13.76 |    275
>   2048        12.39 |    612        25.55 |    290        25.69 |    289
>   4096        12.54 |   1211        26.25 |    564        42.36 |    334
>   8192        12.54 |   2421        26.82 |   1111        51.85 |    485
>  16384        12.61 |   4824        26.91 |   2209        54.26 |    925
>  32768        12.62 |   9652        27.04 |   4404        54.72 |   1942
>  65536        12.64 |  19287        26.95 |   8835        57.30 |   3535
> 131072        12.64 |  38824        26.95 |  17900        58.58 |   7747
> 262144        12.66 |  77610        26.95 |  35743        66.31 |  13801
>
> OPEN QUESTION:
> --------------
>
> The best batch size depends on the hardware, and bigger isn't always better.
> NR_MAX_BATCHED_MIGRATION decides how many pages we move at once.
> Higher batch size can help amortize the setup cost of migrator but
> increases the first-folio latency (the folio is inaccessible for this
> window).

Yes.  This is an important parameter for the batched migration.  We
really need more information from people who have workload information
to make a decision.  Can you try to reach them?

Also, can we find a sweet point where we can archive higher throughput
without increasing latency too much (e.g., < 1ms)?

> Goals could be workload dependent, e.g. higher throughput versus
> same throughput under a bounded latency.
>
> Should this be tunable to accommodate different hardware and goals?

[snip]

---
Best Regards,
Huang, Ying

Re: [PATCH RFC v6 3/5] mm/migrate: add copy offload registration infrastructure

[Huang, Ying]

Shivank Garg <shivankg@amd.com> writes:

[snip]

> diff --git a/mm/migrate.c b/mm/migrate.c
> index 41b732e78a67..4fed3110ca0a 100644
> --- a/mm/migrate.c
> +++ b/mm/migrate.c
> @@ -43,6 +43,7 @@
>  #include <linux/sched/sysctl.h>
>  #include <linux/memory-tiers.h>
>  #include <linux/pagewalk.h>
> +#include <linux/migrate_copy_offload.h>
>  
>  #include <asm/tlbflush.h>
>  
> @@ -51,12 +52,6 @@
>  #include "internal.h"
>  #include "swap.h"
>  
> -/* For now, never offload. Wired up in later patch. */
> -static bool migrate_should_offload(int reason)
> -{
> -	return false;
> -}
> -
>  static const struct movable_operations *offline_movable_ops;
>  static const struct movable_operations *zsmalloc_movable_ops;
>  
> @@ -2050,7 +2045,7 @@ static int migrate_pages_batch(struct list_head *from,
>  
>  	/* Batch-copy eligible folios before the move phase */
>  	if (!list_empty(&unmap_batch))
> -		migrate_folios_mc_copy(&dst_batch, &unmap_batch, nr_batch);
> +		migrate_offload_batch_copy(&dst_batch, &unmap_batch, nr_batch);
>  
>  	retry = 1;
>  	for (pass = 0; pass < nr_pass && retry; pass++) {

Found a difference between v5 and v6 here.  In v5, migrate_pages_batch()
checks the return value of migrate_offload_batch_copy() and acts
accordingly. While in v6, it does not.  Why was this changed?

IIUC, FOLIO_CONTENT_COPIED is now used to indicate a copy failure.  Is
this necessary?  Is it possible that some (but not all) folios fail to
be copied?

[snip]

---
Best Regards,
Huang, Ying

Re: [PATCH RFC v6 3/5] mm/migrate: add copy offload registration infrastructure

[Zi Yan]

On 20 Jul 2026, at 7:19, Huang, Ying wrote:

> Shivank Garg <shivankg@amd.com> writes:
>
> [snip]
>
>> diff --git a/mm/migrate.c b/mm/migrate.c
>> index 41b732e78a67..4fed3110ca0a 100644
>> --- a/mm/migrate.c
>> +++ b/mm/migrate.c
>> @@ -43,6 +43,7 @@
>>  #include <linux/sched/sysctl.h>
>>  #include <linux/memory-tiers.h>
>>  #include <linux/pagewalk.h>
>> +#include <linux/migrate_copy_offload.h>
>>
>>  #include <asm/tlbflush.h>
>>
>> @@ -51,12 +52,6 @@
>>  #include "internal.h"
>>  #include "swap.h"
>>
>> -/* For now, never offload. Wired up in later patch. */
>> -static bool migrate_should_offload(int reason)
>> -{
>> -	return false;
>> -}
>> -
>>  static const struct movable_operations *offline_movable_ops;
>>  static const struct movable_operations *zsmalloc_movable_ops;
>>
>> @@ -2050,7 +2045,7 @@ static int migrate_pages_batch(struct list_head *from,
>>
>>  	/* Batch-copy eligible folios before the move phase */
>>  	if (!list_empty(&unmap_batch))
>> -		migrate_folios_mc_copy(&dst_batch, &unmap_batch, nr_batch);
>> +		migrate_offload_batch_copy(&dst_batch, &unmap_batch, nr_batch);
>>
>>  	retry = 1;
>>  	for (pass = 0; pass < nr_pass && retry; pass++) {
>
> Found a difference between v5 and v6 here.  In v5, migrate_pages_batch()
> checks the return value of migrate_offload_batch_copy() and acts
> accordingly. While in v6, it does not.  Why was this changed?
>
> IIUC, FOLIO_CONTENT_COPIED is now used to indicate a copy failure.  Is
> this necessary?  Is it possible that some (but not all) folios fail to
> be copied?

Based on the discussion[1], FOLIO_CONTENT_COPIED is used to indicate both
1. folio is copied via offloading, and 2. MC copy failure. So the code
no longer needs to store the return value of migrate_offload_batch_copy()
and can just check dst->migrate_info & FOLIO_CONTENT_COPIED.

For 32bit, FOLIO_CONTENT_COPIED is always 0, so all folios are always
copied in the serialized path. Hmm, that might be causing double copying,
since migrate_folios_mc_copy() copies batched folios and later
migrate_folios_move() copies them again since
dst->migrate_info & FOLIO_CONTENT_COPIED is always false. Maybe we could
make migrate_folios_mc_copy() return immediately in 32bit, otherwise,
we will need to find somewhere else to store FOLIO_CONTENT_COPIED.

[1] https://lore.kernel.org/all/c2bfdf39-8caa-43bd-b1ad-2285b4cc767a@amd.com/

>
> [snip]
>
> ---
> Best Regards,
> Huang, Ying


Best Regards,
Yan, Zi

Re: [PATCH RFC v6 3/5] mm/migrate: add copy offload registration infrastructure

[Garg, Shivank]

On 7/20/2026 8:14 PM, Zi Yan wrote:
> On 20 Jul 2026, at 7:19, Huang, Ying wrote:
> 
>> Shivank Garg <shivankg@amd.com> writes:
>>
>> [snip]
>>
>>> diff --git a/mm/migrate.c b/mm/migrate.c
>>> index 41b732e78a67..4fed3110ca0a 100644
>>> --- a/mm/migrate.c
>>> +++ b/mm/migrate.c
>>> @@ -43,6 +43,7 @@
>>>  #include <linux/sched/sysctl.h>
>>>  #include <linux/memory-tiers.h>
>>>  #include <linux/pagewalk.h>
>>> +#include <linux/migrate_copy_offload.h>
>>>
>>>  #include <asm/tlbflush.h>
>>>
>>> @@ -51,12 +52,6 @@
>>>  #include "internal.h"
>>>  #include "swap.h"
>>>
>>> -/* For now, never offload. Wired up in later patch. */
>>> -static bool migrate_should_offload(int reason)
>>> -{
>>> -	return false;
>>> -}
>>> -
>>>  static const struct movable_operations *offline_movable_ops;
>>>  static const struct movable_operations *zsmalloc_movable_ops;
>>>
>>> @@ -2050,7 +2045,7 @@ static int migrate_pages_batch(struct list_head *from,
>>>
>>>  	/* Batch-copy eligible folios before the move phase */
>>>  	if (!list_empty(&unmap_batch))
>>> -		migrate_folios_mc_copy(&dst_batch, &unmap_batch, nr_batch);
>>> +		migrate_offload_batch_copy(&dst_batch, &unmap_batch, nr_batch);
>>>
>>>  	retry = 1;
>>>  	for (pass = 0; pass < nr_pass && retry; pass++) {
>>
>> Found a difference between v5 and v6 here.  In v5, migrate_pages_batch()
>> checks the return value of migrate_offload_batch_copy() and acts
>> accordingly. While in v6, it does not.  Why was this changed?
>>
>> IIUC, FOLIO_CONTENT_COPIED is now used to indicate a copy failure.  Is
>> this necessary?  Is it possible that some (but not all) folios fail to
>> be copied?
> 
> Based on the discussion[1], FOLIO_CONTENT_COPIED is used to indicate both
> 1. folio is copied via offloading, and 2. MC copy failure. So the code
> no longer needs to store the return value of migrate_offload_batch_copy()
> and can just check dst->migrate_info & FOLIO_CONTENT_COPIED.
> 
> For 32bit, FOLIO_CONTENT_COPIED is always 0, so all folios are always
> copied in the serialized path. Hmm, that might be causing double copying,
> since migrate_folios_mc_copy() copies batched folios and later
> migrate_folios_move() copies them again since
> dst->migrate_info & FOLIO_CONTENT_COPIED is always false. Maybe we could
> make migrate_folios_mc_copy() return immediately in 32bit, otherwise,
> we will need to find somewhere else to store FOLIO_CONTENT_COPIED.
> 
> [1] https://lore.kernel.org/all/c2bfdf39-8caa-43bd-b1ad-2285b4cc767a@amd.com/
> 
>>
It is not cause double copying.
migrate_should_offload() /and migrate_offload_enabled
gate if folio can be added to unmap_batch/dst_batch.

migrate_offload_enabled branch is enabled from the offload driver like DCBM (which
again depends on 64_BIT) will call migrate_offload_register().

I think I can add a defensive 64_BIT check in migrate_should_offload() to prevent
any potential bugs.

Thanks,
Shivank










>> [snip]
>>
>> ---
>> Best Regards,
>> Huang, Ying
> 
> 
> Best Regards,
> Yan, Zi

Re: [PATCH RFC v6 3/5] mm/migrate: add copy offload registration infrastructure

[Zi Yan]

On 20 Jul 2026, at 11:32, Garg, Shivank wrote:

> On 7/20/2026 8:14 PM, Zi Yan wrote:
>> On 20 Jul 2026, at 7:19, Huang, Ying wrote:
>>
>>> Shivank Garg <shivankg@amd.com> writes:
>>>
>>> [snip]
>>>
>>>> diff --git a/mm/migrate.c b/mm/migrate.c
>>>> index 41b732e78a67..4fed3110ca0a 100644
>>>> --- a/mm/migrate.c
>>>> +++ b/mm/migrate.c
>>>> @@ -43,6 +43,7 @@
>>>>  #include <linux/sched/sysctl.h>
>>>>  #include <linux/memory-tiers.h>
>>>>  #include <linux/pagewalk.h>
>>>> +#include <linux/migrate_copy_offload.h>
>>>>
>>>>  #include <asm/tlbflush.h>
>>>>
>>>> @@ -51,12 +52,6 @@
>>>>  #include "internal.h"
>>>>  #include "swap.h"
>>>>
>>>> -/* For now, never offload. Wired up in later patch. */
>>>> -static bool migrate_should_offload(int reason)
>>>> -{
>>>> -	return false;
>>>> -}
>>>> -
>>>>  static const struct movable_operations *offline_movable_ops;
>>>>  static const struct movable_operations *zsmalloc_movable_ops;
>>>>
>>>> @@ -2050,7 +2045,7 @@ static int migrate_pages_batch(struct list_head *from,
>>>>
>>>>  	/* Batch-copy eligible folios before the move phase */
>>>>  	if (!list_empty(&unmap_batch))
>>>> -		migrate_folios_mc_copy(&dst_batch, &unmap_batch, nr_batch);
>>>> +		migrate_offload_batch_copy(&dst_batch, &unmap_batch, nr_batch);
>>>>
>>>>  	retry = 1;
>>>>  	for (pass = 0; pass < nr_pass && retry; pass++) {
>>>
>>> Found a difference between v5 and v6 here.  In v5, migrate_pages_batch()
>>> checks the return value of migrate_offload_batch_copy() and acts
>>> accordingly. While in v6, it does not.  Why was this changed?
>>>
>>> IIUC, FOLIO_CONTENT_COPIED is now used to indicate a copy failure.  Is
>>> this necessary?  Is it possible that some (but not all) folios fail to
>>> be copied?
>>
>> Based on the discussion[1], FOLIO_CONTENT_COPIED is used to indicate both
>> 1. folio is copied via offloading, and 2. MC copy failure. So the code
>> no longer needs to store the return value of migrate_offload_batch_copy()
>> and can just check dst->migrate_info & FOLIO_CONTENT_COPIED.
>>
>> For 32bit, FOLIO_CONTENT_COPIED is always 0, so all folios are always
>> copied in the serialized path. Hmm, that might be causing double copying,
>> since migrate_folios_mc_copy() copies batched folios and later
>> migrate_folios_move() copies them again since
>> dst->migrate_info & FOLIO_CONTENT_COPIED is always false. Maybe we could
>> make migrate_folios_mc_copy() return immediately in 32bit, otherwise,
>> we will need to find somewhere else to store FOLIO_CONTENT_COPIED.
>>
>> [1] https://lore.kernel.org/all/c2bfdf39-8caa-43bd-b1ad-2285b4cc767a@amd.com/
>>
>>>
> It is not cause double copying.
> migrate_should_offload() /and migrate_offload_enabled
> gate if folio can be added to unmap_batch/dst_batch.

So migrate_should_offload() is actually migrate_should_offload_batched_copy().
32BIT does not get batched copy nor offloaded copy.

>
> migrate_offload_enabled branch is enabled from the offload driver like DCBM (which
> again depends on 64_BIT) will call migrate_offload_register().
>
> I think I can add a defensive 64_BIT check in migrate_should_offload() to prevent
> any potential bugs.

I think a better name would be helpful and sufficient.

BTW, ideally, batched copy and offloaded copy are two separate features
and could be enabled separately. But considering the hassle of finding a
place to store FOLIO_CONTENT_COPIED on 32BIT, it might be fine to
have a combined batched and offloaded copy.

Best Regards,
Yan, Zi

Re: [PATCH RFC v6 3/5] mm/migrate: add copy offload registration infrastructure

[Huang, Ying]

Zi Yan <ziy@nvidia.com> writes:

> On 20 Jul 2026, at 7:19, Huang, Ying wrote:
>
>> Shivank Garg <shivankg@amd.com> writes:
>>
>> [snip]
>>
>>> diff --git a/mm/migrate.c b/mm/migrate.c
>>> index 41b732e78a67..4fed3110ca0a 100644
>>> --- a/mm/migrate.c
>>> +++ b/mm/migrate.c
>>> @@ -43,6 +43,7 @@
>>>  #include <linux/sched/sysctl.h>
>>>  #include <linux/memory-tiers.h>
>>>  #include <linux/pagewalk.h>
>>> +#include <linux/migrate_copy_offload.h>
>>>
>>>  #include <asm/tlbflush.h>
>>>
>>> @@ -51,12 +52,6 @@
>>>  #include "internal.h"
>>>  #include "swap.h"
>>>
>>> -/* For now, never offload. Wired up in later patch. */
>>> -static bool migrate_should_offload(int reason)
>>> -{
>>> -	return false;
>>> -}
>>> -
>>>  static const struct movable_operations *offline_movable_ops;
>>>  static const struct movable_operations *zsmalloc_movable_ops;
>>>
>>> @@ -2050,7 +2045,7 @@ static int migrate_pages_batch(struct list_head *from,
>>>
>>>  	/* Batch-copy eligible folios before the move phase */
>>>  	if (!list_empty(&unmap_batch))
>>> -		migrate_folios_mc_copy(&dst_batch, &unmap_batch, nr_batch);
>>> +		migrate_offload_batch_copy(&dst_batch, &unmap_batch, nr_batch);
>>>
>>>  	retry = 1;
>>>  	for (pass = 0; pass < nr_pass && retry; pass++) {
>>
>> Found a difference between v5 and v6 here.  In v5, migrate_pages_batch()
>> checks the return value of migrate_offload_batch_copy() and acts
>> accordingly. While in v6, it does not.  Why was this changed?
>>
>> IIUC, FOLIO_CONTENT_COPIED is now used to indicate a copy failure.  Is
>> this necessary?  Is it possible that some (but not all) folios fail to
>> be copied?
>
> Based on the discussion[1], FOLIO_CONTENT_COPIED is used to indicate both
> 1. folio is copied via offloading, and 2. MC copy failure. So the code
> no longer needs to store the return value of migrate_offload_batch_copy()
> and can just check dst->migrate_info & FOLIO_CONTENT_COPIED.
>
> For 32bit, FOLIO_CONTENT_COPIED is always 0, so all folios are always
> copied in the serialized path. Hmm, that might be causing double copying,
> since migrate_folios_mc_copy() copies batched folios and later
> migrate_folios_move() copies them again since
> dst->migrate_info & FOLIO_CONTENT_COPIED is always false. Maybe we could
> make migrate_folios_mc_copy() return immediately in 32bit, otherwise,
> we will need to find somewhere else to store FOLIO_CONTENT_COPIED.
>
> [1] https://lore.kernel.org/all/c2bfdf39-8caa-43bd-b1ad-2285b4cc767a@amd.com/

Thanks a lot for your information!  Although personally I don't think
that it's necessary to optimize for really rare folio_mc_copy() failure
cases, I am fine with it if you think that this is the better way.

>>
>> [snip]
>>

---
Best Regards,
Huang, Ying